issue: #51381
related: #51414
related: #51416
### What this PR does
- Adds Python client E2E coverage for nullable Struct Array
NULL-versus-empty semantics.
- Covers compaction, snapshot restoration, partial updates, drop and
re-add isolation, imports, regex indexes, aggregation, and dynamic
fields.
- Adds exact ANN oracles for Struct Array element and embedding-list
searches without assuming self-hit behavior.
- Covers hybrid search collapse strategies and element identity
preservation.
- Marks known regressions with strict xfail markers linked to their
corresponding issues while keeping unaffected parameters active.
### Why
Struct Array rows can represent NULL, empty arrays, and non-empty
arrays. These states affect predicates, nested indexes, element offsets,
and vector search candidates differently. The added coverage protects
these semantics across write, reload, compaction, import, and snapshot
lifecycle paths.
### Validation
- `ruff check` passed for all changed Python files.
- `ruff format --check` passed for all changed Python files.
- Python compilation passed for all changed Python files.
- Pytest collection passed.
- The nine known regression nodes executed as expected: `9 xfailed`.
- The three unaffected parameterized cases passed.
- Targeted E2E validation used Milvus master build `09c44f2dbb`.
---------
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
## What changed
- Replace Pod deletion with Chaos Mesh `container-kill` in CDC
force-promote and source-down failover tests.
- Kill every regular container in matching Pods while preserving the Pod
objects.
- Verify that Pod UIDs remain unchanged and each container's restart
count increases.
- Clean up the generated `PodChaos` resources after fault injection.
## Why
Deleting Pods changes their identity and tests Pod recreation rather
than container-level recovery. These scenarios need to verify CDC
failover behavior when containers restart inside the existing Pods.
## Impact
CDC failover tests now exercise container failures while explicitly
validating that Kubernetes does not recreate the affected Pods.
## Validation
- `ruff check` passed for all modified files.
- `ruff format --check` passed for all modified files.
- Python compilation passed for all modified files.
- Full CDC E2E was not run locally because it requires a Kubernetes
environment with Chaos Mesh.
---------
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
issue: #51248
## What changed
- correct the omitted-`target_size` case to assert ordinary manual
compaction semantics
- add a physical Force Merge target-size case using persistent segment
IDs and MinIO insert-log sizes
- add deterministic Go boundary coverage for grouping selection at the
configured threshold and threshold + 1
- add target-size memory clamp coverage for cluster and standalone
co-location modes
- extend Optimize format, signed-int64 boundary, loaded-segment refresh,
and live async lifecycle coverage
- add the async Optimize wrapper and update the PyMilvus test dependency
to 3.1.0rc64
## Why
The previous cases did not prove that an explicit Force Merge target
affected physical output, assumed the wrong grouping threshold, and
depended on manual log inspection for algorithm selection. Loaded
refresh and the public async Optimize workflow also lacked live
end-to-end coverage.
The async case exposed milvus-io/pymilvus#3680 on PyMilvus 3.1.0rc62.
Version 3.1.0rc64 contains the tuple-unpacking fix and passes the
unmodified test.
## Validation
- all 10 planned case IDs pass; 0 failed and 0 blocked
- `milvus-dev-cli` Go UT job
`go-ut-local-zhuwenxi-zhuwenxing-i-4082040-30413568` succeeded; both
test functions and all five subtests passed
- physical target-size L3 case passed against a real cluster and MinIO
in 200.68s
- loaded refresh, target format/boundary, and manual compaction cases
passed against the same server version
- async Optimize L3 passed unmodified with PyMilvus 3.1.0rc64 in 56.91s
- 56 Python nodes collect successfully; Python compilation, Ruff, and
`git diff --check` pass
---------
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
## What changed
- Remove the stale `xfail` marker from `test_file_with_bom_and_crlf`,
which covers the UTF-8 BOM regression tracked by
[#49684](https://github.com/milvus-io/milvus/issues/49684) and fixed by
[#49691](https://github.com/milvus-io/milvus/pull/49691).
- Remove the stale `skip` marker from
`test_drop_collection_then_remove_remote_resources_no_panic`, which
covers the FileResource lifecycle panic tracked by
[#49279](https://github.com/milvus-io/milvus/issues/49279) and fixed by
[#49412](https://github.com/milvus-io/milvus/pull/49412).
Both issues are closed. This change makes the existing cases active CI
regressions; it does not change their fixtures, test bodies, assertions,
or server behavior.
## User impact
CI will now detect regressions where stop-word files with a UTF-8 BOM
are parsed incorrectly, or where dropping a collection and removing its
remote analyzer resources can panic the server.
## Validation
The existing cases were previously run against the latest
`upstream/master` commit `03762320e8537f855c18c28133834bf292b45361`
using image `harbor.milvus.io/manta/milvus:master-20260710-0376232`. The
`-p no:skipping` option only bypassed pytest's skip/xfail marker
handling; no case code, fixture, or assertion was changed:
```bash
python -W ignore -m pytest -v -p no:skipping \
milvus_client/test_milvus_client_file_resource.py::TestMilvusClientFileResourceContent::test_file_with_bom_and_crlf \
milvus_client/test_milvus_client_file_resource.py::TestMilvusClientFileResourceLifecycleAdvanced::test_drop_collection_then_remove_remote_resources_no_panic \
--host <temporary-milvus-host> --port 19530 --token 'root:Milvus' \
--minio_host <temporary-minio-host> --minio_bucket fr-closed-issues-master \
--tb=short
```
Result: `2 passed in 2.16s`.
The #49279 case was then repeated ten times with the same marker bypass:
```bash
python -W ignore -m pytest -q -p no:skipping \
milvus_client/test_milvus_client_file_resource.py::TestMilvusClientFileResourceLifecycleAdvanced::test_drop_collection_then_remove_remote_resources_no_panic \
--count=10 \
--host <temporary-milvus-host> --port 19530 --token 'root:Milvus' \
--minio_host <temporary-minio-host> --minio_bucket fr-closed-issues-master \
--tb=short
```
Result: `10 passed in 14.59s`. Milvus remained healthy, pod restart
count was zero, and logs contained no panic, `tokenizer.h:31`, fatal
error, or segmentation fault. The temporary validation instance has been
cleaned up.
The two node IDs also collect successfully after removing the markers:
```bash
python -W ignore -m pytest --collect-only -q \
milvus_client/test_milvus_client_file_resource.py::TestMilvusClientFileResourceContent::test_file_with_bom_and_crlf \
milvus_client/test_milvus_client_file_resource.py::TestMilvusClientFileResourceLifecycleAdvanced::test_drop_collection_then_remove_remote_resources_no_panic
```
Result: `2 tests collected in 0.09s`.
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
## What type of PR is this?
/test
## What this PR does / why we need it
Adds Python client coverage for struct array feature scenarios on
master:
- Covers struct-array vector subfields across JSON, JSONL, CSV, and
Parquet import files.
- Covers LocalBulkWriter generated JSON, JSONL, CSV, and Parquet files
for FLOAT_VECTOR, FLOAT16_VECTOR, BFLOAT16_VECTOR, INT8_VECTOR, and
BINARY_VECTOR struct subfields.
- Aligns BulkWriter test inputs with ordinary vector field input forms,
including FP16/BF16 ndarray inputs and INT8 ndarray inputs.
- Adds hybrid search, range, group-by, element_scope regression
coverage, and bitmap index coverage for struct array element fields.
- Updates Python client test dependency pin to pymilvus 3.1.0rc50 so the
merged BulkWriter JSONL and struct vector fixes are exercised.
## Which issue(s) this PR fixes
None.
## Special notes for your reviewer
The BulkWriter coverage intentionally uses native insert-format inputs
for non-float32 vectors to match ordinary vector field behavior.
## Tests
```bash
python -W ignore -m pytest -v -n 6 --tb=short \
milvus_client/test_milvus_client_struct_array.py::TestMilvusClientStructArrayImport::test_import_struct_array_vector_subfield_with_local_bulk_writer \
--host 10.100.36.207 --port 19530 --token root:Milvus \
--minio_host 10.100.36.206 --minio_bucket struct-array-master-ff3dbeb
```
Result: `20 passed in 120.02s`
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
## What type of PR is this?
/kind test
## What this PR does / why we need it:
Adds RESTful import 2PC coverage for Milvus import lifecycle and
CDC-focused scenarios.
This PR adds:
- REST import job helpers for `/describe`, `/commit`, `/abort`, and
state polling
- pytest options for secondary CDC endpoint/object storage/topology
parameters
- a new REST test suite for import 2PC lifecycle, visibility,
commit/abort, delete semantics, TTL, compaction, formats, data types,
indexes, partitions, DML interleaving, CDC, and fault-tolerance gates
- xfail regression coverage for known issues:
- #50458 REST import commit/abort authorization gap
- #50459 NaN/Inf FloatVector import/query behavior
- #50460 REST `options.auto_commit=null` validation gap
## Which issue(s) this PR fixes:
N/A
## Special notes for your reviewer:
The CDC cases require passing secondary cluster REST/MinIO options and
topology arguments. Without those options, CDC-specific cases skip with
an explicit environment precondition message.
The manual compaction-after-commit case is marked non-strict xfail
because compaction may remain `Executing` on current import 2PC test
builds.
## Test result:
```text
python -m py_compile tests/restful_client_v2/conftest.py tests/restful_client_v2/api/milvus.py tests/restful_client_v2/testcases/test_import_2pc_operation.py
ruff check tests/restful_client_v2/api/milvus.py tests/restful_client_v2/conftest.py tests/restful_client_v2/testcases/test_import_2pc_operation.py
python -m pytest --collect-only -q testcases/test_import_2pc_operation.py
141 tests collected in 2.14s
```
---------
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
### What this PR does
Adds Python client and REST coverage for struct array nullable fields,
including dynamic field scenarios and nullable vector config variants.
### Tests
Not run; PR creation only.
---------
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
issue: #50536
## What changed
This fixes growing-source flush for nullable `VECTOR_ARRAY` fields.
`BuildVectorArrayForChunk` now preserves null logical rows as Arrow null
list entries and maps valid logical rows to compact physical offsets
before reading `ConcurrentVector<VectorArray>`.
A regression test covers nullable vector-array round-trip flushing with
`valid, null, valid, null` rows.
## Why
With StorageV3/Loon and growing-source flush enabled, dropping a
collection can flush growing segment data through
`FlushGrowingSegmentData`. Nullable StructArray ArrayOfVector subfields
can contain null logical rows, but the previous vector-array builder
rejected any `valid_data`, causing `VECTOR_ARRAY does not support null
rows` and a server panic during drop collection.
## Verification
- `git diff --check`
- `milvus-dev-cli cpp-ut local . -b master --allow-dirty --target
all_tests --case
FlushGrowingSegmentTest.FlushNullableVectorArrayRoundTrip -f`
- UT ID: `ut-cpp-local-zhuwenxi-zhuwenxing-b-3e34efd-705e9e04`
- Status: succeeded
- `milvus-dev-cli build local -b master --allow-dirty -f`
- Build ID: `build-local-zhuwenxing-zhuwenxing-bug-m-3e34efd`
- Image:
`harbor.milvus.io/manta/milvus:local-zhuwenxing-zhuwenxing-bug-milvus-panics-when-dropping-nullable-structa-3e34efd`
- Deployed with:
- `common.storage.useLoonFFI=true`
- `common.storage.enableGrowingSourceFlush=true`
- `--mq rocksmq`
- Ran the issue repro with `pymilvus==3.1.0rc39` from TestPyPI:
- `drop_collection succeeded`
- `has_collection_after_drop=False`
- standalone pod restart count stayed `0`
- no `panic: flush growing source data` or `VECTOR_ARRAY does not
support null rows` in standalone logs
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
## What this PR does
When chaos checker assertions fail, the final pytest assertion message
currently only includes normalized error text, which can drop useful
values such as expected/actual row counts. This PR keeps a compact
sample of the original error alongside the normalized error type so the
useful failure detail appears at the end of the pytest output.
## Verification
- python3 -m py_compile tests/python_client/chaos/checker.py
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
Fixes#50107
## What
- Size field-id-indexed bitsets from the maximum user field-id offset
instead of `schema->size()`.
- Apply the helper to search plan involved-fields bitsets and sealed
segment ready bitsets.
- Add C++ and Python regressions for searching a late StructArray vector
subfield after multiple StructArray parent fields.
## Why
StructArray parent fields consume field IDs but are not stored as
regular C++ schema fields. With multiple StructArray parents, later
subfields can have `field_id - START_USER_FIELDID` greater than
`schema->size()`, causing `invalid field id` during search.
## Verification
- `git diff --cached --check`
- `python3 -m py_compile
tests/python_client/milvus_client/test_milvus_client_struct_array.py`
- `milvus-build build local -b master --allow-dirty`:
`build-local-zhuwenxing-fix-50107-bitset-d44e5bc`
- Deployed
`harbor.milvus.io/manta/milvus:local-zhuwenxing-fix-50107-bitset-d44e5bc`
as `issue-50107-master-fix-d44e5bc`: Healthy
- `verify_milvus.py 10.100.36.218 19530`: passed, server
`master-20260528-b7293edf8d`
- `pytest
milvus_client/test_milvus_client_struct_array.py::TestMilvusClientStructArraySearch::test_search_struct_array_last_vector_subfield`:
1 passed
- Imported original `crime_cases_v3` issue data: 600 rows;
`debug_struct_vector.py`: all 6 ANN searches OK, including
`evidence[evidence_vector]`
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
## What
Switch Python test-side Jieba usage from the Python `jieba` package to
`rjieba`, the Python binding for `jieba-rs`.
## Changes
- Replace Python test requirements from `jieba==0.42.1` to
`rjieba==0.1.11`
- Update Python client and REST test helpers to call `rjieba.cut`,
`rjieba.cut_for_search`, and `rjieba.cut_all`
- Update analyzer expected-token helper to use `rjieba` for supported
mode/hmm cases
- Keep custom-dictionary validation focused on analyzer output because
`rjieba` does not expose dynamic dictionary APIs like Python
`jieba.add_word` or `set_dictionary`
## Test
- `git diff --check`
- `python3 -m py_compile tests/python_client/common/common_func.py
tests/python_client/common/phrase_match_generator.py
tests/restful_client_v2/utils/utils.py
tests/python_client/milvus_client/test_milvus_client_analyzer.py`
- `rjieba` API smoke test
- Against `struct-hybrid-master-d44e5bc` at `10.100.36.224:19530`:
- `TestMilvusClientAnalyzer::test_jieba_custom_analyzer`: 6 passed
- `TestMilvusClientAnalyzer::test_analyzer`: 3 passed
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
## What this PR does
Adds Python client correctness coverage for struct array element-level +
normal field hybrid search.
The new case:
- inserts more than 3000 rows with multi-element struct arrays
- computes brute-force ground truth from inserted data
- verifies raw element-level search results before collapse
- verifies normal vector search results
- verifies hybrid RRF results after element-level row collapse
- covers both FLAT and HNSW indexes
## Verification
Ran against a master standalone instance: `master-20260528-d44e5bcdc4`.
```bash
python -m pytest milvus_client/test_milvus_client_struct_array_element_search.py::TestMilvusClientStructArrayElementHybridSearch::test_hybrid_element_filter_collapse_correctness --host 10.100.36.224 --port 19530 --token root:Milvus --tb=short -q
```
Repeated 3 times:
```text
run 1/3: 2 passed in 23.44s
run 2/3: 2 passed in 26.81s
run 3/3: 2 passed in 21.81s
```
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
## What this PR does
Fixes CDC stability checker initialization for partial update.
The shared CDC chaos checker schema includes struct-array fields.
PyMilvus rejects `partial_update=True` when the collection schema
contains struct fields, so `PartialUpdateChecker` should not be
initialized with the shared collection name and shared struct schema.
This PR keeps the master all-datatype schema intact for the other
checkers, including struct-array fields, BM25, MinHash, and nullable
settings. Only `PartialUpdateChecker` gets its own schema with
`enable_struct_array_field=False` and its own collection:
- shared checkers: `_build_checker_schema()`
- partial update checker:
`_build_checker_schema(enable_struct_array_field=False)` and no shared
`collection_name`
## Tests
- `uv run python -m py_compile
cdc/stablity/test_single_request_operation.py`
- `uvx ruff@0.15.11 check
tests/python_client/cdc/stablity/test_single_request_operation.py`
- `uvx ruff@0.15.11 format --check --diff
tests/python_client/cdc/stablity/test_single_request_operation.py`
- Local schema check: shared schema keeps
`struct_fields=["array_struct"]`, BM25, and MinHash; partial-update
schema has `struct_fields=[]` while still keeping BM25 and MinHash
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
## Summary
Add struct array (ArrayOfVector) support to the RESTful v2 API:
- Schema: declare struct fields with sub-fields in `createCollection`
- Data: ingest struct array rows in `insert` / `upsert`, auto-detecting
per-element-object vs. parallel-column row shapes
- Search: query a struct sub-vector via `annsField:
"<struct>.<subField>"`,
with dim/type validated against the schema
- Reject `nullable` / `defaultValue` on sub-fields (not supported by
engine)
## Test plan
- [x] Unit tests in `struct_array_v2_test.go` cover schema validation,
row parsing for both shapes, and search request building
- [ ] Manual e2e via curl against a local standalone
issue: #45496
---------
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
## Summary
- Expand Python MilvusClient external_table E2E coverage for schema
validation, lifecycle, DQL, refresh, indexing, read-only/write-blocked
behavior, and multi-format datasets.
- Cover parquet, lance-table, iceberg-table, and vortex basic format
flows with shared data builders and stricter assertions.
- Add REST v2 external collection coverage for external job APIs,
nullable vectors, custom database flow, and format matrix behavior.
- Move Vortex data generation to the Python 3.12 test environment and
remove the old sidecar helper scripts.
- Add Python 3.12 environment setup in `tests/scripts` so E2E scripts
create and activate the expected test runtime.
## Notes
- Vortex full-matrix coverage excludes fields that the current
`vortex-data==0.56.0` writer or Milvus refresh path cannot represent
consistently (`FixedSizeBinary`-backed vector fields and known
string/JSON fields).
- Chaos fault-injection coverage is included in the PR but was not run
against the shared validation instance because it mutates the
deployment.
## Validation
- `uv venv -p 3.12 .venv`
- `uv pip install --python .venv/bin/python -r requirements.txt`
- `pytest -q -x -n 6 milvus_client/test_milvus_client_external_table.py
--host 10.100.36.207 --port 19530 --minio_host 10.100.36.172
--minio_bucket external-table-pr49378-master`
- `162 passed, 2 skipped in 176.78s`
- `pytest -q -x -n 6 testcases/test_external_collection_operations.py
--endpoint http://10.100.36.207:19530 --token root:Milvus --minio_host
10.100.36.172 --bucket_name external-table-pr49378-master`
- `22 passed, 4 xfailed in 66.04s`
- `git diff --check`
---------
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Summary
- Align snapshot wrapper and test call signatures with the latest
PyMilvus snapshot API.
- Update Python client test dependency to `pymilvus==3.1.0rc5`.
- Make snapshot test cleanup safe under `pytest -n 6` by isolating
cleanup lists and tracking RBAC users/roles per test.
- Skip the `CreateSnapshot`/`DropCollection` race case and link it to
#49761.
- Remove stale xfail marks for snapshot RBAC privilege group cases now
covered by the fixed server behavior.
## Test plan
- `.venv/bin/python -m py_compile base/client_base.py
base/client_v2_base.py milvus_client/test_milvus_client_snapshot.py`
- `git diff --check`
- `.venv/bin/python -m pytest -q
milvus_client/test_milvus_client_snapshot.py --dist loadgroup -n 6
--host 10.100.36.198 --port 19530 --tb=short`
- `119 passed, 1 skipped in 293.48s`
- Cleanup audit after the full run:
- `collections []`
- `users ['root']`
- `roles ['admin', 'public']`
- `databases ['default']`
---------
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
## Summary
End-to-end correctness tests for struct array filter expressions in both
`query()` and `search()`, covering:
- **Index access** `structA[i][sub_field]` — newly introduced by PR
#48987
- **`array_contains` / `array_contains_all` / `array_contains_any`** —
exact
ground-truth `==` assertions to detect false negatives (the existing
Test Case 1 only checks no-false-positives via per-hit walking)
- **`array_length(structA[sub_field])`** — positive cases plus rejection
of `array_length(structA[i][sub_field])` (grammar disallowed)
## Why
Every case asserts **exact ID sets** (no false negatives AND no false
positives) instead of subset / per-hit checks, so any drop in recall or
spurious match fails the test. Coverage spans:
- All comparison operators (`==`, `!=`, `>`, `>=`, `<`, `<=`)
- `IN` / `NOT IN`
- Forward / reverse range syntax (`a < x < b` and `b > x > a`)
- String sub-field
- Compound predicates across positions (`structA[0]... &&
structA[1]...`)
- Doc-level filter combination
- Out-of-bounds index (rows shorter than the requested index)
- Parser-error negative cases (missing field, swapped form, indexed-form
passed to `array_length`)
- **Per-segment consistency**: mirrored controlled rows in sealed vs
growing produce mirrored result sets
- **After `delete`** and **after `upsert`** (both matching and
non-matching rows, cross-segment)
## Layout
- query: `TestMilvusClientStructArrayIndexAccess` (19 cases) in
`test_milvus_client_struct_array_element_query.py`
- search: `TestMilvusClientStructArrayIndexAccessSearch` (12 cases) in
`test_milvus_client_struct_array_element_search.py`
- uses FLAT index + controlled vector layout (controlled rows share
the query vector at COSINE=1.0) so search recall is 1.0 and exact
ID set assertions are valid
Each case uses **>= 3500 rows** (3000 sealed + 500 growing inert
background plus controlled rows split across both segments).
## Test plan
- [x] verified locally on a Milvus standalone instance built from PR
#48987 (commit `0c5a964ea0`): **31 / 31 passed in 387.54s**
- [x] each case scoped to `id < 100` so inert background rows cannot
contaminate strict-equality assertions
issue: #42148
---------
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
## Summary
- Add 40 E2E tests for FileResource CRUD APIs and analyzer integration
(jieba, synonym, stop words, decompounder)
- Verify tokenization correctness with baseline comparison (with vs
without file resources)
- Test data generated at runtime, no static files committed
/kind improvement
---------
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
#### What this PR does / why we need it:
Remove unused GitHub workflow files that are either disabled or related
to legacy chaos/deploy automation, and clean up README links that
pointed to removed workflows.
Removed workflows:
- `.github/workflows/delete_comments.yaml`
- `.github/workflows/deploy-test.yaml`
- `.github/workflows/io-latency-chaos-test.yaml`
- `.github/workflows/mem-stress-chaos-test.yaml`
- `.github/workflows/network-latency-chaos-test.yaml`
- `.github/workflows/network-partition-chaos-test.yaml`
- `.github/workflows/pod-failure-chaos-test.yaml`
- `.github/workflows/pod-kill-chaos-test-kafka-version.yaml`
- `.github/workflows/pod-kill-chaos-test.yaml`
- `.github/workflows/simd-compatibility-test.yaml`
#### Which issue(s) this PR fixes:
issue: #49574
#### Special notes for your reviewer:
This is a GitHub Actions cleanup only. The removed workflow file names
no longer have references under `.github` or `tests`.
#### Verification:
- `git diff --check`
- `rg -n
"delete_comments|deploy-test\.yaml|io-latency-chaos-test|mem-stress-chaos-test|network-latency-chaos-test|network-partition-chaos-test|pod-failure-chaos-test|pod-kill-chaos-test|simd-compatibility-test"
.github tests` returned no matches
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
## Summary
Fix false `Row count mismatch` failures in `SnapshotRestoreChecker` by
reordering the state capture to happen **after** snapshot creation
instead of before.
**Root cause**: The checker captured `count(*)` at timestamp T1, then
created the snapshot at T2 > T1. Under Milvus's streaming architecture,
data that was flushed but not yet visible at T1's guarantee timestamp
could be captured by the snapshot at T2, causing `actual > expected`
mismatches (typically +3 to +16 rows).
**Fix**: Move `_capture_snapshot_state()` to after `create_snapshot()`.
Since:
1. The checker uses an independent collection with no concurrent DML
after flush
2. The count query's guarantee timestamp T2 >= snapshot's timestamp T1
3. No new writes happen between snapshot and count
The count will exactly match the snapshot's data.
**Evidence from chaos-test-cron #24383**: 10 mismatch instances occurred
both before and after chaos injection, confirming this is a checker
timing issue, not a server bug:
```
04:44:54 expected=2960 actual=2970 diff=+10
04:48:08 expected=2990 actual=3001 diff=+11
05:19:02 expected=2991 actual=2998 diff=+7
05:22:30 expected=2976 actual=2992 diff=+16
...
```
## Test plan
- [ ] Run chaos test with `pod-failure` chaos type and verify no `Row
count mismatch` errors from `SnapshotRestoreChecker`
- [ ] Verify `SnapshotRestoreChecker` still correctly detects real data
corruption (e.g., by manually injecting a mismatch)
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
issue: #49163
## Summary
Brings the Go SDK (`client/v2`) struct array feature to parity with
pymilvus and fixes several client-side correctness issues found in the
master surface.
**New capabilities**
- Vector-array column types for struct sub-fields:
`ColumnFloatVectorArray`, `ColumnFloat16VectorArray`,
`ColumnBFloat16VectorArray`, `ColumnBinaryVectorArray`,
`ColumnInt8VectorArray` (new `client/column/vector_array.go`). Response
decoding goes through a shared generic helper with nil-row +
length-multiple-of-width validation.
- EmbeddingList search types: `entity.FloatVectorArray` + new
`Float16VectorArray` / `BFloat16VectorArray` / `BinaryVectorArray` /
`Int8VectorArray`, each dispatched to the correct
`PlaceholderType_EmbList*` in `vector2Placeholder`. Unblocks MAX_SIM /
embedding-list search against struct-array vector sub-fields from Go.
- Row-based insert helper
`(*columnBasedDataOption).WithStructArrayColumn(name,
*entity.StructSchema, []map[string]any)` that infers sub-column types
from the schema.
**Correctness fixes**
- `WithStructArrayColumn` now records construction errors on a
`deferredErr` field and surfaces them through `InsertRequest` /
`UpsertRequest` instead of panicking in the builder chain.
- `StructSchema.Validate(parent)` + `Schema.Validate()` mirror pymilvus
`StructFieldSchema._check_fields` (reject nested `ARRAY` /
`ARRAY_OF_VECTOR` / `STRUCT` sub-fields, `primary_key`, `partition_key`,
`clustering_key`, `nullable`, `auto_id`, `dynamic`, `default_value`,
nested struct schema). `Client.CreateCollection` calls `Validate()` via
interface assertion so bad schemas are rejected before the RPC.
- `columnStructArray.Len()` requires equal lengths across sub-fields and
panics loudly on drift, preventing silently malformed insert payloads.
- `columnStructArray.AppendValue` is now atomic: on sub-field error,
earlier sub-columns are rolled back to their pre-call length via
`Slice()` so the struct stays in lock-step.
- `parseVectorArrayData` validates nil `VectorField` rows, checks
`len(payload) % width == 0` and rejects `dim % 8 != 0` for binary
vectors. The 5 per-type branches are deduplicated via a generic
`splitVectorArrayRows` helper.
## Test plan
- [x] `go test -tags dynamic,test -gcflags="all=-N -l" -count=1 ./...`
in `client/` (all green): new unit tests in `entity/vectors_test.go`,
`entity/field_test.go`, `column/struct_test.go`,
`milvusclient/write_option_test.go` cover each behavior above.
- [x] End-to-end smoke test against a live Milvus server
(`chaos-testing/struct-array-test`, image
`feat-rest-v2-struct-array-20260420-0c5a964`) covering:
- [x] `Schema.Validate()` rejects a nested-struct sub-field
- [x] `Client.CreateCollection` pre-flight rejects the same schema
before the RPC
- [x] Valid schema create / `DescribeCollection` round-trip preserves
`max_capacity` on sub-fields and correctly unwraps `ArrayOfVector` back
to `FloatVector` with `dim` preserved
- [x] `Insert` via `WithStructArrayColumn` with mixed scalar + vector
sub-fields
- [x] Builder no longer panics; missing-key deferred error surfaces on
`InsertRequest`
- [x] `Flush` / `CreateIndex(vec)` / `CreateIndex("clips[clip_emb]")`
(MAX_SIM_COSINE) / `LoadCollection`
- [x] `Query` with `filter=id<3` returns expected rows
- [x] `Search` with `entity.FloatVectorArray` (EmbList Float
placeholder) returns hits on `anns_field="clips[clip_emb]"`
- [x] Integration test cases under `tests/go_client/testcases/`
(struct_array / struct_array_element_*) — added as new test harness;
requires a struct-array-capable server to exercise.
---------
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
## Summary
- Add 15 new test cases for snapshot feature covering lifecycle edge
cases, alias support, and cross-database restore
- Verify fix for #47578 (DropSnapshot during active restore)
- All tests verified on latest master image
(`master-20260224-091b147e-amd64`)
### New Test Classes
| Class | Tests | Level | Description |
|-------|-------|-------|-------------|
| `TestMilvusClientSnapshotDropInvalid` | +1 | L1 | Drop during active
restore (#47578) |
| `TestMilvusClientSnapshotLifecycle` | +8 | L2/L3 | Collection
lifecycle + snapshot interactions |
| `TestMilvusClientSnapshotAlias` | +6 | L2 | Snapshot ops via
collection aliases |
### Test Details
**Lifecycle tests (L2):**
- Drop target collection during restore
- Rename source collection after snapshot creation
- Create snapshot on collection being restored to
- Restore failure leaves no resource leak
- Concurrent drop of same snapshot (idempotent)
- Create snapshot during drop of source collection
- Cross-database snapshot restore
- Drop and restore race condition (L3)
**Alias tests (L2):**
- Create snapshot via alias → describe shows real collection name
- List snapshots via alias == list via real name
- Restore from alias-created snapshot with data integrity
- List restore jobs via alias on restored collection
- Dropped alias → create snapshot fails with "not found"
- Alter alias retarget → list snapshots reflects new target
## Test plan
- [x] All 14 L1/L2 tests pass on standalone
(master-20260224-091b147e-amd64)
- [x] L3 race test works but may timeout on standalone due to resource
exhaustion
---------
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
## Summary
- Add `tests/pyproject.toml` (+ `tests/.python-version`,
`tests/uv.lock`) hosting a
shared ruff configuration that covers all Python under `tests/`
(python_client,
restful_client, restful_client_v2, benchmark, scripts). uv only manages
the ruff
toolchain; each sub-directory keeps its own `requirements.txt` for
runtime deps.
- Register ruff `ruff-check` / `ruff-format` pre-commit hooks scoped to
`tests/**/*.py`, so only staged new/modified code is checked locally.
- Add `.github/workflows/python-lint.yaml`: on pushes to master and on
PRs touching
Python files under `tests/`, run `ruff check` and `ruff format --check`
against
**only the PR-diff changed files**. This intentionally avoids blocking
on the
existing baseline.
This PR only lands the baseline configuration. The 316 existing `.py`
files under
`tests/` are untouched — no mass reformat — and will be cleaned up
incrementally in
follow-up PRs per sub-directory.
Enabled ruff rules: `E`, `F`, `W`, `I`, `UP`. Target Python: `3.10`,
line length `120`.
## Test plan
- [ ] `cd tests && uv sync && uv run ruff --version` works
- [ ] `uv run ruff check scripts/` and `uv run ruff format --check
scripts/` run successfully
- [ ] pre-commit `ruff-check` / `ruff-format` hooks fire on a modified
`tests/**/*.py`
- [ ] The new `Python Lint (tests/)` GitHub Actions job runs green on
this PR
(no `.py` under `tests/` changed, so it should short-circuit to success)
- [ ] Existing CI (Code Checker, E2E, etc.) is unaffected
---------
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
## Summary
- Remove `tests/benchmark/` — long-unmaintained benchmark framework, no
CI references.
## Test plan
- [ ] PR/Nightly CI pipelines pass (no references to removed paths
remain).
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
## Summary
- Refactor snapshot test cases to align with PR #48143 (bind snapshot
lifecycle to collection)
- Update base class helpers: add `collection_name` to
`drop_snapshot`/`describe_snapshot`, add `source_collection_name` to
`restore_snapshot`
- Remove 18 `@pytest.mark.skip` decorators since pymilvus SDK now
supports collection_name
- Update all snapshot API calls (58 drop, 4 describe, 46 restore) to
pass required collection_name
- Rewrite `test_snapshot_list_after_drop_collection` →
`test_snapshot_cascade_delete_on_drop_collection` for new cascade delete
semantics
- Add 3 comprehensive data type tests: struct array + BM25 text match +
MinHash, BFloat16Vector, all 8 array element types
## Test plan
- [x] L0: 2 passed (basic lifecycle + restore)
- [x] L1: 19 passed (invalid params + list/describe + restore state)
- [x] L2: 49 passed (data types, indexes, partitions, concurrency, data
ops, collection properties)
- [x] All 70 tests pass against master-20260416-e87cc36e (the commit
that introduced collection-scoped snapshots)
---------
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Remove per-geometry debug spam in geometry client tests, replace prints
with log.debug, and drop large result dumps in upsert test. Warnings and
errors on invalid WKT are unchanged.
/kind improvement
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
## Summary
- Add multi-replica chaos test (`test_chaos_apply_multi_replicas.py`)
that uses resource group (RG) label-based pod targeting for chaos
injection
- Enhance `test_chaos_multi_replicas.py` with periodic chaos injection
mode for long-run stability testing
- Add round-robin chaos injection strategy and conftest support for
RG-based chaos scenarios
- Update checker.py with streamingnode support in RG chaos targeting
## Test plan
- [ ] Run multi-replica chaos test with 2+ replicas across resource
groups
- [ ] Verify periodic chaos injection mode with configurable intervals
- [ ] Validate round-robin pod selection for chaos injection
---------
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
## Summary
- Set `consistency_level="Strong"` at collection creation time in chaos
`checker.py` (9 places) to ensure data correctness during chaos testing
- Add explicit `consistency_level="Strong"` to all search/query calls in
`test_all_collections_after_chaos.py` (8 places) since it operates on
pre-existing collections
## Test plan
- [x] Pod logs confirm `[consistency_level=Strong]` at CreateCollection
- [x] `describe_collection` returns `consistency_level: 0` (Strong)
- [x] Insert-then-query returns all data immediately without explicit
consistency_level param
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Summary
- Increase the default `maxVectorFieldNum` from **4** to **10** to
accommodate the growing variety of vector types (dense, sparse,
function-based) supported by Milvus
- Update related test constants and hardcoded values in both Go and
Python test suites
## Changes
- `configs/milvus.yaml`: default config value 4 → 10
- `pkg/util/paramtable/component_param.go`: Go param default "4" → "10"
- `tests/go_client/common/consts.go`: Go test constant 4 → 10
- `tests/python_client/common/common_type.py`: Python test constant 4 →
10
- `tests/python_client/milvus_client/test_milvus_client_collection.py`:
replace hardcoded "4" in error message with constant reference
Closes#47402
## Test plan
- [x] Verify collection creation with up to 10 vector fields succeeds
- [x] Verify collection creation with 11+ vector fields fails with
proper error message
- [x] Run existing Go integration tests (`tests/go_client`)
- [x] Run existing Python client tests (`tests/python_client`)
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
## Summary
- Split snapshot testing into two checkers to fix row count mismatch
failures in chaos tests
- **SnapshotChecker** (`Op.snapshot`): lightweight, shares collection
with other checkers, only verifies snapshot create/restore operations
succeed
- **SnapshotRestoreChecker** (`Op.restore_snapshot`): uses independent
collection with internal DML operations, verifies data correctness (row
count + PK) after restore
- Removed `_snapshot_lock` from `Checker` class and all DML checkers
(Insert/Upsert/Delete) to eliminate coupling
## Root Cause
The `SnapshotRestoreChecker` shared a collection with other concurrent
checkers and relied on a global `_snapshot_lock` to prevent data
modifications during snapshot creation. However, some code paths (e.g.,
`Checker.insert_data()` base method,
`DeleteChecker.update_delete_expr()` refill logic) bypassed the lock,
causing row count mismatches after restore. The delta was exactly 3000
(`DELTA_PER_INS`).
Failure log: `expected=158188, actual=155188` (delta=3000)
ref:
https://qa-jenkins.milvus.io/blue/organizations/jenkins/chaos-test-cron/detail/chaos-test-cron/23943/pipeline
## Test plan
- [ ] Run chaos-test-cron pipeline and verify `Op.snapshot` (shared
collection) passes
- [ ] Verify `Op.restore_snapshot` (independent collection) passes with
no row count mismatch
- [ ] Confirm other checkers (insert/upsert/delete) are not impacted by
lock removal
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
## Summary
- Convert `targetSize` from MB to bytes before comparing with
`configMaxSize` in `triggerOneCollection`
- Update error message to display values in MB for better user
understanding
- Update `expectedTargetSize` in `ForceMergeSegmentView` to use
converted bytes value
- Update test cases to use MB values for `targetSize` parameter
- Add new test case `TestTriggerOneCollection_TargetSizeTooSmall` for
boundary validation
## Root Cause
According to the ForceMerge design doc, `targetSize` is specified in MB
(e.g., `target_size=2048` means 2GB). However, `configMaxSize` from
`getExpectedSegmentSize()` returns bytes. The direct comparison caused
valid requests like `compact(target_size=32)` to fail when
`configMaxSize` was 16MB (16777216 bytes).
## Test plan
- [x] Verify existing unit tests pass with updated `targetSize` values
- [x] Verify new boundary test
`TestTriggerOneCollection_TargetSizeTooSmall` passes
- [x] Integration test: Deploy Milvus with `dataCoord.segment.maxSize:
16`, call `compact(target_size=32)` should succeed
issue #47195
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
---------
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
## Summary
- Add mock TEI server utility (`common/mock_tei_server.py`) for testing
text embedding functions
- Add test cases for PR #46984 fix (alter function when other function
is invalid)
## Test Cases
1. `test_alter_function_when_other_function_is_invalid` - Verify that
altering a valid function succeeds even when another function in the
collection is invalid
2. `test_alter_invalid_function_to_valid_endpoint` - Verify that users
can fix an invalid function by altering it to a valid endpoint
## Related Issues
- Issue: https://github.com/milvus-io/milvus/issues/46949
- Fix PR: https://github.com/milvus-io/milvus/pull/46984
## Test Plan
- [x] Verified test fails with unfixed Milvus
(`master-20260112-7bcd3b10`)
- [x] Verified test passes with fixed Milvus (`master-20260112-b39ecb0`)
---------
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
## Summary
- Add `SnapshotRestoreChecker` to test snapshot restore functionality in
chaos testing
- Support concurrent execution with other checkers (shared collection)
- Use `self.get_schema()` to get latest schema for DML operations
- Simplified data verification for concurrent scenarios
## Test plan
- [x] Single run test passed
- [x] Concurrent operation test passed (100% success rate)
- [x] Added to test_concurrent_operation.py
- [x] Added to test_single_request_operation.py
---------
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
## Summary
- Add comprehensive test cases for Milvus snapshot API functionality
- Cover snapshot create, list, describe, drop, restore operations
- Test various data types, index types, partition scenarios
- Add concurrent operation and data integrity tests
- Update pymilvus to 2.7.0rc122 for snapshot API support
## Test Coverage
- L0 smoke tests: basic snapshot lifecycle
- L1 tests: data types, partitions, indexes
- L2 tests: boundary conditions, negative cases, concurrency
## Related Issue
https://github.com/milvus-io/milvus/issues/44358
## Test plan
- [x] Run tests with `-n 6` parallel execution
- [x] Verify all tests pass (67 tests: 62 passed, 4 xfail)
---------
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
## Summary
Add INFO level logging to the `adaptiveGroupSegments` function to show
which grouping algorithm is selected during force merge operations:
- `maxFullSegmentsGrouping`: Used when `segmentCount <=
maxFullSegmentThreshold` (DP algorithm, O(n²))
- `largerGroupingSegments`: Used when `segmentCount >
maxFullSegmentThreshold` (Greedy algorithm, O(n))
### Log output example:
```
[INFO] ["adaptiveGroupSegments: using maxFullSegmentsGrouping algorithm"] [segmentCount=5] [threshold=10] [targetSize=67108864]
```
### Why this change?
This improves observability for force merge operations and helps:
1. Verify that `maxFullSegmentThreshold` configuration is working
correctly
2. Debug unexpected compaction behavior
3. Monitor and understand compaction performance characteristics
issue: #47209
## Test plan
- [ ] Verify log output appears when triggering force merge with segment
count <= threshold
- [ ] Verify log output appears when triggering force merge with segment
count > threshold
- [ ] Verify segmentCount, threshold, and targetSize values are logged
correctly
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
## Summary
- Remove xfail marker from
`test_import_struct_array_with_local_bulk_writer` as the related
pymilvus issue is fixed
- Add test coverage for empty struct_array in bulk writer tests
## Changes
1. Remove `@pytest.mark.xfail(reason="issue:
https://github.com/milvus-io/pymilvus/issues/3050")` marker
2. Add 10% probability to generate empty `struct_array` (`[]`) in test
data
3. Add logging for empty array count to track test coverage
## Related PRs
- pymilvus fix: https://github.com/milvus-io/pymilvus/pull/3182
## Test Plan
- [x] Tested with Milvus 2.6.8 and pymilvus fix
- [x] Both PARQUET and JSON formats pass
- [x] Empty struct arrays are correctly written, imported, and queried
---------
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
/kind improvement
The test was flaky because sort compaction may not complete within the
sleep time. Instead of relying on sleep, wait for index building to
complete after flush, which ensures sort compaction is done before
getting binlog files.
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
/kind improvement
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
- Core invariant: tests' persistence of EventRecords and RequestRecords
must be append-safe under concurrent writers; this PR replaces Parquet
with JSONL and uses per-file locks and explicit buffer flushes to
guarantee atomic, append-safe writes (EventRecords uses event_lock +
append per line; RequestRecords buffers under request_lock and flushes
to file when threshold or on sink()).
- Logic removed/simplified and rationale: DataFrame-based parquet
append/read logic (pyarrow/fastparquet) and implicit parquet buffering
were removed in favor of simple line-oriented JSON writes and explicit
buffer management. The complex Parquet append/merge paths were redundant
because parquet append under concurrent test-writer patterns caused
corruption; JSONL removes the append-mode complexity and the
parquet-specific buffering/serialization code.
- Why no data loss or behavior regression (concrete code paths):
EventRecords.insert writes a complete JSON object per event under
event_lock to /tmp/ci_logs/event_records_*.jsonl and get_records_df
reads every JSON line under the same lock (or returns an empty DataFrame
with the same schema on FileNotFound/Error), preserving all fields
event_name/event_status/event_ts. RequestRecords.insert appends to an
in-memory buffer under request_lock and triggers _flush_buffer() when
len(buffer) >= 100; _flush_buffer() writes each buffered JSON line to
/tmp/ci_logs/request_records_*.jsonl and clears the buffer; sink() calls
_flush_buffer() under request_lock before get_records_df() reads the
file — ensuring all buffered records are persisted before reads. Both
read paths handle FileNotFoundError and exceptions by returning empty
DataFrames with identical column schemas, so external callers see the
same API and no silent record loss.
- Enhancement summary (concrete): Replaces flaky Parquet append/read
with JSONL + explicit locking and deterministic flush semantics,
removing the root cause of parquet append corruption in tests while
keeping the original DataFrame-based analysis consumers unchanged
(get_records_df returns equivalent schemas).
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
/kind improvement
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Tests**
* Updated test suite dependencies to pymilvus 2.7.0rc91.
* Enhanced text highlighting validation in test checkers.
<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
/kind improvement
- Add normalize_error_message function to extract and normalize error
text
- Collect unique error messages during chaos test execution
- Display error details in assertion messages for better debugging
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
/kind improvement
Replace direct self.schema access and describe_collection() calls with
get_schema() method to ensure consistent schema handling with complete
struct_fields information. Also fix FlushChecker error handling and
change schema log level from info to debug.
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
- Refactor connection logic to prioritize uri and token parameters over
host/port/user/password for a more modern connection approach
- Add explicit limit parameter (limit=5) to search and query operations
in chaos checkers to avoid returning unlimited results
- Migrate test_all_collections_after_chaos.py from Collection wrapper to
MilvusClient API style
- Update pytest fixtures in chaos test files to support uri/token params
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
