issue: https://github.com/milvus-io/milvus/issues/50461
ref: https://github.com/milvus-io/milvus/issues/42148
Issue: nullable VectorArray chunks store non-null rows in a compact
physical layout, so logical row indexes can diverge from physical row
indexes when missing/null rows exist. Some access paths use
logical-to-physical translation when reading the vector-array data, but
the offsets returned by VectorArrayOffsets() were still indexed by the
compact physical layout. Sealed segment vector search used logical row
indexes to read those offsets directly, without applying the same
logical-to-physical translation. This could make it read the wrong row
span or read past the valid offsets range, resulting in empty search
results or std::bad_alloc.
Fix: build VectorArrayChunk offsets by logical row count instead of
physical row count. The offsets array now has row_nums + 1 entries and
can be indexed directly by logical row id. Null rows and empty rows both
have a zero-length span, while the existing Chunk valid bitmap preserves
the distinction between null and non-null empty rows.
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
- Add a refreshable proxy.maxCollectionDescriptionLength setting with a
default 1024-byte limit and document it in milvus.yaml.
- Validate collection descriptions during CreateCollection and
AlterCollection PreExecute, including duplicate alter properties, while
leaving existing metadata loading unchanged.
- Document that restore or replication flows that recreate collections
through CreateCollection must satisfy the configured limit or raise it
first.
- Cover byte-length boundaries, CJK byte overflow, create rejection,
alter rejection, refreshability, duplicate alter-property rejection, and
generated YAML consistency in tests.
related: #50173
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
## Summary
Restore the CSV branch in the nullable vector RemoteBulkWriter
regression test.
This removes the previous skip and the explicit `nullkey="null"`
workaround so the test now covers the default CSV import path for
nullable `FLOAT_VECTOR` values.
Fixes#49678
## Test Plan
- [x] `/Users/yanliang.qiao/fork/.venv-py312/bin/python -m pytest
testcases/test_bulk_insert.py::TestBulkInsertNullableVector::test_bulk_writer_nullable_float_vector
--collect-only -q`
- [x] `/Users/yanliang.qiao/fork/.venv-py312/bin/python -m pytest -q -s
'testcases/test_bulk_insert.py::TestBulkInsertNullableVector::test_bulk_writer_nullable_float_vector[60-32-4]'
--host 10.104.30.186 --port 19530 --minio_host 10.104.18.198
--minio_bucket yanliang-mas2`
Note: I also ran the full JSON/CSV/Parquet parametrized test against the
same temporary cluster. JSON and CSV passed; the Parquet parameter later
hit a `get_bulk_insert_state` RPC `DEADLINE_EXCEEDED` while polling
import state, unrelated to this CSV regression change.
Signed-off-by: Yanliang Qiao <yanliang.qiao@zilliz.com>
## Summary
- Bump **Go to 1.26.4** across CI workflows, Docker builders, RPM setup,
and the `go` directive in all 4 modules (+ examples).
- Refresh Go 1.26-compatible deps to latest: `sonic` v1.15.2, `mockey`
v1.4.6, `gopkg` v0.1.4, `sonic/loader` v0.5.1, `cpuid/v2` v2.3.0. sonic
1.14.x fails to build on the 1.26 compiler (bytedance/sonic#895).
- **datacoord tests:** replace the anonymous `struct{ Iface }` mockey
idiom with **named helper types**
(`embeddedHandler`/`embeddedBroadcastAPI`/`embeddedBroker`/`embeddedAllocator`).
## Why the datacoord test change
Go 1.26's `printf` vet pass (run automatically by `go test`) calls
`x/tools/refactor/satisfy.Finder` unconditionally and **panics
`(*ast.StructType)`** on a **method expression of `*struct{ Iface }`** —
an upstream Go toolchain bug. Only `internal/datacoord` used this mockey
idiom (31 sites in 2 test files; verified repo-wide it's the only
place), which is why only `build-ut-cov`/`ut-go` failed on
`internal/datacoord [build failed]`. Switching to a **named type** makes
the method-expression receiver an identifier instead of a struct
literal, so `satisfy.Finder` no longer panics. mockey usage and test
behavior are unchanged. Minimal repro: `type I interface{ Logf(string,
...any) }` + `var _ = (*struct{ I }).Logf` under `go 1.26` → `go vet`
panics; named type does not.
## Notes
- `.env` builder-image tag intentionally not hand-edited (auto-bumped
post-merge by `bump-builder-version`).
- The bug also warrants an upstream report to golang/go (reproducer
above).
## Test plan
- [x] `go mod tidy` clean across all four modules
- [x] sonic 1.15.2 + mockey compile on go 1.26.4
- [x] reproduced the vet panic; verified named-type method expression
avoids it; gofmt-clean
- [ ] Full CI (build / build-ut-cov / ut-go / ut-cpp / integration)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What this PR does
The `AllFiltered` / `NoFilter` fast-path probes in `OffsetMapping`
(`BitsetHasNoFilteredRows` / `BitsetFiltersAllRows`) scanned the filter
bitset **one byte at a time with an early-exit loop**, which cannot be
auto-vectorized. These probes run on the hot search path — once per
segment per filtered query on nullable vector columns — and must scan
the **full** bitset exactly when they succeed, which is the common case
for time/tenant-clustered data (most segments are all-filtered or
not-filtered-at-all).
### Changes
1. **bitset library**: rewrite `ElementWiseBitsetPolicy::op_all/op_none`
middle loops to AND/OR-reduce 64-byte blocks. The fixed-trip inner loop
auto-vectorizes (SSE2/NEON at baseline, AVX2 when enabled) and branches
once per block. Both the element-wise and the vectorized policies pick
this up (`VectorizedElementWiseBitsetPolicy` delegates these two ops),
so `CustomBitset/TargetBitmap::all()/none()` get faster too.
2. **`milvus::BitsetView`**: expose `all()` / `none()`, delegating to
the bitset library for the contiguous path and keeping the per-bit
fallback for `out_ids` views. Any future BitsetView consumer gets the
same primitive instead of hand-rolling scans.
3. **`OffsetMapping.cpp`**: delete the two hand-written scan helpers;
`ShouldSkipBitsetTransform` now uses `bitset.all()/none()` (semantics
preserved — the empty case is handled before the probes, as before).
### Benchmark (1M-bit full scan, op_none, x86-64)
| implementation | time | speedup |
|---|---|---|
| byte early-exit (current master) | 89.1 µs | 1.0x |
| word early-exit | 11.9 µs | 7.5x |
| **blocked 64B (this PR)** | **2.2 µs** | **40x** (~55 GB/s,
memory-bound) |
### Tests
- `Util_Segcore.BitsetViewAllNone`: sweep across byte-tail and 64-byte
block boundaries, trailing-garbage bits ignored, single-bit
perturbations at first/middle/last positions, empty-view semantics.
- `Util_Segcore.TransformBitsetAllFilteredAndNoFilterFastPaths`:
AllFiltered / NoFilter statuses on a non-8/64-aligned row count (130).
- Existing `BitsetTest` all/none coverage exercises the new blocked
loops for the `uint64_t` policy; standalone correctness sweep over
uint8/16/32/64 element types × sizes × offsets passed locally.
### Note
#50447 touches the same two OffsetMapping helpers (byte→word
early-exit). This PR deletes them entirely, so whichever merges second
has a trivial conflict; the OffsetMapping hunk of #50447 is superseded
by this change while its Expr.h optimizations are unaffected.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
issue: #50381
Search results can contain non-empty FieldsData even when there are no
matched rows, because FieldsData may only hold empty output-field
templates.
Use result IDs to detect zero-hit search results before running lexical
or
semantic highlight processing.
Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
#issue: #50182
Storage version compaction used to only care whether a segment had
reached the target storage version. That is not enough once StorageV3
segments can still have different column-group formats, because a
segment may already be V3 but still not match the format DataNode is
configured to write. This change lets DataCoord use the same compaction
path to refresh those segments by format as well.
The implementation adds a separate storage format compaction switch and
enables the policy when either version or format compaction is turned
on. For V3 segments, DataCoord now checks every field binlog format
against the configured target format, schedules only the segments that
actually need a rewrite, and keeps the existing health, L0, importing,
compacting, and snapshot-protection filters in place. External
collections are skipped so table-backed segments are not accidentally
pulled into this internal rewrite flow.
Segment metrics now include a segment format label in addition to state,
level, sorted status, and storage version. The metric update path also
tracks format transitions when binlogs are added or rewritten, including
legacy, unknown, and mixed-format cases, so DataCoord can report how
storage formats are distributed while segments move through refresh and
compaction.
---------
Signed-off-by: jiaqizho <jiaqi.zhou@zilliz.com>
## Summary
- Treat GPU CAGRA indexes with `adapt_for_cpu` enabled as CPU-adapted in
QueryNode resource checks.
- Apply the same GPU requirement check when estimating segment load
resources and collection GPU index flags.
- Keep the Milvus-side OpenBLAS Conan option needed by the current
Knowhere v3.0.3 dependency set.
issue: #49836
## Test plan
- [x] `git diff --check origin/master..HEAD`
- [x] `GO_DIFF_FILES="internal/querynodev2/segments/collection.go
internal/querynodev2/segments/collection_test.go
internal/querynodev2/segments/segment_loader.go
internal/querynodev2/segments/segment_loader_test.go" make fmt`
- [x] `make milvus-gpu > output.txt 2>&1`
- [x] `env
PKG_CONFIG_PATH=/home/ubuntu/data/skills/issue_review/milvus_issue_49836/milvus/internal/core/output/lib/pkgconfig
LD_LIBRARY_PATH=/home/ubuntu/data/skills/issue_review/milvus_issue_49836/milvus/internal/core/output/lib:/home/ubuntu/data/skills/issue_review/milvus_issue_49836/milvus/internal/core/output/lib64
go test -count=1 -tags test ./internal/querynodev2/segments -run
'TestGpuIndexRequiresGpu|TestCollectionManagerSuite/TestGpuIndexFlagWithCagraAdaptForCPU'`
Signed-off-by: marcelo-cjl <marcelo.chen@zilliz.com>
issue: #49046
## What
- Add Go SDK builders for search aggregation and top hits, including
validation and proto request wiring.
- Parse search aggregation responses from `AggBuckets`/`AggTopks` into
`ResultSet.AggregationBuckets`.
- Reject incompatible search options such as legacy group-by, offset,
and search iterator usage.
- Add unit tests, a Go SDK example, and Go E2E coverage for search
aggregation.
## Tests
- `cd client && go test ./... -count=1`
- `cd tests/go_client && GO_CLIENT_SKIP_EXTERNAL_TABLE_DEPS_INSTALL=true
go test ./testcases -run TestSearchAggregation -count=1 -args -addr
http://10.104.18.101:19530`
## Notes
- `make static-check` was attempted locally but blocked by missing LLVM:
`ERROR: Valid LLVM (14-18) not installed. Run: brew install llvm@17`.
Signed-off-by: Yanliang Qiao <yanliang.qiao@zilliz.com>
issue: https://github.com/milvus-io/milvus/issues/48957
## What changed
- Add streaming V3 `LoadEntries` paths for scalar indexes to avoid
reading full index entries into memory before materialization.
- Persist and load auxiliary metadata for sort/string indexes, including
offset maps, valid bitsets, and CSR arrays, so mmap paths can map
metadata directly and memory paths can preallocate buffers.
- Update scalar index resource estimates and related loading tests for
the streaming/mmap load behavior.
## Why
- Reduce peak memory during sealed scalar index load, especially for
large scalar sort, string, and bitmap indexes.
- Avoid recomputing persisted metadata at load time when V3 entries
already contain it.
---------
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
issue: #50344
## Summary
After a `force_promote`, the persisted `SalvageCheckpoint` was
unreachable from any client. Two compounding defects:
### Defect 1 — `GetReplicateInfo` returns early, never reaching
`GetSalvageCheckpoint`
`internal/proxy/impl.go`: the handler called `GetReplicateCheckpoint`
first and returned on any error. On a standalone-primary (post
`force_promote`) the WAL is no longer a secondary, so
`GetReplicateCheckpoint` fails with `STREAMING_CODE_REPLICATE_VIOLATION`
("wal is not a secondary cluster in replicating topology") and the
follow-up `GetSalvageCheckpoint` — the whole point — was never reached.
**Fix:** treat **only** `REPLICATE_VIOLATION` as non-fatal — leave the
live checkpoint `nil` and continue to the salvage checkpoint. All other
errors stay fatal (no blanket swallow).
### Defect 2 — client retries `REPLICATE_VIOLATION` to the deadline
`internal/streamingnode/client/handler/handler_client_impl.go`:
`createHandlerAfterStreamingNodeReady` retried the (permanent) violation
in a backoff loop until the caller's context cancelled, so it surfaced
as `DEADLINE_EXCEEDED` rather than a typed error.
**Fix:** return immediately on
`status.AsStreamingError(err).IsUnrecoverable()` — a category that
**already** includes replicate violation (and is documented as "Stop
resuming retry and report to user"). The retry loop simply wasn't
honoring it.
Fixing Defect 2 is also what lets Defect 1 detect the violation:
otherwise the error would have been swallowed into a deadline before the
proxy could classify it.
## Changes
- `internal/proxy/impl.go`: `GetReplicateInfo` continues to
`GetSalvageCheckpoint` on `REPLICATE_VIOLATION`, returns
`checkpoint=nil` in that case.
- `internal/streamingnode/client/handler/handler_client_impl.go`: stop
retrying unrecoverable errors; return them within RTT.
- `internal/streamingnode/client/handler/handler_client_test.go`: add
`TestHandlerClient_GetReplicateCheckpointReplicateViolation` asserting
an immediate, typed return (no retry loop / `Watch` call).
## Test Plan
- [x] gofmt clean; `internal/util/streamingutil/status` builds
- [x] New unit test mirrors the existing handler-client mock harness
- [ ] CI: `TestHandlerClient_GetReplicateCheckpointReplicateViolation`
passes
- [ ] CI: `replication/data_salvage` integration suite —
`TestGetReplicateInfoOnPrimaryCluster` now returns cleanly on a primary
instead of erroring
> Note: the proxy / streamingnode packages can't be fully built locally
in this environment due to a pre-existing stale C++ artifact mismatch in
the unrelated `internal/storagev2/packed` cgo package (reproducible on a
clean master checkout). The changed packages are pure-Go and use
already-existing `StreamingError` APIs; CI compiles and runs them.
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Signed-off-by: Yihao Dai <yihao.dai@zilliz.com>
issue: #50445
## What this PR does
`UnaryElementFuncForMatch` constructed a `LikePatternMatcher` from the
expression's constant LIKE pattern inside the per-batch kernel.
`PhyExpr::Eval()` runs once per 8192-row batch, so a `LIKE` filter over
a 1M-row segment re-parsed the same pattern and reallocated the
matcher's token structures ~122 times.
Mirror the existing RegexMatch caching convention
(`cached_regex_matcher_` / `EnsureRegexCache()`):
- `UnaryElementFuncForMatch` gains a `const LikePatternMatcher* matcher`
member; `nullptr` falls back to local construction, so call sites that
are not wired keep their current behavior (same design as
`UnaryElementFuncForRegexMatch`).
- `PhyUnaryRangeFilterExpr` gains `cached_like_matcher_` +
`EnsureLikeMatcherCache()`, built once per segment.
- The data-path `Match` dispatch injects the cached matcher.
## Tests
- Incremental core build passes.
- `--gtest_filter="*Match*:*Like*:*StringExpr*"`: 221/221 passed (1
pre-existing skip: `RegexMatcherTest.NewLine`).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
## What this PR does
Make the bitset `Proxy::operator=(bool)` branchless. Found while
profiling random single-bit set/test performance.
The proxy assignment used `if (value) set(); else reset();`. When the
assigned value is data-dependent — expression-evaluation kernels writing
computed predicates per row (JSON / string / array / index fallback
paths, ~65% of expression eval) — the branch mispredicts on
mixed-selectivity data and dominates the cost of the bit write.
Replace the branch with a blend (expand the bool to an all-ones/zero
word, merge under the bit mask). Both original branches performed a
store, so the blend is semantically identical; the conditional-store
compound operators (`|=`, `&=`, `^=`) are intentionally left unchanged.
Microbenchmark, Xeon Gold 6338, 16.7M ops:
| scenario | before | after |
|---|---|---|
| random-index assign, 128KB bitset, 50/50 values | 157 Mops/s | 557
Mops/s (**3.5x**) |
| sequential assign (expr-eval pattern), 50/50 | 147 Mops/s | 257 Mops/s
(**1.75x**) |
| highly predictable values (99%+ same) | — | bounded ~15% regression |
| `set(i)` / `reset(i)` (compile-time constant) | — | identical codegen
(single `or`/`andn`, verified in asm) |
The trade-off is asymmetric: the branchy version's worst case
(unpredictable values) costs 144% extra, the blend's worst case (99%+
predictable runtime values) costs ~15% extra — bounded downside for a
large upside, the standard choice for vectorized engine kernels (cf.
X100/Vectorwise adaptive select).
Correctness: all 328 existing bitset unit tests pass; additionally
cross-validated against an independent reference implementation under
identical 16.7M-op random sequences.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
## What this PR does
Fixes a flaky failure of `TestIndexInspector_inspect/normal_test`
(observed in the `ci-v2/ut-go` run of #50438: `Enqueue: 0 out of 1
expectation(s) met`, `AlterSegmentIndexes: 1 out of 2`).
The test called `inspector.Start()` and pushed the notify **before**
registering the mock expectations. `Start()` synchronously runs
`reloadFromMeta()` and spawns the inspect loop (ticker + notify
channel), so the inspector goroutine can invoke `AllocID` /
`CreateSegmentIndex` / `Enqueue` while `EXPECT()` registration is still
in progress on the test goroutine. A call that lands on a mock with no
matching expectation aborts the indexing flow, the expected calls never
happen, and the test fails at teardown with unmet expectations.
Fix: register all expectations before `Start()`, so no inspector-side
call can race with registration.
## Tests
`go test -race -count=20 -run 'TestIndexInspector_inspect$'
./internal/datacoord/` — all pass.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
## Summary
issue: #50320
Clean downstream-only collections created during CDC force-promote
verification before restoring the topology back to `upstream ->
downstream`.
The affected tests create `*_after` collections while downstream is
force-promoted primary. If teardown restores `A -> B` first, downstream
becomes a replica again and `drop_collection` is rejected, leaving
`test_fp_*_after` collections that fail the final upstream/downstream
diff.
## Changes
- Add targeted cleanup for downstream-only force-promote verification
collections.
- Remove those collections from the generic cleanup list after the
targeted cleanup attempt.
- Add a focused unit test for the cleanup helper behavior.
## Test Plan
- [x] `PYTHONPATH=tests/python_client conda run -n milvus2 python -m
pytest -c /tmp/empty-pytest.ini
--confcutdir=tests/python_client/cdc/testcases
tests/python_client/cdc/testcases/test_force_promote_cleanup.py -q`
- [x] `conda run -n milvus2 python -m py_compile
tests/python_client/cdc/testcases/test_force_promote.py
tests/python_client/cdc/testcases/test_force_promote_cleanup.py`
- [x] `git diff --check`
- [ ] `make static-check` could not complete locally because the CGo
core artifacts in this workspace do not match the latest master headers.
It first failed without `milvus_core.pc`; with
`PKG_CONFIG_PATH=/Users/yihao.dai/workspace/milvus/internal/core/output/lib/pkgconfig`,
it failed at `internal/util/initcore` with missing
`C.CArrowReaderConfig` / `C.InitArrowReaderConfig`.
Signed-off-by: Yihao Dai <yihao.dai@zilliz.com>
issue: https://github.com/milvus-io/milvus/issues/50428
Replace the one-shot jemalloc allocated delta assertion in
TestExportSearchResultAsArrowRecordBatch_NoCMemoryLeak with the same
sliding-window sustained-growth check used by the sibling C memory leak
test.
This avoids false positives from global jemalloc noise while preserving
coverage for real repeated C heap growth across the Arrow export/release
path. Extract the shared jemalloc growth assertion into a helper so both
tests use the same thresholds and measurement strategy.
Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
## Summary
- Cherry-pick #50283 from 2.6 to master.
- Cache nullable vector valid_data sidecar for stream-loaded disk ANN
indexes so offset mapping can be built.
- Add C++ coverage for valid_data slice filtering/cache, multiple
valid_data slices, and stream load mapping.
Related PR: #50283
## Test plan
- [x] `git diff --check upstream/master..HEAD`
- [x] `make cppcheck`
- [ ] C++ unittest not run locally: clean workspace has no built
`internal/core/output/unittest/all_tests`; targeted C++ test needs core
unittest build.
Signed-off-by: marcelo-cjl <marcelo.chen@zilliz.com>
issue: https://github.com/milvus-io/milvus/issues/50291
## What changed
This PR reduces noisy copy segment restore logs by:
- suppressing empty copy segment job/task checker stats logs;
- removing the per-poll `copy segment task not completed` log;
- skipping failed-task cleanup for target segments already Dropped;
- replacing full copy segment source/target/result payload logs with
compact ids
and counters.
## Why
Copy segment polling can run frequently while there is no work or while
tasks
are still in progress. The previous logs were repetitive and sometimes
included
large protobuf payloads, making restore logs harder to scan.
## Validation
- `source ~/.profile && source scripts/setenv.sh && make milvus`
- `reset-milvus; go test -tags dynamic,test -gcflags="all=-N -l"
-ldflags="-r ${RPATH}" github.com/milvus-io/milvus/internal/datacoord
-run 'TestCopySegment|TestAssembleCopySegmentRequest' -count=1 -timeout
300s`
- `source ~/.profile && source scripts/setenv.sh && make lint-fix`
Full `internal/datacoord` coverage was also attempted locally and hit
the known
macOS-only DISKANN mismatch in
`TestCompactionTriggerManagerSuite/TestGetExpectedSegmentSize`.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
## Summary
- add a segmentID index to QueryCoord's SegmentDistManager node snapshot
- make WithSegmentID use the indexed lookup path before applying
remaining filters
- add coverage that segmentID lookups only visit matching segment
replicas
Signed-off-by: sunby <sunbingyi1992@gmail.com>
## What changed
- Add Python client, RESTful v2, and Go SDK regression coverage for
hybrid search on collections with `partitionkey.isolation=true`.
- Verify normal search still rejects unsupported partition-key isolation
filters:
- `tenant in ["tenant_a", "tenant_b"]`
- missing tenant filter
- Keep the current hybrid search server bug as conditional xfail/skip,
so baseline search regressions are not hidden.
- Add RESTful v2 coverage for MinHash + dense hybrid search with
explicit `partitionNames`.
## Issues
Related to #50398
Related to #50396
## Verification
- `ruff check
../tests/python_client/milvus_client/test_milvus_client_partition_key_isolation.py
../tests/restful_client_v2/testcases/test_vector_operations.py`
- `python -m py_compile
tests/python_client/milvus_client/test_milvus_client_partition_key_isolation.py
tests/restful_client_v2/testcases/test_vector_operations.py`
- `git diff --check`
- Python client 50398 case against `10.104.18.101`: `1 xfailed`
- RESTful v2 50398 case against `10.104.18.101`: `1 xfailed`
- Go SDK 50398 case against `10.104.18.101`: reproduces current bug and
conditionally skips
Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
## Summary
- update `test_milvus_client_collection_describe` to expect
`properties.max_field_id` in `describe_collection`
- align `search_aggregation` JSON/dynamic field rejection cases with the
current server error messages
- keep the affected Python client regressions covered without asserting
outdated responses
## Test plan
- [x] `python3 -m py_compile
tests/python_client/milvus_client/test_milvus_client_collection.py
tests/python_client/milvus_client/test_milvus_client_search_aggregation.py`
- [ ] full pytest execution was not run locally in this submission
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: nico <cheng.yuan@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
issue: https://github.com/milvus-io/milvus/issues/45881
## Summary
- Make external user vector fields nullable by default during schema
normalization.
- Keep external system/virtual fields and generated function output
fields
excluded from forced nullable normalization.
- Update schema, REST, and MilvusClient external table coverage so
omitted
vector nullable settings are expected to describe as nullable.
## Why
External collection schemas already make user scalar fields nullable by
default because external files can contain null values that Milvus
cannot
reject at collection creation time. Vector fields were temporarily
excluded
while the nullable vector offset mapping path still depended on eager
load-time mapping.
After lazy offset mapping for nullable vectors, that special-case
exclusion is
no longer needed. External vector fields can now follow the same default
nullable normalization rule as scalar fields.
Related lazy offset mapping PR:
https://github.com/milvus-io/milvus/pull/49168
## Validation
- `git diff --check`
- `python3 -m py_compile
tests/restful_client_v2/testcases/test_external_collection_operations.py
tests/python_client/milvus_client/test_milvus_client_external_table.py`
- `GOTOOLCHAIN=auto go test -tags dynamic,test
github.com/milvus-io/milvus/pkg/v3/util/typeutil -run
TestNormalizeAndValidateExternalCollectionSchema -count=1 -v`
- `GOTOOLCHAIN=auto go test -tags dynamic,test
github.com/milvus-io/milvus/pkg/v3/util/typeutil -count=1`
## Local environment note
- `/reset-milvus` reset etcd/minio before Go test runs, but this
worktree does
not have `bin/milvus`, so standalone Milvus could not be restarted
locally.
The validated tests above are pure schema/typeutil tests.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
https://github.com/milvus-io/milvus/issues/48794
Normalize credential names before building config keys so API key,
AK/SK, and GCP credential lookups work when names contain mixed-case
characters.
Add tests covering mixed-case credential names for all supported
credential types.
Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
Related to #50384
This increases standalone Milvus CPU requests from 2 cores to 3 cores in
E2E and nightly standalone Helm values.
On the Go SDK ARM CI nodes, allocatable CPU is about 7.91 cores. With a
2-core request, three standalone Milvus pods can be scheduled onto the
same 8-core node while each pod can burst up to an 8-core limit. Raising
the request to 3 cores prevents three standalone pods from fitting on
one node and reduces CPU contention during index build / snapshot
restore heavy tests.
Validation:
- `git diff --check origin/master..HEAD`
- Parsed all updated YAML values with Ruby `YAML.load_file`
- Confirmed all updated standalone CPU requests are `"3"`
Signed-off-by: Yanliang Qiao <yanliang.qiao@zilliz.com>
## What
- Set standalone Helm test resource requests to 2 CPU / 8Gi memory.
- Set standalone Helm test resource limits to 8 CPU / 16Gi memory.
- Add the same standalone resources to one-pod values that previously
omitted them.
## Related
Related #50384
## Verification
- ruby -ryaml validation for all 11 standalone values files
- git diff --check HEAD~1..HEAD
Signed-off-by: Yanliang Qiao <yanliang.qiao@zilliz.com>
issue: #49879
## What changed
- Add an independent Python client regression test for search with
`order_by_fields` plus `offset`, using controlled ANN candidates from
#49879.
- Mark the regression as strict xfail because current behavior still
applies offset before scalar ordering.
- Mark `search_aggregation` top_hits sorting by dynamic fields as strict
xfail because dynamic fields are explicitly rejected for search
aggregation today.
## Verification
```bash
/Users/yanliang.qiao/fork/.venv-py312/bin/python -m pytest -s --tb=short --host 10.104.18.101 --port 19530 \
milvus_client/test_milvus_client_search_order.py::TestMilvusClientSearchOrderIndependent::test_milvus_client_search_order_by_with_offset \
milvus_client/test_milvus_client_search_aggregation.py::TestSearchAggregationIndependent::test_search_aggregation_top_hits_sort_by_dynamic_field
```
Result: `2 xfailed in 13.46s`.
Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
issue: https://github.com/milvus-io/milvus/issues/47425
Milvus already asks Conan to build the C++ third-party stack with C++20
for the host profile, but Conan keeps the build profile separate. That
means packages used during the build can still fall back to the default
C++ standard, which caused protobuf and abseil to be compiled as gnu17
in some environments and fail during the dependency build.
This change passes C++20 to the Conan build profile as well, so both the
host context and the build context use the same language level. It
applies the setting to both Linux Conan install branches in the
third-party build script, including the gcc4 ABI path and the normal
libstdc++11 path.
Signed-off-by: jiaqizho <jiaqi.zhou@zilliz.com>
Co-authored-by: EC2 Default User <ec2-user@ip-10-15-6-84.us-west-2.compute.internal>
The boringcrypto-only cgo package pkg/util/fips/openssl_enabled.go calls
libcrypto symbols (OSSL_LIB_CTX_load_config,
EVP_default_properties_is_fips_enabled,
RAND_bytes) but declared no link dependency, so FIPS builds failed at
the Go
external-link step with "libcrypto.so.3: DSO missing from command line".
The link
only used to succeed when a transitive library re-exported those
symbols.
Declare the dependency in the package that needs it via `#cgo LDFLAGS:
-lcrypto`
rather than adding -lcrypto to CGO_LDFLAGS in the Makefile/environment.
A
package-scoped directive leaves the recorded CGO_LDFLAGS build setting
empty, so it
does not change the package hash of other cgo packages and does not
require every
Go plugin to be rebuilt with a matching flag to keep plugin.Open
working.
Verified on a real FIPS build: bin/milvus links libcrypto.so.3 directly,
while the
hook/autoindex/cipher plugins keep empty CGO_LDFLAGS, carry no libcrypto
dependency,
and pass the plugin.Open package-hash check.
issue: #50374
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
issue: #50379Fixes#50379
This PR narrows Kafka chaos selectors to broker pods by adding
`app.kubernetes.io/component: kafka`. The previous selector matched both
broker pods and the Kafka exporter pod because both use
`app.kubernetes.io/name: kafka`.
Changes:
- Add `app.kubernetes.io/component: kafka` to Kafka container-kill
chaos.
- Add the same broker selector to Kafka pod-kill chaos.
- Update Kafka pod-failure chaos from the legacy `component: kafka`
selector to `app.kubernetes.io/component: kafka`.
Verification:
- `git diff --check`
- Parsed the three Kafka chaos YAML files and verified each selector
contains `app.kubernetes.io/component=kafka` with no stale `component`
selector.
Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
issue: #50206
## Summary
- Support dynamically updating access log configuration (formatters,
methods, writer settings) without restarting, by watching all
`proxy.accessLog` config keys instead of only the enable switch.
- Add RESTful API paths (`/search`, `/query`, `/v2/vectordb/entities/*`)
to the default search/query formatter methods, so RESTful requests also
get detailed access log formatting.
- Fix resource leak: close old `RotateWriter` before re-initializing
when config changes.
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
## What was fixed
This fixes import two-phase commit timestamp handling so imported
segments keep the commit timestamp assigned by the import transaction
instead of being made visible at an unrelated later timestamp.
Changes included:
- Preserve import commit timestamps in DataCoord segment metadata.
- Broadcast import commit and rollback through DML channels so
downstream consumers observe the 2PC result consistently.
- Apply delete visibility in sealed segments based on the imported
segment commit timestamp.
- Carry the import transaction DML position needed for commit/rollback
propagation.
- Guard import in replicating clusters behind a disabled-by-default
config switch, and only allow `auto_commit=false` when that switch is
enabled.
- Downgrade only unsupported `GetSalvageCheckpoint` errors for
compatibility with older StreamingCoord implementations.
- Move import jobs to `Committing` before handling per-vchannel commit
acknowledgements, including the race where the vchannel commit is
handled before the DDL ack callback.
- Sync the generated `configs/milvus.yaml` entry for the new import
replication guard.
Fixes#48525
## Test plan
- `source ./scripts/setenv.sh && make SKIP_3RDPARTY=1
build-cpp-with-unittest`
- `make build-go`
- `make static-check`
-
`CLANG_FORMAT=/opt/homebrew/Caskroom/miniconda/base/envs/milvus2/bin/clang-format
make cppcheck`
- `make rustcheck`
- `make fmt`
- `go test ./cmd/tools/config -run TestYamlFile -count=1`
- `go test -tags dynamic,test -gcflags="all=-N -l" ./internal/datacoord
-run 'TestHandleCommitVchannel|TestCommitImportCallback' -count=1`
- Local import commit timestamp tests under
`~/workspace/snippets/tests/test_import_commit_ts`:
- `test_import_delete.py`
- `test_import_auto_commit.py`
- `test_import_basic.py`
- `test_import_mvcc.py`
- `test_import_insert_delete.py`
- `test_import_insert_delete_auto_id.py`
- `test_import_ttl.py`
- Local import + replication tests under
`~/workspace/snippets/tests/test_cdc/import`.
Local Python coverage used StorageV2. StorageV3 needs a separate run
with `common.storage.useLoonFFI=true`.
---------
Signed-off-by: Yihao Dai <yihao.dai@zilliz.com>
issue: #47598
## Summary
The `DumpMessages` streaming RPC (added in #47599 for data salvage) is
exposed by the proto and the low-level `milvuspb` stub, but the
high-level Go SDK client (`client/milvusclient`) had **no wrapper** —
unlike the sibling replicate APIs (`UpdateReplicateConfiguration`,
`GetReplicateConfiguration`, `GetReplicateInfo`,
`CreateReplicateStream`) that already live in `replicate.go`.
This adds `Client.DumpMessages`, mirroring `CreateReplicateStream`: it
returns the server-streaming client whose `Recv()` yields
`DumpMessagesResponse` frames (each carrying either an error status or a
non-system message).
For reference, the Python SDK already exposes this API in
milvus-io/pymilvus#3573.
## Changes
- `client/milvusclient/replicate.go`: add `Client.DumpMessages(ctx, req,
opts...) (milvuspb.MilvusService_DumpMessagesClient, error)`.
- `client/milvusclient/replicate_example_test.go`: add
`ExampleClient_DumpMessages` showing the salvage flow
(`GetReplicateInfo` → salvage checkpoint → `DumpMessages` → iterate to
`io.EOF`).
## Test Plan
- [x] `go build ./milvusclient/` and `go vet ./milvusclient/` pass
- [x] `go test ./milvusclient/ -run TestReplicate` passes
- [x] Example compiles (built as part of the package test binary)
- [x] gofmt clean
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
## Summary
Adds Go SDK RESTful wrappers for the Import 2PC commit/abort endpoints
introduced server-side in #48524, mirroring the existing
`GetImportProgress` wrapper in `client/bulkwriter/bulk_import.go`:
- `CommitImport` → `POST /v2/vectordb/jobs/import/commit`
- `AbortImport` → `POST /v2/vectordb/jobs/import/abort`
Both take a `jobId` (plus optional `clusterId`/`apiKey`) and delegate
status handling to the existing `ResponseBase.CheckStatus` helper. The
2PC opt-in (`auto_commit=false`) continues to flow through the existing
import-create `options` map, so no create-path change is needed.
issue: #48525
### Changes
**Modified:**
- `client/bulkwriter/bulk_import.go`: add
`CommitImportOption`/`AbortImportOption` (+ constructors, `WithAPIKey`,
`GetRequest`), `CommitImportResponse`/`AbortImportResponse`, and the
`CommitImport`/`AbortImport` functions.
- `client/bulkwriter/bulk_import_test.go`: add
`TestCommitImport`/`TestAbortImport` (normal / status-error /
server-closed cases).
### Test Plan
- [x] `go test ./bulkwriter/` — `TestBulkImportAPIs` suite passes (incl.
new `TestCommitImport`/`TestAbortImport`).
- [x] `gofumpt` clean; imports unchanged.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
When a QueryNode does not report `MaximumIndexVersion`, use its
`CurrentIndexVersion` as the conservative maximum for cluster-wide
vector index version resolution. This prevents `targetVecIndexVersion`
from advancing beyond old QueryNodes during rolling upgrade.
issue: #48470
## Tests
- `source ./scripts/setenv.sh && make SKIP_3RDPARTY=1
build-cpp-with-unittest`
- `source ./scripts/setenv.sh && make build-go`
- `source ./scripts/setenv.sh && LOCAL_STORAGE_SIZE=100 go test -v
-gcflags="all=-N -l" -tags dynamic,test ./internal/datacoord -run
"Test_IndexEngineVersionManager_(GetMaximumIndexEngineVersion|ResolveVecIndexVersion)"
-count=1 -ldflags="-r ${RPATH}"`
- `LOCAL_STORAGE_SIZE=100 ./scripts/run_go_unittest.sh -t datacoord`
---------
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
#48986
Replace C++ ReduceSearchResultsAndFillData with a Go reduce pipeline:
- Export per-segment search results as Arrow RecordBatch via C Data
Interface
- HeapMergeReduce: k-way heap merge with PK dedup and GroupBy support
- Late Materialization: single CGO call (FillOutputFieldsOrdered) for
output fields
- ExportSearchResultAsArrow supports extra field IDs for future L0
rerank
Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
## Summary
- Go 1.25.9 -> 1.25.10 and builder Dockerfiles now use Go 1.25.10. This
covers CVE-2026-33811, CVE-2026-33814, CVE-2026-39820, CVE-2026-39823,
CVE-2026-39825, CVE-2026-39826, CVE-2026-39836, CVE-2026-42499.
- golang.org/x/crypto -> v0.52.0 across root/pkg/client/tests_go_client.
This covers CVE-2026-39829, CVE-2026-39830, CVE-2026-39831,
CVE-2026-39832, CVE-2026-39833, CVE-2026-39834, CVE-2026-42508,
CVE-2026-46595, CVE-2026-46597.
- golang.org/x/net -> v0.55.0 across root/pkg/client/tests_go_client.
This covers CVE-2026-33814 and CVE-2026-39821.
- github.com/apache/thrift v0.20.0 -> v0.23.0, including
tests_go_client. This covers CVE-2026-41602.
- Updated tests/go_client/go.mod and go.sum so Go SDK CI has the
required x/* sum entries.
This PR is generated by the automated CVE fix task.
issue: https://github.com/milvus-io/milvus/issues/49032
## Test plan
- go mod tidy in pkg/
- go mod tidy in client/
- go mod tidy in tests/go_client/
- go mod tidy -e at repository root (standard tidy is blocked by
existing missing generated package cmd/tools/migration/legacy/legacypb)
- git diff --check
- go list -tags L0,L1,L2,test ./... in tests/go_client
- go list -m for updated dependencies in root/pkg/client/tests_go_client
- CI validates image build, Go SDK e2e, and scan
---------
Signed-off-by: Li Liu <li.liu@zilliz.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
