## Summary
Avoid direct captures of structured bindings in
`ChunkedSegmentSealedImpl.cpp`, which is compiled with OpenMP enabled.
- Use an init-capture for the field ID in `LoadBatchTextIndexes`.
- Use an init-capture for the field ID in `FinalizeLoadDiffForReopen`
when dropping JSON indexes.
- Keep the inner text-index commit lambda as a normal value capture
because it captures the ordinary init-capture from the outer lambda, not
a structured binding.
issue: #51231
## Test Plan
- [x] `make`
- [x] `ninja -C cmake_build
src/segcore/CMakeFiles/milvus_segcore.dir/ChunkedSegmentSealedImpl.cpp.o`
- [x] `git diff --check`
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
## Problem
During schema evolution, SQN can convert an insert payload with the old
schema and then race with a schema update before ProcessInsert creates
and writes the growing segment. If the old payload still contains a
dropped field, the native collection may already use the new schema and
segcore rejects the write with unexpected new field.
## Solution
- Add a collection-local schema-transition RW lock.
- Insert holds the reader lock from schema snapshot use through payload
conversion and delegator.ProcessInsert, so a growing segment is created
and written in the same schema epoch as its payload.
- Schema update and existing-collection load refresh paths take the
writer lock around native schema mutation and Go schema snapshot
publication.
- Acquire a temporary collection lease before releasing
collectionManager.mut. This keeps the collection alive while the writer
waits, without serializing unrelated collection-manager operations
behind a schema transition.
- Preserve dropped-field replay handling: when DDL has already
completed, payload conversion filters fields absent from the current
schema before inserting into a new-schema growing segment.
## Tests
- Add a deterministic native regression: pause after a v950 payload
containing field 580 is converted, queue the v951 drop-field update,
then resume a real NewSegment plus growing.Insert and verify both insert
and DDL succeed.
- Assert the transition reader covers both payload conversion and
ProcessInsert.
- Cover the DDL-first replay path, where field 580 is filtered before
native insertion.
- Cover both schema writer entry points: UpdateSchema and
existing-collection PutOrRef.
## Validation
- make static-check
- make build-go
- Native pipeline regression tests repeated 20 times with dynamic,test
- Collection schema-transition and lease tests repeated 20 times
- internal/querynodev2/pipeline and internal/querynodev2/segments
package tests
- gofumpt -l and git diff --check
Fixes#51436
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
issue: #51436
### What this PR does
- Propagates a CreateCollection RecoveryStorage observation failure from
the flusher component to WAL dispatch.
- Stops before creating a DataSyncService when the recovery state was
not recorded.
- Adds coverage for the canceled observation path, including the absence
of schema lookup.
### Verification
- `go test -v -tags dynamic,test -gcflags="all=-N -l" -count=1
./internal/streamingnode/server/flusher/flusherimpl -run
"TestWALFlusher.*CreateCollection"`
- `make static-check`
- `gofumpt -l` on changed Go files
- `git diff --check`
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
## Summary
issue: #51052
This PR reduces QueryNode memory pressure by avoiding duplicated or
long-lived segment metadata after sealed segment load/reopen and by
caching frequently reused schema conversion state.
## Storage Mode Scope
- For manifest-backed Storage V3 segments, both Go-side runtime load
info and segcore-side runtime load info are compacted after load/reopen.
- For Storage V2 binlog-mode segments, Go-side sealed runtime load info
is compacted, while segcore keeps the full load info because binlog diff
still depends on binlog/index metadata.
- The optimization primarily targets Storage V3, which is the forward
path.
## Commit Breakdown
1. `9d4dbdf065 enhance: cache text LOB loon arrow schema`
- Cache the Loon Arrow schema generated for the TEXT LOB binary
projection used by Storage V3 load paths.
- Invalidate the cache when schema fields change and clear mutable cache
state on schema assignment.
2. `4f0bb58ef6 enhance: avoid duplicate segment index load info`
- Remove duplicated converted index load info storage.
- Reuse the existing field-index cache for current index identity and
diff/drop decisions.
3. `89e8a15b42 enhance: compact runtime segment load info`
- Compact Go QueryNode sealed segment load info after load/reopen.
- Keep runtime-required lightweight fields and cache related data size
before dropping large metadata.
4. `8a38b2fdbf enhance: skip real count scan without deletes`
- Return row count directly when a segment has no deleted rows.
- Avoid scanning the deleted bitmap on the real-count fast path.
5. `d041837d33 enhance: compact segcore runtime load info`
- Compact manifest-backed segcore runtime load info after load/reopen.
- Preserve lightweight diff identity and local runtime state needed by
schema-only reopen and later diffs.
## Test Plan
- [x] `git diff --check upstream/master...HEAD`
- [x] `gofumpt -l internal/querynodev2/segments/segment.go
internal/querynodev2/segments/segment_loader.go
internal/querynodev2/segments/segment_test.go
internal/querynodev2/segments/utils.go
internal/querynodev2/segments/utils_test.go`
- [x] `/opt/homebrew/opt/llvm@18/bin/clang-format --dry-run --Werror
--style=file <changed C++ files>`
- [x] `ninja -C cmake_build all_tests`
- [x] `./cmake_build/unittest/all_tests
--gtest_filter='SchemaTest.ConvertToLoonArrowSchemaCachesTextLobBinarySchema:SegmentLoadInfoTest.CompactRuntimeInfoForManifest:SegmentLoadInfoTest.CompactRuntimeInfoForManifestKeepsIndexDiffIdentity:SegmentLoadInfoTest.CompactRuntimeInfoForManifestSchemaCopyKeepsIndexIdentity:SegmentLoadInfoTest.ReplaceSchemaForReopenPrunesDroppedFieldRuntimeState:SealedSegmentLoad.LoadPublishesDefaultFilledStateAfterCompact:SealedSegmentReopen.SchemaOnlyReopenPublishesDefaultFilledState:SealedSegmentReopen.SchemaAwareReopenDiscardsOlderSchema:SealedSegmentReopen.SchemaOnlyReopenPreservesCreatedTextIndexState:SealedSegmentReopen.TextIndexCreatedWipedByReopen'`
- [x] `go test -v -count=1 -tags dynamic,test -gcflags="all=-N -l"
-ldflags="-extldflags \"-Wl,-rpath,${MILVUS_WORK_DIR}/cmake_build/lib
-Wl,-rpath,${MILVUS_WORK_DIR}/internal/core/output/lib
-Wl,-rpath,${MILVUS_WORK_DIR}/cmake_build/thirdparty/boost_ext\""
./internal/querynodev2/segments -run
"Test(CompactSegmentLoadInfoForRuntime|GetSegmentRelatedDataSize)$"
-timeout 300s`
- [x] `go test -v -count=1 -tags dynamic,test -gcflags="all=-N -l"
-ldflags="-extldflags \"-Wl,-rpath,${MILVUS_WORK_DIR}/cmake_build/lib
-Wl,-rpath,${MILVUS_WORK_DIR}/internal/core/output/lib
-Wl,-rpath,${MILVUS_WORK_DIR}/cmake_build/thirdparty/boost_ext\""
./internal/querynodev2/segments -timeout 300s` timed out locally after
303.862s while still running MinIO-backed retrieve tests
(`TestRetrieveWithFilter`).
---------
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
## Summary
This PR avoids directly capturing a structured binding in
`ChunkedSegmentSealedImpl::LoadBatchJsonKeyIndexes`.
Clang rejects direct structured-binding captures when compiling this
code path with OpenMP enabled. The lambda now uses a generalized
init-capture for `field_id`, preserving the existing value-capture
behavior while avoiding the unsupported capture form.
issue: #51231
## Test Plan
- [x] `git diff --check`
- [x] `/tmp/milvus-clang-format-15-bin/clang-format-15 --dry-run
--Werror --style=file
internal/core/src/segcore/ChunkedSegmentSealedImpl.cpp`
- [x] `make SKIP_3RDPARTY=1`
- [x] `PATH="/tmp/milvus-clang-format-15-bin:$PATH" make verifiers`
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
## Summary
Fixes#50788
The DataNode compactor executor used the same mutex for task state, slot
accounting, channel enqueueing, completion callbacks, and filesystem
metric publication. When the task queue was full, `Enqueue` could block
while holding the mutex, which also blocked `Slots()` and made DataNode
`QuerySlot` time out.
This PR keeps the executor mutex scoped to in-memory state updates only:
- release the mutex before sending to `taskCh`
- update terminal task state and release slot usage before completion
callbacks
- publish filesystem metrics outside the executor lock
- add regression tests for full task queues and completion callbacks
querying slots
## Test Plan
- `go test -v -count=1 -tags dynamic,test -gcflags="all=-N -l"
-ldflags="-r ${TEST_MILVUS_WORK_DIR}/cmake_build/lib -r
${TEST_MILVUS_WORK_DIR}/internal/core/output/lib"
./internal/datanode/compactor -run "TestCompactionExecutor" -timeout
300s`
- `gofumpt -l internal/datanode/compactor/executor.go
internal/datanode/compactor/executor_test.go`
- `git diff --check`
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
## Summary
Fixes#50793
QueryCoord segment task delta filtering checked only sealed segment
distribution. Streaming/growing segments are tracked from the channel
leader view, so a streaming reduce task could be treated as reflected
immediately after the RPC returned even if the growing segment was still
present in distribution.
This PR makes segment distribution matching scope-aware:
- `DataScope_Streaming` checks channel leader view `GrowingSegments`
- sealed segment scopes keep using `SegmentDistManager`
- segment delta records persist channel name and data scope
- adds regression coverage for streaming reduce deltas staying active
until the growing distribution disappears
## Test Plan
- `gofumpt -l internal/querycoordv2/task/action.go
internal/querycoordv2/task/scheduler.go
internal/querycoordv2/task/task_test.go`
- `git diff --check HEAD~1..HEAD`
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
## Summary
Coalesce in-flight QueryCoord collection info recovery by collection ID
so concurrent tasks for the same collection share the same
`DescribeCollection` request.
This reduces repeated metadata recovery work during bursty
same-collection task submission while still avoiding a long-lived
collection info cache after the shared request completes.
This also fixes a task metric refresh bug so task-count metrics are
updated after scheduler tasks are drained to zero.
Fixes#50800
## Changes
- Add a per-executor singleflight for collection info recovery.
- Keep caller cancellation independent from the shared metadata lookup.
- Add tests covering non-caching behavior and caller cancellation.
- Refresh task metrics on empty scheduler dispatches without publishing
pre-removal task counts.
## Test Plan
- [x] `gofumpt -l internal/querycoordv2/task/executor.go
internal/querycoordv2/task/executor_test.go
internal/querycoordv2/task/scheduler.go`
- [x] `git diff --check upstream/master...HEAD`
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
## Summary
- Update the drop collection field design doc with the final
drop-function semantics introduced by PR
https://github.com/milvus-io/milvus/pull/50471.
- Document the split between detach function and drop function field
behavior.
- Align RootCoord, Proxy, SDK cache invalidation, and testing strategy
descriptions with the current implementation.
issue: #48983
Design doc:
docs/design-docs/design_docs/20260413-drop-collection-field-design.md
## Test Plan
- [x] git diff --check upstream/master..HEAD
- [x] rg stale drop-function wording in
docs/design-docs/design_docs/20260413-drop-collection-field-design.md
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
## Summary
Optimize sealed segment load by reusing the cached metadata already held
by `SegmentLoadInfo` when computing the initial load diff.
This avoids rebuilding converted index metadata from the protobuf
snapshot during `ChunkedSegmentSealedImpl::Load()`, while still
rebuilding the binlog pointer cache for the copied protobuf storage.
issue: #50521
## Test Plan
- `git diff --check upstream/master..HEAD`
- `/opt/homebrew/opt/llvm@18/bin/clang-format --dry-run --Werror
--style=file internal/core/src/segcore/ChunkedSegmentSealedImpl.cpp
internal/core/src/segcore/SegmentLoadInfo.h`
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
## Summary
issue: #50579
Cache estimated index load resources on `LoadIndexInfo` so segment load
paths can reuse the same estimate for raw-data checks and resource
accounting.
## Changes
- Store the cached load resource estimate as
`std::optional<LoadResourceRequest>`.
- Reuse cached load-resource estimates in sealed index loading and V1
translator paths.
- Keep `CheckIndexHasRawData` inline with a direct `IndexFactory`
fallback instead of a separate helper.
- Add unit coverage for optional assignment, copy, and reset behavior.
## Test Plan
- [x] `/opt/homebrew/opt/llvm@18/bin/clang-format --dry-run --Werror
internal/core/src/segcore/SegmentLoadInfo.cpp
internal/core/src/segcore/SegmentLoadInfo.h`
- [x] `git diff --check upstream/master..HEAD`
- [x] `rg "GetIndexLoadResource|has_load_resource_request"
internal/core/src/segcore internal/core/unittest/test_loading.cpp`
- [x] `make run-test-cpp
filter=IndexLoadTest.LoadResourceRequestCacheIsOptional`
- [x] `make SKIP_3RDPARTY=1 build-cpp-with-unittest` passed before the
final inline cleanup; after the cleanup, the same target compiled
`SegmentLoadInfo.cpp` successfully and was then stopped before
completion.
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
#### What this PR does / why we need it:
Improve QueryCoord balance stability by making scheduler segment task
deltas aware of current segment distribution.
When pending task deltas are not aligned with actual segment
distribution, balance policies can repeatedly shift plans based on stale
workload information. This PR tracks segment task delta effects with
action records and filters effects that have already been reflected in
distribution.
This also caches collection row count and delegator count in score
workload status to avoid repeated distribution scans, and applies the
updated segment delta snapshot to score and row-count balance paths.
Round-robin segment assignment keeps the lightweight segment-count only
ordering.
#### Which issue(s) this PR fixes:
Fixes#49860
#### Special notes for your reviewer:
MultiTargetBalancer is intentionally unchanged because it does not
consume scheduler segment task deltas in the original design.
#### Tests
- `gofumpt -l internal/querycoordv2/task/scheduler.go
internal/querycoordv2/assign/assign_policy_score.go
internal/querycoordv2/assign/assign_policy_rowcount.go
internal/querycoordv2/assign/assign_policy_roundrobin.go`
- `git diff --check`
- `go test -v -count=1 -tags dynamic,test -gcflags="all=-N -l"
-ldflags="-r ${MILVUS_WORK_DIR}/cmake_build/lib -r
${MILVUS_WORK_DIR}/internal/core/output/lib"
./internal/querycoordv2/assign -run
"Test(ScoreBasedAssignPolicy|RowCountBasedAssignPolicy|RoundRobinAssignPolicy)"
-timeout 300s`
- `go test -v -count=1 -tags dynamic,test -gcflags="all=-N -l"
-ldflags="-r ${MILVUS_WORK_DIR}/cmake_build/lib -r
${MILVUS_WORK_DIR}/internal/core/output/lib"
./internal/querycoordv2/task -run
"TestTask/(TestCalculateTaskDelta|TestSegmentTaskDeltaWithDistFilter|TestChannelTaskDeltaCache)"
-timeout 300s`
---------
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
Fixes#50424
## Summary
This PR uses the `DropFunctionOutputFields` request flag to let
`AlterCollectionSchema` distinguish two client-side drop paths:
- `drop_collection_function` detaches the function and keeps its output
fields as normal schema fields.
- `drop_function_field` sets the flag and cascades supported
BM25/MinHash output fields.
MinHash detach keeps the output field because the MinHash signature is a
materialized `BinaryVector` field. Insert and compaction paths persist
the generated signature, raw retrieval is only blocked for BM25 function
outputs, and after detach the MinHash output field is kept with
`IsFunctionOutput=false` so it can stand alone as a normal vector field.
The proxy validation and RootCoord schema mutation logic now follow the
requested semantics, with tests covering detach-only and
output-field-drop behavior.
## Tests
- `make generated-proto-without-cpp`
- `make`
- `go test -v -count=1 -tags dynamic,test -gcflags="all=-N -l"
./internal/proxy -run
"TestAlterCollectionSchemaTask_PreExecute|TestValidateDropFunction"
-timeout 300s`
- `go test -v -count=1 -tags dynamic,test -gcflags="all=-N -l"
./internal/rootcoord -run
"TestBuildSchemaForDetachFunction|TestBuildSchemaForDropFunctionField"
-timeout 300s`
- `/Users/zilliz/dev/milvus_py`: `pytest
test_drop_collection_field.py::test_drop_function_field_bm25_cascades_output_field
test_drop_collection_field.py::test_drop_collection_function_detaches_minhash_output_field
test_drop_collection_field.py::test_drop_whole_struct_array_field_basic
-v -s`
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
## Summary
Decouple QueryCoord task dispatch from the dist pull loop.
When a QueryNode holds more and more segments, `GetDataDistribution`
returns a larger distribution and `pullDist` can take longer. If
dispatch only happens after dist pull finishes, QueryCoord task dispatch
frequency drops together with pull latency, which can hurt scheduling
stability. This PR keeps dispatch cadence independent from dist pull
latency.
issue: #50158
## Changes
- Add `queryCoord.dispatchInterval` with a default value of 500ms.
- Start separate dist pull and task dispatch loops in each dist handler.
- Keep `pullDist` and `handleDistResp` focused on updating distribution
state.
- Remove the old dispatch flag from dist response handling.
## Tests
- `gofumpt -l internal/querycoordv2/dist/dist_controller.go
internal/querycoordv2/dist/dist_controller_test.go
internal/querycoordv2/dist/dist_handler.go
internal/querycoordv2/dist/dist_handler_test.go
pkg/util/paramtable/component_param.go`
- `git diff upstream/master..HEAD --check`
- `go test -count=1 ./util/paramtable`
- `go test -v -count=1 -tags dynamic,test -gcflags="all=-N -l"
-ldflags="-r ${MILVUS_WORK_DIR}/cmake_build/lib -r
${MILVUS_WORK_DIR}/internal/core/output/lib"
./internal/querycoordv2/dist -run
"TestDistControllerSuite/TestSyncAll|TestDispatchLoopUsesDispatchInterval"
-timeout 300s` blocked locally because this worktree has no
`internal/core/output/lib/pkgconfig/milvus_core.pc`.
---------
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
issue: https://github.com/milvus-io/milvus/issues/50476
This PR rejects dropping a field when it is referenced by the collection
`ttl_field` property. Users need to drop the property first, which
avoids leaving dangling entity-level TTL metadata after Drop Field.
Validation:
- gofumpt -l internal/proxy/task.go internal/proxy/task_test.go
- git diff --check
- TestValidateDropField
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
## Summary
- Allow AlterCollectionSchema AddRequest to carry a field-only add
request while keeping the added field as a non-function output.
- Support function-only add requests against existing output fields, and
validate the merged function schema through the shared function
validator.
- Keep the BM25 field-plus-function path explicit, reject invalid BM25
add forms, and normalize TIMESTAMPTZ default values before broadcasting
field-only schema changes.
issue: #50119
## Test Plan
- [x] source scripts/setenv.sh && go test -v -count=1 -tags dynamic,test
-gcflags="all=-N -l" -ldflags="-r ${MILVUS_WORK_DIR}/cmake_build/lib -r
${MILVUS_WORK_DIR}/internal/core/output/lib" ./internal/proxy -run
'TestAlterCollectionSchemaTask($|_)' -timeout 300s
- [x] gofumpt -l internal/proxy/function_task.go
internal/proxy/function_task_test.go internal/proxy/task.go
internal/proxy/task_test.go internal/proxy/util.go
internal/proxy/util_test.go
internal/rootcoord/ddl_callbacks_alter_collection_schema.go
internal/rootcoord/ddl_callbacks_alter_collection_schema_test.go
internal/util/function/validator/validator.go
- [x] git diff --check upstream/master..HEAD
- [ ] source scripts/setenv.sh && go test -v -count=1 -tags dynamic,test
-gcflags="all=-N -l" -ldflags="-r ${MILVUS_WORK_DIR}/cmake_build/lib -r
${MILVUS_WORK_DIR}/internal/core/output/lib" ./internal/rootcoord -run
'TestDDLCallbacksBroadcastAlterCollectionSchema|TestDDLCallbacksAlterCollectionSchemaAddSkipsSchemaDropReady'
-timeout 300s (blocked locally at link time by stale C++/Loon symbols
such as _FreeCFieldMemSizeList and _loon_segment_writer_* in local core
libraries)
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
Fixes#50490
## What this PR does
This PR replaces the anonymous embedded `struct{ session.Cluster }`
mockey patch in copy segment tests with the generated
`session.MockCluster`. The tests still mock `QueryCopySegment`, but they
no longer use an anonymous struct method expression that triggers the Go
1.26.4 `printf` analyzer panic during vet.
## Verification
```bash
gofumpt -l internal/datacoord/copy_segment_task_test.go
git diff --check
rg -n "mockey\.Mock\(\(\*struct\{|\(\*struct\{[^\n]*\}\)\." --glob "*.go"
source scripts/setenv.sh && go test -vet=printf -tags dynamic,test -gcflags="all=-N -l" -ldflags="-r ${MILVUS_WORK_DIR}/cmake_build/lib -r ${MILVUS_WORK_DIR}/internal/core/output/lib" -run "^$" -exec /usr/bin/true -count=1 ./internal/datacoord
rm -rf /tmp/milvus2-test-libs && mkdir -p /tmp/milvus2-test-libs && for f in internal/core/output/lib/*.dylib cmake_build/thirdparty/boost_ext/*.dylib; do ln -sf "$PWD/$f" /tmp/milvus2-test-libs/; done && source scripts/setenv.sh && go test -v -vet=off -tags dynamic,test -gcflags="all=-N -l" -ldflags="-r /tmp/milvus2-test-libs" -run "TestCopySegmentTask/TestQueryTaskOnWorker" -count=1 ./internal/datacoord
```
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
## Summary
- Support dropping fields from collection schema via
`AlterCollectionSchema` RPC with `DropRequest`
- Support dropping function fields (cascade delete output fields and
indexes)
- Support disabling dynamic field via `AlterCollectionSchema` (reuse
existing `AlterCollection` logic)
- C++ segcore defense-in-depth: skip dropped fields during segment
loading
issue: #48983
design doc:
docs/design-docs/design_docs/20260413-drop-collection-field-design.md
## Test plan
- [x] Unit tests for rootcoord drop field/function logic
- [x] Unit tests for proxy task validation (drop field constraints)
- [x] Unit tests for datacoord index service (dropped field handling)
- [x] Unit tests for querynode segment loader (dropped field skipping)
- [x] C++ unit tests for SegmentLoadInfo ComputeDiff with dropped fields
- [ ] E2E tests: drop scalar field, drop vector field, drop field with
index, drop function, disable dynamic field (deferred to QA pending
pymilvus RC)
🤖 Generated with [Claude Code](https://claude.ai/code)
---------
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
issue: #49967
## What was changed
- Add `X-Milvus-Trace-Id` to REST responses for `/v1/vector/...` and
`/v2/vectordb/...` routes.
- Create a REST entry trace span before access logging so response
headers and REST logs use the same request trace ID.
- Keep the v2 wrapper from overwriting the entry trace ID when one
already exists.
- Add timeout and early-return coverage for the trace ID header.
## Verification
- `gofumpt -l internal/distributed/proxy/httpserver/handler_v2.go
internal/distributed/proxy/httpserver/traceid_test.go
internal/distributed/proxy/httpserver/utils.go
internal/distributed/proxy/httpserver/constant.go
internal/distributed/proxy/httpserver/timeout_middleware.go
internal/distributed/proxy/service.go`
- `git diff --check`
- `zsh -c 'source scripts/setenv.sh && go test -v -count=1 -tags
dynamic,test -gcflags="all=-N -l" -ldflags="-r
${MILVUS_WORK_DIR}/cmake_build/lib -r
${MILVUS_WORK_DIR}/internal/core/output/lib"
./internal/distributed/proxy/httpserver -run
"TestHTTPReturnWritesTraceIDHeader|TestHTTPReturnSkipsTraceIDHeaderWhenMissing|TestTraceIDHandlerFuncCreatesRequestTraceID|TestTraceIDHandlerFuncCoversV1AndV2RestRoutes|TestTraceIDHandlerFuncCreatesTraceIDWhenUnsampled|TestTraceIDHandlerFuncReturnsUniqueTraceID|TestTraceIDHandlerFuncCoversEarlyAbort|TestWrapperPostBadJSONReturnsTraceIDHeader|TestWrapperPostKeepsRequestTraceID|TestTimeoutMiddlewareReturnsTraceIDHeader"
-timeout 300s'`
- Manual curl verification against local standalone: REST response
header and REST access log trace IDs matched; 10 repeated search
requests returned 10 valid and unique 32-hex trace IDs.
## Note
Full `go test ./internal/distributed/proxy/httpserver` currently hits an
existing unrelated `TestSearchV2/search#09` assertion mismatch: expected
`Mismatch type uint8`, actual contains `Mismatch type []uint8 ...`.
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
#### What this PR does / why we need it:
Stabilizes `TestExternalCollectionManager_CancelTask` by waiting for the
asynchronous manager state update after cancel is observed by the task
function.
`CancelTask` only triggers context cancellation. The task goroutine
records `JobStateFailed` after `taskFunc` returns with
`context.Canceled`, so reading the task info immediately after observing
`ctx.Done()` can race with `UpdateResult` and intermittently see
`JobStateInProgress` with an empty failure reason.
This test now waits until the manager snapshot reports `JobStateFailed`,
then asserts the failure reason separately for clearer failure output.
#### Which issue(s) this PR fixes:
N/A
#### Special notes for your reviewer:
This is a test-only flaky fix.
#### Tests
- `gofumpt -l internal/datanode/external/manager_test.go`
- `git diff --check HEAD~1..HEAD`
- `go test -v -count=1 -tags dynamic,test -gcflags="all=-N -l"
-ldflags="-r ${MILVUS_WORK_DIR}/cmake_build/lib -r
${MILVUS_WORK_DIR}/internal/core/output/lib"
./internal/datanode/external -run
'TestExternalCollectionManager_CancelTask' -timeout 300s`
- `go test -v -count=1 -tags dynamic,test -gcflags="all=-N -l"
-ldflags="-r ${MILVUS_WORK_DIR}/cmake_build/lib -r
${MILVUS_WORK_DIR}/internal/core/output/lib"
./internal/datanode/external -timeout 300s`
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
issue: #49920Fixes#49920
This PR aligns round-robin and row-count based segment assignment with
the score-based policy for forced segment assignment.
Changes:
- Skip common segment node filtering when `forceAssign=true` in
round-robin and row-count policies.
- Apply `queryCoord.balanceSegmentBatchSize` only when
`forceAssign=false`.
- Add unit coverage for balance-limited assignment and forced assignment
behavior.
Verification:
- `gofumpt -l internal/querycoordv2/assign/assign_policy_roundrobin.go
internal/querycoordv2/assign/assign_policy_rowcount.go
internal/querycoordv2/assign/assign_policy_roundrobin_test.go
internal/querycoordv2/assign/assign_policy_rowcount_test.go`
- `git diff --check`
Not run:
- Build and Go tests were not run per local instruction to avoid
compilation in this round.
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
## Summary
This PR optimizes QueryCoord `ChannelDistManager` under the 100K
collections recovery scenario.
The main change is to optimize the ChannelDist write/update path.
`ChannelDistManager.Update` no longer rebuilds a global collection index
while holding the manager write lock. This shortens the write-side
critical section and reduces read-lock holding time during QueryNode
distribution updates, so Scheduler can observe distribution changes and
dispatch recovery tasks sooner.
The expected tradeoff is that part of the cost moves to the read path:
`GetByCollectionAndFilter` is removed, while `GetByFilter` becomes the
unified filtering path and increases in the profile. The increase is
bounded, and the overall 30-minute CPU profile still decreases.
## Profile Result
100K collections, 30-minute MixCoord CPU profile:
| Metric | master | this PR | change |
|---|---:|---:|---:|
| Total CPU samples | `1,672,474` | `1,598,355` | `-74,119 / -4.43%` |
| `ChannelDistManager.Update` | `39,981 / 2.39%` | `20,588 / 1.29%` |
`-48.5%` |
| `ChannelDistManager.GetByCollectionAndFilter` | `22,416 / 1.34%` | not
sampled | cleared |
| `ChannelDistManager.GetByFilter` | `22,771 / 1.36%` | `55,853 / 3.49%`
| `+145.3%` |
## Code Changes
- Remove the global `collectionIndex` from `ChannelDistManager`.
- Remove `GetByCollectionAndFilter` and migrate callers to `GetByFilter`
with `WithCollectionID2Channel`.
- Reuse per-node `nodeChannels.collChannels` for collection-scoped
lookup instead of rebuilding a global index on every update.
- Reduce temporary allocations in ChannelDist and SegmentDist filtering
paths.
- Update QueryCoord callers and tests for the unified ChannelDist filter
path.
Fixes#49511
## Test Plan
- `git diff --check upstream/master...HEAD`
- `gofumpt -l $(git diff --name-only upstream/master...HEAD -- '*.go')`
- Extended `TestGetChannelDistJSON` to cover `GetChannelDist(100)` and
`GetLeaderView(0|100|300)`.
- Updated `TestBalanceChecker_ConstructNormalBalanceQueue_*` mocks to
use the unified `GetByFilter` API.
---------
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
## Summary
- Replace ReplicaManager global locking with per-collection write locks.
- Add lock-free read indexes for replica ID and collection-scoped
replica lookups.
- Keep collection and flat replica indexes consistent across spawn,
move, recover, and removal paths.
issue: #48866
## Test Plan
- [x] Added ReplicaManager unit coverage for index consistency and
partial collection updates.
- [ ] Focused local go test is blocked by local core linker errors for
external segment symbols (`_loon_*`, `_FreeCFieldMemSizeList`,
`_SampleExternalSegmentFieldSizes`).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
issue: #49634
This fixes a flaky WAL adaptor unit test introduced by #49696.
The test mocks RecoverWALFlusher and stores the parameter for assertions
after handleAlterWALFlushingStage returns. The real call passes a
non-escaping RecoverWALFlusherParam composite literal, so keeping the
pointer in the mock can read unstable stack data. Copy the parameter
value inside the mock callback before asserting on it.
Validation:
- go test -v -count=1 -tags dynamic,test -gcflags="all=-N -l"
-ldflags="-r <artifact>/cmake_build/lib -r
<artifact>/internal/core/output/lib"
./internal/streamingnode/server/wal/adaptor -run
^TestHandleAlterWALFlushingStagePassesRateLimitComponent$ -timeout 300s
- go test -v -count=1 -tags dynamic,test -gcflags="all=-N -l"
-ldflags="-r <artifact>/cmake_build/lib -r
<artifact>/internal/core/output/lib"
./internal/streamingnode/server/wal/adaptor -timeout 300s
- gofumpt -l internal/streamingnode/server/wal/adaptor/opener_test.go
- git diff --check
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
## Summary
- Keep QueryCoord waitQueue as the global priority queue while using
node-bucketed indexing for processQueue so Dispatch(node) scans only
related tasks.
- Index multi-node process tasks under all related nodes, including
MoveTask endpoints and LeaderAction worker/leader nodes.
- Dispatch LeaderAction through the leader executor to match the
SyncDistribution RPC target while preserving action.Node() as the worker
payload node.
- Remove the dist handler checkExecutedFlag ticker and avoid eager
task.String() formatting on non-error stale checks.
- Serialize RemoveByNode with scheduling to avoid re-adding tasks to
processQueue during node-down cleanup races.
Fixes#49512
## Test Plan
- git diff --check upstream/master...HEAD
- gofumpt -l internal/querycoordv2/dist/dist_handler.go
internal/querycoordv2/task/scheduler.go
internal/querycoordv2/task/task_test.go
- go test -v -count=1 -tags dynamic,test -gcflags="all=-N -l"
-ldflags="-r ${MILVUS_WORK_DIR}/cmake_build/lib -r
${MILVUS_WORK_DIR}/internal/core/output/lib"
./internal/querycoordv2/task -run
"TestTask/(TestRemoveByNodeWaitsForSchedule|TestLeaderTaskUsesLeaderExecutor|TestNoExecutor|TestTaskQueueRangePriority|TestNodeTaskQueueNodeBucketing|TestNodeTaskQueueMoveTaskDualNode|TestNodeTaskQueueLeaderActionDualNode|TestNodeTaskQueueRangeByNodePriority)"
-timeout 300s
- make static-check
---------
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
Fixes#49476
This fixes a QueryCoord `IndexChecker` panic when a loaded segment has
redundant index metadata but does not have any missing index.
The redundant-index path used to put only the segment ID into
`segmentsToUpdate`. Later, the reopen task was created from
`idSegments[segmentID]`, but `idSegments` is only populated by the
missing-index path. When a segment only had redundant indexes,
`idSegments[segmentID]` was nil and `createSegmentUpdateTask` panicked.
Changes:
- Store `*meta.Segment` directly in `segmentsToUpdate`.
- Use the stored segment when creating reopen tasks.
- Add a unit test for the redundant-index-only path.
Verification:
- `gofumpt -l internal/querycoordv2/checkers/index_checker.go
internal/querycoordv2/checkers/index_checker_test.go`
- `git diff --check HEAD~1..HEAD`
- `go test -v -count=1 -tags dynamic,test -gcflags="all=-N -l"
-ldflags="-r ${MILVUS_WORK_DIR}/cmake_build/lib -r
${MILVUS_WORK_DIR}/internal/core/output/lib"
./internal/querycoordv2/checkers -run
"TestRemoveRedundantIndex|TestIndexChecker" -timeout 300s`
- `make static-check`
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
## Summary
Revert shell env inheritance of `CGO_LDFLAGS` / `CGO_CFLAGS` in
`Makefile` `build-go` and `milvus-gpu` targets. This disabled Go's
default `-O2 -g` CGo flags when both the shell env and the Makefile
variable are empty, compiling DataDog/zstd (indirect dep) and other CGo
code at `-O0 -g` and causing a ~77% regression in QueryNode segment load
wall time.
Users can still pass custom flags via `make CGO_LDFLAGS=... build-go`.
ASan builds remain unaffected.
issue: #49286
## Test plan
- [ ] Build succeeds with default flags
- [ ] Build succeeds with ASan (`USE_ASAN=ON make milvus`)
- [ ] Build succeeds with user-provided CGO flags (`make build-go
CGO_LDFLAGS=-L/some/path`)
- [ ] Segment load wall returns to pre-regression baseline
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
issue: #48722
## Summary
- Move the segment dist existence check (`existsInDist`) in `checkStale`
from running in both `promote` and `preProcess` to only running during
`promote`.
- During `preProcess`, the task's LoadSegments RPC may already be
in-flight. If QueryNode loads the segment and reports it via heartbeat
before the RPC returns, `preProcess` would incorrectly cancel the task —
killing the in-flight RPC with a misleading `context canceled` error.
- Now the dist check only runs during `promote` (waitQueue →
processQueue), where no RPC has been dispatched yet.
## Test plan
- [ ] Existing unit test
`TestTaskStaleBySegmentInDist/SegmentAlreadyInDist` still passes (task
canceled at promote stage)
- [ ] Verify no more spurious `WARN "failed to load segment" context
canceled` logs when segment is loaded successfully
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
## Summary
Optimize standalone MixCoord recovery to reduce startup time by
parallelizing independent operations:
- **Parallelize DataCoord & QueryCoord startup**: both only depend on
RootCoord being ready, run them concurrently via errgroup
- **Parallelize DataCoord sub-meta loading**: load all 6 sub-metas
(indexMeta, analyzeMeta, partitionStatsMeta, compactionTaskMeta,
statsTaskMeta, snapshotMeta) concurrently alongside reloadFromKV
- **Parallelize indexMeta reload**: run ListIndexes and
ListSegmentIndexes concurrently since they update independent data
structures (m.indexes vs m.segmentIndexes/m.segmentBuildInfo)
- **Batch delete TargetManager targets**: replace per-collection
RemoveCollectionTarget calls with single RemoveWithPrefix
- **Batch etcd loading for ListCollections**: replace sequential per-key
reads with single LoadWithPrefix in RootCoord's initMetaTable
issue: #47783
## Test Plan
- [x] Unit tests pass for all changed packages (index_meta, meta,
kv_catalog, target_manager)
- [x] Race detector clean (`go test -race`)
- [x] Coverage: core functions (reloadFromKV, newIndexMeta,
RemoveCollectionTargets) at 100%; batch load functions at 81-93%
---------
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
## Summary
- Replace the hardcoded thread pool max threads limit (16) with a
configurable parameter `common.threadCoreCoefficient.maxThreadsSize`,
default 16, only effective when greater than 0
- Add missing Resize watchers for middle and low priority thread pools
- When `maxThreadsSize` changes dynamically, update the limit first then
resize all pools to ensure correct ordering
issue: #48378
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
## Summary
- macOS `ld` treats each `-r` flag as a single directory path, unlike
Linux `ld` which accepts colon-separated paths
- Since d69bdd288c added `cmake_build/lib` to `DYLD_LIBRARY_PATH`,
`RPATH=$DYLD_LIBRARY_PATH` produces a colon-separated value like
`path1:path2`
- The Makefile's `-r $RPATH` embeds this as a single invalid path in the
binary, causing `Library not loaded: @rpath/libmilvus-storage.dylib` at
runtime
- Fix: emit two separate `-r` flags so the Makefile expands to `-r path1
-r path2`
## Test plan
- [x] Verified `RPATH` variable output: `path1 -r path2` (correct)
- [x] Verified binary loads correctly after fix (`otool -l` shows two
separate `LC_RPATH` entries)
- [x] Verified Milvus standalone starts and serves requests successfully
issue: #47809
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
## Summary
- Restore the hard-coded maximum of 16 threads for the C++ IO thread
pool
- In large file scenarios, CGo threads can consume all available thread
resources, starving Go goroutines
- This reverts commit ffd3435c4f and the related test changes from
a41d170c14
issue: https://github.com/milvus-io/milvus/issues/48060
## Test plan
- [ ] Verify thread pool size is clamped to 16 via C++ unit test
(`ThreadPoolTest.ResizeAboveMaximum`)
- [ ] Test large file load scenarios to confirm Go goroutines are no
longer starved
🤖 Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
issue: https://github.com/milvus-io/milvus/issues/46678
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
- Core invariant: Text index log keys are canonicalized at KV
(serialization) boundaries — etcd stores compressed filename-only
entries, while in-memory and runtime consumers must receive full
object-storage keys so Datanode/QueryNode can load text indexes
directly.
- Logic removed/simplified: ad-hoc reconstruction of full text-log paths
scattered across components (garbage_collector.getTextLogs,
querynodev2.LoadTextIndex, compactor/index task code) was removed;
consumers now use TextIndexStats.Files as-provided (full keys). Path
compression/decompression was centralized into KV marshal/unmarshal
utilities (metautil.ExtractTextLogFilenames in marshalSegmentInfo and
metautil.BuildTextLogPaths in kv_catalog.listSegments), eliminating
redundant, inconsistent prefix-rebuilding logic that broke during
rolling upgrades.
- Why this does NOT cause data loss or regressions: before persist,
marshalSegmentInfo compresses TextStatsLogs.Files to filenames
(metautil.ExtractTextLogFilenames) so stored KV remains compact; on
load, kv_catalog.listSegments calls metautil.BuildTextLogPaths to
restore full paths and includes compatibility logic that leaves
already-full keys unchanged. Thus every persisted filename is
recoverable to a valid full key and consumers receive correct full paths
(see marshalSegmentInfo → KV write path and kv_catalog.listSegments →
reload path), preventing dropped or malformed keys.
- Bug fix (refs #46678): resolves text-log loading failures during
cluster upgrades by centralizing path handling at KV encode/decode and
removing per-component path reconstruction — the immediate fix is
changing consumers to read TextIndexStats.Files directly and relying on
marshal/unmarshal to perform compression/expansion, preventing
mixed-format failures during rolling upgrades.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
issue: https://github.com/milvus-io/milvus/issues/46410
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
- Core invariant: etcd metadata and in-memory Segment/TextIndex records
must store only compact filenames for text-index files; full object keys
are deterministically reconstructed at use-sites from a stable root +
common.TextIndexPath + IDs via metautil.BuildTextLogPaths.
- Bug & fix (issue #46410): the etcd RPC size overflow was caused by
persisting full upload keys in segment/TextIndex metadata. Fix: at
upload/creation sites (internal/datanode/compactor/sort_compaction.go
and internal/datanode/index/task_stats.go) store only filenames using
metautil.ExtractTextLogFilenames; at consumption/use sites
(internal/datacoord/garbage_collector.go,
internal/querynodev2/segments/segment.go, and other GC/loader code)
reconstruct full paths with metautil.BuildTextLogPaths before accessing
object storage.
- Simplified/removed logic: removed the redundant practice of carrying
full object keys through metadata and in-memory structures; callers now
persist compact filenames and perform on-demand path reconstruction.
This eliminates large payloads in etcd and reduces memory pressure while
preserving the same runtime control flow and error handling.
- No data loss / no regression: filename extraction is a deterministic
suffix operation (metautil.ExtractTextLogFilenames) and reloadFromKV
performs backward compatibility (internal/datacoord/meta.go converts
existing full-path entries to filenames before caching). All read paths
reconstruct full paths at runtime (garbage_collector.getTextLogs,
LocalSegment.LoadTextIndex, GC/loader), so no files are modified/deleted
and access semantics remain identical.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
