mirror of
https://github.com/milvus-io/milvus.git
synced 2026-07-21 10:15:43 +00:00
master
1416
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
42f6bcb69a |
fix: add PK stats to growing-source flush (#51532)
issue: #50425 Signed-off-by: luzhang <luzhang@zilliz.com> Co-authored-by: luzhang <luzhang@zilliz.com> |
||
|
yihao.daiandGitHub
|
ddf3e4d558 |
fix: force L0 import to use storage v2 (#51158)
## Summary Fix L0 import storage version selection when `UseLoonFFI` is enabled: - Force DataCoord to assign StorageV2 for L0 import segments. - Force L0 import requests to disable Loon FFI. - Defensively keep DataNode L0 sync tasks on StorageV2 even if the request carries StorageV3. - Add regression coverage for L0 import with `UseLoonFFI=true`. Fixes #51148 ## Known mixed-version window (documented, not fixed here) The `AssignSegments` metadata fix only applies to jobs created on the new version. A job created by a **pre-upgrade** DataCoord with `common.storage.useLoonFFI=true` has already persisted its L0 segments with `StorageVersion=3` in etcd, and segment IDs are reused (not reallocated) when the task is recovered. If such an in-flight job completes **after** the upgrade — either the upgraded DataCoord recovers the task and reassembles the request as V2, or an upgraded DataNode serves the old coordinator's V3 request (`syncDelete` now hard-codes V2) — the deltalogs are written as V2 but the segment stays tagged `StorageVersion=3` with an empty `ManifestPath`: the import completion path does not reconcile `StorageVersion`, and L0 segments are excluded from the storage-version compaction policy, so nothing self-heals the tag. Impact: harmless for delta load and L0 compaction (both branch on the manifest, not the tag), but snapshot restore trusts the tag — `collectSegmentFiles` fails on `storage_version>=3` with an empty manifest. Scope: requires a deployment running the experimental `useLoonFFI=true` with an L0 import job in flight at the moment of the upgrade, later restored via snapshot. Since the pre-upgrade V3 L0 write path always fails (#51148), no successfully-written V3 L0 data exists — only the metadata tag can be stale. If this window ever materializes in practice, the backfill is a one-operator change in the completion path (`UpdateStorageVersionOperator(segID, StorageV2)` for L0). ## Test Plan - [x] `make static-check` - [x] `go test -tags dynamic,test -gcflags="all=-N -l" -count=1 ./internal/datacoord -run TestImportUtil_L0ImportUsesStorageV2WhenLoonFFIEnabled` - [x] `go test -tags dynamic,test -gcflags="all=-N -l" -count=1 ./internal/datanode/importv2 -run 'TestL0Import/TestL0ImportForcesStorageV2ForSyncTask'` Note: after rebasing to latest master, the focused local tests could not be rerun with the reused local C++ output because the local `libmilvus_core` is older than current headers and misses `_SegcoreSetPrefetchThreadPoolNum`. The same focused tests passed before the rebase; CI should build with matching artifacts. Signed-off-by: Yihao Dai <yihao.dai@zilliz.com> |
||
|
|
d6982c74c4 |
enhance: record and expose DataNode task cost_time / cost_cpu_num via QueryTask (#51443)
issue: #49156 ## Summary - Record `cost_time` / `cost_cpu_num` on DataNode index build tasks and surface them through `QueryTask` response `Properties`. - Reserve the unified `cost_time` / `cost_cpu_num` keys under `pkg/taskcommon` so future PRs can wire up the other `CreateTask`-dispatched task types. - Introduce small `internal/datanode/taskcost` helper (`NowMs`, `ElapsedMs`, `EstimateIndexBuildCPUNum`) for index build task cost accounting. - Final state/failReason and execution-end cost are written in a single `TaskManager` critical section, and `QueryTask` reads them from a single snapshot, so state and cost in one response are always consistent. ## Notes This PR is producer-side only. DataCoord / proxy / metrics exporter do not consume these properties yet; downstream consumption should land in a follow-up linked from #49156. `cost_cpu_num` is a coarse, statistics-only estimate for observability (vector index ≈ knowhere build pool size = NumCPU in cluster mode; scalar index = 1). It is not used for coordinator-side scheduling, so the standalone pool-ratio deviation (NumCPU × buildIndexThreadPoolRatio) and pool sharing across concurrent builds are intentionally not modeled. Supersedes #49157 (that PR's head branch was rebased onto latest master and force-pushed, so GitHub no longer allows reopening it). Content is identical, rebased to a single commit on top of current master. ## Test plan - [x] `go test ./taskcommon/...` from `pkg/` - [x] `go test -tags dynamic,test -gcflags="all=-N -l" -count=1 ./internal/datanode/taskcost` - [ ] `go test -tags dynamic,test -gcflags="all=-N -l" -count=1 ./internal/datanode/index -run 'TestIndexTaskSchedulerRecordsIndexTaskCost|Test_statsTaskInfoSuite/Test_IndexTaskCostMethods'` - [ ] Manual sanity: `QueryTask` returns `cost_time` / `cost_cpu_num` for DataNode index build task success and failure paths. --------- Signed-off-by: cai.zhang <cai.zhang@zilliz.com> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
|
0671607431 |
fix: CAS the schema-bump in-place manifest commit to stop dropping a concurrent index (#51376) (#51377)
issue: #51376 ## What this fixes The in-place partial-materialization path of `bump_schema_version_compaction` commits onto a live segment's manifest, and DataCoord adopted the result on a version-newer check only — it never verified the result was built on the segment's *current* manifest. A bump built on a plan-pinned base could therefore be adopted after a concurrent stats/index task advanced the manifest pointer, silently dropping that index (correct version number, wrong content lineage). The loss is permanent and non-self-healing, because the segment meta stats placeholder stops the stats inspector from re-triggering. ## Fix — application-layer adoption CAS (mirrors the stats path) - The schema-bump result carries the plan-pinned base manifest (`CompactionSegment.base_manifest`, like `StatsResult.base_manifest`); the DataNode fills it with the manifest it materialized on. - DataCoord adopts the in-place result only if `result.base_manifest == segment.ManifestPath` (the current pointer); a missing or stale base is rejected (`ErrIllegalCompactionPlan`) so the task re-triggers and rebuilds on the current manifest instead of overwriting the concurrent commit. This mirrors the stats path's `errStatsResultStale`. - The storage commit stays on `OverwriteResolver` (unchanged). Because the CAS is at the adoption layer (against the meta pointer, not storage-latest), a result lost after a DataNode crash self-heals on retry: the pointer stays put, so the retry re-pins the same base and is adopted — no wedge. - The bump policy is restricted to internal collections (matching every sibling compaction policy), so a rejected attempt's orphan is reclaimed by internal dropped-segment GC and bump never churns on external collections. ## Verification Both interleavings converge with no data loss: stats-first → bump's base no longer matches the advanced pointer → reject + retrigger on the current manifest; bump-first → stats' existing base-check rejects and retries. Added meta-level adoption tests (matching/stale base) and updated the in-place adoption tests to carry the base. Signed-off-by: MrPresent-Han <chun.han@gmail.com> Co-authored-by: MrPresent-Han <chun.han@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
wei liuandGitHub
|
09c44f2dbb |
enhance: pass external segment row target from DataCoord (#51352)
issue: #45881 ## What this PR does External refresh previously read the segment row target directly from each DataNode configuration during execution. That made the worker input implicit and allowed task behavior to depend on the selected node. This change: - keeps `dataNode.externalCollection.targetRowsPerSegment` as the existing configuration source; - has DataCoord snapshot the configured value into the refresh task request; - adds `target_rows_per_segment` to the DataCoord-to-DataNode request; - validates the transmitted value in DataNode; - uses the request value for fragment splitting and segment balancing; - preserves the MilvusTable one-fragment-per-segment behavior. ## Validation - `make generated-proto-without-cpp` - `make lint-fix` - `make build-go` - `go test -gcflags="all=-N -l" -race -tags dynamic,test github.com/milvus-io/milvus/pkg/v3/util/paramtable -failfast -count=1 -ldflags="-r ${RPATH}" -run "^TestCachedParam$" -v` - `go test -gcflags="all=-N -l" -race -coverprofile=/tmp/refine-rowcount-config-dc.out -tags dynamic,test github.com/milvus-io/milvus/internal/datacoord -failfast -count=1 -ldflags="-r ${RPATH}" -run "^TestRefreshExternalCollectionTask_CreateTaskOnWorker$" -v` - `go test -gcflags="all=-N -l" -race -coverprofile=/tmp/refine-rowcount-config-dn.out -tags dynamic,test github.com/milvus-io/milvus/internal/datanode/external -failfast -count=1 -ldflags="-r ${RPATH}" -run "^TestRefreshExternalCollectionTaskSuite$" -v` - `git diff --check` Signed-off-by: Wei Liu <wei.liu@zilliz.com> |
||
|
wei liuandGitHub
|
760407a17e |
enhance: support external JSON stats for external tables (#51008)
issue: #45881 ## What changed This PR enables JSON key stats / JSON shredding support for external StorageV3 segments that already have a committed manifest. The main behavior changes are: - Allow DataCoord to schedule and reload `JsonKeyIndexJob` for external StorageV3 manifest-backed segments. - Add `StatsResult.base_manifest` so DataCoord can reject stale Text and JSON stats results when an external refresh has already moved the segment to a newer manifest. - Keep stale-result CAS and task placeholder release in the common stats task path, so a rejected result does not finish the old task forever. - Best-effort cleanup rejected Text/JSON stats files only for external collections; internal collections keep the existing GC ownership rules. - Clear old `TextStatsLogs` and `JsonKeyStats` metadata during external refresh. - Gate manifest JSON stats exposure on DataCoord metadata placeholders, preventing stale manifest stats from being loaded after refresh. - Keep external BM25 inspector jobs skipped. - Update external table design docs to match the new support boundary. ## Why External table refresh can replace the manifest of the same segment while an older JSON stats task is still running. Without a base-manifest check, a late stats result built from the old manifest could overwrite the refreshed manifest or expose stale JSON stats. External collection segments are patched only when the source changes. If the source stays stable, the segment can remain active forever, so rejected external stats files should be cleaned immediately instead of waiting for dropped-segment GC. ## Validation - `make generated-proto-without-cpp` - `gofmt -w` on touched Go files - `git diff --check` - `git diff --cached --check` - `git diff --check milvus/master..HEAD` - staged mockery addition scan Targeted Go test attempted locally: ```bash source ~/.profile && source scripts/setenv.sh && \ go test -tags dynamic,test -gcflags="all=-N -l" -count=1 \ github.com/milvus-io/milvus/internal/datacoord \ -run 'Test_statsTaskSuite/TestQueryTaskOnWorkerDiscardsStaleStatsResult' \ -v ``` It is blocked by the local C++ link environment: `_SegcoreSetPrefetchThreadPoolNum` is missing from the local linked segcore library. `reset-milvus` also cannot fully restart standalone in this worktree because `bin/milvus` is missing. Signed-off-by: Wei Liu <wei.liu@zilliz.com> |
||
|
|
b16c821b36 |
enhance: carry TEXT LOB refs through schema-bump full-rewrite compaction (#51125)
### What & why `bump_schema_version`'s full-rewrite path (`runFullSchemaRewrite`, reached only for a **drop-field** schema bump) builds a from-scratch output manifest via `NewBinlogRecordWriter` and did **not** carry the source segment's TEXT LOB references — leaving dangling refs for an out-of-line (`>=64KB`) TEXT column. This was the `TODO(#50021)`. ### What this does Wire **REUSE_ALL** LOB handling into `runFullSchemaRewrite`, mirroring sort compaction. Both are `1->1` with a single output manifest, so REUSE_ALL is always correct: a schema bump never changes the existing TEXT LOB data — only its references are carried. - `internal/compaction/lob_compaction.go`: `GetForcedStrategy` forces REUSE_ALL for `BumpSchemaVersionCompaction`. - `internal/datanode/compactor/bump_schema_version_compactor.go`: collect the source LOB files into a `LOBCompactionContext`, pass `storage.WithTextRefsAsBinary()`, update per-LOB-file `valid_rows` (`SetSegmentRowStats`), and merge the LOB references into the output manifest (`applyLOBCompaction`) **before** building the text-match index — `createTextIndex` reads the TEXT column through this manifest, so the carried LOB files must already be referenced (matches sort/mix ordering). Removed the `TODO(#50021)`. - Unit tests: forced strategy, init/apply LOB wiring, the applyLOBCompaction-before-createTextIndex ordering (`TestFullRewriteMergesLOBRefsBeforeBuildingTextIndex`), and a real add-nullable-TEXT + drop-field full-rewrite regression driving the actual packed writer (`TestFullRewriteFillsMissingNullableTextAsBinary`). ### Also: guard add_function_field behind StorageV3 (#51167) Adding a function to an existing collection must backfill the new output field into **pre-existing** sealed segments; that backfill runs through `bump_schema_version` compaction, which only works on a StorageV3 segment. A pre-existing V2 segment is only backfilled after the storage-version upgrade compaction rewrites it to V3, and that upgrade runs only when **both** `common.storage.useLoonFFI` and `dataCoord.compaction.storageVersion.enabled` are on. The proxy now rejects add-function unless both are enabled (`validateAddFunctionRequiresStorageV3`, wired into `addCollectionFunctionTask` and `alterCollectionSchemaTask`). `create_collection` with a function is unaffected. Full reasoning in #51167. ### Notes - Reuses the LOB machinery from **#50784**. - The `GenerateEmptyArrayFromSchema` binary-TEXT production fix landed independently on master via **#51124**; this PR keeps the regression test guarding the bump path. issue: #50021, #51167 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: MrPresent-Han <chun.han@gmail.com> Co-authored-by: MrPresent-Han <chun.han@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
XuanYang-cnandGitHub
|
d9bb262d71 |
enhance: simplify tsoutil timestamp composition (#51120)
Make ComposeTSByTime the logical-zero time-based timestamp API, add ComposeTSByTimeWithLogical for explicit logical composition in tests, and remove the duplicate GetCurrentTime wrapper. See also: #51119 Signed-off-by: yangxuan <xuan.yang@zilliz.com> Signed-off-by: yangxuan <xuan.yang@zilliz.com> |
||
|
|
879312922b |
fix: use binary arrays for nullable text lob refs (#51124)
issue: #50425 Signed-off-by: luzhang <luzhang@zilliz.com> Co-authored-by: luzhang <luzhang@zilliz.com> |
||
|
|
65307b9d14 |
fix: sync analyzer runtime options (#50997)
issue: #50931 ## Summary Initialize analyzer runtime options with Lindera download URLs and the default dictionary path. Register runtime config callbacks so analyzer yaml updates are propagated to the Rust analyzer layer. Signed-off-by: aoiasd <zhicheng.yue@zilliz.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
sijie-ni-0214andGitHub
|
331cf54bef |
fix: Avoid blocking compactor executor slot queries (#50789)
## Summary Fixes #50788 The DataNode compactor executor used the same mutex for task state, slot accounting, channel enqueueing, completion callbacks, and filesystem metric publication. When the task queue was full, `Enqueue` could block while holding the mutex, which also blocked `Slots()` and made DataNode `QuerySlot` time out. This PR keeps the executor mutex scoped to in-memory state updates only: - release the mutex before sending to `taskCh` - update terminal task state and release slot usage before completion callbacks - publish filesystem metrics outside the executor lock - add regression tests for full task queues and completion callbacks querying slots ## Test Plan - `go test -v -count=1 -tags dynamic,test -gcflags="all=-N -l" -ldflags="-r ${TEST_MILVUS_WORK_DIR}/cmake_build/lib -r ${TEST_MILVUS_WORK_DIR}/internal/core/output/lib" ./internal/datanode/compactor -run "TestCompactionExecutor" -timeout 300s` - `gofumpt -l internal/datanode/compactor/executor.go internal/datanode/compactor/executor_test.go` - `git diff --check` Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com> |
||
|
aoiasdandGitHub
|
950062c78b |
fix: preserve segment insert log paths for text index (#50959)
issue: #50865 ## Summary Preserve existing insert log paths when building segment insert files for StorageV2 text index builds. Fall back to constructing paths from log IDs when the binlog path is absent. Signed-off-by: aoiasd <zhicheng.yue@zilliz.com> |
||
|
|
ac74b0988e |
enhance: isolate online-write cgo pool from segment loading and add pool metrics (#50812)
issue: #50811 related: #49435 ## What On the QueryNode, the load-time delta-log replay (`LoadDeltaData`, `UseLoad=true`) and the online forward delete (`segment.Delete`) both submit their CGO onto the **same `DynamicPool`** (size `CPUNum`). After a compaction, the multi-partition replay burst exhausts the pool, starving online `Insert`/`Delete`. Because `deleteNode.Operate` runs `ProcessDelete` before `UpdateTSafe`, channel **tSafe freezes for tens of seconds**, and Strong-consistency queries fail with `channel tsafe stalled` (#49435). Evidence from the failing master run (traceID `b31d6e1626bd4d368961c89e6d4a5652`): `update tsafe` / `start to process delete` had a ~19–27s gap exactly overlapping a post-compaction segment load; `milvus_querynode_pool_active_threads{pool_name="DynamicPool"}` was pegged at **8/8 for ~27s** while `LoadPool` sat idle at **0/40**. ## Changes Pool isolation + cleanup (`internal/querynodev2/segments`): - Add a dedicated **MutatePool** for the online-write CGO path (`segment.Insert`/`segment.Delete`), isolated from load/management work. - Route load-time **`LoadDeltaData`** replay and **BM25 stats** load onto **LoadPool**. - Remove orphaned **WarmupPool**; merge away **BM25LoadPool**. - Keep **DeletePool** (it is the `DeleteBatch` dispatch pool, not orphaned). - Make **DynamicPool**/**DeletePool** size-configurable + hot-reloadable (`CPUNum * factor`, default `CPUNum`); add `MutatePoolSizeFactor` / `DynamicPoolSizeFactor` / `DeletePoolSizeFactor`. Observability: - Export `capacity / active_threads / queue_depth` for all QueryNode pools (`milvus_querynode_pool_*`, MutatePool added) and, newly, for the important DataNode pools (`milvus_datanode_pool_*`: compaction-exec, index-build, import-exec, io, stats). ## Notes - This fixes the **resource-exhaustion** half of #49435. The `deleteMut` lock decoupling (moving the worker RPC out of the lock so it only guards the in-memory catch-up→publish boundary) is tracked as follow-up. - The pool "priority" coefficients are pool-size only (no OS scheduling priority); isolation guarantees online-write admission, not CPU preemption under full saturation. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: xiaofanluan <xf@hjjaq.com> Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com> Co-authored-by: xiaofanluan <xf@hjjaq.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
2310c93b15 |
fix: align growing source storage v3 flush (#50738)
issue: #50697 Summary: 1. Align StorageV3 column group layout for growing-source flush by reusing the segment currentSplit/manifest layout, passing a schema-based split pattern to C++, and using the schema_based writer policy instead of always writing a single column group. 2. Project growing flush output to the target segment layout by passing AllowedFieldIDs to C++, so ordinary fields, vector fields, text fields, and BM25 output fields are all filtered to the old segment layout when appending to an existing StorageV3 manifest. 3. Stop retrying non-retryable layout mismatches and preserve V3 layout metadata, so column count/group mismatches do not loop forever and recovery/balance can restore currentSplit for future layout-compatible appends. Signed-off-by: luzhang <luzhang@zilliz.com> Co-authored-by: luzhang <luzhang@zilliz.com> |
||
|
wei liuandGitHub
|
4d24af7063 |
feat: support Milvus snapshot as external table source (#49914)
issue: #45881 ## Summary Add the `milvus-table` external format so external collection refresh can consume Milvus snapshot metadata and StorageV3 segment manifests directly. This change lets refresh resolve source manifest row counts, rebuild target StorageV3 manifests from source column groups, copy source deltalogs with deterministic preallocated IDs, and keep real external primary-key semantics for milvus-table segments. ## Why Snapshot-backed external tables are not generic file tables. The refresh path needs to preserve Milvus StorageV3 manifest structure, deltalogs, and real PK behavior instead of falling back to virtual PK semantics or format-agnostic fragment handling. ## Details - Add milvus-table external spec parsing and snapshot manifest resolution. - Add StorageV3 manifest translation for source Milvus segment manifests. - Copy milvus-table deltalogs into target manifests with deterministic LogIDs. - Wire QueryNode real-PK lookup/load handling for milvus-table segments. - Add unit and Go client coverage for spec parsing, manifest resolution, refresh, and deltalog handling. ## Validation - `git diff --cached --check` - No proto changes detected - No mockery patterns in changed test files - `source ~/.profile && source scripts/setenv.sh && make milvus` - Reset Milvus local standalone before Go tests - `go test -tags dynamic,test -gcflags="all=-N -l" -ldflags="-r ${RPATH}" github.com/milvus-io/milvus/internal/datanode/external github.com/milvus-io/milvus/internal/storagev2/packed -count=1 -timeout 300s` - `source ~/.profile && source scripts/setenv.sh && make lint-fix` Related: [#45881](https://github.com/milvus-io/milvus/issues/45881) --------- Signed-off-by: Wei Liu <wei.liu@zilliz.com> |
||
|
sparknackandGitHub
|
8dad593ba5 |
enhance: stream field data loading (#50478)
- Replace `common.entryStream.streamBudgetRatio` with refreshable `common.loadTransientBudgetBytes`, a process-wide transient load budget shared by scalar index V3 entry streaming and storage v2/v3 field-data loading. `0` keeps the limit disabled. - Gate field-data batch reads on the shared budget, split batches by loading overhead, scale read parallelism from the effective batch budget, and finalize `GroupChunk` cells before pushing them so Arrow tables are released promptly. - Align MCL loading-overhead reservations for storage v1 scalar indexes and storage v2/v3 field data under one load-transient overhead group, with array-field overhead and mmap file usage accounted separately. - Propagate load cancellation into budget waits, field-data batch tasks, and V3 scalar `IndexEntryReader` stream/file reads so canceled loads do not stay blocked behind transient memory pressure. - Update segcore init/config watchers and add C++/Go coverage for budget sizing, cancellation, field-data batching/finalization, V3 entry streaming, and the new config. issue: #49499 --------- Signed-off-by: Shawn Wang <shawn.wang@zilliz.com> |
||
|
|
656b600a46 |
fix: preserve text lob refs in compaction (#50784)
issue: #50770 Signed-off-by: luzhang <luzhang@zilliz.com> Co-authored-by: luzhang <luzhang@zilliz.com> |
||
|
|
b4f8684ce9 |
fix: preserve text lob metadata in merge-sort compaction (#50754)
issue: #50730 Signed-off-by: luzhang <luzhang@zilliz.com> Co-authored-by: luzhang <luzhang@zilliz.com> |
||
|
|
426b7f9260 |
enhance: build text index inline in mixcompaction (#50140)
issue: #50145 ## Summary Make `mixCompactionTask` build text indexes inline for `enable_match` fields (mirroring `sortCompactionTask`) so that mix-compaction output segments arrive at QueryNode already carrying `TextStatsLogs`. This eliminates the CGO_LOAD `CreateTextIndex` fallback path in segcore — observed in production to redundantly rebuild text indexes during segment load for tens of seconds up to 13+ minutes per segment. ## Background For a freshly mix-compacted segment with text-match fields, the load pipeline currently races: 1. Vector index task completes within ~10s (event-triggered) 2. QueryCoord treats the segment as loadable once the vector index is ready 3. The `TextIndexJob` (stats task) is event-less — only the 60s polling ticker submits it 4. The job itself takes ~2 minutes for typical text-heavy segments 5. QueryNode receives the segment before its persistent text index is written; segcore detects `text_stats_logs == nil` and calls `CreateTextIndex` in-place at load time (`[CGO_LOAD]` scope) `sortCompactionTask` already avoids this by building the text index inline as part of compaction (`internal/datanode/compactor/sort_compaction.go:469-503`). `mixCompactionTask` did not — this PR closes the gap. ## What changes - **`internal/datanode/compactor/mix_compactor.go`** — Add `cm storage.ChunkManager` field; constructor takes one extra arg; add a thin `createTextIndex` method wrapper that delegates to the existing package-level `createTextIndex` helper (in `compactor_common.go`, already used by sort compaction); after `applyLOBCompaction`, loop over each non-empty **sorted** output segment, build text indexes, update the V3 manifest, and assign `TextStatsLogs`. Emits a `create_text_index` stage latency metric. (No new shared helper is introduced — this PR reuses `compactor_common.go:createTextIndex` and `sort_compaction.go` is untouched.) - **`internal/datanode/compactor/namespace_compactor.go`** — Forward `cm` to the inner `mixCompactionTask`. - **`internal/datanode/services.go`** — Updated call sites (pass `cm`). - **`internal/datacoord/compaction_task_mix.go`** — Extend the `FileResources` plumbing condition to include `MixCompaction` so custom analyzers (ref mode) reach the inline build. Previously only `SortCompaction` got `FileResources`, which would have caused mix-compacted text indexes to use default tokenization → silent search regressions. - **`internal/datacoord/compaction_task_clustering.go` / `compaction_inspector.go`** — Namespace-enabled `ClusteringCompaction` is routed on the DataNode to `NewNamespaceCompactor`, which delegates to `mixCompactionTask.Compact()` and now builds the text index inline for sorted-by-namespace outputs. Wire `IndexEngineVersionManager` into `clusteringCompactionTask` and, in `BuildCompactionRequest()`, set `CurrentScalarIndexVersion` (otherwise the inline text index metadata is emitted with version 0) and fetch `FileResources` for namespace-enabled clustering plans in ref mode (otherwise the custom analyzer resources are not downloaded and the text index falls back to the default analyzer). Addresses review feedback from @aoiasd. - **`internal/datanode/compactor/mix_compactor_test.go` / `namespace_compactor_test.go`** — Updated existing constructor call sites to pass the new `cm` arg. - **`internal/datanode/compactor/mix_compactor_text_index_test.go` (new)** — Mockey-based unit tests covering: the wrapper's identifier propagation to the package helper, error propagation, and real `Compact()` inline-loop semantics (empty + unsorted segments skipped, `TextStatsLogs` assigned for sorted, V3 manifest rewritten, build/manifest errors abort). - **`internal/datacoord/compaction_task_mix_test.go`** — Test that `MixCompaction` plans receive `FileResources` in ref mode. ## Test plan - [x] `go test -tags dynamic,test -gcflags="all=-N -l" -count=1 -run "TestMixCompaction|TestNamespace|TestSort" ./internal/datanode/compactor/...` — PASS (post-rebase) - [x] `go test -tags dynamic,test -gcflags="all=-N -l" -count=1 -run "TestMixCompaction|TestCompactionTask" ./internal/datacoord/...` — PASS - [x] New unit tests pass: `TestMixCompaction_createTextIndex_Delegates`, `TestMixCompaction_createTextIndex_PropagatesError`, `TestMixCompaction_Compact_InlineTextIndex`, `TestMixCompaction_Compact_InlineTextIndex_BuildError`, `TestMixCompaction_Compact_InlineTextIndex_ManifestError`, and (datacoord) `TestMixCompactionTaskSuite/TestBuildCompactionRequest_MixFileResourcesInRefMode`, `TestClusteringCompactionTaskSuite/TestBuildCompactionRequest_NamespaceFileResourcesInRefMode` - [ ] CI builds against fresh C++ artifacts (local worktree build was skipped because the C++ shared libs in this workspace are stale relative to upstream's recent cgo changes — unrelated to this PR's diff) - [ ] Manual / integration: verify a freshly mix-compacted segment with text-match fields lands on QueryNode with non-empty `text_stats_logs` and does NOT trigger the `[CGO_LOAD] CreateTextIndex` slow path in QueryNode logs 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: cai.zhang <cai.zhang@zilliz.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Zhen YeandGitHub
|
b07c62d53c |
enhance: replace log package with context-aware mlog (#50094)
issue: #35917 - mlog package: move logger initialization, zap core, async buffered writes, field helpers, and scoped logger binding into pkg/mlog while removing pkg/log. - logging callsites: migrate Milvus logging usage to context-aware mlog APIs and simplify redundant With chains across components, utilities, tests, and tools. - trace propagation: replace logutil trace interceptors with mlog/tracer integration and add client_request_id fallback propagation for server stats handlers. --------- Signed-off-by: chyezh <chyezh@outlook.com> |
||
|
XuanYang-cnandGitHub
|
aa8331d4eb |
enhance: size compaction result ID preallocation (#49919)
Derive compaction result segment ID reservations from the expected output segment count instead of fixed ranges. - Estimate output segment counts from total input size and target segment size for single, sort, clustering, storage-version-upgrade, force-merge, and schema-version compactions. - Allocate result segment IDs and task metadata IDs in one contiguous RootCoord block, with the result segment range fixed at the front and metadata IDs consumed from the tail. - Add CompactionView helpers for total input size and collection TTL so scheduler submission paths share the same size-based allocation flow. - Reserve result-segment IDs with dataCoord.compaction.preAllocateIDExpansionFactor so result segment and metadata ID preallocation share the existing compaction ID factor. - Update milvus-proto to e90ee63ad3fc for snapshot export and external restore APIs, and refresh affected service mocks. - Validate nil or external force-merge and schema-version collections before ID allocation, and keep compaction-local oversized allocation rejection before RootCoord AllocN. - Keep schema-version compaction on the current BumpSchemaVersion path and preserve the frozen schema captured by the view. - Cover allocation sizing, range layout, defensive returns, force-merge target counts, schema-version submission, and oversized allocation rejection. See also: #49955 Signed-off-by: yangxuan <xuan.yang@zilliz.com> --------- Signed-off-by: yangxuan <xuan.yang@zilliz.com> |
||
|
congqixiaandGitHub
|
78db266996 |
enhance: set external refresh segment level to L1 (#50525)
External collection refresh created new segments without an explicit level, so the protobuf default marked them as Legacy. Set the level to L1 to align the stored metadata with the normal non-L0 segment semantics and keep metrics/observability consistent. Signed-off-by: Congqi Xia <congqi.xia@zilliz.com> |
||
|
|
e2787d3981 |
enhance: standardize error handling on merr + Sys/Input classification (#50221)
issue: #47420 ## What this PR does Project-wide migration of raw `fmt.Errorf` / `errors.New` in function bodies onto the `merr` framework, plus the Sys-vs-Input error classification and the machinery it drives (retriability, fine-grained metrics, segcore unification), plus the convention docs and a linter that keeps it from regressing. Scope: storage, proxy, coordinators (root/data/query), query node, data node, `pkg/util` & `internal/util`, expression parser, message queue, streaming, and misc packages. Bare raw-error usages went from ~3000 to a ~340 allowlist (package-level sentinels / build-tag / test sites). --- ## How to review this PR It is large but the vast majority is mechanical. Changes fall into three tiers; spend review budget on Part 2 and Part 3. ### Part 1 — Mechanical standardization (low risk, verify by rule) Each converted call follows one of a small fixed set of rules. To review, check that each site obeys the matching rule rather than reading every line: | Pattern | Rule | |---|---| | `fmt.Errorf("...")` originating a new error | → `merr.WrapErrXxxMsg("...")` with a code matching the failure's meaning | | Adding context to an existing typed error | → `merr.Wrap(err, "...")` / `merr.Wrapf(...)` — **preserves** the inner code (never `WrapErr*Err`, which overwrites it) | | Errors inside the streaming subsystem | → `status.New*` factories (StreamingError), **not** merr — this is the component-internal dialect (see `docs/dev/error_handling_guide.md`) | | Low-level / control-flow signal caught by `errors.Is` | → kept as a package-level `errors.New` sentinel (lowercase, same-package) | Conventions are documented in `docs/dev/error_handling_guide.md` (how-to) and `docs/dev/error_sentinel_convention.md` (rules + audit). A `gocritic`/`ruleguard` rule (`rawmerrerror`, in `rules.go`) enforces "no raw `return errors.New/fmt.Errorf`" under `make verifiers`. ### Part 2 — Behavior changes (review these closely) These are the sites where the wire contract or runtime behavior changes, not just the source text. Listed by category; representative locations given, full set in the diff. **A. gRPC wire-code shifts: `UnexpectedError(1)/Code 65535` → typed code.** Where a handler previously returned a raw error (collapsed to `Code=65535` on the wire), it now returns a typed merr, so the client sees a real code. The most common shift is to `IllegalArgument(5)/Code 1100` (ParameterInvalid). Touch points include datanode task handlers (CreateTask/Query/Drop), proxy Upsert, querynode GetMetrics, datacoord CreateIndex, httpserver query-response builder, and typeutil schema validation. One code refinement: an index-param validation moved `1100` → `1101` (ParameterMissing). **Client/SDK assertions and any code that switched on `Code=65535` for these paths must be re-checked** (the go_client e2e assertions were already aligned in this PR). **B. Prometheus `status` label contract change (externally visible).** The proxy metric's coarse `fail` / `rejected` values are split into `fail_input` / `fail_system` and `rejected_user` / `rejected_system` (in `requestutil.ParseMetricLabel`; auth/privilege rejections count as `rejected_user`), so dashboards can attribute a failure to caller vs operator. **Dashboards/alerts querying `status="fail"` must migrate to `status=~"fail_.*"`, and `status="rejected"` to `status=~"rejected_.*"`.** The in-repo Grafana dashboard is already migrated; external dashboards built on the old values silently go empty after upgrade. This is the one change that requires an ops-side migration. **C. Retriability semantics.** - C1: `merr.Status(err)` now forces `Retriable=false` when the error is an `InputError` — a malformed request can never succeed on blind retry, so clients never get the self-contradictory "your input is wrong but you may retry". - C2: `retry.Do` short-circuits an `InputError` (non-retriable) — **but only when the caller did not pass a `RetryErr` predicate**. The check is an `if c.isRetryErr != nil { ... } else if InputError { ... }` *mutually exclusive* branch (`pkg/util/retry/retry.go`): an explicit `RetryErr` takes precedence and bypasses the InputError abort. `retry.Handle` deliberately does **not** apply the InputError abort (its callers signal abort via `shouldRetry=false`). Four flusher startup callsites that must retry through transient "not ready" errors were given explicit `RetryErr` escape hatches. **D. segcore (C++→Go) error classification.** A single shared Go-side table (`pkg/util/merr/segcore.go`) maps each segcore code to a merr sentinel + InputError/signal category, replacing scattered hand-written `if errorCode == ...` switches in the cgo wrappers. **Wire `Code` values change for every segcore pass-through error, not just the remapped ones.** Named sentinels remap (C++ `2003` → merr `2001`, `2033` → `2002`, Folly/Knowhere codes likewise); **all remaining pass-through codes (`2004`–`2043`, previously surfaced to clients as raw C++ enum values) now serialize as `2000`** (`ErrSegcore`), with the original C++ code preserved in the `Reason` text (`segcoreCode=...`); unknown/future codes collapse to `2000` as well (pinned by the `wire_code_projection` test). Transient segcore classes (object storage / file IO / OOM / mmap / FieldNotLoaded — 11 codes) now report `Retriable=true`. **Any client switching on raw segcore codes in the `2004`–`2043` range must be re-checked**; the in-Reason code remains available for diagnostics. Signal codes (PretendFinished / FollyCancel) are recognized centrally. `errors.Is`-based control flow on these (e.g. scheduler skip/retry) is preserved. **E. InputError classification (25 sentinels + dynamic marks).** 25 sentinels in `errors.go` carry `WithErrorType(InputError)` (the Collection / ResourceGroup / Database families, `ErrIndexDuplicate`, `ErrParameterInvalid`, `ErrPrivilegeNotAuthenticated`, `ErrImportFailed`, `ErrQueryPlan`, ...), plus dynamic marks for the 8 segcore input codes (ExprInvalid, DimNotMatch, MetricTypeInvalid, FieldIDInvalid, ...) and `WrapErrAsInputError`. The widest blast radius is `ErrParameterInvalid` (1100): ~2335 `WrapErrParameterInvalid*` callsites now classify as input / non-retriable. Because of C1/C2 this changes retriability for any path that returns these. **The audit to confirm no transient path was mis-marked is the single most important review item** (see Part 3). One reverse correction: storage field-stats parsing moved from `ErrParameterInvalid` (input) to `ErrDataIntegrity` — a corrupted stored stat is data corruption, not user input. ### Part 3 — Known risks & traps (called out proactively) 1. **`merr.Wrap` vs `WrapErr*Err` (code-masking).** `WrapErr*Err` builds a `wrappedMilvusError{sentinel: ErrServiceInternal}` whose `code()` returns the *outer* sentinel — it overwrites the inner typed code and hides the `errors.Is` chain. This is intentional (use it to *deliberately* downgrade), but it was a recurring conversion defect; the rule "add context with `merr.Wrap`, downgrade with `WrapErr*Err`" is enforced by convention and reviewed across the diff. 2. **InputError × `retry.Do` blast radius.** Marking a sentinel `InputError` makes any `retry.Do(...)` without a `RetryErr` predicate stop retrying it. Reviewers should sanity-check that no transient use of the 19 newly-marked sentinels (especially `ErrParameterInvalid`) sits inside a retry loop that needed to keep spinning. The known flusher cases were handled (see C2). 3. **The ~340 raw-error allowlist.** What remains as bare `errors.New` is, by design: package-level sentinels (caught by `errors.Is`), `//go:build test` sites, and out-of-band trees (`cmd/`, `tests/`, codegen, walimpls). The linter only bans the *direct-return* form; assignment-then-return escapes and the full no-exceptions ban are deferred to an AST-based linter (Tier 2, documented). 4. **segcore C++ second step deferred.** This PR unifies classification on the Go side; splitting the dual-semantic C++ codes at the source is a follow-up. --- ## Validation - `make verifiers`: Go side clean (gofmt + static-check across modules, including the new `rawmerrerror` rule with a 0-hit baseline repo-wide). - `make test-go`: passing; the one real regression introduced (a datanode `invalid_task_type` assertion shifting `1` → `5` from a ParameterInvalid conversion) was fixed in-tree. - go_client e2e CreateIndex assertions aligned to the new merr messages. --------- Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
80257b9f97 |
enhance: support text_match/bm25 for text type (#50426)
issue: #48783 Signed-off-by: luzhang <luzhang@zilliz.com> Co-authored-by: luzhang <luzhang@zilliz.com> |
||
|
marcelo-cjlandGitHub
|
6b7eeb09dd |
fix: support CPU adaptation for GPU CAGRA loading (#50096)
## Summary - Treat GPU CAGRA indexes with `adapt_for_cpu` enabled as CPU-adapted in QueryNode resource checks. - Apply the same GPU requirement check when estimating segment load resources and collection GPU index flags. - Keep the Milvus-side OpenBLAS Conan option needed by the current Knowhere v3.0.3 dependency set. issue: #49836 ## Test plan - [x] `git diff --check origin/master..HEAD` - [x] `GO_DIFF_FILES="internal/querynodev2/segments/collection.go internal/querynodev2/segments/collection_test.go internal/querynodev2/segments/segment_loader.go internal/querynodev2/segments/segment_loader_test.go" make fmt` - [x] `make milvus-gpu > output.txt 2>&1` - [x] `env PKG_CONFIG_PATH=/home/ubuntu/data/skills/issue_review/milvus_issue_49836/milvus/internal/core/output/lib/pkgconfig LD_LIBRARY_PATH=/home/ubuntu/data/skills/issue_review/milvus_issue_49836/milvus/internal/core/output/lib:/home/ubuntu/data/skills/issue_review/milvus_issue_49836/milvus/internal/core/output/lib64 go test -count=1 -tags test ./internal/querynodev2/segments -run 'TestGpuIndexRequiresGpu|TestCollectionManagerSuite/TestGpuIndexFlagWithCagraAdaptForCPU'` Signed-off-by: marcelo-cjl <marcelo.chen@zilliz.com> |
||
|
|
59401b0fa4 |
enhance: support minhash schema bump backfill (#50330)
## What this PR does This PR supports end-to-end add-MinHash-function schema evolution by: - allowing RootCoord alter-schema to accept MinHash functions with strict arity validation; - adding a concrete MinHash record materializer in DataNode compactor; - extending bump schema version compaction to materialize missing MinHash BinaryVector outputs while keeping BM25 stats BM25-only. ## Tests - source ./scripts/setenv.sh && go test -v -count=1 -gcflags="all=-N -l" -tags dynamic,test ./internal/datanode/compactor -run 'TestBumpSchemaVersionCompactionTaskSuite/TestBumpSchemaVersionCompactionMaterializesMinHashOutput' - source ./scripts/setenv.sh && go test -v -count=1 -gcflags="all=-N -l" -tags dynamic,test ./internal/datanode/compactor -run 'Test(BumpSchemaVersionCompactionTaskSuite|RecordMaterializer|BM25FunctionMaterializer|MinHashFunctionMaterializer)' - source ./scripts/setenv.sh && go test -v -count=1 -gcflags="all=-N -l" -tags dynamic,test ./internal/rootcoord -run TestDDLCallbacksBroadcastAlterCollectionSchema issue: #48808 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: MrPresent-Han <chun.han@gmail.com> Co-authored-by: MrPresent-Han <chun.han@gmail.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
jiaqizhoandGitHub
|
54acb3e4b4 |
enhance: preserve storage v3 writer formats from manifests when appending (#50227)
issue: #50182 Storage V3 can now keep writing a segment after `dataNode.storage.format` changes. The main problem was that an already-started growing segment may have a committed manifest written with one format, while the current global config has moved to another format. In that case, the next append must follow the format already recorded in the manifest instead of blindly using the new global config. This keeps writer.format as the current global setting, but makes the actual split policy format local to the writer path. QueryNode growing flush resolves writer.split.single.format from the acknowledged manifest when one exists, and falls back to the global config only for a brand-new segment. DataNode Storage V3 flush carries column group formats through SegmentInfo and passes them as writer.split.schema_based.formats, so schema-based writes can append with the formats already known for each column group. The metadata path now records the column group format in FieldBinlog and preserves it through recovery, merge, duplicate filtering, fake external segment binlogs, and V3 compaction output. This lets metacache rebuild the current split with enough information to choose writer formats without reading the manifest on every flush. The storage writer wrappers now accept explicit writer format properties for both regular packed writers and TEXT-aware segment writers. TEXT LOB files still use their own LOB path and storage behavior; the format selection here applies to the segment column groups that store normal data and LOB references, not to changing the LOB payload file layout. Signed-off-by: jiaqizho <jiaqi.zhou@zilliz.com> |
||
|
grootandGitHub
|
965e4b1564 |
build(deps): Upgrade knowhere and manage milvus-common via conan (#49912)
issue: https://github.com/milvus-io/milvus/issues/47425 # Plan: upgrade Milvus to Knowhere with Conan `milvus-common` and Conan OpenBLAS ## Context Milvus currently fetches both `knowhere` and `milvus-common` from source with CMake `FetchContent`. The requested Knowhere commit `0e53e57cd1f67f74decbdf9af42e74d0e3850155` expects `milvus-common` and Linux OpenBLAS from Conan 2.x instead: - `milvus-common/1.0.0-9ca5ea6@milvus/dev#274d428d85f1d3d996e1092f0c9c7144` - `find_package(milvus-common REQUIRED)` and target `milvus-common::milvus-common` - Linux `openblas/0.3.30` with CMake target `OpenBLAS::OpenBLAS` - macOS Apple BLAS/LAPACK via `BLA_VENDOR Apple`, not Conan OpenBLAS Milvus also currently installs system OpenBLAS in builder Dockerfiles, runtime Dockerfiles, dependency scripts, and packaging scripts. The intended outcome is one build-time OpenBLAS source on Linux: Conan. Runtime packaging should use the Conan-built shared libraries instead of relying on distro `libopenblas-dev`/`openblas-devel` where possible. ## Implementation 1. **Update Conan dependency pins in `internal/core/conanfile.py`.** - Add direct requirement `milvus-common/1.0.0-9ca5ea6@milvus/dev#274d428d85f1d3d996e1092f0c9c7144`. - Align dependencies required by Knowhere `0e53e57c`: - `folly/2026.04.20.00@milvus/dev#06852bea5b6449f0c4eb0df002b5779c` - `fmt/11.2.0#eb98daa559c7c59d591f4720dde4cd5c` - `lz4/1.10.0#982d9b673900f665a1da109e09c17cab` - `fast_float/8.0.0@milvus/dev#c7802833c74c5a86ffed70e4af1a795e` - Linux-only `openblas/0.3.30` - Add `openblas/*:dynamic_arch = True` to default options to match Knowhere. 2. **Move Milvus core build settings to C++20.** - Change `internal/core/CMakeLists.txt` from `CMAKE_CXX_STANDARD 17` to `20`. - Keep `OPENTELEMETRY_STL_VERSION=2017` unchanged; it controls OpenTelemetry ABI mode, not the language standard. - Change all `compiler.cppstd=17` Conan install settings in `scripts/3rdparty_build.sh` to `compiler.cppstd=20`. - Review `internal/core/src/storage/gcp-native-storage/CMakeLists.txt`; if it inherits incompatible flags or includes C++20-dependent headers, change its local `CMAKE_CXX_STANDARD 17` to `20` too. 3. **Expose Conan packages to Milvus CMake.** - Add `find_package(milvus-common REQUIRED)` in `internal/core/CMakeLists.txt` near the existing Conan `find_package(...)` block. - Add Linux-only `find_package(OpenBLAS CONFIG REQUIRED)` in `internal/core/CMakeLists.txt` so missing Conan OpenBLAS fails before Knowhere configures. - Remove `-DOpenBLAS_SOURCE=AUTO` from `internal/core/build.sh`; the requested Knowhere commit no longer consumes that flag and directly calls `find_package(OpenBLAS CONFIG REQUIRED)` on non-Apple platforms. 4. **Stop executing source FetchContent for `milvus-common`.** - Remove `add_subdirectory(milvus-common)` from `internal/core/thirdparty/CMakeLists.txt`. - Keep `add_subdirectory(knowhere)`. - Do not keep a compatibility alias to the old source target; all consumers should use the Conan target. - Leave `internal/core/thirdparty/milvus-common/` files in place for the first pass unless cleanup is explicitly desired; they will no longer be active once the subdirectory is removed. 5. **Pin Knowhere to the requested commit.** - Update `internal/core/thirdparty/knowhere/CMakeLists.txt`: - `KNOWHERE_VERSION` -> `0e53e57cd1f67f74decbdf9af42e74d0e3850155`. - Use the full SHA for deterministic FetchContent checkout. 6. **Migrate CMake links/includes from source target to Conan target.** - In `internal/core/src/CMakeLists.txt`: - Remove `${MILVUS_COMMON_INCLUDE_DIR}` from include directories. - Add `milvus-common::milvus-common` to `CONAN_TARGETS` so `milvus_conan_deps` propagates include directories to object libraries. - Remove plain `milvus-common` from `LINK_TARGETS`. - In `internal/core/unittest/CMakeLists.txt`: - Remove `${MILVUS_COMMON_INCLUDE_DIR}` from include directories. - Remove plain `milvus-common` from `all_tests` and `test_json_uint64` link lists; both already link `milvus_conan_deps`, which will carry `milvus-common::milvus-common`. 7. **Adjust build-time system OpenBLAS installs.** - Remove Linux build-time OpenBLAS package installs that can shadow or duplicate Conan OpenBLAS: - `scripts/install_deps.sh`: remove Ubuntu `libopenblas-dev`; remove Rocky/Amazon/CentOS `openblas-devel` where only needed for C++ core build. - `build/docker/builder/cpu/ubuntu20.04/Dockerfile`, `ubuntu22.04/Dockerfile`, `ubuntu24.04/Dockerfile`: remove `libopenblas-dev`. - `build/docker/builder/gpu/ubuntu20.04/Dockerfile`, `gpu/ubuntu22.04/Dockerfile`: remove `libopenblas-dev`. - `build/docker/builder/cpu/amazonlinux2023/Dockerfile`, `rockylinux9/Dockerfile`: remove `openblas-devel` and any OpenBLAS header symlink workaround that only supported system OpenBLAS. - `build/deb/build_deb.sh`: remove build-time `libopenblas-dev` install if no longer needed after Conan generation. - Keep Fortran/toolchain packages needed to build Conan OpenBLAS from source when binary packages are missing. - Do not change `scripts/install_deps_embd.sh` or `scripts/install_deps_msys.sh` in the first pass unless the target build path uses them; they are separate embedded/MSYS dependency paths and may still require manual/system OpenBLAS. - On macOS, remove `openblas` from `.github/workflows/mac.yaml` only after confirming the new Knowhere path uses Apple BLAS/LAPACK and Milvus has no other macOS OpenBLAS consumer. 8. **Adjust runtime/package OpenBLAS handling carefully.** - The runtime should still include `libopenblas.so*` if the Milvus binary or Conan-built Knowhere/Faiss links it dynamically. - Prefer copying the Conan-provided OpenBLAS shared library from `cmake_build/lib` or the Conan package output instead of installing distro development packages in runtime images. - Update packaging sites that currently copy or depend on system OpenBLAS: - `build/deb/build_deb.sh`: change the copied source from `/usr/lib/x86_64-linux-gnu/libopenblas.so.0` to the Conan/build output path if present. - `build/rpm/milvus.spec`: stop assuming `/usr/lib/libopenblas-r0.3.9.so`; use the packaged Conan/build output library. - Runtime Dockerfiles under `build/docker/milvus/**`: remove `libopenblas-dev`/`openblas-devel` only after confirming the image gets the Conan OpenBLAS `.so` with Milvus libs. - `build/docker/milvus/gpu/ubuntu20.04/Dockerfile.base`: review the explicit `milvusdb/openblas` stage; replace it with the Conan-produced library if that image path is still active. - Keep runtime `libgomp1`/`libgomp` and `libaio` if required by OpenMP/libaio-linked dependencies. 9. **Clean stale build state before verification.** - Use a fresh `cmake_build` or remove the old FetchContent checkout/cache before rebuilding; stale `MILVUS_COMMON_INCLUDE_DIR`, old `knowhere-src`, or system OpenBLAS paths can hide integration errors. Signed-off-by: yhmo <yihua.mo@zilliz.com> |
||
|
|
cd8b3b11f0 |
fix: fill missing rows for JSON stats build (#49590)
issue: #49495 Signed-off-by: luzhang <luzhang@zilliz.com> Co-authored-by: luzhang <luzhang@zilliz.com> |
||
|
wei liuandGitHub
|
6615811dee |
enhance: Support add-column refresh for external collections (#50082)
issue: #45881 Design doc: docs/design-docs/design_docs/20260526-external_table_add_column_refresh.md This PR lets external collection refresh handle additive schema changes, especially adding columns to parquet-backed external collections after segments have already been refreshed. What changed: - DataNode detects existing same-fragment external segments that are missing newly added external columns, appends manifest column groups idempotently, rebuilds fake binlogs, and returns same-ID updated segments. - DataCoord applies kept and updated refresh task payloads as validated segment upserts, including both patched existing segments and newly created segments. - Refresh apply intentionally does not use a job/task schema-version gate for the current additive-only scope. An older-schema refresh may complete without the newest fields, and the next refresh self-heals missing external columns. Segment-level validation still rejects schema-version rollback. - Proxy and RootCoord validate external add-field and alter-schema flows after field IDs are resolved, rejecting duplicate external_field mappings, generated output collisions, and unsupported external field types. - StorageV2 packed manifest helpers can read existing column groups and append missing columns through CommitManifestUpdates. - Patched same-fragment segments include function-output bytes in fake binlog MemorySize so memory estimates include generated columns. - Added the add-column refresh design doc and linked it from the external table design doc. - Added a go-client e2e that adds parquet column score after the first refresh and verifies returned query values, including 0..4 -> 0..0.04 and 100..104 -> 1.00..1.04. Validation: - make generated-proto-without-cpp - make milvus - make lint-fix - git diff --check - git diff --check milvus/master...HEAD - No-new-mockery diff scan - reset Milvus before scoped Go test runs - go test -tags dynamic,test -gcflags="all=-N -l" -ldflags="-r ${RPATH}" for targeted typeutil, proxy, datacoord, and storagev2/packed tests - go test -tags dynamic,test -gcflags="all=-N -l" -ldflags="-r ${RPATH}" github.com/milvus-io/milvus/internal/rootcoord -run '^$' - go test -tags dynamic,test -gcflags="all=-N -l" -ldflags="-r ${RPATH}" github.com/milvus-io/milvus/internal/datanode/external -run TestRefreshExternalCollectionTaskSuite -testify.m TestOrganizeSegments_PatchesMissingExternalColumns - go test -tags dynamic,test -gcflags="all=-N -l" -ldflags="-r ${RPATH}" github.com/milvus-io/milvus/internal/storagev2/packed github.com/milvus-io/milvus/pkg/v3/util/typeutil -run 'TestAppendSegmentManifestColumnsValidationPaths|TestAppendSegmentManifestColumnsReadAndNoopPaths|TestNormalizeAndValidateExternalCollectionSchema' - internal/datacoord full package coverage hit the known local macOS DISKANN subcase; targeted refresh task tests passed after reset Signed-off-by: Wei Liu <wei.liu@zilliz.com> |
||
|
|
4fa49058c1 |
enhance: extend arrow IO thread pool config to DataNode init (#50099)
issue: #50098 ## Summary - Apply `common.arrow.ioThreadPoolCoefficient` / `ioThreadPoolMaxCapacity` to DataNode at startup, mirroring the QueryNode wiring from #49208/#49623. Without this, the paramtable keys exist but have no effect on DataNode and the Arrow IO pool stays at Arrow's hard-coded default of 8. - Also call `initcore.InitArrowReaderConfig` so the parquet reader range-coalescing limits (`hole`/`range` size) apply. - Register paramtable watchers so capacity changes hot-reload without restart, matching QueryNode behavior. - Raise `dataCoord.import.fileNumPerSlot` default from 1 to 4 to reduce per-import slot demand by 4x and leave more arrow IO pool headroom for concurrent compactions on the same worker. ## Why See #50098 for the full reproducer and analysis. Briefly: production observed a 1.5M-row sort compaction taking >1 hour while the worker's CPU sat at 12% and network at 14% of baseline — all 5 large SortCompactions on different pods finished at nearly identical wall times (variance < 1%), the classic shared-pool saturation signature. Local bench at 50ms RTT with 100 concurrent readers: | ARROW_IO_THREADS | conc=100 wall_ms | per-task p50_ms | |---|---|---| | 8 (default) | 36,917 | 35,908 | | 32 | 18,174 | 16,074 | | 64 | 17,320 | 15,702 | ## Test plan - [x] `go test -count=1 -run TestComponentParam ./util/paramtable/` (pkg module) - [x] `go test -tags dynamic,test -gcflags="all=-N -l" -count=1 -run TestRegisterArrowIOThreadPoolWatchers ./internal/datanode/index/` - [x] `go test -tags dynamic,test -gcflags="all=-N -l" -count=1 -run "TestImport" ./internal/datacoord/` - [ ] CI 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: cai.zhang <cai.zhang@zilliz.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
sparknackandGitHub
|
d5917c265e |
enhance: Add V3 entry streaming reader (#50073)
issue: #48957 - Add `IndexEntryReader::ReadEntryStream`, `GetEntrySize`, and `HasEntry` for V3 entries. - Stream plain and encrypted entries through ordered slices with incremental CRC32c verification. - Bound transient stream memory with a process-wide entry stream budget. - Add dynamically refreshable `common.entryStream.streamBudgetRatio` to tune the shared entry stream budget for current scalar index loading and later field data loading. - Use 16MB stream slices by default, aligned with the encrypted slice size and existing V3 range size. - Derive the entry stream budget from CPU core count: `CPU cores * streamBudgetRatio * 16MB`. - Harden stream error handling: - keep active futures bounded by the thread pool size - reject zero or too-small explicit slice sizes - use overflow-safe slice count calculation - verify CRC for empty entries - release budget when callbacks throw or task submission fails - wait for active workers before returning on abnormal exits - validate decrypted slice length and include actual/expected CRC in stream errors - Document V3 streaming read behavior and shared budget configuration. --------- Signed-off-by: Shawn Wang <shawn.wang@zilliz.com> |
||
|
|
7841f8a9b0 |
enhance: use bump_schema_version to involve backfill and drop field compaction(#48808) (#49759)
related: https://github.com/milvus-io/milvus/issues/48808 Signed-off-by: MrPresent-Han <chun.han@gmail.com> Co-authored-by: MrPresent-Han <chun.han@gmail.com> |
||
|
sijie-ni-0214andGitHub
|
bad0a82fb1 |
test: stabilize external collection cancel test (#50074)
#### What this PR does / why we need it:
Stabilizes `TestExternalCollectionManager_CancelTask` by waiting for the
asynchronous manager state update after cancel is observed by the task
function.
`CancelTask` only triggers context cancellation. The task goroutine
records `JobStateFailed` after `taskFunc` returns with
`context.Canceled`, so reading the task info immediately after observing
`ctx.Done()` can race with `UpdateResult` and intermittently see
`JobStateInProgress` with an empty failure reason.
This test now waits until the manager snapshot reports `JobStateFailed`,
then asserts the failure reason separately for clearer failure output.
#### Which issue(s) this PR fixes:
N/A
#### Special notes for your reviewer:
This is a test-only flaky fix.
#### Tests
- `gofumpt -l internal/datanode/external/manager_test.go`
- `git diff --check HEAD~1..HEAD`
- `go test -v -count=1 -tags dynamic,test -gcflags="all=-N -l"
-ldflags="-r ${MILVUS_WORK_DIR}/cmake_build/lib -r
${MILVUS_WORK_DIR}/internal/core/output/lib"
./internal/datanode/external -run
'TestExternalCollectionManager_CancelTask' -timeout 300s`
- `go test -v -count=1 -tags dynamic,test -gcflags="all=-N -l"
-ldflags="-r ${MILVUS_WORK_DIR}/cmake_build/lib -r
${MILVUS_WORK_DIR}/internal/core/output/lib"
./internal/datanode/external -timeout 300s`
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
|
||
|
wei liuandGitHub
|
7bdd40d692 |
enhance: [ExternalTable Part10] enable function output fields on external collections (#49307)
issue: #45881 ## Summary Part 10 of the External Table series. Enables BM25 / MinHash / TextEmbedding function output fields on external collections, and makes external-table text_match work with persisted text indexes. Related: [#45881](https://github.com/milvus-io/milvus/issues/45881) (External Collection Lakehouse Integration tracking). ### What's in this PR **Schema & segcore** - Allow `Function` declarations on external schemas; function-output fields skip `external_field_mapping` validation - `Schema` tracks function-output field IDs; `ChunkedSegmentSealedImpl` and `ManifestGroupTranslator` resolve function-output columns by field name - Fast paths for retrieve / search load function-output columns by field name; external columns continue using `external_field_mapping` - `Util.cpp::GetFieldDatasFromManifest` resolves function-output columns by field name so indexbuilder reads the correct packed column **Refresh pipeline (format-agnostic via loon FFI)** 1. `CreateSegmentManifestWithBasePath` creates the input manifest that references external original files as CG0 2. `FFIPackedReader(v1)` streams input columns into `InsertData` via the shared `ArrowRecordToInsertData` helper 3. `embedding.RunAll` populates function-output fields in-place 4. `FFIPackedWriter` writes output fields as a new column group on top of v1 5. `AddStatsToManifest` registers BM25 stats into the final manifest 6. Memory peak stays around one Arrow batch, defaulting to 64 MiB **RootPath isolation** - Function-output input manifests, output manifests, BM25 stats, and text index artifacts now use the same `RootPath/insert_log/<collection>/<partition>/<segment>` layout as normal external table StorageV3 manifests - This avoids writing function-output artifacts under bucket-root `external/...` paths when multiple clusters share the same bucket **Text match** - External text fields with `enable_match` persist text index artifacts during refresh and load them through the regular external segment path - English and Chinese analyzer coverage are both included in the E2E test **Cross-bucket** - `NewPackedFFIReaderWithManifest` accepts `ExternalReaderContext` and injects per-collection `extfs.{collID}.*` aliases when `external_source` is set. Existing callers pass `{}` and are unaffected **VirtualPK** - `VirtualPKChunkedColumn` implements `GetChunk` / `GetAllChunks` so proxy requery filter-by-PK works for external collections with virtual primary keys **Shared embedding runner** - New `internal/util/function/embedding/runner.go` provides canonical `RunAll(ctx, schema, data, opts)` shared by import and external-table refresh paths - Supports BM25 output vector types including Float, BFloat16, Float16, Binary, and Sparse ### Test Plan - [x] Go unit tests for `internal/datanode/external` with 99.8% package coverage - [x] Go unit tests for changed import, packed, embedding, and schema helpers from the existing branch validation - [x] C++ rebuild + `milvus_storage` / `milvus_core` lib install from the existing branch validation - [x] E2E function-output tests in `tests/go_client/testcases/external_table_function_test.go` - `TestExternalTableBM25Function` - `TestExternalTableMinHashFunction` - `TestExternalTableTextEmbeddingFunction` - [x] `TestExternalTableTextMatch` with 10 files x 5000 rows = 50000 rows, English and Chinese text fields, persisted text index object checks in MinIO, load readiness check, and query correctness checks Signed-off-by: Wei Liu <wei.liu@zilliz.com> |
||
|
Ted XuandGitHub
|
c3207d5963 |
fix: make BulkPackWriterV3.Write atomic with single manifest commit (#49711)
issue: #49710 related: #49069 BulkPackWriterV3.Write currently bumps the loon manifest version up to four times per call (inserts, stats, delta, bm25 each open and commit their own transaction). That breaks atomicity for concurrent readers, leaves orphan manifest versions behind on partial failure, and churns loon's manifest history. This PR refactors the V3 sync write path into a do-then-commit shape: Phase 1 (slow, runs once outside the retry loop): writeInserts, writeStats, writeDelta, writeBM25Stasts write all the data files (parquet/LOB via the loon writer, deltalog, stat blobs) and each helper returns its contribution (ColumnGroups, SegmentOutput, StatEntry list, DeltaLogEntry list) without touching shared state. Phase 2 (fast, retried on transient loon errors): Write assembles one packed.ManifestUpdates from the helper returns and calls a single new primitive packed.CommitManifestUpdates, which opens a short loon transaction, stages every change, and commits once. To make this clean the loon FFI writers are decoupled from manifest concerns entirely: FFIPackedWriter and FFISegmentWriter no longer take a baseVersion at construction. Their Close primitives return data carriers (packed.ColumnGroups, packed.SegmentOutput) that own the C output. packed.CommitManifestUpdates is the only Go-level entry point that mutates a manifest. Empty payload short-circuits to the unchanged manifest path without opening a transaction. The same do-then-commit shape applies to the V2-style binlog import writers (PackedManifestRecordWriter, PackedTextManifestRecordWriter): they assemble inserts plus bloom-filter and BM25 stat blobs into one ManifestUpdates and call CommitManifestUpdates once. Internal renames: the low-level adapters packedRecordManifestWriter and packedTextManifestWriter are renamed to packedRecordBatchWriter and packedTextBatchWriter because they don't touch the manifest themselves — they translate storage.Record into arrow.Record and track per-column-group sizes. The interface manifestRecordWriter is renamed to packedBatchWriter to match. New tests in pack_writer_v3_test.go: TestWrite_SingleVersionBumpAcrossSections drives an insert + delta + flush so all four sections fire and asserts the new manifest version equals baseVer + 1. TestWrite_RetryDoesNotLeakVersionBumps mockey-patches CommitManifestUpdates to fail once with packed.ErrLoonTransient and asserts the eventual successful commit still yields baseVer + 1 rather than baseVer + N_retries. All 11 V3 tests pass. Existing packed and storage tests pass. Out of scope (filed as follow-ups in #49710's discussion): backfill_compactor.go's runBm25Function still issues a separate AddStatsToManifest after the column-groups commit, producing two version bumps per backfill. The architecture supports folding the two but the change isn't strictly needed for the V3 sync atomicity fix. FFIPackedWriter lacks a Destroy method; on loon_writer_close failure the handle and cProperties leak. Pre-existing on master. --------- Signed-off-by: Ted Xu <ted.xu@zilliz.com> |
||
|
|
0ca60c5b47 |
feat: add commit_timestamp to SegmentInfo for correct MVCC/TTL/GC on import/CDC segments (#48472)
## What this PR does
Adds `commit_timestamp` (uint64) to `SegmentInfo` and `SegmentLoadInfo`
so that import/CDC segments use their logical commit time — not raw row
timestamps — for all temporal decisions.
**Problem:** When bulk-insert imports rows, the row timestamps generated
during the import process may be slightly older than the actual commit
time. Every time-based check in the system (MVCC snapshot visibility, GC
eligibility, collection-level TTL compaction triggering, delete-buffer
anchoring) sees these outdated timestamps instead of the actual commit
time, producing correctness bugs.
**Solution:**
Two-layer approach:
1. **C++ segcore (load-time overwrite):** During `LoadFieldData`, the
in-memory timestamp column is overwritten to `commit_ts_` when
`commit_ts_ != 0`. This makes existing MVCC/delete logic work correctly
with zero hot-path changes.
2. **Compaction normalization:** During compaction, all compaction paths
(mix/sort/clustering) rewrite row timestamps to `commit_ts` in output
binlogs, then set `CommitTimestamp = 0` on the output segment. After
first compaction, the segment becomes a normal segment with no special
handling needed. `commit_ts` is a temporary state that only exists
between import and first compaction.
**Key design decisions:**
- **TTL field (per-row):** Not affected by commit_ts. TTL field values
represent user-specified expiration intent and are honored as-is.
- **Collection-level TTL:** Uses `max(row_ts, commit_ts)` as the
effective row age to prevent premature expiration.
- **Delete/Upsert before commit_ts:** A delete/upsert with `ts <
commit_ts` does NOT take effect, because the row did not exist at that
time. The original `search_pk(pk, delete_ts)` logic handles this
correctly since `row_ts` is overwritten to `commit_ts`.
- **CommitTimestamp assignment:** Currently TODO in `import_checker.go`,
pending companion PR for 2PC commit flow.
## Changes
- `pkg/proto/data_coord.proto`, `query_coord.proto`, `segcore.proto`:
add `commit_timestamp` field
- `internal/datacoord/segment_info.go`:
`segmentEffectiveTs`/`segmentEffectiveDmlTs`/`effectiveTimestamp`
helpers
- `internal/datacoord/meta.go`: compaction completion mutations set
`CommitTimestamp=0` and normalize position timestamps via
`normalizePositionTimestamp` helper
- `internal/datacoord/compaction_trigger.go`: use `effectiveTimestamp`
for TTL compaction trigger
- `internal/datacoord/handler.go`, `garbage_collector.go`,
`compaction_task_l0.go`: use effective timestamp helpers
- `internal/querycoordv2`: propagate `CommitTimestamp` in
`PackSegmentLoadInfo`
- `internal/querynodev2/delegator`: use `segmentEffectiveTs` for
delete-buffer pin/list anchoring
- `internal/core/src/segcore/ChunkedSegmentSealedImpl`: overwrite
timestamp column at load time when `commit_ts_ != 0`
- `internal/datanode/compactor/timestamp_overwrite.go`: Record/Reader
wrappers for compaction timestamp normalization
-
`internal/datanode/compactor/{mix,merge_sort,sort,clustering}_compactor.go`:
apply timestamp overwrite during compaction
## Tests
- Unit tests: DataCoord helpers, `UpdateCommitTimestamp` operator,
`GenSnapshot` filter, GC eligibility, TTL compaction trigger
- Unit tests: QueryNode `segmentEffectiveTs`, delete-buffer pin at
commit_ts
- Unit tests: `timestamp_overwrite.go` Record/Reader wrappers
- C++ tests: MVCC gate, TTL gate, pre-commit delete not applied, normal
segment unchanged
- Integration tests: MVCC visibility, delete/upsert on import segment,
delete/upsert before commit_ts (should not apply), compaction normalizes
commit_ts to 0
## issue
issue: #48471
design doc:
https://github.com/milvus-io/milvus-design-docs/blob/main/design_docs/20260324-commit-timestamp.md
---------
Signed-off-by: Yihao Dai <yihao.dai@zilliz.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
||
|
aoiasdandGitHub
|
2e2266f1ff |
fix: keep text match stats paths compatible (#49957)
relate: #49907 ## Summary Make TextMatch upload return relative file names consistently, including the unified scalar index path. Restore full object keys in DataNode text stats task results and compaction metadata so mixed-version rolling upgrades remain compatible with older Coord/QueryNode consumers. --------- Signed-off-by: aoiasd <zhicheng.yue@zilliz.com> |
||
|
wei liuandGitHub
|
385df21c70 |
fix: Support external table snapshot restore (#49812)
issue: https://github.com/milvus-io/milvus/issues/45881 issue: https://github.com/milvus-io/milvus/issues/49468 ## What - Restore external-table snapshots by copying StorageV3 manifest-backed segment files to the target collection/partition/segment IDs. - Propagate external-collection identity through `CopySegmentSource.is_external_collection` so restore can preserve fake insert binlogs and row metadata. - Propagate `partitionID` through refresh-external-collection tasks so refreshed external segments write manifest paths with the target partition ID. - Add unit coverage for manifest path rewrite and external fake binlog handling, plus a go-client e2e that restores an external collection and verifies count, query, and search. ## Verification - `git diff --check` - `git diff --cached --check` - `go test -c -gcflags="all=-N -l" -tags dynamic,test -o /tmp/importv2_external_snapshot.test github.com/milvus-io/milvus/internal/datanode/importv2` - `/tmp/importv2_external_snapshot.test -test.run "^(TestGenerateTargetPath|TestTransformManifestPath_ExternalTable|TestTransformFieldBinlogs_SkipsPathMappingForExternalTable|TestCopySegmentAndIndexFiles_ExternalTable)$" -test.v` - `GODEBUG=netdns=go go test -tags dynamic,test ./testcases -run "^TestExternalCollectionSnapshotRestoreAndAccess$" -v -count=1 -timeout 30m -args -addr http://127.0.0.1:19530` Signed-off-by: Wei Liu <wei.liu@zilliz.com> |
||
|
congqixiaandGitHub
|
aedb81f825 |
fix: move L0 V3 manifest commit to datacoord (#49827)
Related to #49706 Transfer storage v3 L0 compaction manifest transaction commit from DataNode to DataCoord so the metadata owner commits deltalogs against the latest segment manifest. DataNode now returns the actual written deltalog path in the compaction result, and DataCoord uses that path to update the manifest before saving segment meta. --------- Signed-off-by: Congqi Xia <congqi.xia@zilliz.com> |
||
|
|
66fa592752 |
enhance: revert index_store_path full-path plumbing to montage assembly (#49805)
## Summary PR #49325 introduced collection-rooted index storage and also added a precomputed `index_store_path` string flowing through the build path. This PR scopes that back down: the collection-rooted layout stays, but the precomputed `index_store_path` string is removed from the write/build path. Build/write path: - DataCoord sends `IndexStorePathVersion` with the build job. - DataNode forwards the version to C++ indexbuilder. - C++ `FileManager` assembles the index object prefix locally from `(path_version, collID, partID, segID, buildID, indexVersion)`. Read/load path: - Go keeps passing full `IndexFilePaths` through DataCoord -> QueryCoord -> QueryNode -> cgo/segcore. - Load-side protos stay on full paths; no `index_file_keys` / `index_store_path_version` split is added there. - C++ load keeps PR #49325's direct full-path open behavior for `index_files/...` and `index_v1/...` paths. The v1 root is intentionally named `index_v1` before release. v0 remains `index_files`. issue: #49086 ## Compatibility note This PR is wire-incompatible with the short-lived #49325-era internal schema that carried `index_store_path`. That is intentional and acceptable because no release has shipped between the merge of #49325 and this rework. In other words, there are no released binaries or released persisted wire data that rely on the #49325-era `index_store_path` field semantics. This PR treats the proto changes as a pre-release schema correction rather than a compatibility migration. Because of that: - `BuildIndexInfo.index_store_path = 18` is replaced by `index_store_path_version = 18`. - `CreateJobRequest.index_store_path = 21` is removed. - `CreateJobRequest.index_store_path_version = 36` is kept. - We do not reserve the removed field numbers/names in this PR; if this schema had crossed a release boundary, we would reserve them and use fresh field numbers instead. ## What changes **Protos** - `worker.proto`: drop `CreateJobRequest.index_store_path`; keep `CreateJobRequest.index_store_path_version`. - `index_cgo_msg.proto`: replace `BuildIndexInfo.index_store_path` with `BuildIndexInfo.index_store_path_version`. - Read/load-side protos keep their full-path shape: no `IndexFilePaths -> IndexFileKeys` rename and no load-side `index_store_path_version` propagation. **C++** - `storage::IndexMeta`: replace `std::string index_store_path` with `IndexStorePathVersion index_store_path_version`. - `FileManager::GetRemoteIndexObjectPrefix()`: assemble v0/v1 prefixes from path version and IDs. - `FileManager::OpenInputStream()`: keep direct full-path open support for read/load paths. - `indexbuilder/index_c.cpp`: pass `index_store_path_version` into `IndexMeta`. - `DiskFileManagerTest`: cover v0 and v1 prefix assembly. **Go** - DataCoord `task_index.go`: stop building/sending `IndexStorePath`; continue sending `IndexStorePathVersion`. - DataNode `index/task_index.go`: forward `IndexStorePathVersion` to C++ build info. - DataNode `index_services.go`: drop logs for the removed `index_store_path` field. - DataCoord FinishTask validation and `SegmentIndex.IndexStorePathVersion` metadata remain unchanged. - GC/copy/import/snapshot/read path keep full-path behavior; only expected v1 root strings are updated from `index_files_v1` to `index_v1`. ## Notes for reviewers - The read/load path intentionally remains full-path based. QueryNode should not convert full paths to basenames or parse path version for load. - `IndexFilePathInfo.IndexStorePathVersion` remains as DataCoord metadata, but QueryCoord does not forward it for loading. - `metautil.IndexPathBuilder` remains needed for Go-side object storage operations such as GC, snapshot, copy-segment, and DataCoord `GetIndexInfos` path construction. - Mixed-version concerns are limited to unreleased #49325-era binaries, which is why this PR can remove/retype `index_store_path` without compatibility scaffolding. ## Test plan - [x] C++ build: `ninja milvus_core` clean. - [x] `DiskAnnFileManagerTest.GetRemoteIndexObjectPrefix_{V0BuildRooted,V1CollectionRooted}` covers v0/v1 prefix assembly. - [x] Updated Go/C++ tests for the `index_v1` root rename. - [ ] CI on this PR: code-check, UT, E2E. Signed-off-by: bigsheeper <yihao.dai@zilliz.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
congqixiaandGitHub
|
d6d274cf88 |
fix: make packed writer close idempotent (#49813)
Related to #49790 Packed writer close transfers ownership of the C++ wrapper to C, which deletes the wrapper before returning. Clear the Go-side C pointer before calling Close/CloseAndTell so cleanup paths cannot pass a freed pointer back into C. Also clear MultiSegmentWriter state after a successful close, preserve the original merge-sort error when cleanup close fails, and add regression coverage for repeated close and rotate allocation failure cleanup. --------- Signed-off-by: Congqi Xia <congqi.xia@zilliz.com> |
||
|
Spade AandGitHub
|
3fb7c9d63b |
feat: impl StructArray --- support null for struct array and vector array (#48553)
issue: https://github.com/milvus-io/milvus/issues/42148 design doc: https://github.com/milvus-io/milvus-design-docs/blob/main/design_docs/20260306-struct.md --------- Signed-off-by: SpadeA <tangchenjie1210@gmail.com> |
||
|
yihao.daiandGitHub
|
e70a4fdefd |
enhance: support collection-rooted index file storage layout (#49325)
Replaces #49087. ## Summary This PR introduces a collection-rooted storage layout for vector/scalar index files while keeping the legacy layout readable. Legacy v0 layout: `index_files/{buildID}/{indexVersion}/{partitionID}/{segmentID}/{fileKey}` New v1 layout: `index_files_v1/{collectionID}/{partitionID}/{segmentID}/{buildID}/{indexVersion}/{fileKey}` Main changes: - Add `IndexStorePathVersion` metadata to record which path layout each segment index uses. - Add `IndexPathBuilder` to centralize v0/v1 index path construction. - Gate v1 index path rollout by the cluster minimum node version, so mixed-version clusters continue writing v0 paths. - Propagate the path version through index build metadata, worker results, segment index metadata, and copy segment metadata. - Update GC to handle both legacy `index_files` paths and new `index_files_v1` paths safely. - Update copy/restore path mapping to resolve vector/scalar index paths according to their recorded path version. issue: #49086 ## Compatibility Existing v0 index files remain readable and continue to use the legacy layout. New v1 index files are only written when the cluster supports the new path layout. After v1 index files exist, rolling DataCoord back to a binary that does not understand `index_files_v1` is unsafe because older GC logic may not preserve those files correctly. Rolling DataNode or QueryNode back independently is expected to keep writing and using v0 paths because DataCoord gates the selected layout by reported node versions. ## Test Plan - [ ] Index path builder tests for v0 and v1 layouts. - [ ] SegmentIndex marshal/unmarshal tests for `IndexStorePathVersion`. - [ ] DataCoord tests for selecting v0/v1 based on cluster node versions. - [ ] GC tests for both `index_files` and `index_files_v1`. - [ ] Copy/restore tests for v0 and v1 index paths. - [ ] Recovery/snapshot index metadata tests for preserving `index_store_path_version`. --------- Signed-off-by: Yihao Dai <yihao.dai@zilliz.com> |
||
|
Ted XuandGitHub
|
bed359b4c5 |
enhance: unify filesystem access through FilesystemCache, remove ArrowFileSystemSingleton (#49521)
See #47520 Eliminate ArrowFileSystemSingleton from milvus so all filesystem access goes through FilesystemCache. This unifies I/O metrics across C++ segcore and Go FFI paths, fixing the issue where metrics showed all zeros for the default filesystem because I/O went through the singleton while metrics were read from the cache (a different FS instance). Key changes: - New GetDefaultArrowFileSystem() helper retrieves default FS from cache via LoonFFIPropertiesSingleton properties - Unified InitArrowFileSystem(CStorageConfig) replaces the split InitLocalArrowFileSystemSingleton/InitRemoteArrowFileSystemSingleton - Fixed Go metrics code to build full FFI properties for cache lookup (was passing empty properties, causing cache misses) - Fixed address normalization in StorageV2FSCache::Get (missing http:// prefix caused different cache key than init path) - Added TestCompactMetrics unit test verifying metrics increase after compaction - Verified end-to-end: standalone milvus Prometheus endpoint shows non-zero milvus_storage_filesystem_write_bytes after insert+flush Signed-off-by: Ted Xu <ted.xu@zilliz.com> |
||
|
|
3bc0d7cdf1 |
fix: restore RowID and limit TEXT flush behavior changes (#49529)
issue: #49495 Signed-off-by: luzhang <luzhang@zilliz.com> Co-authored-by: luzhang <luzhang@zilliz.com> |
||
|
congqixiaandGitHub
|
7311d450a0 |
enhance: bump Go dependencies to v3 modules (#49485)
Related to #49398 Bump Go module references from pkg/v2 and milvus-proto/go-api/v2 to pkg/v3 and milvus-proto/go-api/v3 so the client tracks the Milvus 3.x release line. This prepares the repository for the upcoming 3.x.y release by aligning imports, module dependencies, and proto API references with the new major-version module paths. --------- Signed-off-by: Congqi Xia <congqi.xia@zilliz.com> |
||
|
congqixiaandGitHub
|
2a14f5c346 |
fix: use overwrite resolver for manifest updates (#49467)
Related to #49457 #49462 Use overwrite conflict resolution when adding L0 compaction deltalogs and manifest stats so these updates commit from their base manifest version. Signed-off-by: Congqi Xia <congqi.xia@zilliz.com> |
||
|
wei liuandGitHub
|
d78dc6b3e5 |
enhance: Validate external schema during sample (#49425)
issue: #45691 This PR validates external table Arrow field types while sampling field sizes so schema mismatches fail before refresh planning continues. It passes the collection schema through the sample FFI path, normalizes supported Arrow variants against FieldMeta, and removes implicit integer narrowing for external Arrow input. Tests: - make milvus - /tmp/external.test -test.run TestRefreshExternalCollectionTaskSuite/TestBalanceFragmentsToSegments_SamplingTypeMismatchFails -test.v -test.count=1 - go test -tags dynamic,test github.com/milvus-io/milvus/tests/go_client/testcases -run TestExternalTableIcebergRefreshFailsOnSchemaTypeMismatch -v -count=1 Signed-off-by: Wei Liu <wei.liu@zilliz.com> |
||
|
wei liuandGitHub
|
3ae4cb28f5 |
fix: external table stability pack for alter tuple, refresh, spec validation, and datanode edge cases (#49366)
issue: #49335, #48830, #49338, #49334, #49292, #49336 ## Summary This PR delivers a stability hardening pack for external table lifecycle and spec handling. ### 1) Alter/Refresh metadata correctness - Guard invalid tuple transitions in alter-collection path. - Persist refreshed external tuple on refresh completion. - Disable unsupported `TruncateCollection` on external collections. ### 2) ExternalSpec safety and contract clarity - Require explicit `cloud_provider`; remove scheme-based inference. - Redact credentials from `DescribeCollection`. - Reserve system field names (including `__virtual_pk__`) from user schema. ### 3) Runtime robustness and edge-case handling - Skip anonymous entries in extfs Layer0 zero-init. - Align DataNode explore-manifest sort + format-filter behavior. - Prevent datanode crash on zero-row parquet. - Surface real sampling error in `RefreshFailed`. - Improve compatibility for vortex schemaless view and large Arrow type variants. ## Why External-table alter/refresh flows had validation and persistence gaps, plus multiple edge-case failures in runtime paths. This PR closes those gaps to make behavior safer, deterministic, and debuggable. ## Test Plan - Unit: - `pkg/util/externalspec/external_spec_test.go` - `internal/rootcoord/create_collection_task_test.go` - `internal/rootcoord/ddl_callbacks_alter_collection_properties_test.go` - `internal/core/unittest/test_external_take.cpp` - E2E / go client: - `tests/go_client/testcases/external_table_test.go` - `tests/go_client/testcases/external_table_iceberg_e2e_test.go` --------- Signed-off-by: wei liu <wei.liu@zilliz.com> Signed-off-by: Wei Liu <wei.liu@zilliz.com> |