issue: #50425
Disable growing-source flush by default so TEXT collections use the
writer-buffer path unless the feature is explicitly enabled.
Also write TEXT V3 bloom filter and BM25 stats into the manifest instead
of returning legacy stats logs.
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
issue: #51250
Keep TEXT segments on StorageV3 while making growing-source flush an
explicit allowed mode instead of a mandatory TEXT path.
Propagate create-segment storage versions through WAL flusher and write
buffer, reject storage-version mismatches in DataCoord, and keep raw
segcore chunks sticky for segments that may be flushed from growing
source.
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
issue: #51117
Supersedes #51207 per team decision to ship the skipped-field tolerance
for 3.0.0; the era-consistent replay consume path (#51146) remains a
follow-up.
### Problem
A growing segment created while a field was absent from the schema never
materializes that column: the InsertRecord ctor only allocates fields
present at creation, Reopen fills later-added ordinary fields but skips
function outputs by design, and a field dropped before segment creation
is skipped at consume. Flushing such a segment with a schema snapshot
that still carries the field hit `Assert "data_.find(field_id) !=
data_.end()"` in the growing-source flush path and crash-looped the
StreamingNode (exit 134 / CrashLoopBackOff in the schema-evolution L3
runs).
### Design
Principle: the flush writes exactly `flushSchema ∩ materialized`. A
non-materialized column is legally absent — a field dropped from the
segment's own schema, or a function output recomputed by bump-schema
compaction backfill. Real schema/data inconsistency is segcore's own
concern and is verified inside the flush.
- **Go (layout source of truth)**: trim the column groups to the source
segment's materialized field ids (plus system fields) right after layout
derivation — `Fields` and the parallel `Columns` array in lockstep — so
the writer pattern, binlog meta and metacache current split all describe
the same layout. An empty materialized report is refused. On a
committed-flush ack retry the layout is trimmed to the committed binlogs
(`ChildFields` union), the persisted truth.
- **C++ (segment-internal judgement)**: a column missing from the insert
record (`HasFieldData` semantics: allocated-but-empty counts as missing)
is skipped when the field is gone from the segment's own schema
(dropped) or is a function output; a regular field still present in the
segment schema is materialized by ctor/Reopen by construction, so that
case hard-errors instead of asserting.
- Skipped columns are backfilled by bump-schema compaction via
`RecordMaterializer`.
### Review responses (@liliu-z)
- stale `Columns` / phantom column in writer pattern →
`filterColumnGroupFields` trims `Fields`+`Columns` in lockstep at all
three trim sites
- committed ack-retry currentSplit divergence → layout trimmed to
committed binlogs' `ChildFields` union
- allocated-but-empty function-output column wedging the flush →
materialized report and the flush skip both gate on `HasFieldData`
semantics (empty = not materialized)
- silent skip of a missing regular field → three-way judgement: dropped
from segment schema → skip; function output → skip;
present-in-segment-schema regular field → hard error (`has no field data
in growing segment ... but is present in the segment schema`)
- retryability routing of the missing-field error: the legally-absent
cases no longer produce an error at all; the remaining hard error
indicates real data loss and is construction-unreachable via public APIs
(the consume path asserts regular fields carry data)
### Verification
- Go UT (`growing_source_test.go`): trim semantics — dropped ordinary
field trimmed, `Columns` kept in lockstep, empty materialized report
refused, committed-retry trim by `ChildFields`
- C++ UT (`test_storage.cpp`): flush skips a non-materialized function
output; skips a dropped field carried by a staler flush schema; skips an
allocated-but-empty function-output column
- Local schema-evolution chaos runs (concurrent add/drop field + BM25
function field + DML/query/search with `useLoonFFI` + growing-source
flush enabled, incl. StreamingNode force-kill/restart injections): no
panic, no crash loop, final serial consistency validation passes. The
exact original assert requires an era-mismatch replay not reproduced
locally; that path is covered by the unit tests above.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Make ComposeTSByTime the logical-zero time-based timestamp API, add
ComposeTSByTimeWithLogical for explicit logical composition in tests,
and remove the duplicate GetCurrentTime wrapper.
See also: #51119
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
issue: #50697
Summary:
1. Align StorageV3 column group layout for growing-source flush by
reusing the segment currentSplit/manifest layout, passing a schema-based
split pattern to C++, and using the schema_based writer policy instead
of always writing a single column group.
2. Project growing flush output to the target segment layout by passing
AllowedFieldIDs to C++, so ordinary fields, vector fields, text fields,
and BM25 output fields are all filtered to the old segment layout when
appending to an existing StorageV3 manifest.
3. Stop retrying non-retryable layout mismatches and preserve V3 layout
metadata, so column count/group mismatches do not loop forever and
recovery/balance can restore currentSplit for future layout-compatible
appends.
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
issue: #50750
Move sync task submission out of the writebuffer mutex. The writebuffer
still selects segments, yields buffered payload, records sync
checkpoints, and marks segment syncing while holding the lock, but
submits the already built tasks to syncmgr after releasing it.
This prevents ReleaseCollection from hanging in manual-flush preparation
when syncmgr dispatcher backpressure blocks SyncData submission, because
CheckReleaseManualFlushNeed can still acquire the writebuffer read lock.
Also move growing-source follow-up resync out of the callback critical
section and add a regression test that blocks SyncData submission while
verifying writebuffer readers are not blocked.
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
relate: #49716
pr: https://github.com/milvus-io/milvus/pull/49717
## Summary
Split FunctionRunnerManager lifecycle references between WAL and
delegator keys while still sharing runners by function signature. Add
latest-version lookup support for schemaVersion 0 or nil schema in the
manager; master call sites still pass the current schema version
explicitly.
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
issue: #47420
## What this PR does
Project-wide migration of raw `fmt.Errorf` / `errors.New` in function
bodies onto
the `merr` framework, plus the Sys-vs-Input error classification and the
machinery it drives (retriability, fine-grained metrics, segcore
unification),
plus the convention docs and a linter that keeps it from regressing.
Scope: storage, proxy, coordinators (root/data/query), query node, data
node,
`pkg/util` & `internal/util`, expression parser, message queue,
streaming, and
misc packages. Bare raw-error usages went from ~3000 to a ~340 allowlist
(package-level sentinels / build-tag / test sites).
---
## How to review this PR
It is large but the vast majority is mechanical. Changes fall into three
tiers;
spend review budget on Part 2 and Part 3.
### Part 1 — Mechanical standardization (low risk, verify by rule)
Each converted call follows one of a small fixed set of rules. To
review, check
that each site obeys the matching rule rather than reading every line:
| Pattern | Rule |
|---|---|
| `fmt.Errorf("...")` originating a new error | →
`merr.WrapErrXxxMsg("...")` with a code matching the failure's meaning |
| Adding context to an existing typed error | → `merr.Wrap(err, "...")`
/ `merr.Wrapf(...)` — **preserves** the inner code (never `WrapErr*Err`,
which overwrites it) |
| Errors inside the streaming subsystem | → `status.New*` factories
(StreamingError), **not** merr — this is the component-internal dialect
(see `docs/dev/error_handling_guide.md`) |
| Low-level / control-flow signal caught by `errors.Is` | → kept as a
package-level `errors.New` sentinel (lowercase, same-package) |
Conventions are documented in `docs/dev/error_handling_guide.md`
(how-to) and
`docs/dev/error_sentinel_convention.md` (rules + audit). A
`gocritic`/`ruleguard`
rule (`rawmerrerror`, in `rules.go`) enforces "no raw `return
errors.New/fmt.Errorf`"
under `make verifiers`.
### Part 2 — Behavior changes (review these closely)
These are the sites where the wire contract or runtime behavior changes,
not just
the source text. Listed by category; representative locations given,
full set in
the diff.
**A. gRPC wire-code shifts: `UnexpectedError(1)/Code 65535` → typed
code.**
Where a handler previously returned a raw error (collapsed to
`Code=65535` on the
wire), it now returns a typed merr, so the client sees a real code. The
most
common shift is to `IllegalArgument(5)/Code 1100` (ParameterInvalid).
Touch
points include datanode task handlers (CreateTask/Query/Drop), proxy
Upsert,
querynode GetMetrics, datacoord CreateIndex, httpserver query-response
builder,
and typeutil schema validation. One code refinement: an index-param
validation
moved `1100` → `1101` (ParameterMissing). **Client/SDK assertions and
any code
that switched on `Code=65535` for these paths must be re-checked** (the
go_client
e2e assertions were already aligned in this PR).
**B. Prometheus `status` label contract change (externally visible).**
The proxy metric's coarse `fail` / `rejected` values are split into
`fail_input` / `fail_system` and `rejected_user` / `rejected_system` (in
`requestutil.ParseMetricLabel`; auth/privilege rejections count as
`rejected_user`), so dashboards can attribute a failure to caller vs
operator.
**Dashboards/alerts querying `status="fail"` must migrate to
`status=~"fail_.*"`, and `status="rejected"` to
`status=~"rejected_.*"`.** The
in-repo Grafana dashboard is already migrated; external dashboards built
on the
old values silently go empty after upgrade. This is the one change that
requires an ops-side migration.
**C. Retriability semantics.**
- C1: `merr.Status(err)` now forces `Retriable=false` when the error is
an
`InputError` — a malformed request can never succeed on blind retry, so
clients
never get the self-contradictory "your input is wrong but you may
retry".
- C2: `retry.Do` short-circuits an `InputError` (non-retriable) — **but
only when
the caller did not pass a `RetryErr` predicate**. The check is an
`if c.isRetryErr != nil { ... } else if InputError { ... }` *mutually
exclusive*
branch (`pkg/util/retry/retry.go`): an explicit `RetryErr` takes
precedence and
bypasses the InputError abort. `retry.Handle` deliberately does **not**
apply
the InputError abort (its callers signal abort via `shouldRetry=false`).
Four
flusher startup callsites that must retry through transient "not ready"
errors
were given explicit `RetryErr` escape hatches.
**D. segcore (C++→Go) error classification.**
A single shared Go-side table (`pkg/util/merr/segcore.go`) maps each
segcore code
to a merr sentinel + InputError/signal category, replacing scattered
hand-written
`if errorCode == ...` switches in the cgo wrappers. **Wire `Code` values
change
for every segcore pass-through error, not just the remapped ones.**
Named
sentinels remap (C++ `2003` → merr `2001`, `2033` → `2002`,
Folly/Knowhere codes
likewise); **all remaining pass-through codes (`2004`–`2043`, previously
surfaced to clients as raw C++ enum values) now serialize as `2000`**
(`ErrSegcore`), with the original C++ code preserved in the `Reason`
text
(`segcoreCode=...`); unknown/future codes collapse to `2000` as well
(pinned by
the `wire_code_projection` test). Transient segcore classes (object
storage /
file IO / OOM / mmap / FieldNotLoaded — 11 codes) now report
`Retriable=true`.
**Any client switching on raw segcore codes in the `2004`–`2043` range
must be
re-checked**; the in-Reason code remains available for diagnostics.
Signal
codes (PretendFinished / FollyCancel) are recognized centrally.
`errors.Is`-based
control flow on these (e.g. scheduler skip/retry) is preserved.
**E. InputError classification (25 sentinels + dynamic marks).**
25 sentinels in `errors.go` carry `WithErrorType(InputError)` (the
Collection /
ResourceGroup / Database families, `ErrIndexDuplicate`,
`ErrParameterInvalid`,
`ErrPrivilegeNotAuthenticated`, `ErrImportFailed`, `ErrQueryPlan`, ...),
plus dynamic
marks for the 8 segcore input codes (ExprInvalid, DimNotMatch,
MetricTypeInvalid, FieldIDInvalid, ...) and
`WrapErrAsInputError`. The widest blast radius is `ErrParameterInvalid`
(1100):
~2335 `WrapErrParameterInvalid*` callsites now classify as input /
non-retriable. Because of C1/C2 this changes retriability for
any path that returns these. **The audit to confirm no transient path
was
mis-marked is the single most important review item** (see Part 3). One
reverse
correction: storage field-stats parsing moved from `ErrParameterInvalid`
(input)
to `ErrDataIntegrity` — a corrupted stored stat is data corruption, not
user
input.
### Part 3 — Known risks & traps (called out proactively)
1. **`merr.Wrap` vs `WrapErr*Err` (code-masking).** `WrapErr*Err` builds
a
`wrappedMilvusError{sentinel: ErrServiceInternal}` whose `code()`
returns the
*outer* sentinel — it overwrites the inner typed code and hides the
`errors.Is`
chain. This is intentional (use it to *deliberately* downgrade), but it
was a
recurring conversion defect; the rule "add context with `merr.Wrap`,
downgrade
with `WrapErr*Err`" is enforced by convention and reviewed across the
diff.
2. **InputError × `retry.Do` blast radius.** Marking a sentinel
`InputError` makes
any `retry.Do(...)` without a `RetryErr` predicate stop retrying it.
Reviewers
should sanity-check that no transient use of the 19 newly-marked
sentinels
(especially `ErrParameterInvalid`) sits inside a retry loop that needed
to keep
spinning. The known flusher cases were handled (see C2).
3. **The ~340 raw-error allowlist.** What remains as bare `errors.New`
is, by
design: package-level sentinels (caught by `errors.Is`), `//go:build
test`
sites, and out-of-band trees (`cmd/`, `tests/`, codegen, walimpls). The
linter
only bans the *direct-return* form; assignment-then-return escapes and
the full
no-exceptions ban are deferred to an AST-based linter (Tier 2,
documented).
4. **segcore C++ second step deferred.** This PR unifies classification
on the Go
side; splitting the dual-semantic C++ codes at the source is a
follow-up.
---
## Validation
- `make verifiers`: Go side clean (gofmt + static-check across modules,
including
the new `rawmerrerror` rule with a 0-hit baseline repo-wide).
- `make test-go`: passing; the one real regression introduced (a
datanode
`invalid_task_type` assertion shifting `1` → `5` from a ParameterInvalid
conversion) was fixed in-tree.
- go_client e2e CreateIndex assertions aligned to the new merr messages.
---------
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
relate: #49716
## Summary
Move BM25/MinHash function execution from DN/QN pipeline embedding nodes
to the write-before path before WAL append. Add collection-scoped
function runner caching and keep pipeline-side compatibility fill for
old insert messages without generated function output fields.
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
Related to #49398
Bump Go module references from pkg/v2 and milvus-proto/go-api/v2 to
pkg/v3 and milvus-proto/go-api/v3 so the client tracks the Milvus 3.x
release line.
This prepares the repository for the upcoming 3.x.y release by aligning
imports, module dependencies, and proto API references with the new
major-version module paths.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
## Summary
- PR #48809 added a new \`schemaVersion int32\` parameter to
\`l0WriteBuffer.BufferData\`.
- PR #49218 landed new tests in \`write_buffer_test.go\` after #48809
was merged, but used the old 4-arg signature.
- Result: static-check on master fails with typecheck errors for any PR
that rebases onto the latest master.
## Fix
Append \`0\` as the \`schemaVersion\` argument to the two stale test
call sites (lines 363 and 390).
## Test plan
- [x] \`golangci-lint run --build-tags dynamic,test
./internal/flushcommon/writebuffer/...\` passes locally
- [x] \`go vet -tags dynamic,test
./internal/flushcommon/writebuffer/...\` passes
issue: #45881
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
## Summary
- Add schema version check on insert path in streaming node shard
interceptor, rejecting inserts with mismatched schema version
- Add
`AlterCollection`/`AppendNewCollectionSchema`/`CheckIfCollectionSchemaVersionMatch`
methods to shard manager for schema lifecycle management
- Propagate schema version through segment allocation chain:
`shard_manager_segment` → `partition_manager` → `segment_alloc_worker` →
`CreateSegment` message header
- Add `SchemaVersionMismatch` streaming error type (classified as
unrecoverable to force proxy metadata refresh)
- Track collection schema in `CollectionInfo` with
`CollectionSchemaOfVChannel` for version gating
related: https://github.com/milvus-io/milvus/issues/48808
design doc:
https://github.com/milvus-io/milvus-design-docs/blob/main/design_docs/20260129-add-function-field-design.md
## Test plan
- [x] partition_manager_test.go updated for new `schemaVersion`
parameter
- [x] wal_test.go updated with CollectionSchema in CreateCollection
messages
- [ ] CI: code-check, ut-go, integration-test
🤖 Generated with [Claude Code](https://claude.ai/code)
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Replace all hard-coded `-1` literal used as manifest version with the
named constant `packed.ManifestEarliest` (value `0`) for new segment
creation, making the intent explicit and consistent across the codebase.
Key changes:
- Define `ManifestLatest` (-1) and `ManifestEarliest` (0) constants in
`internal/storagev2/packed/constant.go`
- Update all production code (segment_manager, write_buffer,
importv2/util, binlog_record_writer) to use `packed.ManifestEarliest`
- Update all test code (pack_writer_v3_test, task_test) to use
`packed.ManifestEarliest` instead of raw `-1`
- Fix `CreateManifestForSegment` FFI call: use version `0` (earliest)
instead of `-1` (latest) and use `LOON_TRANSACTION_RESOLVE_OVERWRITE`
resolve strategy
- Fix `FFIPackedWriter.Close` to use
`LOON_TRANSACTION_RESOLVE_OVERWRITE` resolve strategy
issue: #48543
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #47790
Replace 131 occurrences of fmt.Sprint(paramtable.GetNodeID()) with the
existing paramtable.GetStringNodeID() across 40 files. This avoids
redundant fmt.Sprint allocations on every call since GetStringNodeID()
uses the more efficient strconv.FormatInt. In the search hot path alone,
this eliminates 3+ string allocations per segment per query.
Signed-off-by: lyang24 <lanqingy93@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Related to #44956
Introduce StorageV3 constant to distinguish loon manifest format from
StorageV2 packed writer format. When UseLoonFFI is enabled, segments are
now marked with StorageVersion=3 instead of 2.
The V3 format currently shares most V2 logic paths since the underlying
storage format is similar. In a future refactor, V3-specific logic will
be separated from the V2 code path for better readability.
Changes:
- Add StorageV3 constant (value=3) in Go and C++ code
- Set StorageVersion=3 when UseLoonFFI is enabled for:
- Compaction params
- Import segments allocation
- Write buffer growing segments
- Streaming node segment allocation
- Add V3 support to existing V2 code paths (read/write/sync)
- Separate V2 and V3 writer creation in NewBinlogRecordWriter
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #46595
Remove the EnableStorageV2 config option and enforce StorageV2 format
across all write paths including compaction, import, write buffer, and
streaming segment allocation. V1 format write tests are now skipped as
writing V1 format is no longer supported.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
related: #45993
This commit extends nullable vector support to the proxy layer,
querynode,
and adds comprehensive validation, search reduce, and field data
handling
for nullable vectors with sparse storage.
Proxy layer changes:
- Update validate_util.go checkAligned() with getExpectedVectorRows()
helper
to validate nullable vector field alignment using valid data count
- Update checkFloatVectorFieldData/checkSparseFloatVectorFieldData for
nullable vector validation with proper row count expectations
- Add FieldDataIdxComputer in typeutil/schema.go for logical-to-physical
index translation during search reduce operations
- Update search_reduce_util.go reduceSearchResultData to use
idxComputers
for correct field data indexing with nullable vectors
- Update task.go, task_query.go, task_upsert.go for nullable vector
handling
- Update msg_pack.go with nullable vector field data processing
QueryNode layer changes:
- Update segments/result.go for nullable vector result handling
- Update segments/search_reduce.go with nullable vector offset
translation
Storage and index changes:
- Update data_codec.go and utils.go for nullable vector serialization
- Update indexcgowrapper/dataset.go and index.go for nullable vector
indexing
Utility changes:
- Add FieldDataIdxComputer struct with Compute() method for efficient
logical-to-physical index mapping across multiple field data
- Update EstimateEntitySize() and AppendFieldData() with fieldIdxs
parameter
- Update funcutil.go with nullable vector support functions
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Full support for nullable vector fields (float, binary, float16,
bfloat16, int8, sparse) across ingest, storage, indexing, search and
retrieval; logical↔physical offset mapping preserves row semantics.
* Client: compaction control and compaction-state APIs.
* **Bug Fixes**
* Improved validation for adding vector fields (nullable + dimension
checks) and corrected search/query behavior for nullable vectors.
* **Chores**
* Persisted validity maps with indexes and on-disk formats.
* **Tests**
* Extensive new and updated end-to-end nullable-vector tests.
<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: marcelo-cjl <marcelo.chen@zilliz.com>
Related to #44956
This change propagates the useLoonFFI configuration through the import
pipeline to enable LOON FFI usage during data import operations.
Key changes:
- Add use_loon_ffi field to ImportRequest protobuf message
- Add manifest_path field to ImportSegmentInfo for tracking manifest
- Initialize manifest path when creating segments (both import and
growing)
- Pass useLoonFFI flag through NewSyncTask in import tasks
- Simplify pack_writer_v2 by removing GetManifestInfo method and relying
on pre-initialized manifest path from segment creation
- Update segment meta with manifest path after import completion
This allows the import workflow to use the LOON FFI based packed writer
when the common.useLoonFFI configuration is enabled.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #43072, #43289
- manage the schema version at recovery storage.
- update the schema when creating collection or alter schema.
- get schema at write buffer based on version.
- recover the schema when upgrading from 2.5.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
Related to #39173
This PR:
- Upgrade milvus-storage commit to fix filesystem finalized issue
- Add bucket-name as prefix for all fs style access io
- Initial arrow fs on querynodes startup
- Fix timestamp access when loading sealed segment
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
1. DataNode: Skip generating BF during the insert phase (BF will be
regenerated during the sync phase).
2. QueryNode: Skip generating or maintaining BF for growing segments;
deletion checks will be handled in the segcore.
issue: https://github.com/milvus-io/milvus/issues/37630
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
