issue: #51418
- flushcommon syncmgr: skip the write-buffer panic callback for stale
growing-source meta errors while preserving failure metrics
- growing-source tests: cover stale channel and missing segment errors
without invoking the failure callback
---------
Signed-off-by: chyezh <chyezh@outlook.com>
issue: #51250
Keep TEXT segments on StorageV3 while making growing-source flush an
explicit allowed mode instead of a mandatory TEXT path.
Propagate create-segment storage versions through WAL flusher and write
buffer, reject storage-version mismatches in DataCoord, and keep raw
segcore chunks sticky for segments that may be flushed from growing
source.
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
relate: #50304
## Summary
Add the local format design doc, use the storage writer format constant,
and introduce local_format metadata propagation with column-group split
policy support.
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
issue: #51117
Supersedes #51207 per team decision to ship the skipped-field tolerance
for 3.0.0; the era-consistent replay consume path (#51146) remains a
follow-up.
### Problem
A growing segment created while a field was absent from the schema never
materializes that column: the InsertRecord ctor only allocates fields
present at creation, Reopen fills later-added ordinary fields but skips
function outputs by design, and a field dropped before segment creation
is skipped at consume. Flushing such a segment with a schema snapshot
that still carries the field hit `Assert "data_.find(field_id) !=
data_.end()"` in the growing-source flush path and crash-looped the
StreamingNode (exit 134 / CrashLoopBackOff in the schema-evolution L3
runs).
### Design
Principle: the flush writes exactly `flushSchema ∩ materialized`. A
non-materialized column is legally absent — a field dropped from the
segment's own schema, or a function output recomputed by bump-schema
compaction backfill. Real schema/data inconsistency is segcore's own
concern and is verified inside the flush.
- **Go (layout source of truth)**: trim the column groups to the source
segment's materialized field ids (plus system fields) right after layout
derivation — `Fields` and the parallel `Columns` array in lockstep — so
the writer pattern, binlog meta and metacache current split all describe
the same layout. An empty materialized report is refused. On a
committed-flush ack retry the layout is trimmed to the committed binlogs
(`ChildFields` union), the persisted truth.
- **C++ (segment-internal judgement)**: a column missing from the insert
record (`HasFieldData` semantics: allocated-but-empty counts as missing)
is skipped when the field is gone from the segment's own schema
(dropped) or is a function output; a regular field still present in the
segment schema is materialized by ctor/Reopen by construction, so that
case hard-errors instead of asserting.
- Skipped columns are backfilled by bump-schema compaction via
`RecordMaterializer`.
### Review responses (@liliu-z)
- stale `Columns` / phantom column in writer pattern →
`filterColumnGroupFields` trims `Fields`+`Columns` in lockstep at all
three trim sites
- committed ack-retry currentSplit divergence → layout trimmed to
committed binlogs' `ChildFields` union
- allocated-but-empty function-output column wedging the flush →
materialized report and the flush skip both gate on `HasFieldData`
semantics (empty = not materialized)
- silent skip of a missing regular field → three-way judgement: dropped
from segment schema → skip; function output → skip;
present-in-segment-schema regular field → hard error (`has no field data
in growing segment ... but is present in the segment schema`)
- retryability routing of the missing-field error: the legally-absent
cases no longer produce an error at all; the remaining hard error
indicates real data loss and is construction-unreachable via public APIs
(the consume path asserts regular fields carry data)
### Verification
- Go UT (`growing_source_test.go`): trim semantics — dropped ordinary
field trimmed, `Columns` kept in lockstep, empty materialized report
refused, committed-retry trim by `ChildFields`
- C++ UT (`test_storage.cpp`): flush skips a non-materialized function
output; skips a dropped field carried by a staler flush schema; skips an
allocated-but-empty function-output column
- Local schema-evolution chaos runs (concurrent add/drop field + BM25
function field + DML/query/search with `useLoonFFI` + growing-source
flush enabled, incl. StreamingNode force-kill/restart injections): no
panic, no crash loop, final serial consistency validation passes. The
exact original assert requires an era-mismatch replay not reproduced
locally; that path is covered by the unit tests above.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Make ComposeTSByTime the logical-zero time-based timestamp API, add
ComposeTSByTimeWithLogical for explicit logical composition in tests,
and remove the duplicate GetCurrentTime wrapper.
See also: #51119
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
issue: #50808
### What this PR does
The datanode object-storage IO pool (`GetOrCreateIOPool`) bounds how
many binlog **download/upload** operations run concurrently. It is
shared by all **compaction** (mix / L0 / sort / clustering) and
**stats** (sort / text index / json key) tasks on a DataNode.
Its size was hard-capped at **32** with a default of only **16**,
regardless of node size:
```go
capacity := paramtable.Get().DataNodeCfg.IOConcurrency.GetAsInt() // default 16
if capacity > 32 { capacity = 32 } // hard-coded since #19892 (2022)
```
On large nodes this pool — not object-storage bandwidth — became the
throughput ceiling for compaction/stats IO (notably after bulk import,
when many sort/compaction tasks contend for it), and an explicit
`ioConcurrency > 32` was silently clamped back to 32.
### Changes
- `dataNode.dataSync.ioConcurrency` now defaults to **auto (`0` →
`CPU*2`)** instead of `16`.
- Removed the hard-coded `> 32` clamp — an explicit value is honored
as-is.
- Extracted `ioPoolCapacity()` and added a unit test.
`CPU*2` is safe because these goroutines are network-IO bound and mostly
blocked on object storage, so concurrency can exceed the CPU count (same
rationale as the existing stats/multiRead pools in this file).
### Notes / follow-up
Downloads currently materialize whole objects in memory, so concurrency
must still be raised with memory in mind — the original `32` cap traces
back to an OOM fix (#33554). Making downloads streaming / range-based
(so concurrency can be raised further without memory blow-up) is a
follow-up. Related: #50452 (storage v3 import slowdown).
### Tests
- `TestIOPoolCapacity` (new): explicit config honored; `0` → `CPU*2`.
- Existing `TestGetOrCreateIOPool`, `TestResizePools` pass.
---------
Signed-off-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
issue: #50697
Summary:
1. Align StorageV3 column group layout for growing-source flush by
reusing the segment currentSplit/manifest layout, passing a schema-based
split pattern to C++, and using the schema_based writer policy instead
of always writing a single column group.
2. Project growing flush output to the target segment layout by passing
AllowedFieldIDs to C++, so ordinary fields, vector fields, text fields,
and BM25 output fields are all filtered to the old segment layout when
appending to an existing StorageV3 manifest.
3. Stop retrying non-retryable layout mismatches and preserve V3 layout
metadata, so column count/group mismatches do not loop forever and
recovery/balance can restore currentSplit for future layout-compatible
appends.
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
issue: #50750
Move sync task submission out of the writebuffer mutex. The writebuffer
still selects segments, yields buffered payload, records sync
checkpoints, and marks segment syncing while holding the lock, but
submits the already built tasks to syncmgr after releasing it.
This prevents ReleaseCollection from hanging in manual-flush preparation
when syncmgr dispatcher backpressure blocks SyncData submission, because
CheckReleaseManualFlushNeed can still acquire the writebuffer read lock.
Also move growing-source follow-up resync out of the callback critical
section and add a regression test that blocks SyncData submission while
verifying writebuffer readers are not blocked.
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
relate: #49716
pr: https://github.com/milvus-io/milvus/pull/49717
## Summary
Split FunctionRunnerManager lifecycle references between WAL and
delegator keys while still sharing runners by function signature. Add
latest-version lookup support for schemaVersion 0 or nil schema in the
manager; master call sites still pass the current schema version
explicitly.
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
issue: #47420
## What this PR does
Project-wide migration of raw `fmt.Errorf` / `errors.New` in function
bodies onto
the `merr` framework, plus the Sys-vs-Input error classification and the
machinery it drives (retriability, fine-grained metrics, segcore
unification),
plus the convention docs and a linter that keeps it from regressing.
Scope: storage, proxy, coordinators (root/data/query), query node, data
node,
`pkg/util` & `internal/util`, expression parser, message queue,
streaming, and
misc packages. Bare raw-error usages went from ~3000 to a ~340 allowlist
(package-level sentinels / build-tag / test sites).
---
## How to review this PR
It is large but the vast majority is mechanical. Changes fall into three
tiers;
spend review budget on Part 2 and Part 3.
### Part 1 — Mechanical standardization (low risk, verify by rule)
Each converted call follows one of a small fixed set of rules. To
review, check
that each site obeys the matching rule rather than reading every line:
| Pattern | Rule |
|---|---|
| `fmt.Errorf("...")` originating a new error | →
`merr.WrapErrXxxMsg("...")` with a code matching the failure's meaning |
| Adding context to an existing typed error | → `merr.Wrap(err, "...")`
/ `merr.Wrapf(...)` — **preserves** the inner code (never `WrapErr*Err`,
which overwrites it) |
| Errors inside the streaming subsystem | → `status.New*` factories
(StreamingError), **not** merr — this is the component-internal dialect
(see `docs/dev/error_handling_guide.md`) |
| Low-level / control-flow signal caught by `errors.Is` | → kept as a
package-level `errors.New` sentinel (lowercase, same-package) |
Conventions are documented in `docs/dev/error_handling_guide.md`
(how-to) and
`docs/dev/error_sentinel_convention.md` (rules + audit). A
`gocritic`/`ruleguard`
rule (`rawmerrerror`, in `rules.go`) enforces "no raw `return
errors.New/fmt.Errorf`"
under `make verifiers`.
### Part 2 — Behavior changes (review these closely)
These are the sites where the wire contract or runtime behavior changes,
not just
the source text. Listed by category; representative locations given,
full set in
the diff.
**A. gRPC wire-code shifts: `UnexpectedError(1)/Code 65535` → typed
code.**
Where a handler previously returned a raw error (collapsed to
`Code=65535` on the
wire), it now returns a typed merr, so the client sees a real code. The
most
common shift is to `IllegalArgument(5)/Code 1100` (ParameterInvalid).
Touch
points include datanode task handlers (CreateTask/Query/Drop), proxy
Upsert,
querynode GetMetrics, datacoord CreateIndex, httpserver query-response
builder,
and typeutil schema validation. One code refinement: an index-param
validation
moved `1100` → `1101` (ParameterMissing). **Client/SDK assertions and
any code
that switched on `Code=65535` for these paths must be re-checked** (the
go_client
e2e assertions were already aligned in this PR).
**B. Prometheus `status` label contract change (externally visible).**
The proxy metric's coarse `fail` / `rejected` values are split into
`fail_input` / `fail_system` and `rejected_user` / `rejected_system` (in
`requestutil.ParseMetricLabel`; auth/privilege rejections count as
`rejected_user`), so dashboards can attribute a failure to caller vs
operator.
**Dashboards/alerts querying `status="fail"` must migrate to
`status=~"fail_.*"`, and `status="rejected"` to
`status=~"rejected_.*"`.** The
in-repo Grafana dashboard is already migrated; external dashboards built
on the
old values silently go empty after upgrade. This is the one change that
requires an ops-side migration.
**C. Retriability semantics.**
- C1: `merr.Status(err)` now forces `Retriable=false` when the error is
an
`InputError` — a malformed request can never succeed on blind retry, so
clients
never get the self-contradictory "your input is wrong but you may
retry".
- C2: `retry.Do` short-circuits an `InputError` (non-retriable) — **but
only when
the caller did not pass a `RetryErr` predicate**. The check is an
`if c.isRetryErr != nil { ... } else if InputError { ... }` *mutually
exclusive*
branch (`pkg/util/retry/retry.go`): an explicit `RetryErr` takes
precedence and
bypasses the InputError abort. `retry.Handle` deliberately does **not**
apply
the InputError abort (its callers signal abort via `shouldRetry=false`).
Four
flusher startup callsites that must retry through transient "not ready"
errors
were given explicit `RetryErr` escape hatches.
**D. segcore (C++→Go) error classification.**
A single shared Go-side table (`pkg/util/merr/segcore.go`) maps each
segcore code
to a merr sentinel + InputError/signal category, replacing scattered
hand-written
`if errorCode == ...` switches in the cgo wrappers. **Wire `Code` values
change
for every segcore pass-through error, not just the remapped ones.**
Named
sentinels remap (C++ `2003` → merr `2001`, `2033` → `2002`,
Folly/Knowhere codes
likewise); **all remaining pass-through codes (`2004`–`2043`, previously
surfaced to clients as raw C++ enum values) now serialize as `2000`**
(`ErrSegcore`), with the original C++ code preserved in the `Reason`
text
(`segcoreCode=...`); unknown/future codes collapse to `2000` as well
(pinned by
the `wire_code_projection` test). Transient segcore classes (object
storage /
file IO / OOM / mmap / FieldNotLoaded — 11 codes) now report
`Retriable=true`.
**Any client switching on raw segcore codes in the `2004`–`2043` range
must be
re-checked**; the in-Reason code remains available for diagnostics.
Signal
codes (PretendFinished / FollyCancel) are recognized centrally.
`errors.Is`-based
control flow on these (e.g. scheduler skip/retry) is preserved.
**E. InputError classification (25 sentinels + dynamic marks).**
25 sentinels in `errors.go` carry `WithErrorType(InputError)` (the
Collection /
ResourceGroup / Database families, `ErrIndexDuplicate`,
`ErrParameterInvalid`,
`ErrPrivilegeNotAuthenticated`, `ErrImportFailed`, `ErrQueryPlan`, ...),
plus dynamic
marks for the 8 segcore input codes (ExprInvalid, DimNotMatch,
MetricTypeInvalid, FieldIDInvalid, ...) and
`WrapErrAsInputError`. The widest blast radius is `ErrParameterInvalid`
(1100):
~2335 `WrapErrParameterInvalid*` callsites now classify as input /
non-retriable. Because of C1/C2 this changes retriability for
any path that returns these. **The audit to confirm no transient path
was
mis-marked is the single most important review item** (see Part 3). One
reverse
correction: storage field-stats parsing moved from `ErrParameterInvalid`
(input)
to `ErrDataIntegrity` — a corrupted stored stat is data corruption, not
user
input.
### Part 3 — Known risks & traps (called out proactively)
1. **`merr.Wrap` vs `WrapErr*Err` (code-masking).** `WrapErr*Err` builds
a
`wrappedMilvusError{sentinel: ErrServiceInternal}` whose `code()`
returns the
*outer* sentinel — it overwrites the inner typed code and hides the
`errors.Is`
chain. This is intentional (use it to *deliberately* downgrade), but it
was a
recurring conversion defect; the rule "add context with `merr.Wrap`,
downgrade
with `WrapErr*Err`" is enforced by convention and reviewed across the
diff.
2. **InputError × `retry.Do` blast radius.** Marking a sentinel
`InputError` makes
any `retry.Do(...)` without a `RetryErr` predicate stop retrying it.
Reviewers
should sanity-check that no transient use of the 19 newly-marked
sentinels
(especially `ErrParameterInvalid`) sits inside a retry loop that needed
to keep
spinning. The known flusher cases were handled (see C2).
3. **The ~340 raw-error allowlist.** What remains as bare `errors.New`
is, by
design: package-level sentinels (caught by `errors.Is`), `//go:build
test`
sites, and out-of-band trees (`cmd/`, `tests/`, codegen, walimpls). The
linter
only bans the *direct-return* form; assignment-then-return escapes and
the full
no-exceptions ban are deferred to an AST-based linter (Tier 2,
documented).
4. **segcore C++ second step deferred.** This PR unifies classification
on the Go
side; splitting the dual-semantic C++ codes at the source is a
follow-up.
---
## Validation
- `make verifiers`: Go side clean (gofmt + static-check across modules,
including
the new `rawmerrerror` rule with a 0-hit baseline repo-wide).
- `make test-go`: passing; the one real regression introduced (a
datanode
`invalid_task_type` assertion shifting `1` → `5` from a ParameterInvalid
conversion) was fixed in-tree.
- go_client e2e CreateIndex assertions aligned to the new merr messages.
---------
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
relate: #49716
## Summary
Move BM25/MinHash function execution from DN/QN pipeline embedding nodes
to the write-before path before WAL append. Add collection-scoped
function runner caching and keep pipeline-side compatibility fill for
old insert messages without generated function output fields.
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
issue: #50182
Storage V3 can now keep writing a segment after
`dataNode.storage.format` changes. The main problem was that an
already-started growing segment may have a committed manifest written
with one format, while the current global config has moved to another
format. In that case, the next append must follow the format already
recorded in the manifest instead of blindly using the new global config.
This keeps writer.format as the current global setting, but makes the
actual split policy format local to the writer path. QueryNode growing
flush resolves writer.split.single.format from the acknowledged manifest
when one exists, and falls back to the global config only for a
brand-new segment. DataNode Storage V3 flush carries column group
formats through SegmentInfo and passes them as
writer.split.schema_based.formats, so schema-based writes can append
with the formats already known for each column group.
The metadata path now records the column group format in FieldBinlog and
preserves it through recovery, merge, duplicate filtering, fake external
segment binlogs, and V3 compaction output. This lets metacache rebuild
the current split with enough information to choose writer formats
without reading the manifest on every flush.
The storage writer wrappers now accept explicit writer format properties
for both regular packed writers and TEXT-aware segment writers. TEXT LOB
files still use their own LOB path and storage behavior; the format
selection here applies to the segment column groups that store normal
data and LOB references, not to changing the LOB payload file layout.
Signed-off-by: jiaqizho <jiaqi.zhou@zilliz.com>
issue: #49710
related: #49069
BulkPackWriterV3.Write currently bumps the loon manifest version up to
four times per call (inserts, stats, delta, bm25 each open and commit
their own transaction). That breaks atomicity for concurrent readers,
leaves orphan manifest versions behind on partial failure, and churns
loon's manifest history.
This PR refactors the V3 sync write path into a do-then-commit shape:
Phase 1 (slow, runs once outside the retry loop): writeInserts,
writeStats, writeDelta, writeBM25Stasts write all the data files
(parquet/LOB via the loon writer, deltalog, stat blobs) and each
helper returns its contribution (ColumnGroups, SegmentOutput,
StatEntry list, DeltaLogEntry list) without touching shared state.
Phase 2 (fast, retried on transient loon errors): Write assembles
one packed.ManifestUpdates from the helper returns and calls a
single new primitive packed.CommitManifestUpdates, which opens a
short loon transaction, stages every change, and commits once.
To make this clean the loon FFI writers are decoupled from manifest
concerns entirely:
FFIPackedWriter and FFISegmentWriter no longer take a baseVersion
at construction. Their Close primitives return data carriers
(packed.ColumnGroups, packed.SegmentOutput) that own the C output.
packed.CommitManifestUpdates is the only Go-level entry point that
mutates a manifest. Empty payload short-circuits to the unchanged
manifest path without opening a transaction.
The same do-then-commit shape applies to the V2-style binlog import
writers (PackedManifestRecordWriter, PackedTextManifestRecordWriter):
they assemble inserts plus bloom-filter and BM25 stat blobs into one
ManifestUpdates and call CommitManifestUpdates once.
Internal renames: the low-level adapters packedRecordManifestWriter
and packedTextManifestWriter are renamed to packedRecordBatchWriter
and packedTextBatchWriter because they don't touch the manifest
themselves — they translate storage.Record into arrow.Record and
track per-column-group sizes. The interface manifestRecordWriter is
renamed to packedBatchWriter to match.
New tests in pack_writer_v3_test.go:
TestWrite_SingleVersionBumpAcrossSections drives an insert + delta
+ flush so all four sections fire and asserts the new manifest
version equals baseVer + 1.
TestWrite_RetryDoesNotLeakVersionBumps mockey-patches
CommitManifestUpdates to fail once with packed.ErrLoonTransient
and asserts the eventual successful commit still yields baseVer + 1
rather than baseVer + N_retries.
All 11 V3 tests pass. Existing packed and storage tests pass.
Out of scope (filed as follow-ups in #49710's discussion):
backfill_compactor.go's runBm25Function still issues a separate
AddStatsToManifest after the column-groups commit, producing two
version bumps per backfill. The architecture supports folding the
two but the change isn't strictly needed for the V3 sync atomicity
fix.
FFIPackedWriter lacks a Destroy method; on loon_writer_close
failure the handle and cProperties leak. Pre-existing on master.
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
See #47520
Eliminate ArrowFileSystemSingleton from milvus so all filesystem access
goes through FilesystemCache. This unifies I/O metrics across C++
segcore
and Go FFI paths, fixing the issue where metrics showed all zeros for
the default filesystem because I/O went through the singleton while
metrics were read from the cache (a different FS instance).
Key changes:
- New GetDefaultArrowFileSystem() helper retrieves default FS from cache
via LoonFFIPropertiesSingleton properties
- Unified InitArrowFileSystem(CStorageConfig) replaces the split
InitLocalArrowFileSystemSingleton/InitRemoteArrowFileSystemSingleton
- Fixed Go metrics code to build full FFI properties for cache lookup
(was passing empty properties, causing cache misses)
- Fixed address normalization in StorageV2FSCache::Get (missing http://
prefix caused different cache key than init path)
- Added TestCompactMetrics unit test verifying metrics increase after
compaction
- Verified end-to-end: standalone milvus Prometheus endpoint shows
non-zero milvus_storage_filesystem_write_bytes after insert+flush
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
Related to #49398
Bump Go module references from pkg/v2 and milvus-proto/go-api/v2 to
pkg/v3 and milvus-proto/go-api/v3 so the client tracks the Milvus 3.x
release line.
This prepares the repository for the upcoming 3.x.y release by aligning
imports, module dependencies, and proto API references with the new
major-version module paths.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
## Summary
- PR #48809 added a new \`schemaVersion int32\` parameter to
\`l0WriteBuffer.BufferData\`.
- PR #49218 landed new tests in \`write_buffer_test.go\` after #48809
was merged, but used the old 4-arg signature.
- Result: static-check on master fails with typecheck errors for any PR
that rebases onto the latest master.
## Fix
Append \`0\` as the \`schemaVersion\` argument to the two stale test
call sites (lines 363 and 390).
## Test plan
- [x] \`golangci-lint run --build-tags dynamic,test
./internal/flushcommon/writebuffer/...\` passes locally
- [x] \`go vet -tags dynamic,test
./internal/flushcommon/writebuffer/...\` passes
issue: #45881
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
## Summary
- Add schema version check on insert path in streaming node shard
interceptor, rejecting inserts with mismatched schema version
- Add
`AlterCollection`/`AppendNewCollectionSchema`/`CheckIfCollectionSchemaVersionMatch`
methods to shard manager for schema lifecycle management
- Propagate schema version through segment allocation chain:
`shard_manager_segment` → `partition_manager` → `segment_alloc_worker` →
`CreateSegment` message header
- Add `SchemaVersionMismatch` streaming error type (classified as
unrecoverable to force proxy metadata refresh)
- Track collection schema in `CollectionInfo` with
`CollectionSchemaOfVChannel` for version gating
related: https://github.com/milvus-io/milvus/issues/48808
design doc:
https://github.com/milvus-io/milvus-design-docs/blob/main/design_docs/20260129-add-function-field-design.md
## Test plan
- [x] partition_manager_test.go updated for new `schemaVersion`
parameter
- [x] wal_test.go updated with CollectionSchema in CreateCollection
messages
- [ ] CI: code-check, ut-go, integration-test
🤖 Generated with [Claude Code](https://claude.ai/code)
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Replace keyLock-based blocking with per-key FIFO queues so that Submit()
is non-blocking to the caller. Different keys execute concurrently; same
key tasks run serially via queue.
Key changes:
- Add syncutil.Semaphore with dynamic SetCapacity and context-aware
Acquire for backpressure and graceful shutdown.
- Use goroutine in dispatchLocked to avoid deadlock when chaining tasks
from within a worker's completion path.
- Guard cleanup with sync.Once to handle the race between normal task
completion and pool rejection during shutdown.
- Explicit Close() drains all remaining queued tasks.
- resizeHandler adjusts both pool size and semaphore capacity.
issue: #49077
---------
Signed-off-by: chyezh <chyezh@outlook.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
See #48914
After delta logs migrated from SegmentInfo to manifests (V3), the
Deltalogs field in SegmentInfo is always empty for V3 segments. All
compaction trigger logic reads from `segment.GetDeltalogs()`, so
compaction triggers never fire for V3 segments.
Fix: when V3 delta data is produced (flush or L0 compaction), also
write a delta summary entry (`LogID` + `EntriesNum` + `MemorySize`,
no path) to `SegmentInfo.Deltalogs`. All existing consumer code
works unchanged.
- `pack_writer_v3.go`: `writeDelta()` returns delta summary
- `l0_compactor.go`: V3 `CompactionSegment` includes delta summary
- `compaction_task_l0.go`: always write deltalogs via
`AddBinlogsOperator`
- `l0_compaction_test.go`: IT verifying Deltalogs on V3 L1 segment
after L0 compaction
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
Related to #48885
Sync tasks running on StorageV3 segments crash datanode/streamingnode
when loon's FailResolver reports a concurrent transaction (e.g.
ghost-write after a network drop): the error escapes pack_writer_v3.go
and the conc.Pool worker re-panics, taking the process down. The four
manifest steps in BulkPackWriterV3.Write previously had no retry
coverage above writeInserts, so any conflict on writeStats / writeDelta
/ writeBM25Stasts was fatal.
Wrap the whole Write in retry.Do, restarting each attempt from the
manifest version observed at entry. To keep retries idempotent, defer
the metaCache RollStats / MergeBm25Stats mutations into a pending list
and drain it via defer only when Write returns nil — failed syncs no
longer pollute bloom filter / BM25 history. Merged-stats serialization
on flush takes the current batch explicitly through new
serializeMergedPkStatsWith / serializeMergedBM25StatsWith helpers so it
does not depend on the metaCache having been rolled yet
(SegmentBM25Stats.Clone added for the same reason).
The inner retry.Do in writeInserts is removed to avoid attempts^2
amplification under the new outer loop. Loon FFI failures are wrapped
with a new packed.ErrLoonTransient sentinel so classifyLoonErr can retry
them and let everything else short-circuit through retry.Unrecoverable.
The sentinel is intentionally coarse for now: milvus-storage does not
yet expose structured error codes, so all loon errors are treated as
retryable and the bounded retry budget keeps the worst case finite. A
TODO marks the follow-up to narrow the sentinel once error codes land.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Replace all hard-coded `-1` literal used as manifest version with the
named constant `packed.ManifestEarliest` (value `0`) for new segment
creation, making the intent explicit and consistent across the codebase.
Key changes:
- Define `ManifestLatest` (-1) and `ManifestEarliest` (0) constants in
`internal/storagev2/packed/constant.go`
- Update all production code (segment_manager, write_buffer,
importv2/util, binlog_record_writer) to use `packed.ManifestEarliest`
- Update all test code (pack_writer_v3_test, task_test) to use
`packed.ManifestEarliest` instead of raw `-1`
- Fix `CreateManifestForSegment` FFI call: use version `0` (earliest)
instead of `-1` (latest) and use `LOON_TRANSACTION_RESOLVE_OVERWRITE`
resolve strategy
- Fix `FFIPackedWriter.Close` to use
`LOON_TRANSACTION_RESOLVE_OVERWRITE` resolve strategy
issue: #48543
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
## Summary
- Implement virtual primary key (VirtualPK) for external collections
without user-defined PK
- Add ExternalSegmentCandidate for PK oracle to support segment pruning
on external segments
- Support loading external segments via manifest-based sealed segment
loader
- Add ManifestGroupTranslator with external field name fallback for
column mapping
- Add lance-table and vortex format support with HTTP scheme fix for
MinIO endpoints
- Fix extfs boolean property type mismatch by disabling extfs.* bool
properties
- Add lance and vortex format E2E tests with full query/search coverage
- Add Python data generation scripts and CI Docker Python environment
issue: #45881
design doc:
https://github.com/milvus-io/milvus-design-docs/blob/main/design_docs/20260105-external_table.md
## Test plan
- [x] Unit tests pass (pkoracle, segments, delegator, datanode/external)
- [x] E2E tests pass (13/13): TestCreateExternal*, TestRefreshExternal*,
TestExternalCollectionLanceFormat, TestExternalCollectionVortexFormat
- [x] `make milvus` compiles successfully
- [x] `make lint-fix` passes with no auto-fixes
- [ ] CI checks pass
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
## Summary
- **TestChannelCPUpdater/TestUpdate**: increase `Eventually` timeout
from 10s to 30s — the test adds 100k tasks with 10ms mock sleep per
batch call, which can exceed 10s on busy CI machines
- **TestEtcdKV/TestRevisionBytes**: use `GreaterOrEqual` instead of
`Equal` for etcd `resp.Header.Revision` check — the header revision
reflects the cluster's global revision which can be higher than
`revision+1` when other tests write to etcd concurrently
## Test plan
- [x] `TestChannelCPUpdater` 5/5 passes
- [x] `TestEtcdKV/TestRevisionBytes` 5/5 passes
issue: https://github.com/milvus-io/milvus/issues/39893🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: Li Liu <li.liu@zilliz.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
### Summary
Follow-up to #48152 which applied denylist retry to parquet/json/csv
imports but missed two other paths.
- **fix(High)**: `pack_writer.go` `writeLog` now skips retry only for
non-retryable errors (permission denied, bucket not found, invalid
credentials, etc.), matching the denylist strategy in
`retryable_reader.go`.
- **fix(Medium)**: Binlog import's `WithDownloader` callbacks now use
`multiReadWithRetry`, skipping retry only for non-retryable errors.
Previously all transient failures were not retried.
- **fix(Low)**: `IsMilvusError` in `merr/utils.go` switched from
`errors.Cause` (root only) to `errors.As` (full chain traversal).
### Out of Scope
- `pack_writer_v2.go` / `pack_writer_v3.go` — same retry pattern but
different code path (multi-part upload); separate fix.
- `writeDelta` — no retry wrapper; separate concern.
issue: #48153
---------
Signed-off-by: Yihao Dai <yihao.dai@zilliz.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Related to #48120
When pchannel migrates between streaming nodes, growing segments lose
their ManifestPath because it was only generated locally in the write
buffer metacache, never persisted to DataCoord meta. The new node gets
an empty ManifestPath from GetSegmentInfo, causing "unexpected end of
JSON input" errors on sync.
Generate ManifestPath in openNewSegmentWithGivenSegmentID when
StorageVersion is V3, so segments are born with a valid manifest path in
DataCoord meta.
Also fix typo: UnmarshalManfestPath → UnmarshalManifestPath across
codebase.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #44956
Previously, V3 deltalogs were written to
{rootPath}/delta_log/{collID}/{partID}/{segID}/{logID}, separate from
the segment's basePath. The manifest used complex "../" relative paths
to bridge the two locations. This change writes V3 deltalogs directly to
{basePath}/_delta/{logID}, aligning with the C loon library's native
_delta/ convention and simplifying the manifest relative path to just
the logID filename. Legacy V1 segments and existing manifests remain
backward compatible.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
issue: #47790
Replace 131 occurrences of fmt.Sprint(paramtable.GetNodeID()) with the
existing paramtable.GetStringNodeID() across 40 files. This avoids
redundant fmt.Sprint allocations on every call since GetStringNodeID()
uses the more efficient strconv.FormatInt. In the search hot path alone,
this eliminates 3+ string allocations per segment per query.
Signed-off-by: lyang24 <lanqingy93@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
See #47520
This update refactors the StorageV2FSCache to convert the Key structure
into api::Properties, allowing for a unified filesystem cache that
integrates metrics tracking. The changes improve the management of
filesystem instances and enhance logging for filesystem retrieval
operations. Additionally, new filesystem metrics are introduced to
monitor read/write operations and other relevant statistics.
- Converted Key to api::Properties in StorageV2FSCache::Get
- Introduced metrics for filesystem operations
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
This PR:
1. Modifies the import mechanism to use actual timestamps from the
imported data for segment positions instead of using the channel
checkpoint.
2. Fix writeDelta() to extract actual min/max timestamps from
deltaData.Tss instead of using pack.tsFrom/tsTo.
3. Integration test TestBinlogImport updated to verify L1 and L0 segment
positions.
issue: https://github.com/milvus-io/milvus/issues/47275
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Related to #44956
See also #46976
This PR:
- Add `BulkPackWriterV3` for v3 segment sync task
- Remove not needed params for legacy usage
- Fix import task still use bucketname concatenation
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
