issue: #51497
After a successful WAL switch via `POST /management/wal/alter`, `GET
/management/config/get?keys=mq.type` kept returning the pre-switch value
with `"source":"RuntimeSource"` until the coordinator restarted, even
though the ack callback had already persisted the new mq.type into etcd
(and linearizably refreshed the local EtcdSource). External operators
polling that endpoint to confirm switch completion never saw the new
value.
### Root cause
`InitAndSelectWALName()` saved the startup-resolved WAL name into the
config manager's **runtime overlay**, and `Manager.GetConfig` serves the
overlay with top priority — permanently shadowing the EtcdSource for
every reader in the process. The overlay write existed only to feed
`ProcessImmutableConfigs`, which pins `GetConfig`'s value into etcd
create-if-absent at first boot (without it, the literal `"default"`
would be persisted).
### Fix
Replace the overlay bridge with an explicit per-key **renderer** on the
persistence path, so mq.type reads fall through to the etcd source and
naturally follow a WAL switch:
- `ProcessImmutableConfigs(renderers map[string]func(raw string)
string)`: a renderer converts a placeholder raw value (mq.type's literal
`"default"`) into the concrete value at create-if-absent persist time.
It never runs when etcd already holds the key (a completed switch
survives restart), and it also covers the key-absent-from-all-sources
case (raw is passed as `""`). Generic: any future immutable config with
a placeholder value can plug in its own renderer.
- `InitAndSelectWALName()` no longer writes the runtime overlay and
returns the resolved `message.WALName`; the coordinator startup passes a
mq.type renderer backed by that name.
- `dependency.HealthCheck` resolves the literal `"default"` to the
actually-enabled MQ before probing (this gap predates the overlay Save:
since #36822, deployments running with `mq.type: default` probed nothing
and always reported unhealthy).
This also fixes two other readers of the shadowed value:
- the `HandleAlterWAL` same-type short-circuit: re-posting the completed
target no longer re-broadcasts, and a rollback to the original MQ is no
longer silently swallowed with "already configured";
- the proxy MQ health check (`/_cluster/dependencies`) no longer keeps
probing the old MQ forever after a switch.
### Behavior note
`MQCfg.Type.GetValue()` now follows etcd at runtime instead of being
frozen at process start: the coordinator sees a switch immediately
(linearizable refresh in the ack callback); other nodes converge within
one EtcdSource poll cycle (default `refreshInterval` 5s). Audited
readers: `mustSelectMQType` / `MustSelectWALName` resolve `"default"`
themselves and are startup-scoped; the AlterWAL idempotency check and
the proxy health check are exactly the readers this change fixes.
### Verification
- New tests (all watched failing first): renderer converts placeholder
before first persist / existing etcd value never overwritten nor
re-rendered / no-renderer keys unchanged / key absent from all sources
still pinned via renderer (`pkg/config`); `InitAndSelectWALName` leaves
mq.type served from its original source, not `RuntimeSource`
(`streamingutil/util`); `HealthCheck("default")` resolves to the enabled
MQ (`dependency`).
- Full package runs green locally: `pkg/config`,
`internal/util/streamingutil/util`, `internal/util/dependency` (with
`-tags dynamic,test -gcflags="all=-N -l"`).
- `cmd/roles`, `internal/coordinator`, `internal/distributed/streaming`
type-checked via `go vet` locally (no current-master segcore build on
this machine); relying on CI for full compilation. A live-cluster alter
→ config/get end-to-end pass is still pending — the switch-visibility
chain (etcd write → `GetConfig` returns the new value) is covered at
unit level by the existing `TestAlterConfigsInEtcd`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
## What this PR does
Adds an integration test (`TestWpRetentionTruncateRead`) for the
Woodpecker (wp)
WAL implementation pinning the read semantics around truncation that
Milvus
relies on but had no explicit coverage for.
Milvus truncates the wp WAL aggressively right after a flush, but
truncation
only moves a metadata watermark — it does **not** delete data. Data
at/under the
watermark stays readable until the retention-TTL GC physically removes
it. The
test pins this with three sub-cases:
1. **TruncatedDataStillReadable** — after truncating to the latest id,
an
explicit `StartFrom` at any position in `(earliest, latest]` (spanning
segment boundaries) still returns every message, including the watermark
id
itself. As a contrast, a fresh `DeliverPolicy_All` reader is parked past
the
watermark and returns nothing — proving the watermark really moved while
the
data underneath is intact.
2. **SeekForwardWithinSegment** / **SeekForwardCrossSegment** — a reader
asking
for data older than the earliest deliverable position
(`DeliverPolicy_All`
after truncation) is moved forward to the first available position
instead of
erroring or stalling. The two sub-cases exercise both branches of
`adjustPendingReadPointIfTruncated`: a truncation point inside segment 0
(within-segment) vs. in a later segment (cross-segment id jump).
## Notes
- **Deterministic by design**: large retention (`72h`) so GC never
fires, a
small `segmentRollingPolicy.maxSize` so writes span several segments,
single-threaded writes so ids are monotonic, and all assertions are
message-id based. Run logs confirm each sub-case hits its intended
adjustment
branch (`truncatedSegmentId` 0 vs >0).
- **Scope**: covers the open-time read-position adjustment after
truncation. It
does NOT exercise the physical retention-TTL GC + runtime skip-forward
path —
that needs waiting for the async GC sweep and is intentionally left out
to
keep the test deterministic.
- Runs against local fs storage; the truncate/read logic under test
lives in the
woodpecker log handle/reader and is storage-backend agnostic.
issue: #43638
---------
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
issue: #43638
update woodpecker to [release
v0.1.31](https://github.com/zilliztech/woodpecker/releases/tag/v0.1.31)
* fix(dashboard): add server runtime metrics
* fix(meta): honor configured etcd metadata prefix
* fix: make condition write auto mode fail on verification errors
* feat: split cluster, region, and az topology metadata
update woodpecker to [release
v0.1.32](https://github.com/zilliztech/woodpecker/releases/tag/v0.1.32)
* enhance: gate bucket check/create on createBucket for service mode
* enhance(docker): run server images as root by default
* enhance(docker): bundle wp (wpcli) into server images for zero-config
in-pod ops
* fix(objectstorage): silence idle-spin DEBUG logs in MinioFileWriter
sync loop , **related issue:** #48976
* fix: cut read-path DEBUG log volume (per-read verbosity + idle
tail-read spin)
* feat(quorum): make client-side quorum config runtime-tunable via
Dynamic[T]
* feat(metrics): active-segment node metric + client dashboard panels
* feat: delete-log lifecycle with async local reclaim
update woodpecker to [release
v0.1.33](https://github.com/zilliztech/woodpecker/releases/tag/v0.1.33)
* fix(reader): allow reopening at a truncated-but-not-yet-GC'd position
instead of force-advancing to the truncation point
---------
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
issue: #43638
related wp issue:
https://github.com/zilliztech/woodpecker/compare/v0.1.28...v0.1.29
- stagedstorage: replace per-writer sync goroutines with a shared sync
scheduler and bounded worker pool while preserving per-writer flush
ordering
- fix: harden writer lifecycle transitions to reject late writes during
finalize/close/fence/storage-failure states
- enhance: support Azure managed identity authentication for the storage
client
- fix(storage): enable sealed segment direct read by default with
on-demand reader lifecycle management and config validation
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
issue: #48247
Cluster mode requires shared storage across nodes, so woodpecker with
local storage type is not supported.
Add validation in WAL selection (both explicit and auto-select paths) to
panic early at startup with a clear error message.
A hidden config is provided to override this check if needed.
# BREAKING CHANGE
> [!CAUTION]
> **Cluster mode requires shared storage across nodes.**
> Woodpecker configurations using the `local` storage type are **not
supported by default** in cluster mode to prevent data inconsistency.
### Impact & Validation
- **Startup Panic:** We've added strict validation in the WAL selection
(covering both explicit and auto-select paths). The service will now
**panic early at startup** with a clear error message if an incompatible
storage configuration is detected in a clustered environment.
- **Override Option:** A hidden configuration is provided to override
this check if you have a specialized, single-node cluster setup or
understand the specific risks involved.
---
---
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
issue: #44369
woodpecker related[ issue:
#59](https://github.com/zilliztech/woodpecker/issues/59)
Refactor the WAL retention logic in Milvus StreamingNode:
- Remove the simple sampling-based truncation mechanism.
- After flush, WAL data is directly truncated.
- The retention control is now delegated to the underlying message queue
(MQ) implementation.
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
#44892 fix etcd request context contamination by RBAC auth info
```
When RBAC is enabled, Milvus uses the gRPC metadata library to inject RBAC authentication information into the request context (ctx).
Since etcd’s authentication mechanism also relies on the same metadata library, if the same ctx is passed down to the etcd request, the RBAC auth info from Milvus contaminates the auth information used by etcd.
This causes the etcd server to report an invalid auth token error when RBAC is enabled but etcd auth is disabled.
```
#43638 upgrade wp to v0.1.10
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
#43810
Fixed the issue where the result err returned by append timeout was
empty when objectstorage was unavailable, causing the client to
mistakenly believe that the write was successful.
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
#43638#43810
add internal writer without session lock;
refactor and unify read state and log entry
refactor data reading related methods;
fix bug where a closed writer is reused for finalize;
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
#43638
update wp to v0.1.3.
- Fix the goroutine leak of adv file reader.
- Refactor the log reader time wait logic.
- The server segment file reuses the reader singleton.
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
#43574#43604#43431#43603
Fix wp metrics not registered bug;
Update the version dependent on wp to v0.1.2-rc1;
improve advanced reader with concurrent prefetch blks;
add the segment rolling policy based on the number of blocks;
improve concurrent compaction
release lock failed bug
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
#41846#41894
Resolve SN OOM issue during small file loading in Woodpecker;
Correct WP fence/close execution order;
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
issue: #35917
This PR refines the meta-related APIs in datacoord to allow the ctx to
be passed down to the catalog operation interfaces
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
issue: #35917
This PR refine the querycoord meta related interfaces to ensure that
each method includes a ctx parameter.
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
issue: #35917
Before enhancing log trace information, it's necessary to pass the
context to the method entry point.
This PR first refine the rootcoord/metatable interfaces to ensure that
each method includes a ctx parameter.
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
