mirror of
https://github.com/milvus-io/milvus.git
synced 2026-07-21 10:15:43 +00:00
d500dc90d6
issue: #51094 (ported from #51096, targeting the poc_2.6.17 production branch where the incident occurred) ### What Segments created in the same batch (bulk import, DR initial sync, restore) accumulate deltalogs at nearly the same rate, so they cross the single-compaction hard thresholds almost simultaneously. The result is a synchronized rewrite avalanche: in a production incident, ~1,400 segments became eligible within ~36 hours and the cluster rewrote multiple TB against a rate-limited object store, degrading queries for days. Two independent, individually-disableable, refreshable mechanisms: 1. **Deterministic per-segment threshold jitter** (`dataCoord.compaction.single.thresholdJitter`, default 0.25). Each segment gets a stable multiplier in `[1, 1+J]` (splitmix64 hash of segment ID — no persisted state, consistent across restarts/nodes) applied to every accumulation dimension (deltalog count/size, deleted-rows ratio, expired-entities ratio/size). Applies to **all** single-compaction trigger axes and spreads a cohort's crossing times. 2. **Token-bucket admission at plan generation** (`dataCoord.compaction.single.rateLimitTokens` / `rateLimitInterval`, default 256 per 60s) for **delete-accumulation candidates only**, dirtiest-first, with a jittered bounded-deferral escape. ### Scope of each mechanism (precise guarantees) - The **token-bucket rate ceiling applies to delete accumulation only.** Its dirtiest-first ordering and its bounded-deferral escape both use a *normalized delete pressure* = `max` over the three delete axes (deltalog count, deleted-rows ratio, delete-log size) of `actual / threshold`, so a segment triggered by any single axis (e.g. large varchar-PK deletes concentrated in a few files) is ordered and bounded on that axis — not only on file count. - **Expiry accumulation, strict age-based TTL, and index rebuild bypass admission** (they are retention / correctness obligations that must not be paced behind a delete backlog). They are **de-synchronized by jitter only**, not rate-capped. A TTL cliff is therefore smeared by jitter, not converted into a token-bucket-paced stream. - The bounded-deferral **escape is itself jittered** (`pressure >= 4x` scaled by the segment's jitter multiplier) and does not consume a token, so a deferred count-triggered cohort does not re-synchronize at the `4x` threshold and avalanche through the bypass together. The same jitter means a threshold config *decrease* raises pressure into a per-segment spread of escape points rather than releasing the cohort in one round. - An axis whose threshold is non-positive is skipped, so a misconfigured (`<= 0`) threshold disables that axis rather than turning the escape into a free bypass. - `thresholdJitter=0` plus `rateLimitTokens=0` restore exact legacy behavior; force/manual compaction bypasses admission. ### Adversarial-review fixes folded in (three rounds on #51096) - **Starvation of non-delete reasons** — only delete accumulation is admission-gated; expiry accumulation, strict age-TTL and index rebuild bypass pacing. - **Bounded deferral on every delete axis** (not just file count) via the normalized-pressure escape; dirtiest-first ordering likewise covers all axes. - **Jittered escape** so a deferred count-triggered cohort cannot re-synchronize and avalanche through the bypass. - **Sub-one token budget soft-deadlock** — a positive `rateLimitTokens` below 1 clamps to 1. - **Non-positive threshold** no longer disables pacing (per-axis skip). - **Throttle-warning masking** — the sustained-throttle counter is cleared only when the bucket has genuinely refilled. - **Tokens vs backpressure** — both the L1 trigger and the L2 policy refund tokens for admission-gated segments whose plans are dropped before enqueue. The single shared bucket is intentional: it bounds *aggregate* rewrite load against the object store (which is source-agnostic), and refills every round; per-caller buckets would multiply the aggregate ceiling by the caller count and defeat the protection. Sustained one-sided saturation is surfaced by the throttle warning. ### Test plan - New `compaction_admission_test.go`: multiplier determinism / range / zero-jitter disable; disabled-bucket passthrough, dirtiest-first limiting, jittered escape bypass, token refill with injected clock; plus dedicated cases for each review fix (sub-one budget, delete-size-axis reaches the escape, non-positive threshold still paces, refund, throttle-counter survival on a drained bucket, reason classification). - Existing `Test_compactionTrigger_*`, `TestSingleCompactionPolicySuite`, `TestCompactionTriggerManager` pass; two cases asserting the exact legacy `201 vs 200` deltalog boundary pin `thresholdJitter=0`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: zhenshan.cao <caozhenshan007@gmail.com> Co-authored-by: zhenshan.cao <caozhenshan007@gmail.com>