mirror of https://github.com/milvus-io/milvus.git synced 2026-07-21 02:05:41 +00:00

Files

T

5eebaa9ad4 enhance: add WAL trace propagation (#50796 )

issue: #47404

- message trace context: add trace context serialization and
restore/inject helpers for WAL messages and msgstream conversion
- WAL append trace: normalize WAL spans for autocommit, txn, broadcast,
append, appendimpl, and broadcast callback paths
- trace propagation: restore message trace context in producer,
broadcast retry, replicate primary/secondary, recovery, flusher, and
query/data flowgraph consumers

---------

Signed-off-by: chyezh <chyezh@outlook.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-07-15 02:14:37 +08:00

3.4 KiB

Raw Blame History

Milvus Streaming System

How to use this knowledge base: This README provides the architecture overview of the WAL system. Each component name is a link to its detailed doc. When your task involves a specific component, read the linked doc to get implementation details, interfaces, and code locations before making changes.

Architecture Overview

Milvus uses a log-structured WAL (Write-Ahead Log) as its single source of truth for all data mutations and metadata changes. The WAL spans multiple PChannels distributed across StreamingNodes, coordinated by StreamingCoord, and accessed by other components through a StreamingClient library.

Channel Model

The WAL is partitioned into PChannels (physical), mapped to VChannels (logical, per-shard-of-collection), with a singleton CChannel (control) for cluster-wide ordering. See channel/channel.md for details.

Message Model

Every WAL entry is a Message representing a system event, with TimeTick as PChannel-level monotonically increasing log sequence number.

Data Flow

DML (Insert/Delete, etc.): Client → Proxy → StreamingClient.Append → StreamingNode → WAL Backend.
DDL/DCL (CreateCollection, RBAC, etc.): Client → Proxy → StreamingClient.Broadcast → StreamingCoord.Broadcaster → StreamingNodes (all relevant PChannels atomically) → WAL Backend.
WALInternal (TimeTick, Flush, CreateSegment, Txn): Self-generated by WAL System, not from external clients.
Replicated: Primary WAL → CDC ChannelReplicator → Secondary Proxy → Secondary WAL (Replicate Interceptor) → WAL Backend.
Consume & Persist: WAL Backend → RecoveryStorage (checkpoint, metadata, segment data persistence) + Broadcaster ACK (confirms per-PChannel broadcast delivery back to StreamingCoord).

Components

StreamingCoord (singleton, runs inside RootCoord process):

Channel Management: PChannel-to-StreamingNode assignment, node health monitoring, VChannel/CChannel allocation.
Broadcaster: Cross-PChannel atomic broadcast (DDL/DCL) with resource locking and ACK tracking.

StreamingNode (multiple instances, each manages a subset of PChannels):

TimeTick & Transaction: TimeTick allocation/confirmation, transaction lifecycle, LastConfirmedMessageID.
Lock: Exclusive/shared append access at VChannel or PChannel scope.
Shard Management: Per-PChannel collection/partition/segment metadata and segment assignment.
RecoveryStorage: Checkpoint, metadata and data persistence and WAL-based state recovery.
WAL Tracing: Trace span semantics for append, consume, broadcast, transaction, and replication paths.

StreamingClient: In-process Append/Read/Broadcast API with service discovery and auto-reconnect.

Replication & CDC: Star-topology cross-cluster WAL replication and role management.

WAL Backend: Durable storage layer (Kafka/Pulsar/Woodpecker/RMQ), one topic per PChannel.

3.4 KiB Raw Blame History