milvus/docs/design-docs/design_docs/20260131-client_side_telemetry.md

MEP: Client-Side Telemetry with Heartbeat and Server Command Support

• Created: 2026-01-31
• Author(s): @xiaofanluan
• Status: Draft
• Component: SDK | Proxy | Coordinator
• Related Issues: #46934
• Released: [TBD]

Summary

This MEP introduces a client-side telemetry system for the Go SDK that collects operational metrics, sends periodic heartbeats to the server, and supports bidirectional communication through server-pushed commands. The system provides visibility into client behavior, enables real-time monitoring through a WebUI dashboard, and allows server-initiated configuration changes.

Motivation

Currently, Milvus lacks visibility into client-side operations and behavior. Operators cannot:

• Monitor which clients are connected to the cluster
• Understand client-side performance characteristics (latency, error rates)
• Identify problematic clients or usage patterns
• Push configuration changes to connected clients dynamically

This feature addresses these gaps by implementing a comprehensive client telemetry system that:

1. Collects client-side metrics (request counts, latencies, errors)
2. Reports telemetry data to the server via heartbeats
3. Enables server-to-client command channels for dynamic configuration
4. Provides a WebUI for operators to monitor and manage clients

Public Interfaces

Go SDK APIs

// TelemetryConfig holds configurable settings for client telemetry
type TelemetryConfig struct {
    Enabled           bool          // Enable/disable telemetry collection
    HeartbeatInterval time.Duration // Heartbeat frequency (default: 30s)
    SamplingRate      float64       // Sampling rate 0.0-1.0 (default: 1.0)
    ErrorMaxCount     int           // Max errors to track (default: 100)
}

// ClientConfig gains a new field
type ClientConfig struct {
    // ... existing fields ...
    TelemetryConfig *TelemetryConfig
}

HTTP REST APIs (Proxy)

GET  /api/v1/telemetry/clients          - List connected clients with optional filtering
POST /api/v1/telemetry/commands         - Push commands to clients
DELETE /api/v1/telemetry/commands/{id}  - Delete a command

gRPC APIs (RootCoord)

service RootCoord {
    // Client heartbeat with metrics
    rpc ClientHeartbeat(ClientHeartbeatRequest) returns (ClientHeartbeatResponse);

    // Query connected clients
    rpc GetClientTelemetry(GetClientTelemetryRequest) returns (GetClientTelemetryResponse);

    // Push commands to clients
    rpc PushClientCommand(PushClientCommandRequest) returns (PushClientCommandResponse);

    // Delete commands
    rpc DeleteClientCommand(DeleteClientCommandRequest) returns (DeleteClientCommandResponse);
}

Design Details

Architecture Overview

┌─────────────────────────────────────────────────────────────────────────────┐
│                              Client (Go SDK)                                 │
│  ┌───────────────────────────────────────────────────────────────────────┐  │
│  │                    ClientTelemetryManager                              │  │
│  │  ┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐       │  │
│  │  │ Operation       │  │ Error           │  │ Command         │       │  │
│  │  │ Metrics         │  │ Collector       │  │ Handler         │       │  │
│  │  │ Collector       │  │ (Ring Buffer)   │  │ Router          │       │  │
│  │  └────────┬────────┘  └────────┬────────┘  └────────┬────────┘       │  │
│  │           │                    │                    │                 │  │
│  │           └────────────────────┼────────────────────┘                 │  │
│  │                                ▼                                       │  │
│  │                    ┌───────────────────────┐                          │  │
│  │                    │   Heartbeat Loop      │───────── 30s interval    │  │
│  │                    │   (Background)        │                          │  │
│  │                    └───────────┬───────────┘                          │  │
│  └────────────────────────────────┼──────────────────────────────────────┘  │
└───────────────────────────────────┼─────────────────────────────────────────┘
                                    │
                          ClientHeartbeat RPC
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                              Server (Milvus)                                 │
│  ┌────────────────┐          ┌───────────────────────────────────────────┐  │
│  │     Proxy      │◄────────►│              RootCoord                     │  │
│  │                │          │  ┌─────────────────────────────────────┐  │  │
│  │  HTTP API      │          │  │      Telemetry Manager              │  │  │
│  │  /telemetry/*  │          │  │  ┌───────────┐  ┌───────────────┐   │  │  │
│  │                │          │  │  │ Client    │  │ Command       │   │  │  │
│  │  WebUI         │          │  │  │ Store     │  │ Store         │   │  │  │
│  │  telemetry.html│          │  │  └───────────┘  └───────────────┘   │  │  │
│  └────────────────┘          │  └─────────────────────────────────────┘  │  │
│                              └───────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────────────────┘

Client-Side Components

1. ClientTelemetryManager

The central component managing telemetry collection and heartbeat communication.

type ClientTelemetryManager struct {
    config         *TelemetryConfig
    client         *Client
    clientID       string                              // Stable UUID
    collectors     map[string]*OperationMetricsCollector
    errorCollector *ErrorCollectorImpl
    commandHandlers map[string]CommandHandler

    // Heartbeat management
    stopCh         chan struct{}
    wg             sync.WaitGroup
}

Key behaviors:

• Generates stable client UUID on creation
• Starts background heartbeat loop on Start()
• Sends first heartbeat immediately, then every HeartbeatInterval
• Collects and resets metrics atomically during snapshot creation

2. OperationMetricsCollector

Per-operation metrics collection with global and per-collection breakdown.

type OperationMetricsCollector struct {
    // Global metrics
    requestCount int64
    successCount int64
    errorCount   int64
    totalLatency int64  // microseconds
    maxLatency   int64

    // P99 calculation (ring buffer of 1000 samples)
    latencySamples []int64
    totalSamples   int64

    // Per-collection metrics
    collectionMetrics map[string]*CollectionMetrics
}

Metrics tracked:

• Request count
• Success/error counts
• Average latency (ms)
• P99 latency (ms) - calculated from 1000 sample buffer
• Max latency (ms)

3. ErrorCollectorImpl

Ring buffer implementation for tracking recent errors.

type ErrorCollectorImpl struct {
    errors   []*ErrorInfo
    maxCount int
    index    int  // Ring buffer index
}

type ErrorInfo struct {
    Timestamp  int64
    Operation  string
    ErrorMsg   string
    Collection string
    RequestID  string
}

4. Command Handler System

Supports server-pushed commands with extensible handler registration.

type CommandHandler func(cmd *ClientCommand) string  // Returns error message or ""

// Built-in commands:
const (
    CmdSetSamplingRate   = "set_sampling_rate"        // Adjust sampling rate
    CmdEnableCollections = "enable_collections"       // Enable specific collections
    CmdUpdateConfig      = "update_config"            // Update telemetry config
)

Server-Side Components

1. Telemetry Manager (RootCoord)

Central server-side storage for client telemetry data.

type TelemetryManager struct {
    clients      map[string]*ClientTelemetry  // clientID -> telemetry
    commandStore *CommandStore
}

type ClientTelemetry struct {
    ClientInfo        *commonpb.ClientInfo
    LastHeartbeatTime int64
    Status            string  // "active" or "inactive"
    Databases         []string
    Metrics           []*commonpb.OperationMetrics
}

2. Command Store

Manages pending commands for clients with support for persistent and one-time commands.

type CommandStore struct {
    commands     []*commonpb.ClientCommand  // One-time commands
    persistent   []*commonpb.ClientCommand  // Persistent commands
}

Command targeting:

• TargetScope = "*" - All clients
• TargetScope = "client_id:xxx" - Specific client
• TargetScope = "db:database_name" - Clients using specific database

3. HTTP Handlers (Proxy)

REST API endpoints for WebUI and external integrations.

Endpoint	Method	Description
/api/v1/telemetry/clients	GET	List connected clients
/api/v1/telemetry/clients?database=X	GET	Filter by database
/api/v1/telemetry/clients?include_metrics=true	GET	Include operation metrics
/api/v1/telemetry/commands	POST	Push command to clients
/api/v1/telemetry/commands/{id}	DELETE	Remove a command

Authentication: Uses existing Milvus Basic Auth when authorization is enabled.

Heartbeat Protocol

Client                                      Server
   │                                           │
   │──── ClientHeartbeatRequest ──────────────►│
   │     - ClientInfo (ID, SDK version, host)  │
   │     - Metrics (per-operation, per-coll)   │
   │     - CommandReplies                      │
   │     - ConfigHash (for change detection)   │
   │                                           │
   │◄─── ClientHeartbeatResponse ─────────────│
   │     - Commands (pending for this client)  │
   │     - NewConfigHash (if config changed)   │
   │                                           │

Heartbeat interval: 30 seconds (configurable, server can override)

Config hash: SHA-256 of client configuration, used to detect when server pushes new config.

Data Flow

1. Metrics Collection:

   • Each SDK operation (Search, Insert, Query, etc.) records metrics
   • Sampling rate determines if operation is tracked (default: 100%)
   • Metrics stored in per-operation collectors

2. Heartbeat Cycle:

   • Background goroutine wakes every 30 seconds
   • Creates atomic snapshot of all metrics (resets counters)
   • Sends snapshot to RootCoord via ClientHeartbeat RPC
   • Receives and processes any pending commands

3. Command Processing:

   • Server pushes commands via heartbeat response
   • Client executes command handler
   • Reply sent in next heartbeat request
   • Persistent commands re-sent until explicitly deleted

WebUI Dashboard

A new telemetry dashboard (/webui/telemetry.html) provides:

• Client List: Active/inactive clients with connection details
• Metrics View: Per-client operation metrics with latency charts
• Command Interface: Send commands to specific clients or broadcast
• Filtering: By database, client ID, or status

Compatibility, Deprecation, and Migration Plan

• Backward Compatible: Telemetry is opt-in and disabled by default can be enabled
• No Breaking Changes: Existing SDK usage remains unchanged
• Server Compatibility: Old clients work with new servers (no heartbeats sent)
• Client Compatibility: New clients work with old servers (heartbeats fail silently)

Test Plan

Unit Tests

• telemetry_test.go: Metrics collection, P99 calculation, ring buffer
• telemetry_http_handler_test.go: HTTP API handlers
• command_router_test.go: Command routing and handling

Integration Tests

• telemetry_integration_test.go: End-to-end heartbeat flow
• Multi-client scenarios with different configurations
• Command push and execution verification

Manual Testing

• WebUI functionality verification
• Performance impact measurement under load
• Network failure and recovery scenarios

Rejected Alternatives

1. Streaming Telemetry (Rejected)

Using streaming RPC instead of periodic heartbeats.

• Rejected because: Higher resource usage, more complex failure handling

2. External Metrics System (Rejected)

Push metrics to external systems (Prometheus, etc.) from client.

• Rejected because: Adds external dependencies, complicates deployment

3. Pull-Based Model (Rejected)

Server polls clients for metrics.

• Rejected because: Doesn't scale, requires client to expose endpoints

References

• Milvus existing telemetry: internal/proxy/impl.go
• gRPC health checking: https://github.com/grpc/grpc/blob/master/doc/health-checking.md
• OpenTelemetry SDK patterns: https://opentelemetry.io/docs/instrumentation/go/