mirror of
https://github.com/milvus-io/milvus.git
synced 2026-07-21 10:15:43 +00:00
2c0c3ed5f5
## Issue issue: #50102 ## Problem With etcd auth enabled and `mixCoord.enableActiveStandby: true`, a long-running **standby** MixCoord panics when promoted to active: ``` panic: rpc error: code = Unauthenticated desc = etcdserver: invalid auth token ProxyWatcher.startWatchEtcd (proxy_watcher.go:146) ``` This is **not** a credential misconfiguration (the same config works for the active node and all other components, and a restart of the same pod recovers). Root cause: - etcd's default `simple` auth token is stateful and server-local: GC'd after idle TTL (`--auth-token-ttl`, default 300s), member-local, and lost on member restart. A long-idle standby's token gets invalidated server-side without the client knowing. - When an **already-established watch stream** then hits `Unauthenticated`, clientv3 classifies it as a halt error (`isHaltErr` — anything that isn't `Unavailable`/`Internal`) and tears the stream down **without** refreshing the token. clientv3's token refresh only covers unary calls and a stream's first recv, not a live watch stream. - `ProxyWatcher.startWatchEtcd` and `sessionutil.handleWatchErr` only handled `ErrCompacted` and `panic()`ed on every other watch error. ## Fix Treat auth-token watch errors as recoverable and re-establish the watch with a bounded retry, instead of panicking. Re-watching issues a unary request first (`getSessionsOnEtcd` / `GetSessions`), which refreshes the auth token via clientv3's unary retry interceptor, so the new watch stream recovers. - New shared predicate `etcd.IsRetriableWatchErr` classifies recoverable watch errors (`ErrCompacted` + auth-token errors: `ErrInvalidAuthToken` / `ErrUserEmpty` / `ErrAuthOldRevision` / raw `Unauthenticated`). - `ProxyWatcher.startWatchEtcd` and `sessionutil.handleWatchErr` now re-watch on recoverable errors via `retry.Do(..., retry.RetryErr(etcd.IsRetriableWatchErr))`. The retry is bounded by `retry.Do`'s default attempt count / backoff; non-recoverable errors still fail fast. The CDC controller and streaming session discoverer already self-heal (return-and-re-list on watch error), so they are unchanged. > Operational note for affected users: configuring etcd with **JWT** auth tokens (`--auth-token jwt,...`) instead of the default `simple` token avoids the unexpected server-side invalidation entirely (JWT is stateless — valid across members, survives restart, no idle GC). ## Test - Added `TestIsRetriableWatchErr` (pkg/util/etcd) — predicate classification. - Added subtests in `TestWatcherHandleWatchResp` (sessionutil) — auth-token error triggers re-watch (not channel close), non-retriable error closes the channel. - Existing `TestProxyManager_ErrCompacted` still passes (non-retriable bad-session-data fails fast). > Note: the watcher integration tests link the C++ core; they were not runnable in my local/remote env at PR time and will be validated in CI. Signed-off-by: xiaofanluan <xf@hjjaq.com> Co-authored-by: xiaofanluan <xf@hjjaq.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>