furyhawk/milvus
1
0
Fork 0
mirror of https://github.com/milvus-io/milvus.git synced 2026-07-21 10:15:43 +00:00
Code Issues Packages Projects Releases Wiki Activity

enhance: add compaction protection for snapshots (#48227)

Browse Source 
## Summary
- Add `compaction_protection_seconds` parameter to snapshot creation
API, allowing users to protect referenced segments from compaction for a
specified duration
- Implement dual-layer compaction protection: collection-level
fail-closed blocking (during RefIndex loading) + segment-level
expiry-based protection (after RefIndex loads)
- Protect all compaction paths: single, clustering, force-merge,
storage-version, and compaction trigger candidates

issue: https://github.com/milvus-io/milvus/issues/44358
design doc:
https://github.com/milvus-io/milvus-design-docs/blob/main/design_docs/20251114-snapshot_design.md

## Test plan
- [x] Unit tests for `IsCollectionCompactionBlocked` (7 subtests
covering blocked/unblocked/expired/multi-collection scenarios)
- [x] Unit tests for `IsSegmentCompactionProtected` and segment
protection rebuild
- [x] Unit tests for `CreateSnapshot` with
`compaction_protection_seconds` parameter
- [x] Unit tests for proxy-side validation (negative values, exceeding
max 7 days)
- [x] DDL callback tests with compaction protection parameter
- [x] Build verification (`make milvus` passes)
- [x] Lint verification (`make lint-fix` passes)
- [ ] E2E: Create snapshot with compaction protection → verify segments
are not compacted during protection window
- [ ] E2E: Verify compaction resumes after protection expires

🤖 Generated with [Claude Code](https://claude.com/claude-code)

pr: https://github.com/milvus-io/milvus-proto/pull/574

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
This commit is contained in:
wei liu
2026-04-14 20:13:42 +08:00
committed by GitHub
parent 4e8483c55d
commit 80af30b353
33 changed files with 3646 additions and 1968 deletions
Download Patch File Download Diff File Expand all files Collapse all files
configs/milvus.yaml
+1
View File
@@ -729,6 +729,7 @@ dataCoord:
slowDownCPUUsageThreshold: 0.6 # The CPU usage threshold at which the garbage collection will be slowed down
snapshot:
pendingTimeout: 60 # Timeout in minutes for pending snapshots before GC cleanup
maxCompactionProtectionSeconds: 604800 # Maximum allowed compaction protection duration in seconds (default 604800 = 7 days)
enableActiveStandby: false
brokerTimeout: 5000 # 5000ms, dataCoord broker rpc timeout
autoBalance: true # Enable auto balance
docs/user_guides/snapshot_user_guide.md
+7 -2
View File
@@ -619,13 +619,18 @@ Use consistent and descriptive naming:
- This prevents data loss but increases storage consumption
**GC Protection Mechanism:**
- **Segment-level protection**: The garbage collector checks `GetSnapshotBySegment()` before deleting any segment
- **Segment-level protection**: The garbage collector checks `IsSegmentGCBlocked()` before deleting any segment
- Segments referenced by any snapshot are skipped during GC
- Protection applies to both metadata (Etcd) and data files (S3)
- **Build-level protection**: The garbage collector checks `GetSnapshotByBuildID()` before deleting index files
- **Build-level protection**: The garbage collector checks `IsBuildIDGCBlocked()` before deleting index files
- Index files referenced by snapshots (identified by buildID) are preserved even after drop index operations
- BuildID uniquely identifies a specific index build task, providing precise file-level protection
- Ensures index data availability during snapshot restore
- **Fail-closed coarse block**: Both checks embed a collection-level fail-closed guard —
if any snapshot's RefIndex has not yet loaded from S3, the entire collection is
protected until the loader finishes. This prevents data loss during DataCoord startup.
- **O(1) query cost**: Protection state is precomputed by `rebuildAllSegmentProtection` and
stored in in-memory sets, so per-file/per-buildID checks are constant-time set lookups.
- **Implementation location**: `internal/datacoord/garbage_collector.go`
**Storage Cost Considerations:**
go.mod
+1 -1
View File
@@ -19,7 +19,7 @@ require (
github.com/google/btree v1.1.2
github.com/grpc-ecosystem/go-grpc-middleware v1.3.0
github.com/klauspost/compress v1.18.0
github.com/milvus-io/milvus-proto/go-api/v2 v2.6.6-0.20260309063517-8b5776d31f2b
github.com/milvus-io/milvus-proto/go-api/v2 v2.6.6-0.20260312114919-fe1b25e57c40
github.com/minio/minio-go/v7 v7.0.73
github.com/panjf2000/ants/v2 v2.11.3 // indirect
github.com/pingcap/log v1.1.1-0.20221015072633-39906604fb81 // indirect
go.sum
+2 -2
View File
@@ -812,8 +812,8 @@ github.com/milvus-io/cgosymbolizer v0.0.0-20250318084424-114f4050c3a6 h1:YHMFI6L
github.com/milvus-io/cgosymbolizer v0.0.0-20250318084424-114f4050c3a6/go.mod h1:DvXTE/K/RtHehxU8/GtDs4vFtfw64jJ3PaCnFri8CRg=
github.com/milvus-io/gorocksdb v0.0.0-20220624081344-8c5f4212846b h1:TfeY0NxYxZzUfIfYe5qYDBzt4ZYRqzUjTR6CvUzjat8=
github.com/milvus-io/gorocksdb v0.0.0-20220624081344-8c5f4212846b/go.mod h1:iwW+9cWfIzzDseEBCCeDSN5SD16Tidvy8cwQ7ZY8Qj4=
github.com/milvus-io/milvus-proto/go-api/v2 v2.6.6-0.20260309063517-8b5776d31f2b h1:tUp6a72FFeosfjN+8jy/Hax7BqVbtgsaMMVrAfzKnEc=
github.com/milvus-io/milvus-proto/go-api/v2 v2.6.6-0.20260309063517-8b5776d31f2b/go.mod h1:/6UT4zZl6awVeXLeE7UGDWZvXj3IWkRsh3mqsn0DiAs=
github.com/milvus-io/milvus-proto/go-api/v2 v2.6.6-0.20260312114919-fe1b25e57c40 h1:nGDPhwxHmn4wwgrzT/jhq5pOa3zu6f5lGchFP6jkwL4=
github.com/milvus-io/milvus-proto/go-api/v2 v2.6.6-0.20260312114919-fe1b25e57c40/go.mod h1:/6UT4zZl6awVeXLeE7UGDWZvXj3IWkRsh3mqsn0DiAs=
github.com/minio/asm2plan9s v0.0.0-20200509001527-cdd76441f9d8 h1:AMFGa4R4MiIpspGNG7Z948v4n35fFGB3RR3G/ry4FWs=
github.com/minio/asm2plan9s v0.0.0-20200509001527-cdd76441f9d8/go.mod h1:mC1jAcsrzbxHt8iiaC+zU4b1ylILSosueou12R++wfY=
github.com/minio/c2goasm v0.0.0-20190812172519-36a3d3bbc4f3 h1:+n/aFZefKZp7spd8DFdX7uMikMLXX4oubIzJF4kv/wI=
internal/datacoord/compaction_policy_clustering.go
+7 -1
View File
@@ -64,6 +64,11 @@ func (policy *clusteringCompactionPolicy) Trigger(ctx context.Context) (map[Comp
log.Info("skip clustering compaction for external collection", zap.Int64("collectionID", collection.ID))
continue
}
if policy.meta.isCollectionCompactionBlocked(collection.ID) {
log.Info("skip clustering compaction for collection due to unloaded protected snapshot RefIndex",
zap.Int64("collectionID", collection.ID))
continue
}
collectionViews, _, err := policy.triggerOneCollection(ctx, collection.ID, false)
if err != nil {
// not throw this error because no need to fail because of one collection
@@ -139,7 +144,8 @@ func (policy *clusteringCompactionPolicy) triggerOneCollection(ctx context.Conte
!segment.GetIsImporting() && // not importing now
segment.GetLevel() != datapb.SegmentLevel_L0 && // ignore level zero segments
!segment.GetIsInvisible() &&
(!namespaceEnabled || segment.GetIsSortedByNamespace())
(!namespaceEnabled || segment.GetIsSortedByNamespace()) &&
!policy.meta.isSegmentCompactionProtected(segment.GetID()) // not protected by snapshot
}))
views := make([]CompactionView, 0)
internal/datacoord/compaction_policy_forcemerge.go
+7 -1
View File
@@ -72,6 +72,11 @@ func (policy *forceMergeCompactionPolicy) triggerOneCollection(
log := log.Ctx(ctx).With(
zap.Int64("collectionID", collectionID),
zap.Int64("targetSize", targetSize))
if policy.meta.isCollectionCompactionBlocked(collectionID) {
log.Info("skip force merge compaction for collection due to unloaded protected snapshot RefIndex",
zap.Int64("collectionID", collectionID))
return nil, 0, nil
}
collection, err := policy.handler.GetCollection(ctx, collectionID)
if err != nil {
return nil, 0, err
@@ -106,7 +111,8 @@ func (policy *forceMergeCompactionPolicy) triggerOneCollection(
isFlushed(segment) &&
!segment.isCompacting &&
!segment.GetIsImporting() &&
segment.GetLevel() != datapb.SegmentLevel_L0
segment.GetLevel() != datapb.SegmentLevel_L0 &&
!policy.meta.isSegmentCompactionProtected(segment.GetID())
}))
if len(segments) == 0 {
internal/datacoord/compaction_policy_single.go
+14 -2
View File
@@ -62,6 +62,11 @@ func (policy *singleCompactionPolicy) Trigger(ctx context.Context) (map[Compacti
log.Ctx(ctx).Info("skip single compaction trigger for external collection", zap.Int64("collectionID", collection.ID))
continue
}
if policy.meta.isCollectionCompactionBlocked(collection.ID) {
log.Ctx(ctx).Info("skip single compaction for collection due to unloaded protected snapshot RefIndex",
zap.Int64("collectionID", collection.ID))
continue
}
collectionViews, collectionSortViews, _, err := policy.triggerOneCollection(ctx, collection.ID, false)
if err != nil {
// not throw this error because no need to fail because of one collection
@@ -115,6 +120,11 @@ func (policy *singleCompactionPolicy) triggerSegmentSortCompaction(
zap.Bool("isInvisible", segment.GetIsInvisible()))
return nil
}
if policy.meta.isSegmentCompactionProtected(segment.GetID()) {
log.Info("skip sort compaction for snapshot-protected segment",
zap.Int64("segmentID", segment.GetID()))
return nil
}
collectionTTL, err := common.GetCollectionTTLFromMap(collection.Properties)
if err != nil {
@@ -170,7 +180,8 @@ func (policy *singleCompactionPolicy) triggerSortCompaction(
}
triggerableSegments := policy.meta.SelectSegments(ctx, WithCollection(collectionID),
SegmentFilterFunc(func(seg *SegmentInfo) bool {
return canTriggerSortCompaction(seg)
return canTriggerSortCompaction(seg) &&
!policy.meta.isSegmentCompactionProtected(seg.GetID())
}))
if len(triggerableSegments) == 0 {
log.RatedInfo(20, "no triggerable segments")
@@ -263,7 +274,8 @@ func (policy *singleCompactionPolicy) triggerOneCollection(ctx context.Context,
!segment.isCompacting && // not compacting now
!segment.GetIsImporting() && // not importing now
segment.GetLevel() == datapb.SegmentLevel_L2 && // only support L2 for now
!segment.GetIsInvisible()
!segment.GetIsInvisible() &&
!policy.meta.isSegmentCompactionProtected(segment.GetID()) // not protected by snapshot
}))
for _, group := range partSegments {
internal/datacoord/compaction_policy_storage_version.go
+7 -1
View File
@@ -91,6 +91,11 @@ func (policy *storageVersionUpgradePolicy) Trigger(ctx context.Context) (map[Com
if policy.currentCount >= maxCount {
break
}
if policy.meta.isCollectionCompactionBlocked(collection.ID) {
log.Info("skip storage version compaction for collection due to unloaded protected snapshot RefIndex",
zap.Int64("collectionID", collection.ID))
continue
}
collectionViews, err := policy.triggerOneCollection(ctx, collection.ID, maxCount)
if err != nil {
// not throw this error because no need to fail because of one collection
@@ -135,7 +140,8 @@ func (policy *storageVersionUpgradePolicy) triggerOneCollection(ctx context.Cont
!segment.isCompacting &&
!segment.GetIsImporting() &&
segment.GetLevel() != datapb.SegmentLevel_L0 &&
segment.GetStorageVersion() != targetVersion
segment.GetStorageVersion() != targetVersion &&
!policy.meta.isSegmentCompactionProtected(segment.GetID())
}))
views := make([]CompactionView, 0, len(segments))
internal/datacoord/compaction_trigger.go
+10 -1
View File
@@ -549,6 +549,14 @@ func (t *compactionTrigger) generatePlans(segments []*SegmentInfo, signal *compa
// since non-major compaction happens under channel+partition level
// the selected segments are grouped into these categories.
func (t *compactionTrigger) getCandidates(signal *compactionSignal) ([]chanPartSegments, error) {
// Fail-closed: if any protected snapshot's RefIndex hasn't loaded yet,
// block compaction for the entire collection.
if signal.collectionID > 0 && t.meta.isCollectionCompactionBlocked(signal.collectionID) {
log.Info("skip compaction candidates for collection due to unloaded protected snapshot RefIndex",
zap.Int64("collectionID", signal.collectionID))
return nil, nil
}
// default filter, select segments which could be compacted
filters := []SegmentFilter{
SegmentFilterFunc(func(segment *SegmentInfo) bool {
@@ -559,7 +567,8 @@ func (t *compactionTrigger) getCandidates(signal *compactionSignal) ([]chanPartS
segment.GetLevel() != datapb.SegmentLevel_L0 && // ignore level zero segments
segment.GetLevel() != datapb.SegmentLevel_L2 && // ignore l2 segment
!segment.GetIsInvisible() &&
(segment.GetIsSorted() || segment.GetIsSortedByNamespace())
(segment.GetIsSorted() || segment.GetIsSortedByNamespace()) &&
!t.meta.isSegmentCompactionProtected(segment.GetID()) // not protected by snapshot
}),
}
internal/datacoord/ddl_callbacks_snapshot.go
+1 -1
View File
@@ -36,7 +36,7 @@ func (s *DDLCallbacks) createSnapshotV2AckCallback(ctx context.Context, result m
log.Info("createSnapshotV2AckCallback received")
// Create snapshot - ID is allocated inside CreateSnapshot
snapshotID, err := s.snapshotManager.CreateSnapshot(ctx, header.CollectionId, header.Name, header.Description)
snapshotID, err := s.snapshotManager.CreateSnapshot(ctx, header.CollectionId, header.Name, header.Description, header.CompactionProtectionSeconds)
if err != nil {
log.Error("failed to create snapshot via DDL callback", zap.Error(err))
return err
internal/datacoord/ddl_callbacks_snapshot_test.go
+7 -3
View File
@@ -48,11 +48,13 @@ func TestDDLCallbacks_CreateSnapshotV2AckCallback_Success(t *testing.T) {
ctx context.Context,
collectionID int64,
name, description string,
compactionProtectionSeconds int64,
) (int64, error) {
createSnapshotCalled = true
assert.Equal(t, int64(100), collectionID)
assert.Equal(t, "test_snapshot", name)
assert.Equal(t, "test description", description)
assert.Equal(t, int64(3600), compactionProtectionSeconds)
return 1001, nil
}).Build()
defer mockCreateSnapshot.UnPatch()
@@ -66,9 +68,10 @@ func TestDDLCallbacks_CreateSnapshotV2AckCallback_Success(t *testing.T) {
// Create test broadcast result using message builder
broadcastMsg := message.NewCreateSnapshotMessageBuilderV2().
WithHeader(&message.CreateSnapshotMessageHeader{
CollectionId: 100,
Name: "test_snapshot",
Description: "test description",
CollectionId: 100,
Name: "test_snapshot",
Description: "test description",
CompactionProtectionSeconds: 3600,
}).
WithBody(&message.CreateSnapshotMessageBody{}).
WithBroadcast([]string{"control_channel"}).
@@ -100,6 +103,7 @@ func TestDDLCallbacks_CreateSnapshotV2AckCallback_CreateError(t *testing.T) {
ctx context.Context,
collectionID int64,
name, description string,
compactionProtectionSeconds int64,
) (int64, error) {
return 0, expectedErr
}).Build()
internal/datacoord/garbage_collector.go
+51 -127
View File
@@ -558,41 +558,18 @@ func (gc *garbageCollector) recycleUnusedBinLogWithChecker(ctx context.Context,
// isSnapshotProtected checks if a segment should be skipped from GC due to snapshot references.
// Returns true if the segment is protected (should NOT be deleted).
// Caches per-collection loaded state and per-segment query results to avoid repeated calls.
//
// Delegates to snapshotMeta.IsSegmentGCBlocked, which is O(1) and handles the
// fail-closed layering (unloaded RefIndex → coarse collection-level block, else
// point query on the pre-computed segmentReferencedByGC set). The per-call caching
// that used to be needed here is no longer necessary because the lookups are now
// direct set/map reads.
snapshotMeta := gc.meta.GetSnapshotMeta()
var snapshotAllLoaded bool
if snapshotMeta != nil {
snapshotAllLoaded = snapshotMeta.IsAllRefIndexLoaded()
}
collectionLoadedCache := make(map[int64]bool)
segmentProtectedCache := make(map[int64]bool)
isSnapshotProtected := func(segmentID, collectionID int64) bool {
if snapshotMeta == nil {
return false
}
// Check loaded state: per-collection when known, global otherwise.
if collectionID >= 0 {
if loaded, ok := collectionLoadedCache[collectionID]; ok {
if !loaded {
return true
}
} else {
loaded = snapshotMeta.IsRefIndexLoadedForCollection(collectionID)
collectionLoadedCache[collectionID] = loaded
if !loaded {
return true
}
}
} else if !snapshotAllLoaded {
return true
}
// Check segment cache, then query.
if protected, ok := segmentProtectedCache[segmentID]; ok {
return protected
}
protected := len(snapshotMeta.GetSnapshotBySegment(ctx, collectionID, segmentID)) > 0
segmentProtectedCache[segmentID] = protected
return protected
return snapshotMeta.IsSegmentGCBlocked(collectionID, segmentID)
}
futures := make([]*conc.Future[struct{}], 0)
@@ -808,22 +785,16 @@ func (gc *garbageCollector) recycleDroppedSegments(ctx context.Context, signal <
continue
}
// Check if snapshot RefIndex is loaded before querying snapshot references
// If not loaded, skip this segment and try again in next GC cycle
// Skip segments protected by snapshot references. IsSegmentGCBlocked is O(1)
// and embeds the "RefIndex not loaded → fail-closed" check, so we don't need
// a separate loaded-state probe.
if snapshotMeta := gc.meta.GetSnapshotMeta(); snapshotMeta != nil {
if !snapshotMeta.IsRefIndexLoadedForCollection(segment.GetCollectionID()) {
log.Info("skip GC segment since snapshot RefIndex is not loaded yet",
zap.Int64("collectionID", segment.GetCollectionID()))
continue
}
if snapshotIDs := snapshotMeta.GetSnapshotBySegment(ctx, segment.GetCollectionID(), segmentID); len(snapshotIDs) > 0 {
log.Info("skip GC segment since it is referenced by snapshot",
if snapshotMeta.IsSegmentGCBlocked(segment.GetCollectionID(), segmentID) {
log.Info("skip GC segment since it is protected by snapshot",
zap.Int64("collectionID", segment.GetCollectionID()),
zap.Int64("partitionID", segment.GetPartitionID()),
zap.String("channel", segInsertChannel),
zap.Int64("segmentID", segmentID),
zap.Int64s("snapshotIDs", snapshotIDs))
zap.Int64("segmentID", segmentID))
continue
}
}
@@ -1113,19 +1084,13 @@ func (gc *garbageCollector) recycleUnusedSegIndexes(ctx context.Context, signal
zap.Int64("nodeID", segIdx.NodeID),
zap.Int("indexFiles", len(indexFiles)))
// Check if snapshot RefIndex is loaded before querying snapshot references
// If not loaded, skip this index and try again in next GC cycle
// Skip buildIDs protected by snapshot references. IsBuildIDGCBlocked is O(1)
// and embeds the "RefIndex not loaded → fail-closed" check.
if snapshotMeta := gc.meta.GetSnapshotMeta(); snapshotMeta != nil {
if !snapshotMeta.IsRefIndexLoadedForCollection(segIdx.CollectionID) {
log.Info("skip GC segment index since snapshot RefIndex is not loaded yet",
zap.Int64("collectionID", segIdx.CollectionID))
continue
}
if snapshotIDs := snapshotMeta.GetSnapshotByBuildID(segIdx.BuildID); len(snapshotIDs) > 0 {
log.Info("skip GC segment index since buildID is referenced by snapshot",
zap.Int64("buildID", segIdx.BuildID),
zap.Int64s("snapshotIDs", snapshotIDs))
if snapshotMeta.IsBuildIDGCBlocked(segIdx.CollectionID, segIdx.BuildID) {
log.Info("skip GC segment index since buildID is protected by snapshot",
zap.Int64("collectionID", segIdx.CollectionID),
zap.Int64("buildID", segIdx.BuildID))
continue
}
}
@@ -1156,13 +1121,9 @@ func (gc *garbageCollector) recycleUnusedIndexFiles(ctx context.Context) {
prefix := path.Join(gc.option.cli.RootPath(), common.SegmentIndexPath) + "/"
// Pre-fetch snapshot meta once before the walk to avoid repeated calls per buildID.
// Use IsAllRefIndexLoaded because orphan buildIDs (segIdx==nil) have no collection context.
// Resolve snapshotMeta once. Both IsBuildIDGCBlocked paths below are O(1) so
// no caching of intermediate state is needed.
snapshotMeta := gc.meta.GetSnapshotMeta()
var snapshotAllLoaded bool
if snapshotMeta != nil {
snapshotAllLoaded = snapshotMeta.IsAllRefIndexLoaded()
}
// list dir first
keyCount := 0
@@ -1186,18 +1147,12 @@ func (gc *garbageCollector) recycleUnusedIndexFiles(ctx context.Context) {
return true
}
if segIdx == nil {
// buildID no longer exists in meta. Check if any snapshot references this buildID.
if snapshotMeta != nil {
if !snapshotAllLoaded {
logger.Info("skip GC index files since snapshot RefIndex not fully loaded")
return true
}
if snapshotIDs := snapshotMeta.GetSnapshotByBuildID(buildID); len(snapshotIDs) > 0 {
logger.Info("skip GC index files since buildID is referenced by snapshot",
zap.Int64("buildID", buildID),
zap.Int64s("snapshotIDs", snapshotIDs))
return true
}
// buildID no longer exists in meta. Orphan buildID walk: no collection context,
// so IsBuildIDGCBlocked(-1, buildID) fail-closes on ANY unloaded RefIndex globally.
if snapshotMeta != nil && snapshotMeta.IsBuildIDGCBlocked(-1, buildID) {
logger.Info("skip GC index files since buildID is protected by snapshot",
zap.Int64("buildID", buildID))
return true
}
// buildID no longer exists in meta, remove all index files
@@ -1211,21 +1166,13 @@ func (gc *garbageCollector) recycleUnusedIndexFiles(ctx context.Context) {
return true
}
// Check if snapshot RefIndex is loaded before querying snapshot references
// If not loaded, skip this index and try again in next GC cycle
if snapshotMeta := gc.meta.GetSnapshotMeta(); snapshotMeta != nil {
if !snapshotMeta.IsRefIndexLoadedForCollection(segIdx.CollectionID) {
logger.Info("skip GC index files since snapshot RefIndex is not loaded yet",
zap.Int64("collectionID", segIdx.CollectionID))
return true
}
// Check if this build is referenced by any snapshot
// If snapshots reference this buildID, do not delete the index files
if snapshotIDs := snapshotMeta.GetSnapshotByBuildID(segIdx.BuildID); len(snapshotIDs) > 0 {
logger.Info("skip GC index files since buildID is referenced by snapshot",
zap.Int64("buildID", segIdx.BuildID),
zap.Int64s("snapshotIDs", snapshotIDs))
// Skip buildIDs protected by snapshot references. IsBuildIDGCBlocked is O(1)
// and embeds the "RefIndex not loaded → fail-closed" check.
if snapshotMeta != nil {
if snapshotMeta.IsBuildIDGCBlocked(segIdx.CollectionID, segIdx.BuildID) {
logger.Info("skip GC index files since buildID is protected by snapshot",
zap.Int64("collectionID", segIdx.CollectionID),
zap.Int64("buildID", segIdx.BuildID))
return true
}
}
@@ -1389,20 +1336,13 @@ func (gc *garbageCollector) recycleUnusedTextIndexFiles(ctx context.Context, sig
continue
}
// Check if segment is referenced by any snapshot before deleting text index files
if snapshotMeta != nil {
if !snapshotMeta.IsRefIndexLoadedForCollection(seg.GetCollectionID()) {
log.Info("skip GC text index files since snapshot RefIndex is not loaded yet",
zap.Int64("segmentID", seg.GetID()),
zap.Int64("collectionID", seg.GetCollectionID()))
continue
}
if snapshotIDs := snapshotMeta.GetSnapshotBySegment(ctx, seg.GetCollectionID(), seg.GetID()); len(snapshotIDs) > 0 {
log.Info("skip GC text index files since segment is referenced by snapshot",
zap.Int64("segmentID", seg.GetID()),
zap.Int64s("snapshotIDs", snapshotIDs))
continue
}
// Skip segments whose files are still referenced by snapshots. IsSegmentGCBlocked
// is O(1) and embeds the "RefIndex not loaded → fail-closed" check.
if snapshotMeta != nil && snapshotMeta.IsSegmentGCBlocked(seg.GetCollectionID(), seg.GetID()) {
log.Info("skip GC text index files since segment is protected by snapshot",
zap.Int64("segmentID", seg.GetID()),
zap.Int64("collectionID", seg.GetCollectionID()))
continue
}
gc.ackSignal(signal)
@@ -1478,20 +1418,12 @@ func (gc *garbageCollector) recycleUnusedJSONStatsFiles(ctx context.Context, sig
continue
}
// Check if segment is referenced by any snapshot before deleting JSON stats files
if snapshotMeta != nil {
if !snapshotMeta.IsRefIndexLoadedForCollection(seg.GetCollectionID()) {
log.Info("skip GC JSON stats files since snapshot RefIndex is not loaded yet",
zap.Int64("segmentID", seg.GetID()),
zap.Int64("collectionID", seg.GetCollectionID()))
continue
}
if snapshotIDs := snapshotMeta.GetSnapshotBySegment(ctx, seg.GetCollectionID(), seg.GetID()); len(snapshotIDs) > 0 {
log.Info("skip GC JSON stats files since segment is referenced by snapshot",
zap.Int64("segmentID", seg.GetID()),
zap.Int64s("snapshotIDs", snapshotIDs))
continue
}
// Skip segments whose files are still referenced by snapshots.
if snapshotMeta != nil && snapshotMeta.IsSegmentGCBlocked(seg.GetCollectionID(), seg.GetID()) {
log.Info("skip GC JSON stats files since segment is protected by snapshot",
zap.Int64("segmentID", seg.GetID()),
zap.Int64("collectionID", seg.GetCollectionID()))
continue
}
gc.ackSignal(signal)
@@ -1602,22 +1534,14 @@ func (gc *garbageCollector) recycleUnusedJSONIndexFiles(ctx context.Context, sig
continue
}
// Check if segment is referenced by any snapshot before deleting JSON index files
snapshotMeta := gc.meta.GetSnapshotMeta()
if snapshotMeta != nil {
// If RefIndex is not loaded yet, skip to avoid incorrectly deleting snapshot-referenced files
if !snapshotMeta.IsRefIndexLoadedForCollection(seg.GetCollectionID()) {
log.Info("skip GC JSON index files since snapshot RefIndex is not loaded yet",
// Skip segments whose files are still referenced by snapshots.
if snapshotMeta := gc.meta.GetSnapshotMeta(); snapshotMeta != nil {
if snapshotMeta.IsSegmentGCBlocked(seg.GetCollectionID(), seg.GetID()) {
log.Info("skip GC JSON index files since segment is protected by snapshot",
zap.Int64("segmentID", seg.GetID()),
zap.Int64("collectionID", seg.GetCollectionID()))
continue
}
if snapshotIDs := snapshotMeta.GetSnapshotBySegment(ctx, seg.GetCollectionID(), seg.GetID()); len(snapshotIDs) > 0 {
log.Info("skip GC JSON index files since segment is referenced by snapshot",
zap.Int64("segmentID", seg.GetID()),
zap.Int64s("snapshotIDs", snapshotIDs))
continue
}
}
gc.ackSignal(signal)
internal/datacoord/garbage_collector_test.go
+93 -136
View File
@@ -595,10 +595,9 @@ func TestGarbageCollector_recycleUnusedSegIndexes(t *testing.T) {
gc := newGarbageCollector(createMetaForRecycleUnusedSegIndexes(catalog), nil, GcOption{
cli: mockChunkManager,
})
mockIsRefIndexLoaded := mockey.Mock((*snapshotMeta).IsRefIndexLoadedForCollection).Return(true).Build()
defer mockIsRefIndexLoaded.UnPatch()
mockGetSnapshotByBuildID := mockey.Mock((*snapshotMeta).GetSnapshotByBuildID).Return([]UniqueID{}).Build()
defer mockGetSnapshotByBuildID.UnPatch()
// Snapshot layer transparent: no buildID is blocked.
mockIsBuildIDBlocked := mockey.Mock((*snapshotMeta).IsBuildIDGCBlocked).Return(false).Build()
defer mockIsBuildIDBlocked.UnPatch()
gc.recycleUnusedSegIndexes(context.TODO(), nil)
})
@@ -617,10 +616,8 @@ func TestGarbageCollector_recycleUnusedSegIndexes(t *testing.T) {
gc := newGarbageCollector(createMetaForRecycleUnusedSegIndexes(catalog), nil, GcOption{
cli: mockChunkManager,
})
mockIsRefIndexLoaded := mockey.Mock((*snapshotMeta).IsRefIndexLoadedForCollection).Return(true).Build()
defer mockIsRefIndexLoaded.UnPatch()
mockGetSnapshotByBuildID := mockey.Mock((*snapshotMeta).GetSnapshotByBuildID).Return([]UniqueID{}).Build()
defer mockGetSnapshotByBuildID.UnPatch()
mockIsBuildIDBlocked := mockey.Mock((*snapshotMeta).IsBuildIDGCBlocked).Return(false).Build()
defer mockIsBuildIDBlocked.UnPatch()
gc.recycleUnusedSegIndexes(context.TODO(), nil)
})
}
@@ -1410,10 +1407,9 @@ func TestGarbageCollector_clearETCD(t *testing.T) {
cli: cm,
dropTolerance: 1,
})
mockIsRefIndexLoaded := mockey.Mock((*snapshotMeta).IsRefIndexLoadedForCollection).Return(true).Build()
defer mockIsRefIndexLoaded.UnPatch()
mockGetSnapshotBySegment := mockey.Mock((*snapshotMeta).GetSnapshotBySegment).Return([]UniqueID{}).Build()
defer mockGetSnapshotBySegment.UnPatch()
// Snapshot layer transparent: no segment is blocked.
mockIsSegBlocked := mockey.Mock((*snapshotMeta).IsSegmentGCBlocked).Return(false).Build()
defer mockIsSegBlocked.UnPatch()
gc.recycleDroppedSegments(context.TODO(), signal)
/*
@@ -1895,15 +1891,12 @@ func TestGarbageCollector_recycleDroppedSegments_SnapshotReference(t *testing.T)
mock1 := mockey.Mock(meta.GetSnapshotMeta).Return(smMeta).Build()
defer mock1.UnPatch()
mockIsRefIndexLoaded := mockey.Mock((*snapshotMeta).IsRefIndexLoadedForCollection).Return(true).Build()
defer mockIsRefIndexLoaded.UnPatch()
mock2 := mockey.Mock((*snapshotMeta).GetSnapshotBySegment).To(func(ctx context.Context, collectionID, segmentID int64) []int64 {
if segmentID == 1001 {
return []int64{1, 2} // Segment 1001 is referenced by snapshots 1 and 2
}
return []int64{} // Segment 1002 is not referenced
}).Build()
// New O(1) API: IsSegmentGCBlocked encodes both "loaded state" and "is referenced".
// Block segment 1001; leave 1002 unblocked.
mock2 := mockey.Mock((*snapshotMeta).IsSegmentGCBlocked).To(
func(sm *snapshotMeta, collID, segID int64) bool {
return segID == 1001
}).Build()
defer mock2.UnPatch()
mock3 := mockey.Mock((*ServerHandler).ListLoadedSegments).To(func(h *ServerHandler, ctx context.Context) ([]int64, error) {
@@ -2017,15 +2010,12 @@ func TestGarbageCollector_recycleUnusedSegIndexes_SnapshotReference(t *testing.T
}
// Setup mocks
mockIsRefIndexLoaded := mockey.Mock((*snapshotMeta).IsRefIndexLoadedForCollection).Return(true).Build()
defer mockIsRefIndexLoaded.UnPatch()
mock1 := mockey.Mock((*snapshotMeta).GetSnapshotByBuildID).To(func(buildID int64) []int64 {
if buildID == 401 {
return []int64{3, 4} // BuildID 401 is referenced by snapshots 3 and 4
}
return []int64{} // BuildID 402 is not referenced
}).Build()
// New O(1) API: IsBuildIDGCBlocked encodes both "loaded state" and "is referenced".
// Block buildID 401; leave 402 unblocked.
mock1 := mockey.Mock((*snapshotMeta).IsBuildIDGCBlocked).To(
func(sm *snapshotMeta, collID, buildID int64) bool {
return buildID == 401
}).Build()
defer mock1.UnPatch()
mock2 := mockey.Mock((*datacoord.Catalog).ListSegmentIndexes).To(func(c *datacoord.Catalog, ctx context.Context, collectionID int64) ([]*model.SegmentIndex, error) {
@@ -2149,17 +2139,9 @@ func TestGarbageCollector_recycleUnusedBinlogFiles_SnapshotReference(t *testing.
mockParseSegID := mockey.Mock(storage.ParseSegmentIDByBinlog).Return(int64(1001), nil).Build()
defer mockParseSegID.UnPatch()
// Mock IsRefIndexLoadedForCollection to return true (per-collection check for known segments)
mockCollLoaded := mockey.Mock((*snapshotMeta).IsRefIndexLoadedForCollection).Return(true).Build()
defer mockCollLoaded.UnPatch()
// Mock IsAllRefIndexLoaded (used as fallback for segments not in meta)
mockAllLoaded := mockey.Mock((*snapshotMeta).IsAllRefIndexLoaded).Return(true).Build()
defer mockAllLoaded.UnPatch()
// Mock GetSnapshotBySegment to return snapshot IDs (makes segment referenced)
mockGetSnapshotBySegment := mockey.Mock((*snapshotMeta).GetSnapshotBySegment).Return([]int64{1, 2}).Build()
defer mockGetSnapshotBySegment.UnPatch()
// New O(1) API: IsSegmentGCBlocked directly returns "protected".
mockIsSegBlocked := mockey.Mock((*snapshotMeta).IsSegmentGCBlocked).Return(true).Build()
defer mockIsSegBlocked.UnPatch()
// Execute
gc.recycleUnusedBinlogFiles(ctx)
@@ -2293,11 +2275,14 @@ func TestGarbageCollector_recycleUnusedIndexFiles_SnapshotReference(t *testing.T
collectionID := int64(100)
indexID := int64(301)
// Setup mock for WalkWithPrefix to find index directory
// Setup mock for WalkWithPrefix to find index directory.
// Note: the outer walk runs against common.SegmentIndexPath ("index_files"), not "indexes".
// We match that prefix so the walker callback actually fires and exercises the
// segIdx != nil + IsBuildIDGCBlocked branch below.
mockWalk := mockey.Mock((*storage.LocalChunkManager).WalkWithPrefix).To(func(cm *storage.LocalChunkManager, ctx context.Context, prefix string, recursive bool, fn storage.ChunkObjectWalkFunc) error {
if strings.Contains(prefix, "indexes") {
if strings.Contains(prefix, common.SegmentIndexPath) {
chunkInfo := &storage.ChunkObjectInfo{
FilePath: fmt.Sprintf("gc/indexes/%d/1/", buildID),
FilePath: fmt.Sprintf("gc/index_files/%d/1/", buildID),
ModifyTime: time.Now().Add(-time.Hour),
}
fn(chunkInfo)
@@ -2319,17 +2304,9 @@ func TestGarbageCollector_recycleUnusedIndexFiles_SnapshotReference(t *testing.T
mockGetSnapMeta := mockey.Mock(meta.GetSnapshotMeta).Return(smMeta).Build()
defer mockGetSnapMeta.UnPatch()
// Mock IsAllRefIndexLoaded (pre-fetched outside the walk loop)
mockAllLoaded := mockey.Mock((*snapshotMeta).IsAllRefIndexLoaded).Return(true).Build()
defer mockAllLoaded.UnPatch()
// Mock IsRefIndexLoadedForCollection
mockIsRefLoaded := mockey.Mock((*snapshotMeta).IsRefIndexLoadedForCollection).Return(true).Build()
defer mockIsRefLoaded.UnPatch()
// Mock GetSnapshotByBuildID to return snapshot IDs (buildID is referenced by snapshot)
mockGetSnapshotByBuildID := mockey.Mock((*snapshotMeta).GetSnapshotByBuildID).Return([]int64{1, 2}).Build()
defer mockGetSnapshotByBuildID.UnPatch()
// New O(1) API: IsBuildIDGCBlocked returns true (buildID protected by snapshot).
mockIsBuildIDBlocked := mockey.Mock((*snapshotMeta).IsBuildIDGCBlocked).Return(true).Build()
defer mockIsBuildIDBlocked.UnPatch()
removeWithPrefixCalled := false
mockRemoveWithPrefix := mockey.Mock((*storage.LocalChunkManager).RemoveWithPrefix).To(func(cm *storage.LocalChunkManager, ctx context.Context, prefix string) error {
@@ -2400,14 +2377,17 @@ func TestGarbageCollector_recycleUnusedTextIndexFiles_SnapshotReference(t *testi
}).Build()
defer mockRemove.UnPatch()
// Mock WalkWithPrefix to simulate text index files
// Mock WalkWithPrefix to simulate text index files. Production builds the prefix via
// metautil.BuildTextIndexPrefix which uses common.TextIndexPath ("text_log"), so the
// substring match MUST reference that constant — a human-readable string like
// "text_index" would silently never match and hide regressions.
mockWalk := mockey.Mock((*storage.LocalChunkManager).WalkWithPrefix).To(
func(cm *storage.LocalChunkManager, ctx context.Context, prefix string,
recursive bool, fn storage.ChunkObjectWalkFunc,
) error {
if strings.Contains(prefix, "text_index") {
if strings.Contains(prefix, common.TextIndexPath) {
chunkInfo := &storage.ChunkObjectInfo{
FilePath: "gc/text_index/401/1/100/10/1001/101/file1",
FilePath: "gc/text_log/401/1/100/10/1001/101/file1",
ModifyTime: time.Now().Add(-time.Hour),
}
fn(chunkInfo)
@@ -2416,14 +2396,9 @@ func TestGarbageCollector_recycleUnusedTextIndexFiles_SnapshotReference(t *testi
}).Build()
defer mockWalk.UnPatch()
// Mock IsRefIndexLoadedForCollection to return true (RefIndex is loaded)
mockIsRefIndexLoaded := mockey.Mock((*snapshotMeta).IsRefIndexLoadedForCollection).Return(true).Build()
defer mockIsRefIndexLoaded.UnPatch()
// Mock GetSnapshotBySegment to return snapshot IDs (segment is protected)
mockGetSnapshotBySegment := mockey.Mock((*snapshotMeta).GetSnapshotBySegment).
Return([]int64{1, 2}).Build()
defer mockGetSnapshotBySegment.UnPatch()
// New O(1) API: IsSegmentGCBlocked returns true (segment protected by snapshot).
mockIsSegBlocked := mockey.Mock((*snapshotMeta).IsSegmentGCBlocked).Return(true).Build()
defer mockIsSegBlocked.UnPatch()
// Execute
gc.recycleUnusedTextIndexFiles(ctx, nil)
@@ -2488,14 +2463,16 @@ func TestGarbageCollector_recycleUnusedJSONIndexFiles_SnapshotReference(t *testi
}).Build()
defer mockRemove.UnPatch()
// Mock WalkWithPrefix to simulate JSON index files
// Mock WalkWithPrefix to simulate JSON index files. Production builds the prefix using
// common.JSONIndexPath ("json_key_index_log"), so the substring match MUST reference
// that constant — a shorthand like "json_index" would silently never match.
mockWalk := mockey.Mock((*storage.LocalChunkManager).WalkWithPrefix).To(
func(cm *storage.LocalChunkManager, ctx context.Context, prefix string,
recursive bool, fn storage.ChunkObjectWalkFunc,
) error {
if strings.Contains(prefix, "json_index") {
if strings.Contains(prefix, common.JSONIndexPath) {
chunkInfo := &storage.ChunkObjectInfo{
FilePath: "gc/json_index/402/1/100/10/1002/102/json_file1",
FilePath: "gc/json_key_index_log/402/1/100/10/1002/102/json_file1",
ModifyTime: time.Now().Add(-time.Hour),
}
fn(chunkInfo)
@@ -2504,14 +2481,9 @@ func TestGarbageCollector_recycleUnusedJSONIndexFiles_SnapshotReference(t *testi
}).Build()
defer mockWalk.UnPatch()
// Mock IsRefIndexLoadedForCollection to return true (RefIndex is loaded)
mockIsRefIndexLoaded := mockey.Mock((*snapshotMeta).IsRefIndexLoadedForCollection).Return(true).Build()
defer mockIsRefIndexLoaded.UnPatch()
// Mock GetSnapshotBySegment to return snapshot IDs (segment is protected)
mockGetSnapshotBySegment := mockey.Mock((*snapshotMeta).GetSnapshotBySegment).
Return([]int64{1, 2}).Build()
defer mockGetSnapshotBySegment.UnPatch()
// New O(1) API: IsSegmentGCBlocked returns true (segment protected by snapshot).
mockIsSegBlocked := mockey.Mock((*snapshotMeta).IsSegmentGCBlocked).Return(true).Build()
defer mockIsSegBlocked.UnPatch()
// Execute
gc.recycleUnusedJSONIndexFiles(ctx, nil)
@@ -2522,7 +2494,8 @@ func TestGarbageCollector_recycleUnusedJSONIndexFiles_SnapshotReference(t *testi
}
// TestGarbageCollector_recycleUnusedBinlogFiles_SkipWhenRefIndexNotLoaded tests that binlog GC
// is skipped when IsRefIndexLoadedForCollection returns false (startup window).
// is skipped when the unified IsSegmentGCBlocked returns true due to an unloaded RefIndex
// (fail-closed coarse block during the startup window).
func TestGarbageCollector_recycleUnusedBinlogFiles_SkipWhenRefIndexNotLoaded(t *testing.T) {
ctx := context.Background()
@@ -2577,15 +2550,10 @@ func TestGarbageCollector_recycleUnusedBinlogFiles_SkipWhenRefIndexNotLoaded(t *
mockParseSegID := mockey.Mock(storage.ParseSegmentIDByBinlog).Return(int64(1001), nil).Build()
defer mockParseSegID.UnPatch()
// Mock IsAllRefIndexLoaded (pre-fetched before walk, used for segments not in meta)
mockAllLoaded := mockey.Mock((*snapshotMeta).IsAllRefIndexLoaded).Return(false).Build()
defer mockAllLoaded.UnPatch()
// Mock IsRefIndexLoadedForCollection to return false (per-collection check since segment is in meta)
mockCollLoaded := mockey.Mock((*snapshotMeta).IsRefIndexLoadedForCollection).Return(false).Build()
defer mockCollLoaded.UnPatch()
// Do NOT mock GetSnapshotBySegment - it should not be called
// New O(1) API: IsSegmentGCBlocked returns true because the collection's RefIndex
// is unloaded (fail-closed coarse block during startup window).
mockIsSegBlocked := mockey.Mock((*snapshotMeta).IsSegmentGCBlocked).Return(true).Build()
defer mockIsSegBlocked.UnPatch()
gc.recycleUnusedBinlogFiles(ctx)
@@ -2593,7 +2561,7 @@ func TestGarbageCollector_recycleUnusedBinlogFiles_SkipWhenRefIndexNotLoaded(t *
}
// TestGarbageCollector_recycleUnusedTextIndexFiles_SkipWhenRefIndexNotLoaded tests that text index GC
// is skipped when IsRefIndexLoadedForCollection returns false (startup window).
// is skipped when the unified IsSegmentGCBlocked returns true due to an unloaded RefIndex.
func TestGarbageCollector_recycleUnusedTextIndexFiles_SkipWhenRefIndexNotLoaded(t *testing.T) {
ctx := context.Background()
@@ -2645,13 +2613,15 @@ func TestGarbageCollector_recycleUnusedTextIndexFiles_SkipWhenRefIndexNotLoaded(
}).Build()
defer mockRemove.UnPatch()
// Substring match references common.TextIndexPath ("text_log") to stay in sync with
// production metautil.BuildTextIndexPrefix — a shorthand would silently never match.
mockWalk := mockey.Mock((*storage.LocalChunkManager).WalkWithPrefix).To(
func(cm *storage.LocalChunkManager, ctx context.Context, prefix string,
recursive bool, fn storage.ChunkObjectWalkFunc,
) error {
if strings.Contains(prefix, "text_index") {
if strings.Contains(prefix, common.TextIndexPath) {
chunkInfo := &storage.ChunkObjectInfo{
FilePath: "gc/text_index/401/1/100/10/1001/101/file1",
FilePath: "gc/text_log/401/1/100/10/1001/101/file1",
ModifyTime: time.Now().Add(-time.Hour),
}
fn(chunkInfo)
@@ -2660,11 +2630,9 @@ func TestGarbageCollector_recycleUnusedTextIndexFiles_SkipWhenRefIndexNotLoaded(
}).Build()
defer mockWalk.UnPatch()
// Mock IsRefIndexLoadedForCollection to return false (RefIndex not loaded yet)
mockIsRefIndexLoaded := mockey.Mock((*snapshotMeta).IsRefIndexLoadedForCollection).Return(false).Build()
defer mockIsRefIndexLoaded.UnPatch()
// Do NOT mock GetSnapshotBySegment - it should not be called
// New O(1) API: IsSegmentGCBlocked returns true due to unloaded RefIndex (fail-closed).
mockIsSegBlocked := mockey.Mock((*snapshotMeta).IsSegmentGCBlocked).Return(true).Build()
defer mockIsSegBlocked.UnPatch()
gc.recycleUnusedTextIndexFiles(ctx, nil)
@@ -2672,7 +2640,7 @@ func TestGarbageCollector_recycleUnusedTextIndexFiles_SkipWhenRefIndexNotLoaded(
}
// TestGarbageCollector_recycleUnusedJSONIndexFiles_SkipWhenRefIndexNotLoaded tests that JSON index GC
// is skipped when IsRefIndexLoadedForCollection returns false (startup window).
// is skipped when the unified IsSegmentGCBlocked returns true due to an unloaded RefIndex.
func TestGarbageCollector_recycleUnusedJSONIndexFiles_SkipWhenRefIndexNotLoaded(t *testing.T) {
ctx := context.Background()
@@ -2724,13 +2692,15 @@ func TestGarbageCollector_recycleUnusedJSONIndexFiles_SkipWhenRefIndexNotLoaded(
}).Build()
defer mockRemove.UnPatch()
// Substring match references common.JSONIndexPath ("json_key_index_log") to stay in
// sync with production — a shorthand like "json_index" would silently never match.
mockWalk := mockey.Mock((*storage.LocalChunkManager).WalkWithPrefix).To(
func(cm *storage.LocalChunkManager, ctx context.Context, prefix string,
recursive bool, fn storage.ChunkObjectWalkFunc,
) error {
if strings.Contains(prefix, "json_index") {
if strings.Contains(prefix, common.JSONIndexPath) {
chunkInfo := &storage.ChunkObjectInfo{
FilePath: "gc/json_index/402/1/100/10/1002/102/json_file1",
FilePath: "gc/json_key_index_log/402/1/100/10/1002/102/json_file1",
ModifyTime: time.Now().Add(-time.Hour),
}
fn(chunkInfo)
@@ -2739,11 +2709,9 @@ func TestGarbageCollector_recycleUnusedJSONIndexFiles_SkipWhenRefIndexNotLoaded(
}).Build()
defer mockWalk.UnPatch()
// Mock IsRefIndexLoadedForCollection to return false (RefIndex not loaded yet)
mockIsRefIndexLoaded := mockey.Mock((*snapshotMeta).IsRefIndexLoadedForCollection).Return(false).Build()
defer mockIsRefIndexLoaded.UnPatch()
// Do NOT mock GetSnapshotBySegment - it should not be called
// New O(1) API: IsSegmentGCBlocked returns true due to unloaded RefIndex (fail-closed).
mockIsSegBlocked := mockey.Mock((*snapshotMeta).IsSegmentGCBlocked).Return(true).Build()
defer mockIsSegBlocked.UnPatch()
gc.recycleUnusedJSONIndexFiles(ctx, nil)
@@ -2803,13 +2771,13 @@ func TestGarbageCollector_recycleUnusedIndexFiles_SegIdxNil_SnapshotProtection(t
mockGetSnapMeta := mockey.Mock(meta.GetSnapshotMeta).Return(smMeta).Build()
defer mockGetSnapMeta.UnPatch()
// Mock IsAllRefIndexLoaded to return true (all RefIndexes loaded)
mockAllLoaded := mockey.Mock((*snapshotMeta).IsAllRefIndexLoaded).Return(true).Build()
defer mockAllLoaded.UnPatch()
// Mock GetSnapshotByBuildID to return snapshot IDs (buildID is referenced by snapshot)
mockGetSnapshotByBuildID := mockey.Mock((*snapshotMeta).GetSnapshotByBuildID).Return([]int64{1}).Build()
defer mockGetSnapshotByBuildID.UnPatch()
// New O(1) API: mock IsBuildIDGCBlocked to return true for our buildID (the orphan
// branch passes collectionID = -1 since it has no segIdx context).
mockIsBuildIDBlocked := mockey.Mock((*snapshotMeta).IsBuildIDGCBlocked).To(
func(sm *snapshotMeta, collID, bid int64) bool {
return bid == buildID
}).Build()
defer mockIsBuildIDBlocked.UnPatch()
removeWithPrefixCalled := false
mockRemoveWithPrefix := mockey.Mock((*storage.LocalChunkManager).RemoveWithPrefix).To(
@@ -2878,13 +2846,12 @@ func TestGarbageCollector_recycleUnusedBinlogFiles_SegmentNil_SnapshotProtection
mockParseSegID := mockey.Mock(storage.ParseSegmentIDByBinlog).Return(int64(1001), nil).Build()
defer mockParseSegID.UnPatch()
// Mock IsAllRefIndexLoaded to return true (all RefIndexes loaded)
mockAllLoaded := mockey.Mock((*snapshotMeta).IsAllRefIndexLoaded).Return(true).Build()
defer mockAllLoaded.UnPatch()
// Mock GetSnapshotBySegment to return snapshot IDs (segment is protected)
mockGetSnapshotBySegment := mockey.Mock((*snapshotMeta).GetSnapshotBySegment).Return([]int64{1, 2}).Build()
defer mockGetSnapshotBySegment.UnPatch()
// New O(1) API: IsSegmentGCBlocked returns true so the file is kept.
// The caller passes collectionID = -1 (segment is nil in this test), which exercises
// the "global fail-closed" branch of the new API — but here we short-circuit to true
// directly to simulate a snapshot referencing segment 1001.
mockIsSegBlocked := mockey.Mock((*snapshotMeta).IsSegmentGCBlocked).Return(true).Build()
defer mockIsSegBlocked.UnPatch()
gc.recycleUnusedBinlogFiles(ctx)
@@ -2960,14 +2927,9 @@ func TestGarbageCollector_recycleUnusedJSONStatsFiles_SnapshotReference(t *testi
}).Build()
defer mockWalk.UnPatch()
// Mock IsRefIndexLoadedForCollection to return true
mockIsRefIndexLoaded := mockey.Mock((*snapshotMeta).IsRefIndexLoadedForCollection).Return(true).Build()
defer mockIsRefIndexLoaded.UnPatch()
// Mock GetSnapshotBySegment to return snapshot IDs (segment is protected)
mockGetSnapshotBySegment := mockey.Mock((*snapshotMeta).GetSnapshotBySegment).
Return([]int64{1, 2}).Build()
defer mockGetSnapshotBySegment.UnPatch()
// New O(1) API: IsSegmentGCBlocked returns true for the referenced segment.
mockIsSegBlocked := mockey.Mock((*snapshotMeta).IsSegmentGCBlocked).Return(true).Build()
defer mockIsSegBlocked.UnPatch()
gc.recycleUnusedJSONStatsFiles(ctx, nil)
@@ -3093,10 +3055,9 @@ func TestGarbageCollector_recycleDroppedSegments_V3(t *testing.T) {
removeObjectFilesCalled := false
droppedSegmentIDs := []int64{}
mockIsRefIndexLoaded := mockey.Mock((*snapshotMeta).IsRefIndexLoadedForCollection).Return(true).Build()
defer mockIsRefIndexLoaded.UnPatch()
mockGetSnapshotBySegment := mockey.Mock((*snapshotMeta).GetSnapshotBySegment).Return([]int64{}).Build()
defer mockGetSnapshotBySegment.UnPatch()
// Snapshot layer transparent: no segment is blocked.
mockIsSegBlocked := mockey.Mock((*snapshotMeta).IsSegmentGCBlocked).Return(false).Build()
defer mockIsSegBlocked.UnPatch()
mockListLoaded := mockey.Mock((*ServerHandler).ListLoadedSegments).Return([]int64{}, nil).Build()
defer mockListLoaded.UnPatch()
mockChannelExists := mockey.Mock((*datacoord.Catalog).ChannelExists).Return(true).Build()
@@ -3178,13 +3139,9 @@ func TestGarbageCollector_recycleUnusedBinlogFiles_SkipV3(t *testing.T) {
removedFiles := []string{}
// Mock snapshot methods to avoid nil pointer on snapshotMeta internals
mockAllLoaded := mockey.Mock((*snapshotMeta).IsAllRefIndexLoaded).Return(true).Build()
defer mockAllLoaded.UnPatch()
mockCollLoaded := mockey.Mock((*snapshotMeta).IsRefIndexLoadedForCollection).Return(true).Build()
defer mockCollLoaded.UnPatch()
mockGetSnapshotBySegment := mockey.Mock((*snapshotMeta).GetSnapshotBySegment).Return([]int64{}).Build()
defer mockGetSnapshotBySegment.UnPatch()
// Snapshot layer transparent: no segment is blocked.
mockIsSegBlocked := mockey.Mock((*snapshotMeta).IsSegmentGCBlocked).Return(false).Build()
defer mockIsSegBlocked.UnPatch()
// Mock WalkWithPrefix to return V3 files under insert_log
mockWalk := mockey.Mock((*storage.LocalChunkManager).WalkWithPrefix).To(
internal/datacoord/meta.go
+60 -4
View File
@@ -2030,6 +2030,43 @@ func (m *meta) ValidateSegmentStateBeforeCompleteCompactionMutation(t *datapb.Co
m.segMu.RLock()
defer m.segMu.RUnlock()
// Snapshot compaction protection exists to keep the sealed-segment list stable during
// backfill — if an L1/L2 segment gets merged away mid-backfill, the backfill breaks.
// L0 segments are transient delete-log carriers, not part of that stable list, and
// L0 compaction only appends deltalogs to L1/L2 targets without touching L1/L2 binlogs.
// So L0 delete compaction is outside the protection's concern and must not be blocked.
if t.GetType() != datapb.CompactionType_Level0DeleteCompaction {
// Check if compaction is blocked for this collection (snapshot pending or RefIndex not loaded).
if m.isCollectionCompactionBlocked(t.GetCollectionID()) {
log.Info("compaction rejected: collection has pending snapshot or unloaded RefIndex",
zap.Int64("planID", t.GetPlanID()),
zap.String("type", t.GetType().String()),
zap.Int64("collectionID", t.GetCollectionID()),
zap.String("channel", t.GetChannel()),
zap.Int64s("inputSegments", t.GetInputSegments()),
)
return merr.WrapErrCompactionBlocked(
fmt.Sprintf("collection %d has pending snapshot or unloaded snapshot RefIndex",
t.GetCollectionID()))
}
// Check if any input segment is protected by a snapshot.
for _, segmentID := range t.GetInputSegments() {
if m.isSegmentCompactionProtected(segmentID) {
log.Info("compaction rejected: input segment is protected by snapshot",
zap.Int64("planID", t.GetPlanID()),
zap.String("type", t.GetType().String()),
zap.Int64("collectionID", t.GetCollectionID()),
zap.String("channel", t.GetChannel()),
zap.Int64("segmentID", segmentID),
zap.Int64s("inputSegments", t.GetInputSegments()),
)
return merr.WrapErrCompactionBlocked(
fmt.Sprintf("input segment %d is protected by a snapshot", segmentID))
}
}
}
for _, segmentID := range t.GetInputSegments() {
segment := m.segments.GetSegment(segmentID)
if !isSegmentHealthy(segment) {
@@ -2225,6 +2262,29 @@ func (m *meta) GcConfirm(ctx context.Context, collectionID, partitionID UniqueID
return m.catalog.GcConfirm(ctx, collectionID, partitionID)
}
// isSegmentCompactionProtected checks if a segment is protected from compaction by a snapshot.
func (m *meta) isSegmentCompactionProtected(segmentID int64) bool {
if m.snapshotMeta == nil {
return false
}
return m.snapshotMeta.IsSegmentCompactionProtected(segmentID)
}
// isCollectionCompactionBlocked checks if compaction is blocked for a collection
// because a protected snapshot's RefIndex hasn't been loaded yet (fail-closed).
func (m *meta) isCollectionCompactionBlocked(collectionID int64) bool {
if m.snapshotMeta == nil {
return false
}
return m.snapshotMeta.IsCollectionCompactionBlocked(collectionID)
}
// GetCompactableSegmentGroupByCollection returns sealed segments grouped by collection.
// This is consumed exclusively by the L0 compaction policy, which only acts on L0 segments.
// Snapshot compaction protection targets L1/L2 segments referenced by snapshots, so it must
// NOT filter segments here: doing so would prevent L0 delete-log compaction and cause
// delta log accumulation, query latency spikes, and write stalls on collections with
// active snapshots.
func (m *meta) GetCompactableSegmentGroupByCollection() map[int64][]*SegmentInfo {
allSegs := m.SelectSegments(m.ctx, SegmentFilterFunc(func(segment *SegmentInfo) bool {
return isSegmentHealthy(segment) &&
@@ -2235,10 +2295,6 @@ func (m *meta) GetCompactableSegmentGroupByCollection() map[int64][]*SegmentInfo
ret := make(map[int64][]*SegmentInfo)
for _, seg := range allSegs {
if _, ok := ret[seg.CollectionID]; !ok {
ret[seg.CollectionID] = make([]*SegmentInfo, 0)
}
ret[seg.CollectionID] = append(ret[seg.CollectionID], seg)
}
internal/datacoord/meta_test.go
+185
View File
@@ -22,6 +22,7 @@ import (
"math/rand"
"sync/atomic"
"testing"
"time"
"github.com/cockroachdb/errors"
"github.com/samber/lo"
@@ -683,6 +684,190 @@ func (suite *MetaBasicSuite) TestCompleteCompactionMutation() {
})
}
func (suite *MetaBasicSuite) TestValidateSegmentState_BlockedBySnapshot() {
latestSegments := NewSegmentsInfo()
for segID, segment := range map[UniqueID]*SegmentInfo{
1: {SegmentInfo: &datapb.SegmentInfo{
ID: 1,
CollectionID: 100,
PartitionID: 10,
State: commonpb.SegmentState_Flushed,
}},
} {
latestSegments.SetSegment(segID, segment)
}
task := &datapb.CompactionTask{
PlanID: 999,
InputSegments: []UniqueID{1},
CollectionID: 100,
Type: datapb.CompactionType_MixCompaction,
}
suite.Run("rejected by snapshot pending collection", func() {
sm := createTestSnapshotMetaLoaded(suite.T())
sm.SetSnapshotPending(100)
m := &meta{
segments: latestSegments,
snapshotMeta: sm,
}
err := m.ValidateSegmentStateBeforeCompleteCompactionMutation(task)
suite.Error(err)
suite.Contains(err.Error(), "compaction blocked")
})
suite.Run("rejected by segment protection", func() {
sm := createTestSnapshotMetaLoaded(suite.T())
futureTs := uint64(time.Now().Unix()) + 3600
sm.segmentProtectionMu.Lock()
sm.segmentProtectionUntil[1] = futureTs
sm.segmentProtectionMu.Unlock()
m := &meta{
segments: latestSegments,
snapshotMeta: sm,
}
err := m.ValidateSegmentStateBeforeCompleteCompactionMutation(task)
suite.Error(err)
suite.Contains(err.Error(), "segment 1")
})
suite.Run("passes when no snapshot", func() {
sm := createTestSnapshotMetaLoaded(suite.T())
m := &meta{
segments: latestSegments,
snapshotMeta: sm,
}
err := m.ValidateSegmentStateBeforeCompleteCompactionMutation(task)
suite.NoError(err)
})
suite.Run("passes when snapshotMeta is nil", func() {
m := &meta{
segments: latestSegments,
}
err := m.ValidateSegmentStateBeforeCompleteCompactionMutation(task)
suite.NoError(err)
})
suite.Run("rejected when only middle segment is protected in multi-segment task", func() {
multiSegments := NewSegmentsInfo()
for segID, segment := range map[UniqueID]*SegmentInfo{
1: {SegmentInfo: &datapb.SegmentInfo{ID: 1, CollectionID: 100, PartitionID: 10, State: commonpb.SegmentState_Flushed}},
2: {SegmentInfo: &datapb.SegmentInfo{ID: 2, CollectionID: 100, PartitionID: 10, State: commonpb.SegmentState_Flushed}},
3: {SegmentInfo: &datapb.SegmentInfo{ID: 3, CollectionID: 100, PartitionID: 10, State: commonpb.SegmentState_Flushed}},
} {
multiSegments.SetSegment(segID, segment)
}
multiTask := &datapb.CompactionTask{
PlanID: 998,
InputSegments: []UniqueID{1, 2, 3},
CollectionID: 100,
Type: datapb.CompactionType_MixCompaction,
}
sm := createTestSnapshotMetaLoaded(suite.T())
futureTs := uint64(time.Now().Unix()) + 3600
// Only protect segment 2
sm.segmentProtectionMu.Lock()
sm.segmentProtectionUntil[2] = futureTs
sm.segmentProtectionMu.Unlock()
m := &meta{
segments: multiSegments,
snapshotMeta: sm,
}
err := m.ValidateSegmentStateBeforeCompleteCompactionMutation(multiTask)
suite.Error(err)
suite.Contains(err.Error(), "segment 2")
})
suite.Run("passes when protection expired", func() {
sm := createTestSnapshotMetaLoaded(suite.T())
pastTs := uint64(time.Now().Unix()) - 100
sm.segmentProtectionMu.Lock()
sm.segmentProtectionUntil[1] = pastTs
sm.segmentProtectionMu.Unlock()
m := &meta{
segments: latestSegments,
snapshotMeta: sm,
}
err := m.ValidateSegmentStateBeforeCompleteCompactionMutation(task)
suite.NoError(err)
})
// Regression: L0 delete compaction must bypass snapshot protection checks entirely.
// Snapshots only reference sealed L1/L2 segments; blocking L0 here would cause
// delta log accumulation, query latency spikes, and write stalls.
suite.Run("L0 compaction passes even when collection is snapshot-pending", func() {
sm := createTestSnapshotMetaLoaded(suite.T())
sm.SetSnapshotPending(100)
m := &meta{
segments: latestSegments,
snapshotMeta: sm,
}
l0Task := &datapb.CompactionTask{
PlanID: 8001,
InputSegments: []UniqueID{1},
CollectionID: 100,
Type: datapb.CompactionType_Level0DeleteCompaction,
}
err := m.ValidateSegmentStateBeforeCompleteCompactionMutation(l0Task)
suite.NoError(err)
})
suite.Run("L0 compaction passes even when input segment is snapshot-protected", func() {
sm := createTestSnapshotMetaLoaded(suite.T())
futureTs := uint64(time.Now().Unix()) + 3600
sm.segmentProtectionMu.Lock()
sm.segmentProtectionUntil[1] = futureTs
sm.segmentProtectionMu.Unlock()
m := &meta{
segments: latestSegments,
snapshotMeta: sm,
}
l0Task := &datapb.CompactionTask{
PlanID: 8002,
InputSegments: []UniqueID{1},
CollectionID: 100,
Type: datapb.CompactionType_Level0DeleteCompaction,
}
err := m.ValidateSegmentStateBeforeCompleteCompactionMutation(l0Task)
suite.NoError(err)
})
// Regression: collection-level block should produce ErrCompactionBlocked,
// not ErrServiceInternal, so SRE alerting does not treat it as a P0 fault.
suite.Run("rejection uses ErrCompactionBlocked, not ErrServiceInternal", func() {
sm := createTestSnapshotMetaLoaded(suite.T())
sm.SetSnapshotPending(100)
m := &meta{
segments: latestSegments,
snapshotMeta: sm,
}
err := m.ValidateSegmentStateBeforeCompleteCompactionMutation(task)
suite.Error(err)
suite.True(errors.Is(err, merr.ErrCompactionBlocked))
suite.False(errors.Is(err, merr.ErrServiceInternal))
})
}
func (suite *MetaBasicSuite) TestSetSegment() {
meta := suite.meta
catalog := mocks2.NewDataCoordCatalog(suite.T())
internal/datacoord/services.go
+18 -4
View File
@@ -2075,7 +2075,20 @@ func (s *Server) CreateSnapshot(ctx context.Context, req *datapb.CreateSnapshotR
}
log.Info("receive CreateSnapshot request", zap.String("name", req.GetName()),
zap.String("description", req.GetDescription()))
zap.String("description", req.GetDescription()),
zap.Int64("compactionProtectionSeconds", req.GetCompactionProtectionSeconds()))
// Defense-in-depth: re-validate compaction_protection_seconds on the DataCoord side.
// Proxy also validates this, but a buggy or malicious client could bypass Proxy by
// calling this RPC directly. Range validation must be enforced by the owner of the feature.
if req.GetCompactionProtectionSeconds() < 0 {
return merr.Status(merr.WrapErrParameterInvalidMsg("compaction_protection_seconds must be non-negative")), nil
}
maxCompactionProtectionSeconds := paramtable.Get().DataCoordCfg.SnapshotMaxCompactionProtectionSeconds.GetAsInt64()
if req.GetCompactionProtectionSeconds() > maxCompactionProtectionSeconds {
return merr.Status(merr.WrapErrParameterInvalidMsg(
fmt.Sprintf("compaction_protection_seconds must not exceed %d", maxCompactionProtectionSeconds))), nil
}
// Check if snapshot name already exists
if _, err := s.snapshotManager.GetSnapshot(ctx, req.GetName()); err == nil {
@@ -2112,9 +2125,10 @@ func (s *Server) CreateSnapshot(ctx context.Context, req *datapb.CreateSnapshotR
// Snapshot ID is allocated in the callback
if _, err := broadcaster.Broadcast(ctx, message.NewCreateSnapshotMessageBuilderV2().
WithHeader(&message.CreateSnapshotMessageHeader{
CollectionId: req.GetCollectionId(),
Name: req.GetName(),
Description: req.GetDescription(),
CollectionId: req.GetCollectionId(),
Name: req.GetName(),
Description: req.GetDescription(),
CompactionProtectionSeconds: req.GetCompactionProtectionSeconds(),
}).
WithBody(&message.CreateSnapshotMessageBody{}).
WithBroadcast([]string{streaming.WAL().ControlChannel()}).
internal/datacoord/services_test.go
+41
View File
@@ -2914,6 +2914,47 @@ func TestServer_CreateSnapshot_AdditionalCases(t *testing.T) {
assert.True(t, errors.Is(merr.Error(resp), merr.ErrParameterInvalid))
assert.Contains(t, resp.GetReason(), "already exists")
})
// Regression for the defense-in-depth validation added in Server.CreateSnapshot.
// Proxy already validates this range, but a misbehaving client could bypass Proxy
// by calling DataCoord directly, so the owner of the feature must re-check.
t.Run("compaction_protection_seconds_negative", func(t *testing.T) {
ctx := context.Background()
server := &Server{}
server.stateCode.Store(commonpb.StateCode_Healthy)
resp, err := server.CreateSnapshot(ctx, &datapb.CreateSnapshotRequest{
Name: "bad_request",
CollectionId: 100,
CompactionProtectionSeconds: -1,
})
assert.NoError(t, err)
assert.Error(t, merr.Error(resp))
assert.True(t, errors.Is(merr.Error(resp), merr.ErrParameterInvalid))
assert.Contains(t, resp.GetReason(), "non-negative")
})
t.Run("compaction_protection_seconds_over_max", func(t *testing.T) {
ctx := context.Background()
server := &Server{}
server.stateCode.Store(commonpb.StateCode_Healthy)
maxSec := paramtable.Get().DataCoordCfg.SnapshotMaxCompactionProtectionSeconds.GetAsInt64()
resp, err := server.CreateSnapshot(ctx, &datapb.CreateSnapshotRequest{
Name: "bad_request",
CollectionId: 100,
CompactionProtectionSeconds: maxSec + 1,
})
assert.NoError(t, err)
assert.Error(t, merr.Error(resp))
assert.True(t, errors.Is(merr.Error(resp), merr.ErrParameterInvalid))
assert.Contains(t, resp.GetReason(), "must not exceed")
})
}
// --- Test BatchUpdateManifest ---
internal/datacoord/snapshot_manager.go
+19 -2
View File
@@ -99,7 +99,7 @@ type SnapshotManager interface {
// Returns:
// - snapshotID: Allocated snapshot ID (0 on error)
// - error: If name already exists, allocation fails, or save fails
CreateSnapshot(ctx context.Context, collectionID int64, name, description string) (int64, error)
CreateSnapshot(ctx context.Context, collectionID int64, name, description string, compactionProtectionSeconds int64) (int64, error)
// DropSnapshot deletes an existing snapshot by name.
// It removes the snapshot from memory cache, etcd, and S3 storage.
@@ -344,19 +344,31 @@ func (sm *snapshotManager) CreateSnapshot(
ctx context.Context,
collectionID int64,
name, description string,
compactionProtectionSeconds int64,
) (int64, error) {
// Lock to prevent TOCTOU race on snapshot name uniqueness check
sm.createSnapshotMu.Lock()
defer sm.createSnapshotMu.Unlock()
log := log.Ctx(ctx).With(zap.Int64("collectionID", collectionID), zap.String("name", name))
log.Info("create snapshot request received", zap.String("description", description))
log.Info("create snapshot request received",
zap.String("description", description),
zap.Int64("compactionProtectionSeconds", compactionProtectionSeconds))
// Validate snapshot name uniqueness (protected by createSnapshotMu)
if _, err := sm.snapshotMeta.GetSnapshot(ctx, name); err == nil {
return 0, merr.WrapErrParameterInvalidMsg("snapshot name %s already exists", name)
}
// Block compaction commit for this collection during snapshot creation.
// This MUST be unconditional (not gated on compactionProtectionSeconds): even when
// the user requests zero long-term protection, the snapshot must still be atomic
// within the GenSnapshot → SaveSnapshot window, otherwise concurrent compaction
// could drop segments that the in-flight snapshot is about to reference, leaving
// the freshly-created snapshot immediately broken.
sm.snapshotMeta.SetSnapshotPending(collectionID)
defer sm.snapshotMeta.ClearSnapshotPending(collectionID)
// Allocate snapshot ID
snapshotID, err := sm.allocator.AllocID(ctx)
if err != nil {
@@ -376,6 +388,11 @@ func (sm *snapshotManager) CreateSnapshot(
snapshotData.SnapshotInfo.Name = name
snapshotData.SnapshotInfo.Description = description
// Set compaction protection if requested
if compactionProtectionSeconds > 0 {
snapshotData.SnapshotInfo.CompactionExpireTime = uint64(time.Now().Unix()) + uint64(compactionProtectionSeconds)
}
// Save to storage
if err := sm.snapshotMeta.SaveSnapshot(ctx, snapshotData); err != nil {
log.Error("failed to save snapshot", zap.Error(err))
internal/datacoord/snapshot_manager_test.go
+231 -12
View File
@@ -79,11 +79,13 @@ func TestSnapshotManager_CreateSnapshot_Success(t *testing.T) {
}).Build()
defer mockSaveSnapshot.UnPatch()
// Create snapshot manager
// Create snapshot manager. We need a properly-initialized snapshotMeta so that
// the unconditional SetSnapshotPending / ClearSnapshotPending calls (required for
// GenSnapshot → SaveSnapshot atomicity) don't panic on uninitialized maps.
sm := NewSnapshotManager(
nil, // meta
&snapshotMeta{}, // snapshotMeta
nil, // copySegmentMeta
nil, // meta
createTestSnapshotMetaLoaded(t), // snapshotMeta
nil, // copySegmentMeta
mockAllocator,
mockHandler,
nil, // broker
@@ -91,13 +93,64 @@ func TestSnapshotManager_CreateSnapshot_Success(t *testing.T) {
)
// Execute
snapshotID, err := sm.CreateSnapshot(ctx, 100, "test_snapshot", "test description")
snapshotID, err := sm.CreateSnapshot(ctx, 100, "test_snapshot", "test description", 0)
// Verify
assert.NoError(t, err)
assert.Equal(t, int64(1001), snapshotID)
}
func TestSnapshotManager_CreateSnapshot_WithCompactionProtection(t *testing.T) {
ctx := context.Background()
// Setup mocks
mockAllocator := allocator.NewMockAllocator(t)
mockHandler := NewNMockHandler(t)
mockAllocator.EXPECT().AllocID(mock.Anything).Return(int64(2001), nil).Once()
snapshotData := &SnapshotData{
SnapshotInfo: &datapb.SnapshotInfo{
CollectionId: 100,
},
Segments: []*datapb.SegmentDescription{
{SegmentId: 1, NumOfRows: 100},
},
}
mockHandler.EXPECT().GenSnapshot(mock.Anything, int64(100)).Return(snapshotData, nil).Once()
mockGetSnapshot := mockey.Mock((*snapshotMeta).GetSnapshot).To(func(sm *snapshotMeta, ctx context.Context, name string) (*datapb.SnapshotInfo, error) {
return nil, errors.New("not found")
}).Build()
defer mockGetSnapshot.UnPatch()
mockSaveSnapshot := mockey.Mock((*snapshotMeta).SaveSnapshot).To(func(sm *snapshotMeta, ctx context.Context, data *SnapshotData) error {
// Verify compaction expire time is set
assert.True(t, data.SnapshotInfo.CompactionExpireTime > 0)
return nil
}).Build()
defer mockSaveSnapshot.UnPatch()
snapshotMetaInstance := createTestSnapshotMetaLoaded(t)
sm := NewSnapshotManager(
nil,
snapshotMetaInstance,
nil,
mockAllocator,
mockHandler,
nil,
nil,
)
snapshotID, err := sm.CreateSnapshot(ctx, 100, "protected_snap", "with protection", 3600)
// Verify snapshot pending intent is cleared after CreateSnapshot completes
assert.False(t, snapshotMetaInstance.IsCollectionCompactionBlocked(100))
assert.NoError(t, err)
assert.Equal(t, int64(2001), snapshotID)
}
func TestSnapshotManager_CreateSnapshot_DuplicateName(t *testing.T) {
ctx := context.Background()
@@ -118,7 +171,7 @@ func TestSnapshotManager_CreateSnapshot_DuplicateName(t *testing.T) {
)
// Execute
snapshotID, err := sm.CreateSnapshot(ctx, 100, "existing_snapshot", "description")
snapshotID, err := sm.CreateSnapshot(ctx, 100, "existing_snapshot", "description", 0)
// Verify
assert.Error(t, err)
@@ -143,7 +196,7 @@ func TestSnapshotManager_CreateSnapshot_AllocatorError(t *testing.T) {
sm := NewSnapshotManager(
nil,
&snapshotMeta{},
createTestSnapshotMetaLoaded(t),
nil,
mockAllocator,
nil,
@@ -152,7 +205,7 @@ func TestSnapshotManager_CreateSnapshot_AllocatorError(t *testing.T) {
)
// Execute
snapshotID, err := sm.CreateSnapshot(ctx, 100, "test_snapshot", "description")
snapshotID, err := sm.CreateSnapshot(ctx, 100, "test_snapshot", "description", 0)
// Verify
assert.Error(t, err)
@@ -180,7 +233,7 @@ func TestSnapshotManager_CreateSnapshot_GenSnapshotError(t *testing.T) {
sm := NewSnapshotManager(
nil,
&snapshotMeta{},
createTestSnapshotMetaLoaded(t),
nil,
mockAllocator,
mockHandler,
@@ -189,7 +242,7 @@ func TestSnapshotManager_CreateSnapshot_GenSnapshotError(t *testing.T) {
)
// Execute
snapshotID, err := sm.CreateSnapshot(ctx, 100, "test_snapshot", "description")
snapshotID, err := sm.CreateSnapshot(ctx, 100, "test_snapshot", "description", 0)
// Verify
assert.Error(t, err)
@@ -225,7 +278,7 @@ func TestSnapshotManager_CreateSnapshot_SaveError(t *testing.T) {
sm := NewSnapshotManager(
nil,
&snapshotMeta{},
createTestSnapshotMetaLoaded(t),
nil,
mockAllocator,
mockHandler,
@@ -234,7 +287,7 @@ func TestSnapshotManager_CreateSnapshot_SaveError(t *testing.T) {
)
// Execute
snapshotID, err := sm.CreateSnapshot(ctx, 100, "test_snapshot", "description")
snapshotID, err := sm.CreateSnapshot(ctx, 100, "test_snapshot", "description", 0)
// Verify
assert.Error(t, err)
@@ -242,6 +295,172 @@ func TestSnapshotManager_CreateSnapshot_SaveError(t *testing.T) {
assert.Equal(t, expectedErr, err)
}
func TestSnapshotManager_CreateSnapshot_ClearsSnapshotPendingOnGenSnapshotError(t *testing.T) {
ctx := context.Background()
mockAllocator := allocator.NewMockAllocator(t)
mockHandler := NewNMockHandler(t)
mockAllocator.EXPECT().AllocID(mock.Anything).Return(int64(1001), nil).Once()
expectedErr := errors.New("gen snapshot error")
mockHandler.EXPECT().GenSnapshot(mock.Anything, int64(100)).Return(nil, expectedErr).Once()
mockGetSnapshot := mockey.Mock((*snapshotMeta).GetSnapshot).To(func(sm *snapshotMeta, ctx context.Context, name string) (*datapb.SnapshotInfo, error) {
return nil, errors.New("not found")
}).Build()
defer mockGetSnapshot.UnPatch()
snapshotMetaInstance := createTestSnapshotMetaLoaded(t)
sm := NewSnapshotManager(
nil,
snapshotMetaInstance,
nil,
mockAllocator,
mockHandler,
nil,
nil,
)
_, err := sm.CreateSnapshot(ctx, 100, "test_snap", "desc", 3600)
assert.Error(t, err)
// Verify snapshot pending intent is cleared even on error
assert.False(t, snapshotMetaInstance.IsCollectionCompactionBlocked(100))
}
func TestSnapshotManager_CreateSnapshot_ClearsSnapshotPendingOnSaveError(t *testing.T) {
ctx := context.Background()
mockAllocator := allocator.NewMockAllocator(t)
mockHandler := NewNMockHandler(t)
mockAllocator.EXPECT().AllocID(mock.Anything).Return(int64(1001), nil).Once()
snapshotData := &SnapshotData{
SnapshotInfo: &datapb.SnapshotInfo{CollectionId: 100},
}
mockHandler.EXPECT().GenSnapshot(mock.Anything, int64(100)).Return(snapshotData, nil).Once()
mockGetSnapshot := mockey.Mock((*snapshotMeta).GetSnapshot).To(func(sm *snapshotMeta, ctx context.Context, name string) (*datapb.SnapshotInfo, error) {
return nil, errors.New("not found")
}).Build()
defer mockGetSnapshot.UnPatch()
expectedErr := errors.New("save error")
mockSaveSnapshot := mockey.Mock((*snapshotMeta).SaveSnapshot).To(func(sm *snapshotMeta, ctx context.Context, data *SnapshotData) error {
// While inside SaveSnapshot, snapshot pending should be active
assert.True(t, sm.IsCollectionCompactionBlocked(data.SnapshotInfo.GetCollectionId()))
return expectedErr
}).Build()
defer mockSaveSnapshot.UnPatch()
snapshotMetaInstance := createTestSnapshotMetaLoaded(t)
sm := NewSnapshotManager(
nil,
snapshotMetaInstance,
nil,
mockAllocator,
mockHandler,
nil,
nil,
)
_, err := sm.CreateSnapshot(ctx, 100, "test_snap", "desc", 3600)
assert.Error(t, err)
// Verify snapshot pending intent is cleared after save failure
assert.False(t, snapshotMetaInstance.IsCollectionCompactionBlocked(100))
}
// Regression for PR #48227 review comment #4: even when the user requests zero
// long-term compaction protection, CreateSnapshot must hold SetSnapshotPending
// across the GenSnapshot → SaveSnapshot window. Otherwise concurrent compaction
// could drop segments that the in-flight snapshot is about to reference, leaving
// the snapshot immediately broken. Before the fix the SetSnapshotPending call was
// gated on compactionProtectionSeconds > 0.
func TestSnapshotManager_CreateSnapshot_PendingHeldEvenWithoutLongTermProtection(t *testing.T) {
ctx := context.Background()
mockAllocator := allocator.NewMockAllocator(t)
mockHandler := NewNMockHandler(t)
mockAllocator.EXPECT().AllocID(mock.Anything).Return(int64(1001), nil).Once()
snapshotData := &SnapshotData{
SnapshotInfo: &datapb.SnapshotInfo{CollectionId: 100},
Segments: []*datapb.SegmentDescription{{SegmentId: 1}},
}
mockHandler.EXPECT().GenSnapshot(mock.Anything, int64(100)).Return(snapshotData, nil).Once()
mockGetSnapshot := mockey.Mock((*snapshotMeta).GetSnapshot).To(func(sm *snapshotMeta, ctx context.Context, name string) (*datapb.SnapshotInfo, error) {
return nil, errors.New("not found")
}).Build()
defer mockGetSnapshot.UnPatch()
snapshotMetaInstance := createTestSnapshotMetaLoaded(t)
// While SaveSnapshot is in flight, the collection MUST be marked as blocked so
// concurrent compaction commits see the TOCTOU guard and back off. We observe
// this by intercepting SaveSnapshot and asserting the block is visible at that point.
mockSaveSnapshot := mockey.Mock((*snapshotMeta).SaveSnapshot).To(func(sm *snapshotMeta, ctx context.Context, data *SnapshotData) error {
assert.True(t, sm.IsCollectionCompactionBlocked(data.SnapshotInfo.GetCollectionId()),
"collection must be blocked during SaveSnapshot even with protection=0")
return nil
}).Build()
defer mockSaveSnapshot.UnPatch()
sm := NewSnapshotManager(
nil,
snapshotMetaInstance,
nil,
mockAllocator,
mockHandler,
nil,
nil,
)
_, err := sm.CreateSnapshot(ctx, 100, "test_snap", "desc", 0) // compactionProtectionSeconds = 0
assert.NoError(t, err)
// After CreateSnapshot returns, the deferred ClearSnapshotPending must have run.
assert.False(t, snapshotMetaInstance.IsCollectionCompactionBlocked(100),
"block must be released once CreateSnapshot finishes")
}
func TestSnapshotManager_CreateSnapshot_ClearsSnapshotPendingOnAllocError(t *testing.T) {
ctx := context.Background()
mockAllocator := allocator.NewMockAllocator(t)
expectedErr := errors.New("alloc error")
mockAllocator.EXPECT().AllocID(mock.Anything).Return(int64(0), expectedErr).Once()
mockGetSnapshot := mockey.Mock((*snapshotMeta).GetSnapshot).To(func(sm *snapshotMeta, ctx context.Context, name string) (*datapb.SnapshotInfo, error) {
return nil, errors.New("not found")
}).Build()
defer mockGetSnapshot.UnPatch()
snapshotMetaInstance := createTestSnapshotMetaLoaded(t)
sm := NewSnapshotManager(
nil,
snapshotMetaInstance,
nil,
mockAllocator,
nil,
nil,
nil,
)
_, err := sm.CreateSnapshot(ctx, 100, "test_snap", "desc", 3600)
assert.Error(t, err)
// Verify snapshot pending intent is cleared after alloc failure
assert.False(t, snapshotMetaInstance.IsCollectionCompactionBlocked(100))
}
// --- Test DropSnapshot ---
func TestSnapshotManager_DropSnapshot_Success(t *testing.T) {
internal/datacoord/snapshot_meta.go
+428 -124
View File
@@ -67,8 +67,8 @@ const (
// SnapshotRefIndex holds segment and build IDs loaded from S3.
// Loading state is managed per-snapshot to support retry on failure.
//
// CONCURRENCY: Protected by RWMutex since SetLoaded (async loader) and
// ContainsSegment/ContainsBuildID (GC) may run concurrently.
// CONCURRENCY: Protected by RWMutex since the async loader writes via SetLoaded/SetFailed
// while snapshotMeta.rebuildAllSegmentProtection reads via GetSegmentIDs/GetBuildIDs.
type SnapshotRefIndex struct {
mu sync.RWMutex
loadState RefIndexLoadState
@@ -140,6 +140,27 @@ func (r *SnapshotRefIndex) IsFailed() bool {
return r.loadState == RefIndexStateFailed
}
// GetSegmentIDs returns a copy of segment IDs referenced by this snapshot.
func (r *SnapshotRefIndex) GetSegmentIDs() []int64 {
r.mu.RLock()
defer r.mu.RUnlock()
if r.segmentIDs == nil {
return nil
}
return r.segmentIDs.Collect()
}
// GetBuildIDs returns a copy of the index buildIDs referenced by this snapshot.
// Used by rebuildAllSegmentProtection to populate buildIDReferencedByGC.
func (r *SnapshotRefIndex) GetBuildIDs() []int64 {
r.mu.RLock()
defer r.mu.RUnlock()
if r.buildIDs == nil {
return nil
}
return r.buildIDs.Collect()
}
// snapshotMeta manages snapshot metadata both in memory and persistent storage.
//
// This is the central coordinator for snapshot operations, maintaining:
@@ -168,6 +189,49 @@ type snapshotMeta struct {
// Uses RWMutex to allow concurrent reads (ListSnapshots) while blocking writes.
collectionIndexMu sync.RWMutex
// Protection state, maintained by rebuildAllSegmentProtection under segmentProtectionMu.
//
// Compaction and GC share the same fail-closed → converge pattern (coarse collection-level
// block at startup, progressively narrowed to precise segment/buildID sets as the loader
// goroutine reads RefIndexes from S3). They differ in TTL semantics:
//
// - Compaction protection is TTL-bound: only snapshots whose CompactionExpireTime is
// still in the future contribute. Once TTL expires, the segment is eligible for
// compaction again even if the snapshot still exists.
//
// - GC protection is unconditional: any existing snapshot's referenced segments and
// buildIDs must not be deleted, because the snapshot needs them for PIT recovery
// regardless of CompactionExpireTime.
//
// These are therefore maintained as two independent sets of precise state under the
// same lock, rebuilt atomically by rebuildAllSegmentProtection.
segmentProtectionMu sync.RWMutex
// ===== Compaction protection (TTL-bound) =====
// segmentProtectionUntil: segmentID -> latest active protection expiry (Unix seconds).
// When multiple active-TTL snapshots reference the same segment, the latest expiry wins.
segmentProtectionUntil map[int64]uint64
// compactionBlockedCollections: collections with active compaction-protected snapshots
// whose RefIndex hasn't been loaded yet. Fail-closed coarse block.
compactionBlockedCollections typeutil.UniqueSet
// ===== GC protection (unconditional on TTL) =====
// gcBlockedCollections: collections with any snapshot whose RefIndex hasn't been loaded.
// Fail-closed coarse block for GC queries: we cannot know which segments/buildIDs an
// unloaded RefIndex references, so every candidate in the collection must be kept.
gcBlockedCollections typeutil.UniqueSet
// segmentReferencedByGC: set of segment IDs referenced by at least one loaded snapshot.
// Precise-grained GC protection: once we know a snapshot's referenced segments, we can
// point-query this set instead of scanning all snapshots per candidate.
segmentReferencedByGC typeutil.UniqueSet
// buildIDReferencedByGC: set of index buildIDs referenced by at least one loaded snapshot.
buildIDReferencedByGC typeutil.UniqueSet
// snapshotPendingCollections: collections currently in the process of creating a snapshot.
// Blocks compaction commit for these collections to prevent segment state changes
// between GenSnapshot (segment list capture) and SaveSnapshot (protection setup).
// This is independent of the rebuild pattern above — it is a short-lived flag set by
// CreateSnapshot and cleared by its defer.
snapshotPendingCollections typeutil.UniqueSet
// Background RefIndex loader goroutine control
loaderCtx context.Context
loaderCancel context.CancelFunc
@@ -198,15 +262,21 @@ type snapshotMeta struct {
func newSnapshotMeta(ctx context.Context, catalog metastore.DataCoordCatalog, chunkManager storage.ChunkManager) (*snapshotMeta, error) {
loaderCtx, loaderCancel := context.WithCancel(context.Background())
sm := &snapshotMeta{
catalog: catalog,
snapshotID2Info: typeutil.NewConcurrentMap[UniqueID, *datapb.SnapshotInfo](),
snapshotID2RefIndex: typeutil.NewConcurrentMap[UniqueID, *SnapshotRefIndex](),
snapshotName2ID: typeutil.NewConcurrentMap[string, UniqueID](),
collectionID2Snapshots: typeutil.NewConcurrentMap[UniqueID, typeutil.UniqueSet](),
loaderCtx: loaderCtx,
loaderCancel: loaderCancel,
reader: NewSnapshotReader(chunkManager),
writer: NewSnapshotWriter(chunkManager),
catalog: catalog,
snapshotID2Info: typeutil.NewConcurrentMap[UniqueID, *datapb.SnapshotInfo](),
snapshotID2RefIndex: typeutil.NewConcurrentMap[UniqueID, *SnapshotRefIndex](),
snapshotName2ID: typeutil.NewConcurrentMap[string, UniqueID](),
collectionID2Snapshots: typeutil.NewConcurrentMap[UniqueID, typeutil.UniqueSet](),
segmentProtectionUntil: make(map[int64]uint64),
compactionBlockedCollections: typeutil.NewUniqueSet(),
gcBlockedCollections: typeutil.NewUniqueSet(),
segmentReferencedByGC: typeutil.NewUniqueSet(),
buildIDReferencedByGC: typeutil.NewUniqueSet(),
snapshotPendingCollections: typeutil.NewUniqueSet(),
loaderCtx: loaderCtx,
loaderCancel: loaderCancel,
reader: NewSnapshotReader(chunkManager),
writer: NewSnapshotWriter(chunkManager),
}
// Reload all snapshots from catalog to populate in-memory cache
@@ -216,7 +286,24 @@ func newSnapshotMeta(ctx context.Context, catalog metastore.DataCoordCatalog, ch
return nil, err
}
// Start background RefIndex loader goroutine
// Synchronously populate compaction protection state before exposing snapshotMeta
// to the rest of DataCoord. This is a pure in-memory operation:
//
// At this point every RefIndex is in Pending state (freshly created by reload()),
// so rebuildAllSegmentProtection will conservatively imprint a collection-level
// block for every snapshot with an active CompactionExpireTime (fail-closed).
// Segment-level precision is not required on startup — the coarser collection-level
// block is a strict superset of the correct segment-level protection.
//
// The background loader goroutine below will then read RefIndexes from S3 and
// progressively transition each collection from coarse collection-level block to
// precise segment-level protection. This keeps startup O(N) in-memory instead of
// O(N × S3_RTT) I/O-bound, while preserving fail-closed semantics.
sm.rebuildAllSegmentProtection()
// Start background RefIndex loader goroutine. It runs an immediate first load
// (to narrow the coarse collection-level blocks to precise segment-level protection
// as soon as possible) and then periodically retries any RefIndex still in Failed state.
sm.loaderWg.Add(1)
go sm.refIndexLoaderLoop()
@@ -277,7 +364,14 @@ func (sm *snapshotMeta) reload(ctx context.Context) error {
return nil
}
// refIndexLoaderLoop periodically loads unloaded RefIndexes in background.
// refIndexLoaderLoop loads unloaded RefIndexes in the background.
//
// newSnapshotMeta imprints a coarse collection-level block for every protected
// snapshot before returning (fail-closed, no I/O). This loop is responsible for
// converging from that coarse state to precise segment-level protection:
// - first tick: runs immediately, loads RefIndexes from S3, narrows the blocks
// - subsequent ticks: periodically retry any RefIndex still in Failed state
//
// It runs until loaderCtx is cancelled.
func (sm *snapshotMeta) refIndexLoaderLoop() {
defer sm.loaderWg.Done()
@@ -294,8 +388,15 @@ func (sm *snapshotMeta) refIndexLoaderLoop() {
return interval
}
// Load immediately on startup.
sm.loadUnloadedRefIndexes()
runOnce := func() {
if sm.loadUnloadedRefIndexes() {
sm.rebuildAllSegmentProtection()
}
}
// Run immediately on startup so convergence from coarse collection-level blocks
// to precise segment-level protection does not wait a full interval.
runOnce()
timer := time.NewTimer(getInterval())
defer timer.Stop()
@@ -306,7 +407,7 @@ func (sm *snapshotMeta) refIndexLoaderLoop() {
log.Info("RefIndex loader goroutine stopped")
return
case <-timer.C:
sm.loadUnloadedRefIndexes()
runOnce()
// Reset using the latest refreshable interval.
timer.Reset(getInterval())
}
@@ -314,7 +415,19 @@ func (sm *snapshotMeta) refIndexLoaderLoop() {
}
// loadUnloadedRefIndexes loads all RefIndexes that are Pending or Failed.
func (sm *snapshotMeta) loadUnloadedRefIndexes() {
// Returns true if any RefIndex state changed (loaded or failed), indicating
// that callers should rebuild dependent state like segment protection.
//
// Each ReadSnapshot call is bounded by a per-call timeout (dataCoord.snapshot.refIndexLoadTimeout)
// derived from sm.loaderCtx. WITHOUT this timeout, a single hung S3 read would block
// the entire Range, no other RefIndex would ever be loaded, rebuildAllSegmentProtection
// would never be triggered, and every collection with a snapshot would stay in the
// startup fail-closed coarse block — leaking storage on every collection with snapshots
// until DataCoord restarts. On timeout the RefIndex is marked Failed and will be
// retried on the next loader tick.
func (sm *snapshotMeta) loadUnloadedRefIndexes() bool {
changed := false
timeout := paramtable.Get().DataCoordCfg.SnapshotRefIndexLoadTimeout.GetAsDurationByParse()
sm.snapshotID2RefIndex.Range(func(id UniqueID, refIndex *SnapshotRefIndex) bool {
if refIndex.IsLoaded() {
return true // Already loaded, skip
@@ -325,11 +438,14 @@ func (sm *snapshotMeta) loadUnloadedRefIndexes() {
return true // Snapshot deleted
}
snapshotData, err := sm.reader.ReadSnapshot(sm.loaderCtx, info.GetS3Location(), false)
readCtx, cancel := context.WithTimeout(sm.loaderCtx, timeout)
snapshotData, err := sm.reader.ReadSnapshot(readCtx, info.GetS3Location(), false)
cancel()
if err != nil {
log.Warn("failed to load RefIndex from S3",
zap.String("name", info.GetName()),
zap.Int64("id", id),
zap.Duration("timeout", timeout),
zap.Error(err))
refIndex.SetFailed()
} else {
@@ -338,9 +454,11 @@ func (sm *snapshotMeta) loadUnloadedRefIndexes() {
zap.String("name", info.GetName()),
zap.Int64("id", id))
}
changed = true
return true
})
return changed
}
// Close stops the background RefIndex loader goroutine.
@@ -433,7 +551,16 @@ func (sm *snapshotMeta) SaveSnapshot(ctx context.Context, snapshot *SnapshotData
// Step 4: Phase 2 (Commit) - Update catalog to COMMITTED state
snapshot.SnapshotInfo.State = datapb.SnapshotState_SnapshotStateCommitted
// Insert into in-memory cache (two maps)
// Phase 2 window (these 4 in-memory insertions) — invariant note:
// Between the first Insert below and registerSnapshotProtection, this snapshot is
// visible in snapshotID2Info but has not yet contributed to segmentReferencedByGC /
// buildIDReferencedByGC. A concurrent GC in this window would find the segments
// unprotected and could delete the files. We rely on snapshotManager.CreateSnapshot
// having called SetSnapshotPending(collectionID) BEFORE entering SaveSnapshot and
// deferring ClearSnapshotPending until after SaveSnapshot returns; SetSnapshotPending
// blocks compaction commit for the collection, which in turn prevents any of this
// snapshot's segments from transitioning to Dropped (the only state where GC would
// act on them). Therefore this nanosecond-scale window is not actually observable.
sm.snapshotID2Info.Insert(snapshot.SnapshotInfo.GetId(), snapshot.SnapshotInfo)
sm.snapshotID2RefIndex.Insert(snapshot.SnapshotInfo.GetId(),
NewLoadedSnapshotRefIndex(segmentIDs, buildIDs))
@@ -441,13 +568,34 @@ func (sm *snapshotMeta) SaveSnapshot(ctx context.Context, snapshot *SnapshotData
// Build secondary indexes for O(1) lookup
sm.addToSecondaryIndexes(snapshot.SnapshotInfo)
// Register both compaction protection (TTL-bound) and GC protection (unconditional)
// for the referenced segments/buildIDs. Must run for ANY snapshot, including those
// with CompactionExpireTime==0, because GC protection has no TTL — otherwise the
// freshly-created snapshot's referenced files could be deleted by GC.
sm.registerSnapshotProtection(snapshot.SnapshotInfo, segmentIDs, buildIDs)
// Update catalog with COMMITTED state
if err := sm.catalog.SaveSnapshot(ctx, snapshot.SnapshotInfo); err != nil {
// Phase 2 failed, but S3 data is written. GC will eventually cleanup.
// Remove from memory cache and secondary indexes since commit failed
// Phase 2 commit failed, but S3 data and PENDING record are already written.
// GC will eventually clean up the PENDING record and its S3 files.
//
// Roll back all in-memory state that the earlier success-path insertions built:
// - snapshotID2Info / snapshotID2RefIndex / secondary indexes
// - registerSnapshotProtection's contributions to BOTH protection dimensions:
// compaction (segmentProtectionUntil) AND GC (segmentReferencedByGC /
// buildIDReferencedByGC). rebuildAllSegmentProtection recomputes all 5
// pieces of state from the remaining snapshots under a single lock, so
// the rollback is atomic and cannot leak stale GC pins.
//
// NOTE: between the Remove calls and rebuildAllSegmentProtection, readers can
// observe a stale-TRUE protection state (snapshot gone from snapshotID2Info
// but its contributions still in the protection sets). This is a safe error:
// GC/compaction will conservatively skip a round, and the next rebuild fully
// reconciles. The window is nanoseconds, no I/O.
sm.snapshotID2Info.Remove(snapshot.SnapshotInfo.GetId())
sm.snapshotID2RefIndex.Remove(snapshot.SnapshotInfo.GetId())
sm.removeFromSecondaryIndexes(snapshot.SnapshotInfo)
sm.rebuildAllSegmentProtection()
log.Error("failed to update snapshot to committed state, pending record left for GC cleanup",
zap.Error(err))
return fmt.Errorf("failed to update snapshot to committed state: %w", err)
@@ -502,7 +650,21 @@ func (sm *snapshotMeta) DropSnapshot(ctx context.Context, snapshotName string) e
sm.snapshotID2RefIndex.Remove(snapshot.GetId())
sm.removeFromSecondaryIndexes(snapshot)
// Step 4: Delete S3 data (may fail, GC will retry)
// Step 4: Rebuild protection state UNCONDITIONALLY. Must not be gated on
// CompactionExpireTime > 0 because:
//
// 1. GC protection (segmentReferencedByGC/buildIDReferencedByGC) has no TTL —
// even a TTL=0 snapshot contributes, so every drop can leave stale entries.
// 2. Even a TTL>0 snapshot with a Failed/Pending RefIndex imprinted a coarse
// collection-level block (compactionBlockedCollections or gcBlockedCollections)
// that a targeted rebuild cannot clear.
//
// A single rebuildAllSegmentProtection under one lock recomputes all five pieces
// of state consistently. Incremental "unregister" is avoided on purpose because
// it cannot tell whether another snapshot still references the same segment/buildID.
sm.rebuildAllSegmentProtection()
// Step 5: Delete S3 data (may fail, GC will retry)
if err := sm.writer.Drop(ctx, snapshot.GetS3Location()); err != nil {
log.Warn("S3 delete failed, will be cleaned by GC",
zap.Int64("snapshotID", snapshot.GetId()),
@@ -512,7 +674,7 @@ func (sm *snapshotMeta) DropSnapshot(ctx context.Context, snapshotName string) e
return nil
}
// Step 5: Delete catalog record (final cleanup)
// Step 6: Delete catalog record (final cleanup)
if err := sm.catalog.DropSnapshot(ctx, snapshot.GetCollectionId(), snapshot.GetId()); err != nil {
log.Warn("failed to drop snapshot from catalog after S3 cleanup",
zap.Int64("snapshotID", snapshot.GetId()),
@@ -695,46 +857,6 @@ func (sm *snapshotMeta) getSnapshotByName(ctx context.Context, snapshotName stri
return info, nil
}
// GetSnapshotBySegment returns all snapshot IDs that reference a given segment.
//
// This is used for garbage collection safety: before deleting a segment, check if
// any snapshots reference it. If snapshots exist, the segment cannot be deleted.
//
// When collectionID >= 0, only snapshots for that collection are checked.
// When collectionID < 0, all snapshots across all collections are checked.
//
// Caller should check IsAllRefIndexLoaded (when collectionID < 0) or
// IsRefIndexLoadedForCollection (when collectionID >= 0) first to ensure the result is reliable.
//
// Parameters:
// - ctx: Context for cancellation (currently unused but reserved for future)
// - collectionID: Collection ID to filter snapshots, or negative to check all collections
// - segmentID: Segment ID to search for in snapshots
//
// Returns:
// - []UniqueID: List of snapshot IDs that contain this segment (empty if none)
func (sm *snapshotMeta) GetSnapshotBySegment(ctx context.Context, collectionID, segmentID UniqueID) []UniqueID {
snapshotIDs := make([]UniqueID, 0)
sm.snapshotID2Info.Range(func(id UniqueID, info *datapb.SnapshotInfo) bool {
if collectionID >= 0 && info.GetCollectionId() != collectionID {
return true // Skip, different collection
}
refIndex, exists := sm.snapshotID2RefIndex.Get(id)
if !exists {
return true // RefIndex not found (should not happen)
}
if refIndex.ContainsSegment(segmentID) {
snapshotIDs = append(snapshotIDs, id)
}
return true
})
return snapshotIDs
}
// GetPendingSnapshots returns all snapshots that are in PENDING state.
// This is used by GC to find orphaned snapshots that need cleanup.
// Only returns snapshots that have exceeded the pending timeout.
@@ -870,78 +992,260 @@ func (sm *snapshotMeta) removeFromSecondaryIndexes(snapshotInfo *datapb.Snapshot
}
}
// GetSnapshotByBuildID returns all snapshot IDs that reference a given build ID.
//
// This is used for garbage collection safety: before deleting index files for a build,
// check if any snapshots reference it. If snapshots exist, the files cannot be deleted.
//
// BuildID identifies a specific index build task, providing precise file-level protection.
//
// Caller should check IsAllRefIndexLoaded or IsRefIndexLoadedForCollection first.
//
// Parameters:
// - buildID: Build ID to search for in snapshots
//
// Returns:
// - []UniqueID: List of snapshot IDs that contain this build ID (empty if none)
func (sm *snapshotMeta) GetSnapshotByBuildID(buildID UniqueID) []UniqueID {
snapshotIDs := make([]UniqueID, 0)
sm.snapshotID2RefIndex.Range(func(id UniqueID, refIndex *SnapshotRefIndex) bool {
if refIndex.ContainsBuildID(buildID) {
snapshotIDs = append(snapshotIDs, id)
}
return true
})
return snapshotIDs
// IsCollectionCompactionBlocked checks if compaction is blocked for a collection.
// Returns true if:
// - A protected snapshot's RefIndex hasn't been loaded yet (fail-closed), OR
// - The collection is currently in the process of creating a snapshot (intent-based blocking).
func (sm *snapshotMeta) IsCollectionCompactionBlocked(collectionID int64) bool {
sm.segmentProtectionMu.RLock()
defer sm.segmentProtectionMu.RUnlock()
return sm.compactionBlockedCollections.Contain(collectionID) ||
sm.snapshotPendingCollections.Contain(collectionID)
}
// IsAllRefIndexLoaded checks if all RefIndexes across all snapshots are loaded.
// Used by GC when querying snapshot references without a specific collectionID.
func (sm *snapshotMeta) IsAllRefIndexLoaded() bool {
allLoaded := true
sm.snapshotID2Info.Range(func(id UniqueID, info *datapb.SnapshotInfo) bool {
refIndex, exists := sm.snapshotID2RefIndex.Get(id)
if !exists || !refIndex.IsLoaded() {
allLoaded = false
return false // Stop iteration
}
return true
})
return allLoaded
// SetSnapshotPending marks a collection as having a pending snapshot creation.
// This blocks compaction commit for the collection to prevent segment state changes
// during the window between GenSnapshot and SaveSnapshot.
func (sm *snapshotMeta) SetSnapshotPending(collectionID int64) {
sm.segmentProtectionMu.Lock()
defer sm.segmentProtectionMu.Unlock()
sm.snapshotPendingCollections.Insert(collectionID)
log.Info("collection marked as snapshot pending, compaction blocked",
zap.Int64("collectionID", collectionID))
}
// IsRefIndexLoadedForCollection checks if RefIndexes for a collection are loaded.
// Used by GC to check if it's safe to query snapshot references for a specific collection.
func (sm *snapshotMeta) IsRefIndexLoadedForCollection(collectionID int64) bool {
sm.collectionIndexMu.RLock()
snapshotIDs, ok := sm.collectionID2Snapshots.Get(collectionID)
if !ok {
sm.collectionIndexMu.RUnlock()
return true // No snapshots for this collection, consider as loaded
// ClearSnapshotPending removes the pending snapshot mark for a collection.
// Called after SaveSnapshot completes (success or failure) to unblock compaction.
func (sm *snapshotMeta) ClearSnapshotPending(collectionID int64) {
sm.segmentProtectionMu.Lock()
defer sm.segmentProtectionMu.Unlock()
sm.snapshotPendingCollections.Remove(collectionID)
log.Info("collection snapshot pending mark cleared, compaction unblocked",
zap.Int64("collectionID", collectionID))
}
// isProtectionActive returns true if the given expiry timestamp represents
// active (non-zero and not yet expired) compaction protection.
func isProtectionActive(protectionUntil uint64, now uint64) bool {
return protectionUntil > 0 && now < protectionUntil
}
// upsertMaxProtection sets the protection expiry for a segment to the maximum
// of the existing and new values. Must be called with segmentProtectionMu held.
func (sm *snapshotMeta) upsertMaxProtection(segID int64, protectionUntil uint64) {
if existing, ok := sm.segmentProtectionUntil[segID]; !ok || protectionUntil > existing {
sm.segmentProtectionUntil[segID] = protectionUntil
}
}
// IsSegmentCompactionProtected checks if a segment is protected from compaction
// by any active snapshot. Returns true if the segment should be excluded from compaction.
func (sm *snapshotMeta) IsSegmentCompactionProtected(segmentID int64) bool {
sm.segmentProtectionMu.RLock()
defer sm.segmentProtectionMu.RUnlock()
expiryTs, exists := sm.segmentProtectionUntil[segmentID]
if !exists {
return false
}
return uint64(time.Now().Unix()) < expiryTs
}
// IsSegmentGCBlocked reports whether a segment must be kept by GC because of snapshot
// references, without requiring a separate "is loaded" check by the caller.
//
// Fail-closed semantics, layered from coarse to precise:
// 1. If collectionID is non-negative and that collection is in gcBlockedCollections
// (some snapshot's RefIndex has not been loaded from S3 yet), return true — we
// cannot know the precise referenced set, so we keep the segment.
// 2. If collectionID is negative (orphan file walk with no collection context), any
// unloaded RefIndex in any collection triggers a fail-closed return.
// 3. Otherwise, return whether the segment is in segmentReferencedByGC.
//
// collectionID == 0 takes the non-negative branch (same as any positive ID). Since
// Milvus never assigns 0 as a real collection ID, that branch will never match a real
// snapshot and simply falls through to the precise segmentReferencedByGC check. If the
// caller truly has no collection context, pass a negative value to hit branch 2.
//
// GC semantics are unconditional on TTL: snapshots whose CompactionExpireTime has
// elapsed still contribute to the precise set, because PIT recovery still needs the
// underlying files.
//
// Cost: O(1) — all state is precomputed by rebuildAllSegmentProtection.
func (sm *snapshotMeta) IsSegmentGCBlocked(collectionID, segmentID int64) bool {
sm.segmentProtectionMu.RLock()
defer sm.segmentProtectionMu.RUnlock()
if collectionID < 0 {
if sm.gcBlockedCollections.Len() > 0 {
return true
}
} else if sm.gcBlockedCollections.Contain(collectionID) {
return true
}
return sm.segmentReferencedByGC.Contain(segmentID)
}
// IsBuildIDGCBlocked reports whether an index buildID must be kept by GC because of
// snapshot references. Same fail-closed layering as IsSegmentGCBlocked, including
// the collectionID semantics: < 0 means "orphan walk, fail-closed globally", and any
// non-negative value (including 0) narrows the fail-closed path to that collection.
//
// BuildIDs are global (an index build is identified by its unique buildID, not scoped
// to a specific collection). Pass collectionID < 0 when the caller is walking orphan
// buildIDs without a collection context; pass a specific collectionID when the caller
// already has per-segment-index context (in which case only that collection's load
// state gates the fail-closed path).
//
// Cost: O(1).
func (sm *snapshotMeta) IsBuildIDGCBlocked(collectionID, buildID int64) bool {
sm.segmentProtectionMu.RLock()
defer sm.segmentProtectionMu.RUnlock()
if collectionID < 0 {
if sm.gcBlockedCollections.Len() > 0 {
return true
}
} else if sm.gcBlockedCollections.Contain(collectionID) {
return true
}
return sm.buildIDReferencedByGC.Contain(buildID)
}
// registerSnapshotProtection incrementally registers a newly saved snapshot into both
// protection dimensions under a single lock:
//
// - GC protection (unconditional on TTL): the snapshot's segments and buildIDs are
// inserted into segmentReferencedByGC / buildIDReferencedByGC so the underlying
// files are pinned against GC, regardless of CompactionExpireTime.
// - Compaction protection (TTL-bound): if CompactionExpireTime is still active, the
// segments' entries in segmentProtectionUntil are upserted to the max expiry.
//
// This is the incremental counterpart to rebuildAllSegmentProtection — we use it on
// the SaveSnapshot success path to avoid a full O(N) rebuild for a single new snapshot
// while still keeping both dimensions in sync. rebuildAllSegmentProtection remains the
// canonical path when state needs to be recomputed (startup, loader progress, rollback,
// drop), because incremental updates cannot remove another snapshot's contribution.
func (sm *snapshotMeta) registerSnapshotProtection(
info *datapb.SnapshotInfo, segmentIDs, buildIDs []int64,
) {
sm.segmentProtectionMu.Lock()
defer sm.segmentProtectionMu.Unlock()
// GC protection (unconditional) — runs for every snapshot, TTL=0 included.
for _, segID := range segmentIDs {
sm.segmentReferencedByGC.Insert(segID)
}
for _, buildID := range buildIDs {
sm.buildIDReferencedByGC.Insert(buildID)
}
allLoaded := true
snapshotIDs.Range(func(snapshotID int64) bool {
refIndex, exists := sm.snapshotID2RefIndex.Get(snapshotID)
// Safety first:
// - If snapshotInfo exists but refIndex is missing/not loaded, treat as not loaded to avoid unsafe GC.
// - If snapshotInfo doesn't exist (already deleted/cleaned), ignore this snapshotID.
if !exists {
if _, ok := sm.snapshotID2Info.Get(snapshotID); ok {
allLoaded = false
return false
// Compaction protection (TTL-bound) — only snapshots with active CompactionExpireTime.
protectionUntil := info.GetCompactionExpireTime()
if isProtectionActive(protectionUntil, uint64(time.Now().Unix())) {
for _, segID := range segmentIDs {
sm.upsertMaxProtection(segID, protectionUntil)
}
}
log.Info("registered snapshot protection",
zap.Int64("snapshotID", info.GetId()),
zap.Int64("collectionID", info.GetCollectionId()),
zap.Uint64("protectionUntil", protectionUntil),
zap.Int("numSegments", len(segmentIDs)),
zap.Int("numBuildIDs", len(buildIDs)))
}
// rebuildAllSegmentProtection atomically rebuilds BOTH compaction protection and GC
// protection state from all snapshots. Called whenever snapshot state changes (on
// startup, after RefIndex loader makes progress, after SaveSnapshot rollback, after
// DropSnapshot).
//
// A single Range over snapshotID2Info produces 5 pieces of state under one lock,
// keeping the two protection dimensions consistent:
//
// 1. segmentProtectionUntil — TTL-bound segment compaction protection
// 2. compactionBlockedCollections — fail-closed coarse block for compaction
// 3. segmentReferencedByGC — precise segment GC protection (no TTL)
// 4. buildIDReferencedByGC — precise buildID GC protection (no TTL)
// 5. gcBlockedCollections — fail-closed coarse block for GC
//
// Fail-closed design (both dimensions): if a snapshot's RefIndex hasn't been loaded,
// we don't know which segments/buildIDs it references. In that case the entire
// collection is coarsely blocked until the loader goroutine finishes, at which point
// the next rebuild will narrow the coarse block to precise per-segment/buildID sets.
//
// TTL asymmetry between the two dimensions:
// - Compaction only protects snapshots whose CompactionExpireTime is still active.
// Expired-TTL snapshots contribute nothing to segmentProtectionUntil /
// compactionBlockedCollections.
// - GC protects segments/buildIDs referenced by ANY existing snapshot, regardless
// of TTL, because a snapshot always needs its underlying files to serve PIT
// recovery. Expired-TTL snapshots still contribute to the GC sets.
func (sm *snapshotMeta) rebuildAllSegmentProtection() {
sm.segmentProtectionMu.Lock()
defer sm.segmentProtectionMu.Unlock()
// Reset all protection state — the rebuild produces a complete, consistent snapshot.
// NOTE: snapshotPendingCollections is intentionally NOT reset here. It is a short-lived
// intent flag owned by SetSnapshotPending/ClearSnapshotPending (scoped to a single
// CreateSnapshot call), independent from the snapshot→collection state that this
// rebuild derives from snapshotID2Info. Clearing it here would let a concurrent
// compaction slip through the GenSnapshot→SaveSnapshot window.
sm.segmentProtectionUntil = make(map[int64]uint64)
compactionBlocked := typeutil.NewUniqueSet()
gcBlocked := typeutil.NewUniqueSet()
segRefGC := typeutil.NewUniqueSet()
buildRefGC := typeutil.NewUniqueSet()
now := uint64(time.Now().Unix())
sm.snapshotID2Info.Range(func(id UniqueID, info *datapb.SnapshotInfo) bool {
refIndex, exists := sm.snapshotID2RefIndex.Get(id)
loaded := exists && refIndex.IsLoaded()
// ===== GC protection (unconditional on TTL) =====
if !loaded {
// Fail-closed: we don't know which segments/buildIDs this snapshot references.
// Coarse-block the entire collection for GC until the RefIndex loads.
gcBlocked.Insert(info.GetCollectionId())
} else {
// Precise: record every referenced segment/buildID for O(1) lookups.
for _, segID := range refIndex.GetSegmentIDs() {
segRefGC.Insert(segID)
}
for _, buildID := range refIndex.GetBuildIDs() {
buildRefGC.Insert(buildID)
}
}
// ===== Compaction protection (TTL-bound) =====
protectionUntil := info.GetCompactionExpireTime()
if !isProtectionActive(protectionUntil, now) {
// TTL expired or never set — no compaction protection contribution.
return true
}
if !refIndex.IsLoaded() {
allLoaded = false
return false // Stop iteration
if !loaded {
// Fail-closed: active TTL but RefIndex not loaded.
compactionBlocked.Insert(info.GetCollectionId())
log.Info("blocking compaction for collection due to unloaded protected snapshot RefIndex",
zap.Int64("snapshotID", id),
zap.Int64("collectionID", info.GetCollectionId()),
zap.Uint64("protectionUntil", protectionUntil))
return true
}
for _, segID := range refIndex.GetSegmentIDs() {
sm.upsertMaxProtection(segID, protectionUntil)
}
return true
})
sm.collectionIndexMu.RUnlock()
return allLoaded
sm.compactionBlockedCollections = compactionBlocked
sm.gcBlockedCollections = gcBlocked
sm.segmentReferencedByGC = segRefGC
sm.buildIDReferencedByGC = buildRefGC
log.Info("rebuilt all snapshot protection state",
zap.Int("compactionProtectedSegments", len(sm.segmentProtectionUntil)),
zap.Int("compactionBlockedCollections", sm.compactionBlockedCollections.Len()),
zap.Int("gcReferencedSegments", sm.segmentReferencedByGC.Len()),
zap.Int("gcReferencedBuildIDs", sm.buildIDReferencedByGC.Len()),
zap.Int("gcBlockedCollections", sm.gcBlockedCollections.Len()))
}
internal/datacoord/snapshot_meta_test.go
+1093 -355
View File
File diff suppressed because it is too large Load Diff
internal/proxy/task_snapshot.go
+13 -3
View File
@@ -99,6 +99,15 @@ func (cst *createSnapshotTask) PreExecute(ctx context.Context) error {
return err
}
// Validate compaction protection duration
maxCompactionProtectionSeconds := paramtable.Get().DataCoordCfg.SnapshotMaxCompactionProtectionSeconds.GetAsInt64()
if cst.req.GetCompactionProtectionSeconds() < 0 {
return merr.WrapErrParameterInvalidMsg("compaction_protection_seconds must be non-negative")
}
if cst.req.GetCompactionProtectionSeconds() > maxCompactionProtectionSeconds {
return merr.WrapErrParameterInvalidMsg("compaction_protection_seconds must not exceed %d", maxCompactionProtectionSeconds)
}
collectionID, err := globalMetaCache.GetCollectionID(ctx, cst.req.GetDbName(), cst.req.GetCollectionName())
if err != nil {
return err
@@ -120,9 +129,10 @@ func (cst *createSnapshotTask) Execute(ctx context.Context) error {
Base: commonpbutil.NewMsgBase(
commonpbutil.WithMsgType(commonpb.MsgType_CreateSnapshot),
),
Name: cst.req.GetName(),
Description: cst.req.GetDescription(),
CollectionId: cst.collectionID,
Name: cst.req.GetName(),
Description: cst.req.GetDescription(),
CollectionId: cst.collectionID,
CompactionProtectionSeconds: cst.req.GetCompactionProtectionSeconds(),
})
if err = merr.CheckRPCCall(cst.result, err); err != nil {
return err
internal/proxy/task_snapshot_test.go
+67
View File
@@ -112,6 +112,73 @@ func TestCreateSnapshotTask_PreExecute_CollectionNotFound(t *testing.T) {
assert.Contains(t, err.Error(), "collection not found")
}
func TestCreateSnapshotTask_PreExecute_ProtectionNegative(t *testing.T) {
task := &createSnapshotTask{
req: &milvuspb.CreateSnapshotRequest{
Name: "test_snapshot",
DbName: "default",
CollectionName: "test_collection",
CompactionProtectionSeconds: -1,
},
}
err := task.PreExecute(context.Background())
assert.Error(t, err)
assert.True(t, strings.Contains(err.Error(), "non-negative"))
}
func TestCreateSnapshotTask_PreExecute_ProtectionExceedsMax(t *testing.T) {
task := &createSnapshotTask{
req: &milvuspb.CreateSnapshotRequest{
Name: "test_snapshot",
DbName: "default",
CollectionName: "test_collection",
CompactionProtectionSeconds: 7*24*3600 + 1, // 7 days + 1 second
},
}
err := task.PreExecute(context.Background())
assert.Error(t, err)
assert.True(t, strings.Contains(err.Error(), "must not exceed"))
}
func TestCreateSnapshotTask_PreExecute_ProtectionZero(t *testing.T) {
task := &createSnapshotTask{
req: &milvuspb.CreateSnapshotRequest{
Name: "test_snapshot",
DbName: "default",
CollectionName: "test_collection",
CompactionProtectionSeconds: 0,
},
}
globalMetaCache = &MetaCache{}
mockGetCollectionID := mockey.Mock((*MetaCache).GetCollectionID).Return(int64(100), nil).Build()
defer mockGetCollectionID.UnPatch()
err := task.PreExecute(context.Background())
assert.NoError(t, err)
}
func TestCreateSnapshotTask_PreExecute_ProtectionValid(t *testing.T) {
task := &createSnapshotTask{
req: &milvuspb.CreateSnapshotRequest{
Name: "test_snapshot",
DbName: "default",
CollectionName: "test_collection",
CompactionProtectionSeconds: 3600, // 1 hour
},
}
globalMetaCache = &MetaCache{}
mockGetCollectionID := mockey.Mock((*MetaCache).GetCollectionID).Return(int64(100), nil).Build()
defer mockGetCollectionID.UnPatch()
err := task.PreExecute(context.Background())
assert.NoError(t, err)
assert.Equal(t, int64(100), task.collectionID)
}
func TestCreateSnapshotTask_Execute_Success(t *testing.T) {
mockMixCoord := NewMixCoordMock()
task := &createSnapshotTask{
pkg/go.mod
+1 -1
View File
@@ -22,7 +22,7 @@ require (
github.com/jolestar/go-commons-pool/v2 v2.1.2
github.com/json-iterator/go v1.1.13-0.20220915233716-71ac16282d12
github.com/klauspost/compress v1.18.0
github.com/milvus-io/milvus-proto/go-api/v2 v2.6.6-0.20260309063517-8b5776d31f2b
github.com/milvus-io/milvus-proto/go-api/v2 v2.6.6-0.20260312114919-fe1b25e57c40
github.com/minio/minio-go/v7 v7.0.73
github.com/panjf2000/ants/v2 v2.11.3
github.com/prometheus/client_golang v1.20.5
pkg/go.sum
+2 -2
View File
@@ -515,8 +515,8 @@ github.com/milvus-io/cgosymbolizer v0.0.0-20250318084424-114f4050c3a6 h1:YHMFI6L
github.com/milvus-io/cgosymbolizer v0.0.0-20250318084424-114f4050c3a6/go.mod h1:DvXTE/K/RtHehxU8/GtDs4vFtfw64jJ3PaCnFri8CRg=
github.com/milvus-io/gorocksdb v0.0.0-20220624081344-8c5f4212846b h1:TfeY0NxYxZzUfIfYe5qYDBzt4ZYRqzUjTR6CvUzjat8=
github.com/milvus-io/gorocksdb v0.0.0-20220624081344-8c5f4212846b/go.mod h1:iwW+9cWfIzzDseEBCCeDSN5SD16Tidvy8cwQ7ZY8Qj4=
github.com/milvus-io/milvus-proto/go-api/v2 v2.6.6-0.20260309063517-8b5776d31f2b h1:tUp6a72FFeosfjN+8jy/Hax7BqVbtgsaMMVrAfzKnEc=
github.com/milvus-io/milvus-proto/go-api/v2 v2.6.6-0.20260309063517-8b5776d31f2b/go.mod h1:/6UT4zZl6awVeXLeE7UGDWZvXj3IWkRsh3mqsn0DiAs=
github.com/milvus-io/milvus-proto/go-api/v2 v2.6.6-0.20260312114919-fe1b25e57c40 h1:nGDPhwxHmn4wwgrzT/jhq5pOa3zu6f5lGchFP6jkwL4=
github.com/milvus-io/milvus-proto/go-api/v2 v2.6.6-0.20260312114919-fe1b25e57c40/go.mod h1:/6UT4zZl6awVeXLeE7UGDWZvXj3IWkRsh3mqsn0DiAs=
github.com/minio/md5-simd v1.1.2 h1:Gdi1DZK69+ZVMoNHRXJyNcxrMA4dSxoYHZSQbirFg34=
github.com/minio/md5-simd v1.1.2/go.mod h1:MzdKDxYpY2BT9XQFocsiZf/NKVtR7nkE4RoEpN+20RM=
github.com/minio/minio-go/v7 v7.0.73 h1:qr2vi96Qm7kZ4v7LLebjte+MQh621fFWnv93p12htEo=
pkg/proto/data_coord.proto
+4 -2
View File
@@ -1472,6 +1472,7 @@ message CreateSnapshotRequest {
string name = 2; // user-defined snapshot name
string description = 3; // user-defined snapshot description
int64 collection_id= 4; // collection name
int64 compaction_protection_seconds = 5; // duration in seconds to protect referenced segments from compaction, 0 = no protection
}
message DropSnapshotRequest {
@@ -1532,6 +1533,8 @@ message SnapshotInfo {
string s3_location = 7; // s3 location of the snapshot meta file if snapshot is exported to s3.
SnapshotState state = 8; // snapshot state for 2PC commit (Pending/Committed)
int64 pending_start_time = 9; // timestamp when pending started, for GC timeout
uint64 compaction_expire_time = 10; // absolute expiry timestamp (Unix seconds) for compaction protection, 0 = no protection
reserved 11;
}
// StorageV2SegmentManifest maps a segment ID to its StorageV2 manifest file path.
@@ -1557,8 +1560,7 @@ message SnapshotMetadata {
repeated string manifest_list = 5;
repeated StorageV2SegmentManifest storagev2_manifest_list = 6;
repeated int64 segment_ids = 7;
reserved 8; // was index_ids, no longer needed (buildIDs provides more precise GC protection)
repeated int64 build_ids = 9;
repeated int64 build_ids = 8;
}
message RestoreSnapshotInfo {
pkg/proto/datapb/data_coord.pb.go
+925 -902
View File
File diff suppressed because it is too large Load Diff
pkg/proto/messages.proto
+1
View File
@@ -578,6 +578,7 @@ message CreateSnapshotMessageHeader {
int64 collection_id = 1;
string name = 2;
string description = 3;
int64 compaction_protection_seconds = 4; // duration in seconds to protect referenced segments from compaction
}
// CreateSnapshotMessageBody is the body of create snapshot message.
pkg/proto/messagespb/messages.pb.go
+281 -269
View File
@@ -4755,9 +4755,10 @@ type CreateSnapshotMessageHeader struct {
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
CollectionId int64 `protobuf:"varint,1,opt,name=collection_id,json=collectionId,proto3" json:"collection_id,omitempty"`
Name string `protobuf:"bytes,2,opt,name=name,proto3" json:"name,omitempty"`
Description string `protobuf:"bytes,3,opt,name=description,proto3" json:"description,omitempty"`
CollectionId int64 `protobuf:"varint,1,opt,name=collection_id,json=collectionId,proto3" json:"collection_id,omitempty"`
Name string `protobuf:"bytes,2,opt,name=name,proto3" json:"name,omitempty"`
Description string `protobuf:"bytes,3,opt,name=description,proto3" json:"description,omitempty"`
CompactionProtectionSeconds int64 `protobuf:"varint,4,opt,name=compaction_protection_seconds,json=compactionProtectionSeconds,proto3" json:"compaction_protection_seconds,omitempty"` // duration in seconds to protect referenced segments from compaction
}
func (x *CreateSnapshotMessageHeader) Reset() {
@@ -4813,6 +4814,13 @@ func (x *CreateSnapshotMessageHeader) GetDescription() string {
return ""
}
func (x *CreateSnapshotMessageHeader) GetCompactionProtectionSeconds() int64 {
if x != nil {
return x.CompactionProtectionSeconds
}
return 0
}
// CreateSnapshotMessageBody is the body of create snapshot message.
// Empty - all info is in header.
type CreateSnapshotMessageBody struct {
@@ -6772,275 +6780,279 @@ var file_messages_proto_rawDesc = []byte{
0x69, 0x6e, 0x64, 0x65, 0x78, 0x5f, 0x69, 0x64, 0x73, 0x18, 0x02, 0x20, 0x03, 0x28, 0x03, 0x52,
0x08, 0x69, 0x6e, 0x64, 0x65, 0x78, 0x49, 0x64, 0x73, 0x22, 0x16, 0x0a, 0x14, 0x44, 0x72, 0x6f,
0x70, 0x49, 0x6e, 0x64, 0x65, 0x78, 0x4d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x42, 0x6f, 0x64,
0x79, 0x22, 0x78, 0x0a, 0x1b, 0x43, 0x72, 0x65, 0x61, 0x74, 0x65, 0x53, 0x6e, 0x61, 0x70, 0x73,
0x68, 0x6f, 0x74, 0x4d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x48, 0x65, 0x61, 0x64, 0x65, 0x72,
0x12, 0x23, 0x0a, 0x0d, 0x63, 0x6f, 0x6c, 0x6c, 0x65, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x5f, 0x69,
0x64, 0x18, 0x01, 0x20, 0x01, 0x28, 0x03, 0x52, 0x0c, 0x63, 0x6f, 0x6c, 0x6c, 0x65, 0x63, 0x74,
0x69, 0x6f, 0x6e, 0x49, 0x64, 0x12, 0x12, 0x0a, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x02, 0x20,
0x01, 0x28, 0x09, 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x12, 0x20, 0x0a, 0x0b, 0x64, 0x65, 0x73,
0x63, 0x72, 0x69, 0x70, 0x74, 0x69, 0x6f, 0x6e, 0x18, 0x03, 0x20, 0x01, 0x28, 0x09, 0x52, 0x0b,
0x64, 0x65, 0x73, 0x63, 0x72, 0x69, 0x70, 0x74, 0x69, 0x6f, 0x6e, 0x22, 0x1b, 0x0a, 0x19, 0x43,
0x72, 0x65, 0x61, 0x74, 0x65, 0x53, 0x6e, 0x61, 0x70, 0x73, 0x68, 0x6f, 0x74, 0x4d, 0x65, 0x73,
0x73, 0x61, 0x67, 0x65, 0x42, 0x6f, 0x64, 0x79, 0x22, 0x2f, 0x0a, 0x19, 0x44, 0x72, 0x6f, 0x70,
0x53, 0x6e, 0x61, 0x70, 0x73, 0x68, 0x6f, 0x74, 0x4d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x48,
0x65, 0x61, 0x64, 0x65, 0x72, 0x12, 0x12, 0x0a, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x01, 0x20,
0x01, 0x28, 0x09, 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x22, 0x19, 0x0a, 0x17, 0x44, 0x72, 0x6f,
0x70, 0x53, 0x6e, 0x61, 0x70, 0x73, 0x68, 0x6f, 0x74, 0x4d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65,
0x42, 0x6f, 0x64, 0x79, 0x22, 0x7f, 0x0a, 0x1c, 0x52, 0x65, 0x73, 0x74, 0x6f, 0x72, 0x65, 0x53,
0x6e, 0x61, 0x70, 0x73, 0x68, 0x6f, 0x74, 0x4d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x48, 0x65,
0x61, 0x64, 0x65, 0x72, 0x12, 0x23, 0x0a, 0x0d, 0x73, 0x6e, 0x61, 0x70, 0x73, 0x68, 0x6f, 0x74,
0x5f, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x0c, 0x73, 0x6e, 0x61,
0x70, 0x73, 0x68, 0x6f, 0x74, 0x4e, 0x61, 0x6d, 0x65, 0x12, 0x23, 0x0a, 0x0d, 0x63, 0x6f, 0x6c,
0x6c, 0x65, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x5f, 0x69, 0x64, 0x18, 0x02, 0x20, 0x01, 0x28, 0x03,
0x52, 0x0c, 0x63, 0x6f, 0x6c, 0x6c, 0x65, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x49, 0x64, 0x12, 0x15,
0x0a, 0x06, 0x6a, 0x6f, 0x62, 0x5f, 0x69, 0x64, 0x18, 0x03, 0x20, 0x01, 0x28, 0x03, 0x52, 0x05,
0x6a, 0x6f, 0x62, 0x49, 0x64, 0x22, 0x1c, 0x0a, 0x1a, 0x52, 0x65, 0x73, 0x74, 0x6f, 0x72, 0x65,
0x53, 0x6e, 0x61, 0x70, 0x73, 0x68, 0x6f, 0x74, 0x4d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x42,
0x6f, 0x64, 0x79, 0x22, 0xea, 0x01, 0x0a, 0x15, 0x41, 0x6c, 0x74, 0x65, 0x72, 0x57, 0x41, 0x4c,
0x4d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x48, 0x65, 0x61, 0x64, 0x65, 0x72, 0x12, 0x44, 0x0a,
0x0f, 0x74, 0x61, 0x72, 0x67, 0x65, 0x74, 0x5f, 0x77, 0x61, 0x6c, 0x5f, 0x6e, 0x61, 0x6d, 0x65,
0x18, 0x01, 0x20, 0x01, 0x28, 0x0e, 0x32, 0x1c, 0x2e, 0x6d, 0x69, 0x6c, 0x76, 0x75, 0x73, 0x2e,
0x70, 0x72, 0x6f, 0x74, 0x6f, 0x2e, 0x63, 0x6f, 0x6d, 0x6d, 0x6f, 0x6e, 0x2e, 0x57, 0x41, 0x4c,
0x4e, 0x61, 0x6d, 0x65, 0x52, 0x0d, 0x74, 0x61, 0x72, 0x67, 0x65, 0x74, 0x57, 0x61, 0x6c, 0x4e,
0x61, 0x6d, 0x65, 0x12, 0x50, 0x0a, 0x06, 0x63, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x18, 0x02, 0x20,
0x03, 0x28, 0x0b, 0x32, 0x38, 0x2e, 0x6d, 0x69, 0x6c, 0x76, 0x75, 0x73, 0x2e, 0x70, 0x72, 0x6f,
0x74, 0x6f, 0x2e, 0x6d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x73, 0x2e, 0x41, 0x6c, 0x74, 0x65,
0x72, 0x57, 0x41, 0x4c, 0x4d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x48, 0x65, 0x61, 0x64, 0x65,
0x72, 0x2e, 0x43, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x52, 0x06, 0x63,
0x6f, 0x6e, 0x66, 0x69, 0x67, 0x1a, 0x39, 0x0a, 0x0b, 0x43, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x45,
0x6e, 0x74, 0x72, 0x79, 0x12, 0x10, 0x0a, 0x03, 0x6b, 0x65, 0x79, 0x18, 0x01, 0x20, 0x01, 0x28,
0x09, 0x52, 0x03, 0x6b, 0x65, 0x79, 0x12, 0x14, 0x0a, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x18,
0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x3a, 0x02, 0x38, 0x01,
0x22, 0x15, 0x0a, 0x13, 0x41, 0x6c, 0x74, 0x65, 0x72, 0x57, 0x41, 0x4c, 0x4d, 0x65, 0x73, 0x73,
0x61, 0x67, 0x65, 0x42, 0x6f, 0x64, 0x79, 0x22, 0xdb, 0x01, 0x0a, 0x26, 0x52, 0x65, 0x66, 0x72,
0x65, 0x73, 0x68, 0x45, 0x78, 0x74, 0x65, 0x72, 0x6e, 0x61, 0x6c, 0x43, 0x6f, 0x6c, 0x6c, 0x65,
0x79, 0x22, 0xbc, 0x01, 0x0a, 0x1b, 0x43, 0x72, 0x65, 0x61, 0x74, 0x65, 0x53, 0x6e, 0x61, 0x70,
0x73, 0x68, 0x6f, 0x74, 0x4d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x48, 0x65, 0x61, 0x64, 0x65,
0x72, 0x12, 0x23, 0x0a, 0x0d, 0x63, 0x6f, 0x6c, 0x6c, 0x65, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x5f,
0x69, 0x64, 0x18, 0x01, 0x20, 0x01, 0x28, 0x03, 0x52, 0x0c, 0x63, 0x6f, 0x6c, 0x6c, 0x65, 0x63,
0x74, 0x69, 0x6f, 0x6e, 0x49, 0x64, 0x12, 0x12, 0x0a, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x02,
0x20, 0x01, 0x28, 0x09, 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x12, 0x20, 0x0a, 0x0b, 0x64, 0x65,
0x73, 0x63, 0x72, 0x69, 0x70, 0x74, 0x69, 0x6f, 0x6e, 0x18, 0x03, 0x20, 0x01, 0x28, 0x09, 0x52,
0x0b, 0x64, 0x65, 0x73, 0x63, 0x72, 0x69, 0x70, 0x74, 0x69, 0x6f, 0x6e, 0x12, 0x42, 0x0a, 0x1d,
0x63, 0x6f, 0x6d, 0x70, 0x61, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x5f, 0x70, 0x72, 0x6f, 0x74, 0x65,
0x63, 0x74, 0x69, 0x6f, 0x6e, 0x5f, 0x73, 0x65, 0x63, 0x6f, 0x6e, 0x64, 0x73, 0x18, 0x04, 0x20,
0x01, 0x28, 0x03, 0x52, 0x1b, 0x63, 0x6f, 0x6d, 0x70, 0x61, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x50,
0x72, 0x6f, 0x74, 0x65, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x53, 0x65, 0x63, 0x6f, 0x6e, 0x64, 0x73,
0x22, 0x1b, 0x0a, 0x19, 0x43, 0x72, 0x65, 0x61, 0x74, 0x65, 0x53, 0x6e, 0x61, 0x70, 0x73, 0x68,
0x6f, 0x74, 0x4d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x42, 0x6f, 0x64, 0x79, 0x22, 0x2f, 0x0a,
0x19, 0x44, 0x72, 0x6f, 0x70, 0x53, 0x6e, 0x61, 0x70, 0x73, 0x68, 0x6f, 0x74, 0x4d, 0x65, 0x73,
0x73, 0x61, 0x67, 0x65, 0x48, 0x65, 0x61, 0x64, 0x65, 0x72, 0x12, 0x12, 0x0a, 0x04, 0x6e, 0x61,
0x6d, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x22, 0x19,
0x0a, 0x17, 0x44, 0x72, 0x6f, 0x70, 0x53, 0x6e, 0x61, 0x70, 0x73, 0x68, 0x6f, 0x74, 0x4d, 0x65,
0x73, 0x73, 0x61, 0x67, 0x65, 0x42, 0x6f, 0x64, 0x79, 0x22, 0x7f, 0x0a, 0x1c, 0x52, 0x65, 0x73,
0x74, 0x6f, 0x72, 0x65, 0x53, 0x6e, 0x61, 0x70, 0x73, 0x68, 0x6f, 0x74, 0x4d, 0x65, 0x73, 0x73,
0x61, 0x67, 0x65, 0x48, 0x65, 0x61, 0x64, 0x65, 0x72, 0x12, 0x23, 0x0a, 0x0d, 0x73, 0x6e, 0x61,
0x70, 0x73, 0x68, 0x6f, 0x74, 0x5f, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09,
0x52, 0x0c, 0x73, 0x6e, 0x61, 0x70, 0x73, 0x68, 0x6f, 0x74, 0x4e, 0x61, 0x6d, 0x65, 0x12, 0x23,
0x0a, 0x0d, 0x63, 0x6f, 0x6c, 0x6c, 0x65, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x5f, 0x69, 0x64, 0x18,
0x02, 0x20, 0x01, 0x28, 0x03, 0x52, 0x0c, 0x63, 0x6f, 0x6c, 0x6c, 0x65, 0x63, 0x74, 0x69, 0x6f,
0x6e, 0x49, 0x64, 0x12, 0x15, 0x0a, 0x06, 0x6a, 0x6f, 0x62, 0x5f, 0x69, 0x64, 0x18, 0x03, 0x20,
0x01, 0x28, 0x03, 0x52, 0x05, 0x6a, 0x6f, 0x62, 0x49, 0x64, 0x22, 0x1c, 0x0a, 0x1a, 0x52, 0x65,
0x73, 0x74, 0x6f, 0x72, 0x65, 0x53, 0x6e, 0x61, 0x70, 0x73, 0x68, 0x6f, 0x74, 0x4d, 0x65, 0x73,
0x73, 0x61, 0x67, 0x65, 0x42, 0x6f, 0x64, 0x79, 0x22, 0xea, 0x01, 0x0a, 0x15, 0x41, 0x6c, 0x74,
0x65, 0x72, 0x57, 0x41, 0x4c, 0x4d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x48, 0x65, 0x61, 0x64,
0x65, 0x72, 0x12, 0x44, 0x0a, 0x0f, 0x74, 0x61, 0x72, 0x67, 0x65, 0x74, 0x5f, 0x77, 0x61, 0x6c,
0x5f, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0e, 0x32, 0x1c, 0x2e, 0x6d, 0x69,
0x6c, 0x76, 0x75, 0x73, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x2e, 0x63, 0x6f, 0x6d, 0x6d, 0x6f,
0x6e, 0x2e, 0x57, 0x41, 0x4c, 0x4e, 0x61, 0x6d, 0x65, 0x52, 0x0d, 0x74, 0x61, 0x72, 0x67, 0x65,
0x74, 0x57, 0x61, 0x6c, 0x4e, 0x61, 0x6d, 0x65, 0x12, 0x50, 0x0a, 0x06, 0x63, 0x6f, 0x6e, 0x66,
0x69, 0x67, 0x18, 0x02, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x38, 0x2e, 0x6d, 0x69, 0x6c, 0x76, 0x75,
0x73, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x2e, 0x6d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x73,
0x2e, 0x41, 0x6c, 0x74, 0x65, 0x72, 0x57, 0x41, 0x4c, 0x4d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65,
0x48, 0x65, 0x61, 0x64, 0x65, 0x72, 0x2e, 0x43, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x45, 0x6e, 0x74,
0x72, 0x79, 0x52, 0x06, 0x63, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x1a, 0x39, 0x0a, 0x0b, 0x43, 0x6f,
0x6e, 0x66, 0x69, 0x67, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x12, 0x10, 0x0a, 0x03, 0x6b, 0x65, 0x79,
0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x03, 0x6b, 0x65, 0x79, 0x12, 0x14, 0x0a, 0x05, 0x76,
0x61, 0x6c, 0x75, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x05, 0x76, 0x61, 0x6c, 0x75,
0x65, 0x3a, 0x02, 0x38, 0x01, 0x22, 0x15, 0x0a, 0x13, 0x41, 0x6c, 0x74, 0x65, 0x72, 0x57, 0x41,
0x4c, 0x4d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x42, 0x6f, 0x64, 0x79, 0x22, 0xdb, 0x01, 0x0a,
0x26, 0x52, 0x65, 0x66, 0x72, 0x65, 0x73, 0x68, 0x45, 0x78, 0x74, 0x65, 0x72, 0x6e, 0x61, 0x6c,
0x43, 0x6f, 0x6c, 0x6c, 0x65, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x4d, 0x65, 0x73, 0x73, 0x61, 0x67,
0x65, 0x48, 0x65, 0x61, 0x64, 0x65, 0x72, 0x12, 0x23, 0x0a, 0x0d, 0x63, 0x6f, 0x6c, 0x6c, 0x65,
0x63, 0x74, 0x69, 0x6f, 0x6e, 0x5f, 0x69, 0x64, 0x18, 0x01, 0x20, 0x01, 0x28, 0x03, 0x52, 0x0c,
0x63, 0x6f, 0x6c, 0x6c, 0x65, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x49, 0x64, 0x12, 0x27, 0x0a, 0x0f,
0x63, 0x6f, 0x6c, 0x6c, 0x65, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x5f, 0x6e, 0x61, 0x6d, 0x65, 0x18,
0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x0e, 0x63, 0x6f, 0x6c, 0x6c, 0x65, 0x63, 0x74, 0x69, 0x6f,
0x6e, 0x4e, 0x61, 0x6d, 0x65, 0x12, 0x15, 0x0a, 0x06, 0x6a, 0x6f, 0x62, 0x5f, 0x69, 0x64, 0x18,
0x03, 0x20, 0x01, 0x28, 0x03, 0x52, 0x05, 0x6a, 0x6f, 0x62, 0x49, 0x64, 0x12, 0x27, 0x0a, 0x0f,
0x65, 0x78, 0x74, 0x65, 0x72, 0x6e, 0x61, 0x6c, 0x5f, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x18,
0x04, 0x20, 0x01, 0x28, 0x09, 0x52, 0x0e, 0x65, 0x78, 0x74, 0x65, 0x72, 0x6e, 0x61, 0x6c, 0x53,
0x6f, 0x75, 0x72, 0x63, 0x65, 0x12, 0x23, 0x0a, 0x0d, 0x65, 0x78, 0x74, 0x65, 0x72, 0x6e, 0x61,
0x6c, 0x5f, 0x73, 0x70, 0x65, 0x63, 0x18, 0x05, 0x20, 0x01, 0x28, 0x09, 0x52, 0x0c, 0x65, 0x78,
0x74, 0x65, 0x72, 0x6e, 0x61, 0x6c, 0x53, 0x70, 0x65, 0x63, 0x22, 0x26, 0x0a, 0x24, 0x52, 0x65,
0x66, 0x72, 0x65, 0x73, 0x68, 0x45, 0x78, 0x74, 0x65, 0x72, 0x6e, 0x61, 0x6c, 0x43, 0x6f, 0x6c,
0x6c, 0x65, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x4d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x42, 0x6f,
0x64, 0x79, 0x22, 0x67, 0x0a, 0x10, 0x43, 0x61, 0x63, 0x68, 0x65, 0x45, 0x78, 0x70, 0x69, 0x72,
0x61, 0x74, 0x69, 0x6f, 0x6e, 0x73, 0x12, 0x53, 0x0a, 0x11, 0x63, 0x61, 0x63, 0x68, 0x65, 0x5f,
0x65, 0x78, 0x70, 0x69, 0x72, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x73, 0x18, 0x01, 0x20, 0x03, 0x28,
0x0b, 0x32, 0x26, 0x2e, 0x6d, 0x69, 0x6c, 0x76, 0x75, 0x73, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f,
0x2e, 0x6d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x73, 0x2e, 0x43, 0x61, 0x63, 0x68, 0x65, 0x45,
0x78, 0x70, 0x69, 0x72, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x52, 0x10, 0x63, 0x61, 0x63, 0x68, 0x65,
0x45, 0x78, 0x70, 0x69, 0x72, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x73, 0x22, 0xa0, 0x01, 0x0a, 0x0f,
0x43, 0x61, 0x63, 0x68, 0x65, 0x45, 0x78, 0x70, 0x69, 0x72, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x12,
0x83, 0x01, 0x0a, 0x22, 0x6c, 0x65, 0x67, 0x61, 0x63, 0x79, 0x5f, 0x70, 0x72, 0x6f, 0x78, 0x79,
0x5f, 0x63, 0x6f, 0x6c, 0x6c, 0x65, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x5f, 0x6d, 0x65, 0x74, 0x61,
0x5f, 0x63, 0x61, 0x63, 0x68, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x35, 0x2e, 0x6d,
0x69, 0x6c, 0x76, 0x75, 0x73, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x2e, 0x6d, 0x65, 0x73, 0x73,
0x61, 0x67, 0x65, 0x73, 0x2e, 0x4c, 0x65, 0x67, 0x61, 0x63, 0x79, 0x50, 0x72, 0x6f, 0x78, 0x79,
0x43, 0x6f, 0x6c, 0x6c, 0x65, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x4d, 0x65, 0x74, 0x61, 0x43, 0x61,
0x63, 0x68, 0x65, 0x48, 0x00, 0x52, 0x1e, 0x6c, 0x65, 0x67, 0x61, 0x63, 0x79, 0x50, 0x72, 0x6f,
0x78, 0x79, 0x43, 0x6f, 0x6c, 0x6c, 0x65, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x4d, 0x65, 0x74, 0x61,
0x43, 0x61, 0x63, 0x68, 0x65, 0x42, 0x07, 0x0a, 0x05, 0x63, 0x61, 0x63, 0x68, 0x65, 0x22, 0xe7,
0x01, 0x0a, 0x1e, 0x4c, 0x65, 0x67, 0x61, 0x63, 0x79, 0x50, 0x72, 0x6f, 0x78, 0x79, 0x43, 0x6f,
0x6c, 0x6c, 0x65, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x4d, 0x65, 0x74, 0x61, 0x43, 0x61, 0x63, 0x68,
0x65, 0x12, 0x17, 0x0a, 0x07, 0x64, 0x62, 0x5f, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x01, 0x20, 0x01,
0x28, 0x09, 0x52, 0x06, 0x64, 0x62, 0x4e, 0x61, 0x6d, 0x65, 0x12, 0x27, 0x0a, 0x0f, 0x63, 0x6f,
0x6c, 0x6c, 0x65, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x5f, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x02, 0x20,
0x01, 0x28, 0x09, 0x52, 0x0e, 0x63, 0x6f, 0x6c, 0x6c, 0x65, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x4e,
0x61, 0x6d, 0x65, 0x12, 0x23, 0x0a, 0x0d, 0x63, 0x6f, 0x6c, 0x6c, 0x65, 0x63, 0x74, 0x69, 0x6f,
0x6e, 0x5f, 0x69, 0x64, 0x18, 0x03, 0x20, 0x01, 0x28, 0x03, 0x52, 0x0c, 0x63, 0x6f, 0x6c, 0x6c,
0x65, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x49, 0x64, 0x12, 0x25, 0x0a, 0x0e, 0x70, 0x61, 0x72, 0x74,
0x69, 0x74, 0x69, 0x6f, 0x6e, 0x5f, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x04, 0x20, 0x01, 0x28, 0x09,
0x52, 0x0d, 0x70, 0x61, 0x72, 0x74, 0x69, 0x74, 0x69, 0x6f, 0x6e, 0x4e, 0x61, 0x6d, 0x65, 0x12,
0x37, 0x0a, 0x08, 0x6d, 0x73, 0x67, 0x5f, 0x74, 0x79, 0x70, 0x65, 0x18, 0x05, 0x20, 0x01, 0x28,
0x0e, 0x32, 0x1c, 0x2e, 0x6d, 0x69, 0x6c, 0x76, 0x75, 0x73, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f,
0x2e, 0x63, 0x6f, 0x6d, 0x6d, 0x6f, 0x6e, 0x2e, 0x4d, 0x73, 0x67, 0x54, 0x79, 0x70, 0x65, 0x52,
0x07, 0x6d, 0x73, 0x67, 0x54, 0x79, 0x70, 0x65, 0x22, 0x3b, 0x0a, 0x18, 0x4d, 0x61, 0x6e, 0x75,
0x61, 0x6c, 0x46, 0x6c, 0x75, 0x73, 0x68, 0x45, 0x78, 0x74, 0x72, 0x61, 0x52, 0x65, 0x73, 0x70,
0x6f, 0x6e, 0x73, 0x65, 0x12, 0x1f, 0x0a, 0x0b, 0x73, 0x65, 0x67, 0x6d, 0x65, 0x6e, 0x74, 0x5f,
0x69, 0x64, 0x73, 0x18, 0x01, 0x20, 0x03, 0x28, 0x03, 0x52, 0x0a, 0x73, 0x65, 0x67, 0x6d, 0x65,
0x6e, 0x74, 0x49, 0x64, 0x73, 0x22, 0x17, 0x0a, 0x15, 0x46, 0x6c, 0x75, 0x73, 0x68, 0x41, 0x6c,
0x6c, 0x4d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x48, 0x65, 0x61, 0x64, 0x65, 0x72, 0x22, 0x15,
0x0a, 0x13, 0x46, 0x6c, 0x75, 0x73, 0x68, 0x41, 0x6c, 0x6c, 0x4d, 0x65, 0x73, 0x73, 0x61, 0x67,
0x65, 0x42, 0x6f, 0x64, 0x79, 0x22, 0x5a, 0x0a, 0x0a, 0x54, 0x78, 0x6e, 0x43, 0x6f, 0x6e, 0x74,
0x65, 0x78, 0x74, 0x12, 0x15, 0x0a, 0x06, 0x74, 0x78, 0x6e, 0x5f, 0x69, 0x64, 0x18, 0x01, 0x20,
0x01, 0x28, 0x03, 0x52, 0x05, 0x74, 0x78, 0x6e, 0x49, 0x64, 0x12, 0x35, 0x0a, 0x16, 0x6b, 0x65,
0x65, 0x70, 0x61, 0x6c, 0x69, 0x76, 0x65, 0x5f, 0x6d, 0x69, 0x6c, 0x6c, 0x69, 0x73, 0x65, 0x63,
0x6f, 0x6e, 0x64, 0x73, 0x18, 0x02, 0x20, 0x01, 0x28, 0x03, 0x52, 0x15, 0x6b, 0x65, 0x65, 0x70,
0x61, 0x6c, 0x69, 0x76, 0x65, 0x4d, 0x69, 0x6c, 0x6c, 0x69, 0x73, 0x65, 0x63, 0x6f, 0x6e, 0x64,
0x73, 0x22, 0xc4, 0x01, 0x0a, 0x10, 0x52, 0x4d, 0x51, 0x4d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65,
0x4c, 0x61, 0x79, 0x6f, 0x75, 0x74, 0x12, 0x18, 0x0a, 0x07, 0x70, 0x61, 0x79, 0x6c, 0x6f, 0x61,
0x64, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0c, 0x52, 0x07, 0x70, 0x61, 0x79, 0x6c, 0x6f, 0x61, 0x64,
0x12, 0x57, 0x0a, 0x0a, 0x70, 0x72, 0x6f, 0x70, 0x65, 0x72, 0x74, 0x69, 0x65, 0x73, 0x18, 0x02,
0x20, 0x03, 0x28, 0x0b, 0x32, 0x37, 0x2e, 0x6d, 0x69, 0x6c, 0x76, 0x75, 0x73, 0x2e, 0x70, 0x72,
0x6f, 0x74, 0x6f, 0x2e, 0x6d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x73, 0x2e, 0x52, 0x4d, 0x51,
0x4d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x4c, 0x61, 0x79, 0x6f, 0x75, 0x74, 0x2e, 0x50, 0x72,
0x6f, 0x70, 0x65, 0x72, 0x74, 0x69, 0x65, 0x73, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x52, 0x0a, 0x70,
0x72, 0x6f, 0x70, 0x65, 0x72, 0x74, 0x69, 0x65, 0x73, 0x1a, 0x3d, 0x0a, 0x0f, 0x50, 0x72, 0x6f,
0x70, 0x65, 0x72, 0x74, 0x69, 0x65, 0x73, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x12, 0x10, 0x0a, 0x03,
0x6b, 0x65, 0x79, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x03, 0x6b, 0x65, 0x79, 0x12, 0x14,
0x0a, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x05, 0x76,
0x61, 0x6c, 0x75, 0x65, 0x3a, 0x02, 0x38, 0x01, 0x22, 0xbb, 0x01, 0x0a, 0x0f, 0x42, 0x72, 0x6f,
0x61, 0x64, 0x63, 0x61, 0x73, 0x74, 0x48, 0x65, 0x61, 0x64, 0x65, 0x72, 0x12, 0x21, 0x0a, 0x0c,
0x62, 0x72, 0x6f, 0x61, 0x64, 0x63, 0x61, 0x73, 0x74, 0x5f, 0x69, 0x64, 0x18, 0x01, 0x20, 0x01,
0x28, 0x04, 0x52, 0x0b, 0x62, 0x72, 0x6f, 0x61, 0x64, 0x63, 0x61, 0x73, 0x74, 0x49, 0x64, 0x12,
0x1c, 0x0a, 0x09, 0x76, 0x63, 0x68, 0x61, 0x6e, 0x6e, 0x65, 0x6c, 0x73, 0x18, 0x02, 0x20, 0x03,
0x28, 0x09, 0x52, 0x09, 0x76, 0x63, 0x68, 0x61, 0x6e, 0x6e, 0x65, 0x6c, 0x73, 0x12, 0x47, 0x0a,
0x0d, 0x52, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x5f, 0x6b, 0x65, 0x79, 0x73, 0x18, 0x03,
0x20, 0x03, 0x28, 0x0b, 0x32, 0x22, 0x2e, 0x6d, 0x69, 0x6c, 0x76, 0x75, 0x73, 0x2e, 0x70, 0x72,
0x6f, 0x74, 0x6f, 0x2e, 0x6d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x73, 0x2e, 0x52, 0x65, 0x73,
0x6f, 0x75, 0x72, 0x63, 0x65, 0x4b, 0x65, 0x79, 0x52, 0x0c, 0x52, 0x65, 0x73, 0x6f, 0x75, 0x72,
0x63, 0x65, 0x4b, 0x65, 0x79, 0x73, 0x12, 0x1e, 0x0a, 0x0b, 0x61, 0x63, 0x6b, 0x5f, 0x73, 0x79,
0x6e, 0x63, 0x5f, 0x75, 0x70, 0x18, 0x04, 0x20, 0x01, 0x28, 0x08, 0x52, 0x09, 0x61, 0x63, 0x6b,
0x53, 0x79, 0x6e, 0x63, 0x55, 0x70, 0x22, 0x83, 0x02, 0x0a, 0x0f, 0x52, 0x65, 0x70, 0x6c, 0x69,
0x63, 0x61, 0x74, 0x65, 0x48, 0x65, 0x61, 0x64, 0x65, 0x72, 0x12, 0x1d, 0x0a, 0x0a, 0x63, 0x6c,
0x75, 0x73, 0x74, 0x65, 0x72, 0x5f, 0x69, 0x64, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x09,
0x63, 0x6c, 0x75, 0x73, 0x74, 0x65, 0x72, 0x49, 0x64, 0x12, 0x3d, 0x0a, 0x0a, 0x6d, 0x65, 0x73,
0x73, 0x61, 0x67, 0x65, 0x5f, 0x69, 0x64, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x1e, 0x2e,
0x6d, 0x69, 0x6c, 0x76, 0x75, 0x73, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x2e, 0x63, 0x6f, 0x6d,
0x6d, 0x6f, 0x6e, 0x2e, 0x4d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x49, 0x44, 0x52, 0x09, 0x6d,
0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x49, 0x64, 0x12, 0x59, 0x0a, 0x19, 0x6c, 0x61, 0x73, 0x74,
0x5f, 0x63, 0x6f, 0x6e, 0x66, 0x69, 0x72, 0x6d, 0x65, 0x64, 0x5f, 0x6d, 0x65, 0x73, 0x73, 0x61,
0x67, 0x65, 0x5f, 0x69, 0x64, 0x18, 0x03, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x1e, 0x2e, 0x6d, 0x69,
0x6c, 0x76, 0x75, 0x73, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x2e, 0x63, 0x6f, 0x6d, 0x6d, 0x6f,
0x6e, 0x2e, 0x4d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x49, 0x44, 0x52, 0x16, 0x6c, 0x61, 0x73,
0x74, 0x43, 0x6f, 0x6e, 0x66, 0x69, 0x72, 0x6d, 0x65, 0x64, 0x4d, 0x65, 0x73, 0x73, 0x61, 0x67,
0x65, 0x49, 0x64, 0x12, 0x1b, 0x0a, 0x09, 0x74, 0x69, 0x6d, 0x65, 0x5f, 0x74, 0x69, 0x63, 0x6b,
0x18, 0x04, 0x20, 0x01, 0x28, 0x04, 0x52, 0x08, 0x74, 0x69, 0x6d, 0x65, 0x54, 0x69, 0x63, 0x6b,
0x12, 0x1a, 0x0a, 0x08, 0x76, 0x63, 0x68, 0x61, 0x6e, 0x6e, 0x65, 0x6c, 0x18, 0x05, 0x20, 0x01,
0x28, 0x09, 0x52, 0x08, 0x76, 0x63, 0x68, 0x61, 0x6e, 0x6e, 0x65, 0x6c, 0x22, 0x76, 0x0a, 0x0b,
0x52, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x4b, 0x65, 0x79, 0x12, 0x3d, 0x0a, 0x06, 0x64,
0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0e, 0x32, 0x25, 0x2e, 0x6d, 0x69,
0x6c, 0x76, 0x75, 0x73, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x2e, 0x6d, 0x65, 0x73, 0x73, 0x61,
0x67, 0x65, 0x73, 0x2e, 0x52, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x44, 0x6f, 0x6d, 0x61,
0x69, 0x6e, 0x52, 0x06, 0x64, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x12, 0x10, 0x0a, 0x03, 0x6b, 0x65,
0x79, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x03, 0x6b, 0x65, 0x79, 0x12, 0x16, 0x0a, 0x06,
0x73, 0x68, 0x61, 0x72, 0x65, 0x64, 0x18, 0x03, 0x20, 0x01, 0x28, 0x08, 0x52, 0x06, 0x73, 0x68,
0x61, 0x72, 0x65, 0x64, 0x22, 0x88, 0x01, 0x0a, 0x0c, 0x43, 0x69, 0x70, 0x68, 0x65, 0x72, 0x48,
0x65, 0x61, 0x64, 0x65, 0x72, 0x12, 0x13, 0x0a, 0x05, 0x65, 0x7a, 0x5f, 0x69, 0x64, 0x18, 0x01,
0x20, 0x01, 0x28, 0x03, 0x52, 0x04, 0x65, 0x7a, 0x49, 0x64, 0x12, 0x23, 0x0a, 0x0d, 0x63, 0x6f,
0x6c, 0x6c, 0x65, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x5f, 0x69, 0x64, 0x18, 0x02, 0x20, 0x01, 0x28,
0x03, 0x52, 0x0c, 0x63, 0x6f, 0x6c, 0x6c, 0x65, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x49, 0x64, 0x12,
0x19, 0x0a, 0x08, 0x73, 0x61, 0x66, 0x65, 0x5f, 0x6b, 0x65, 0x79, 0x18, 0x03, 0x20, 0x01, 0x28,
0x0c, 0x52, 0x07, 0x73, 0x61, 0x66, 0x65, 0x4b, 0x65, 0x79, 0x12, 0x23, 0x0a, 0x0d, 0x70, 0x61,
0x79, 0x6c, 0x6f, 0x61, 0x64, 0x5f, 0x62, 0x79, 0x74, 0x65, 0x73, 0x18, 0x04, 0x20, 0x01, 0x28,
0x03, 0x52, 0x0c, 0x70, 0x61, 0x79, 0x6c, 0x6f, 0x61, 0x64, 0x42, 0x79, 0x74, 0x65, 0x73, 0x22,
0x7c, 0x0a, 0x1f, 0x54, 0x72, 0x75, 0x6e, 0x63, 0x61, 0x74, 0x65, 0x43, 0x6f, 0x6c, 0x6c, 0x65,
0x63, 0x74, 0x69, 0x6f, 0x6e, 0x4d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x48, 0x65, 0x61, 0x64,
0x65, 0x72, 0x12, 0x13, 0x0a, 0x05, 0x64, 0x62, 0x5f, 0x69, 0x64, 0x18, 0x01, 0x20, 0x01, 0x28,
0x03, 0x52, 0x04, 0x64, 0x62, 0x49, 0x64, 0x12, 0x23, 0x0a, 0x0d, 0x63, 0x6f, 0x6c, 0x6c, 0x65,
0x63, 0x74, 0x69, 0x6f, 0x6e, 0x5f, 0x69, 0x64, 0x18, 0x02, 0x20, 0x01, 0x28, 0x03, 0x52, 0x0c,
0x63, 0x6f, 0x6c, 0x6c, 0x65, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x49, 0x64, 0x12, 0x1f, 0x0a, 0x0b,
0x73, 0x65, 0x67, 0x6d, 0x65, 0x6e, 0x74, 0x5f, 0x69, 0x64, 0x73, 0x18, 0x03, 0x20, 0x03, 0x28,
0x03, 0x52, 0x0a, 0x73, 0x65, 0x67, 0x6d, 0x65, 0x6e, 0x74, 0x49, 0x64, 0x73, 0x22, 0x1f, 0x0a,
0x1d, 0x54, 0x72, 0x75, 0x6e, 0x63, 0x61, 0x74, 0x65, 0x43, 0x6f, 0x6c, 0x6c, 0x65, 0x63, 0x74,
0x69, 0x6f, 0x6e, 0x4d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x42, 0x6f, 0x64, 0x79, 0x22, 0x47,
0x0a, 0x20, 0x42, 0x61, 0x74, 0x63, 0x68, 0x55, 0x70, 0x64, 0x61, 0x74, 0x65, 0x4d, 0x61, 0x6e,
0x69, 0x66, 0x65, 0x73, 0x74, 0x4d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x48, 0x65, 0x61, 0x64,
0x65, 0x72, 0x12, 0x23, 0x0a, 0x0d, 0x63, 0x6f, 0x6c, 0x6c, 0x65, 0x63, 0x74, 0x69, 0x6f, 0x6e,
0x5f, 0x69, 0x64, 0x18, 0x01, 0x20, 0x01, 0x28, 0x03, 0x52, 0x0c, 0x63, 0x6f, 0x6c, 0x6c, 0x65,
0x63, 0x74, 0x69, 0x6f, 0x6e, 0x49, 0x64, 0x12, 0x27, 0x0a, 0x0f, 0x63, 0x6f, 0x6c, 0x6c, 0x65,
0x63, 0x74, 0x69, 0x6f, 0x6e, 0x5f, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09,
0x52, 0x0e, 0x63, 0x6f, 0x6c, 0x6c, 0x65, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x4e, 0x61, 0x6d, 0x65,
0x12, 0x15, 0x0a, 0x06, 0x6a, 0x6f, 0x62, 0x5f, 0x69, 0x64, 0x18, 0x03, 0x20, 0x01, 0x28, 0x03,
0x52, 0x05, 0x6a, 0x6f, 0x62, 0x49, 0x64, 0x12, 0x27, 0x0a, 0x0f, 0x65, 0x78, 0x74, 0x65, 0x72,
0x6e, 0x61, 0x6c, 0x5f, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x18, 0x04, 0x20, 0x01, 0x28, 0x09,
0x52, 0x0e, 0x65, 0x78, 0x74, 0x65, 0x72, 0x6e, 0x61, 0x6c, 0x53, 0x6f, 0x75, 0x72, 0x63, 0x65,
0x12, 0x23, 0x0a, 0x0d, 0x65, 0x78, 0x74, 0x65, 0x72, 0x6e, 0x61, 0x6c, 0x5f, 0x73, 0x70, 0x65,
0x63, 0x18, 0x05, 0x20, 0x01, 0x28, 0x09, 0x52, 0x0c, 0x65, 0x78, 0x74, 0x65, 0x72, 0x6e, 0x61,
0x6c, 0x53, 0x70, 0x65, 0x63, 0x22, 0x26, 0x0a, 0x24, 0x52, 0x65, 0x66, 0x72, 0x65, 0x73, 0x68,
0x45, 0x78, 0x74, 0x65, 0x72, 0x6e, 0x61, 0x6c, 0x43, 0x6f, 0x6c, 0x6c, 0x65, 0x63, 0x74, 0x69,
0x6f, 0x6e, 0x4d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x42, 0x6f, 0x64, 0x79, 0x22, 0x67, 0x0a,
0x10, 0x43, 0x61, 0x63, 0x68, 0x65, 0x45, 0x78, 0x70, 0x69, 0x72, 0x61, 0x74, 0x69, 0x6f, 0x6e,
0x73, 0x12, 0x53, 0x0a, 0x11, 0x63, 0x61, 0x63, 0x68, 0x65, 0x5f, 0x65, 0x78, 0x70, 0x69, 0x72,
0x61, 0x74, 0x69, 0x6f, 0x6e, 0x73, 0x18, 0x01, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x26, 0x2e, 0x6d,
0x69, 0x6c, 0x76, 0x75, 0x73, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x2e, 0x6d, 0x65, 0x73, 0x73,
0x61, 0x67, 0x65, 0x73, 0x2e, 0x43, 0x61, 0x63, 0x68, 0x65, 0x45, 0x78, 0x70, 0x69, 0x72, 0x61,
0x74, 0x69, 0x6f, 0x6e, 0x52, 0x10, 0x63, 0x61, 0x63, 0x68, 0x65, 0x45, 0x78, 0x70, 0x69, 0x72,
0x61, 0x74, 0x69, 0x6f, 0x6e, 0x73, 0x22, 0xa0, 0x01, 0x0a, 0x0f, 0x43, 0x61, 0x63, 0x68, 0x65,
0x45, 0x78, 0x70, 0x69, 0x72, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x12, 0x83, 0x01, 0x0a, 0x22, 0x6c,
0x65, 0x67, 0x61, 0x63, 0x79, 0x5f, 0x70, 0x72, 0x6f, 0x78, 0x79, 0x5f, 0x63, 0x6f, 0x6c, 0x6c,
0x65, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x5f, 0x6d, 0x65, 0x74, 0x61, 0x5f, 0x63, 0x61, 0x63, 0x68,
0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x35, 0x2e, 0x6d, 0x69, 0x6c, 0x76, 0x75, 0x73,
0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x2e, 0x6d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x73, 0x2e,
0x4c, 0x65, 0x67, 0x61, 0x63, 0x79, 0x50, 0x72, 0x6f, 0x78, 0x79, 0x43, 0x6f, 0x6c, 0x6c, 0x65,
0x63, 0x74, 0x69, 0x6f, 0x6e, 0x4d, 0x65, 0x74, 0x61, 0x43, 0x61, 0x63, 0x68, 0x65, 0x48, 0x00,
0x52, 0x1e, 0x6c, 0x65, 0x67, 0x61, 0x63, 0x79, 0x50, 0x72, 0x6f, 0x78, 0x79, 0x43, 0x6f, 0x6c,
0x6c, 0x65, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x4d, 0x65, 0x74, 0x61, 0x43, 0x61, 0x63, 0x68, 0x65,
0x42, 0x07, 0x0a, 0x05, 0x63, 0x61, 0x63, 0x68, 0x65, 0x22, 0xe7, 0x01, 0x0a, 0x1e, 0x4c, 0x65,
0x67, 0x61, 0x63, 0x79, 0x50, 0x72, 0x6f, 0x78, 0x79, 0x43, 0x6f, 0x6c, 0x6c, 0x65, 0x63, 0x74,
0x69, 0x6f, 0x6e, 0x4d, 0x65, 0x74, 0x61, 0x43, 0x61, 0x63, 0x68, 0x65, 0x12, 0x17, 0x0a, 0x07,
0x64, 0x62, 0x5f, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x06, 0x64,
0x62, 0x4e, 0x61, 0x6d, 0x65, 0x12, 0x27, 0x0a, 0x0f, 0x63, 0x6f, 0x6c, 0x6c, 0x65, 0x63, 0x74,
0x69, 0x6f, 0x6e, 0x5f, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x0e,
0x63, 0x6f, 0x6c, 0x6c, 0x65, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x4e, 0x61, 0x6d, 0x65, 0x12, 0x23,
0x0a, 0x0d, 0x63, 0x6f, 0x6c, 0x6c, 0x65, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x5f, 0x69, 0x64, 0x18,
0x03, 0x20, 0x01, 0x28, 0x03, 0x52, 0x0c, 0x63, 0x6f, 0x6c, 0x6c, 0x65, 0x63, 0x74, 0x69, 0x6f,
0x6e, 0x49, 0x64, 0x12, 0x25, 0x0a, 0x0e, 0x70, 0x61, 0x72, 0x74, 0x69, 0x74, 0x69, 0x6f, 0x6e,
0x5f, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x04, 0x20, 0x01, 0x28, 0x09, 0x52, 0x0d, 0x70, 0x61, 0x72,
0x74, 0x69, 0x74, 0x69, 0x6f, 0x6e, 0x4e, 0x61, 0x6d, 0x65, 0x12, 0x37, 0x0a, 0x08, 0x6d, 0x73,
0x67, 0x5f, 0x74, 0x79, 0x70, 0x65, 0x18, 0x05, 0x20, 0x01, 0x28, 0x0e, 0x32, 0x1c, 0x2e, 0x6d,
0x69, 0x6c, 0x76, 0x75, 0x73, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x2e, 0x63, 0x6f, 0x6d, 0x6d,
0x6f, 0x6e, 0x2e, 0x4d, 0x73, 0x67, 0x54, 0x79, 0x70, 0x65, 0x52, 0x07, 0x6d, 0x73, 0x67, 0x54,
0x79, 0x70, 0x65, 0x22, 0x3b, 0x0a, 0x18, 0x4d, 0x61, 0x6e, 0x75, 0x61, 0x6c, 0x46, 0x6c, 0x75,
0x73, 0x68, 0x45, 0x78, 0x74, 0x72, 0x61, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12,
0x1f, 0x0a, 0x0b, 0x73, 0x65, 0x67, 0x6d, 0x65, 0x6e, 0x74, 0x5f, 0x69, 0x64, 0x73, 0x18, 0x01,
0x20, 0x03, 0x28, 0x03, 0x52, 0x0a, 0x73, 0x65, 0x67, 0x6d, 0x65, 0x6e, 0x74, 0x49, 0x64, 0x73,
0x22, 0x17, 0x0a, 0x15, 0x46, 0x6c, 0x75, 0x73, 0x68, 0x41, 0x6c, 0x6c, 0x4d, 0x65, 0x73, 0x73,
0x61, 0x67, 0x65, 0x48, 0x65, 0x61, 0x64, 0x65, 0x72, 0x22, 0x15, 0x0a, 0x13, 0x46, 0x6c, 0x75,
0x73, 0x68, 0x41, 0x6c, 0x6c, 0x4d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x42, 0x6f, 0x64, 0x79,
0x22, 0x5a, 0x0a, 0x0a, 0x54, 0x78, 0x6e, 0x43, 0x6f, 0x6e, 0x74, 0x65, 0x78, 0x74, 0x12, 0x15,
0x0a, 0x06, 0x74, 0x78, 0x6e, 0x5f, 0x69, 0x64, 0x18, 0x01, 0x20, 0x01, 0x28, 0x03, 0x52, 0x05,
0x74, 0x78, 0x6e, 0x49, 0x64, 0x12, 0x35, 0x0a, 0x16, 0x6b, 0x65, 0x65, 0x70, 0x61, 0x6c, 0x69,
0x76, 0x65, 0x5f, 0x6d, 0x69, 0x6c, 0x6c, 0x69, 0x73, 0x65, 0x63, 0x6f, 0x6e, 0x64, 0x73, 0x18,
0x02, 0x20, 0x01, 0x28, 0x03, 0x52, 0x15, 0x6b, 0x65, 0x65, 0x70, 0x61, 0x6c, 0x69, 0x76, 0x65,
0x4d, 0x69, 0x6c, 0x6c, 0x69, 0x73, 0x65, 0x63, 0x6f, 0x6e, 0x64, 0x73, 0x22, 0xc4, 0x01, 0x0a,
0x10, 0x52, 0x4d, 0x51, 0x4d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x4c, 0x61, 0x79, 0x6f, 0x75,
0x74, 0x12, 0x18, 0x0a, 0x07, 0x70, 0x61, 0x79, 0x6c, 0x6f, 0x61, 0x64, 0x18, 0x01, 0x20, 0x01,
0x28, 0x0c, 0x52, 0x07, 0x70, 0x61, 0x79, 0x6c, 0x6f, 0x61, 0x64, 0x12, 0x57, 0x0a, 0x0a, 0x70,
0x72, 0x6f, 0x70, 0x65, 0x72, 0x74, 0x69, 0x65, 0x73, 0x18, 0x02, 0x20, 0x03, 0x28, 0x0b, 0x32,
0x37, 0x2e, 0x6d, 0x69, 0x6c, 0x76, 0x75, 0x73, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x2e, 0x6d,
0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x73, 0x2e, 0x52, 0x4d, 0x51, 0x4d, 0x65, 0x73, 0x73, 0x61,
0x67, 0x65, 0x4c, 0x61, 0x79, 0x6f, 0x75, 0x74, 0x2e, 0x50, 0x72, 0x6f, 0x70, 0x65, 0x72, 0x74,
0x69, 0x65, 0x73, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x52, 0x0a, 0x70, 0x72, 0x6f, 0x70, 0x65, 0x72,
0x74, 0x69, 0x65, 0x73, 0x1a, 0x3d, 0x0a, 0x0f, 0x50, 0x72, 0x6f, 0x70, 0x65, 0x72, 0x74, 0x69,
0x65, 0x73, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x12, 0x10, 0x0a, 0x03, 0x6b, 0x65, 0x79, 0x18, 0x01,
0x20, 0x01, 0x28, 0x09, 0x52, 0x03, 0x6b, 0x65, 0x79, 0x12, 0x14, 0x0a, 0x05, 0x76, 0x61, 0x6c,
0x75, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x3a,
0x02, 0x38, 0x01, 0x22, 0xbb, 0x01, 0x0a, 0x0f, 0x42, 0x72, 0x6f, 0x61, 0x64, 0x63, 0x61, 0x73,
0x74, 0x48, 0x65, 0x61, 0x64, 0x65, 0x72, 0x12, 0x21, 0x0a, 0x0c, 0x62, 0x72, 0x6f, 0x61, 0x64,
0x63, 0x61, 0x73, 0x74, 0x5f, 0x69, 0x64, 0x18, 0x01, 0x20, 0x01, 0x28, 0x04, 0x52, 0x0b, 0x62,
0x72, 0x6f, 0x61, 0x64, 0x63, 0x61, 0x73, 0x74, 0x49, 0x64, 0x12, 0x1c, 0x0a, 0x09, 0x76, 0x63,
0x68, 0x61, 0x6e, 0x6e, 0x65, 0x6c, 0x73, 0x18, 0x02, 0x20, 0x03, 0x28, 0x09, 0x52, 0x09, 0x76,
0x63, 0x68, 0x61, 0x6e, 0x6e, 0x65, 0x6c, 0x73, 0x12, 0x47, 0x0a, 0x0d, 0x52, 0x65, 0x73, 0x6f,
0x75, 0x72, 0x63, 0x65, 0x5f, 0x6b, 0x65, 0x79, 0x73, 0x18, 0x03, 0x20, 0x03, 0x28, 0x0b, 0x32,
0x22, 0x2e, 0x6d, 0x69, 0x6c, 0x76, 0x75, 0x73, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x2e, 0x6d,
0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x73, 0x2e, 0x52, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65,
0x4b, 0x65, 0x79, 0x52, 0x0c, 0x52, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x4b, 0x65, 0x79,
0x73, 0x12, 0x1e, 0x0a, 0x0b, 0x61, 0x63, 0x6b, 0x5f, 0x73, 0x79, 0x6e, 0x63, 0x5f, 0x75, 0x70,
0x18, 0x04, 0x20, 0x01, 0x28, 0x08, 0x52, 0x09, 0x61, 0x63, 0x6b, 0x53, 0x79, 0x6e, 0x63, 0x55,
0x70, 0x22, 0x83, 0x02, 0x0a, 0x0f, 0x52, 0x65, 0x70, 0x6c, 0x69, 0x63, 0x61, 0x74, 0x65, 0x48,
0x65, 0x61, 0x64, 0x65, 0x72, 0x12, 0x1d, 0x0a, 0x0a, 0x63, 0x6c, 0x75, 0x73, 0x74, 0x65, 0x72,
0x5f, 0x69, 0x64, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x09, 0x63, 0x6c, 0x75, 0x73, 0x74,
0x65, 0x72, 0x49, 0x64, 0x12, 0x3d, 0x0a, 0x0a, 0x6d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x5f,
0x69, 0x64, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x1e, 0x2e, 0x6d, 0x69, 0x6c, 0x76, 0x75,
0x73, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x2e, 0x63, 0x6f, 0x6d, 0x6d, 0x6f, 0x6e, 0x2e, 0x4d,
0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x49, 0x44, 0x52, 0x09, 0x6d, 0x65, 0x73, 0x73, 0x61, 0x67,
0x65, 0x49, 0x64, 0x12, 0x59, 0x0a, 0x19, 0x6c, 0x61, 0x73, 0x74, 0x5f, 0x63, 0x6f, 0x6e, 0x66,
0x69, 0x72, 0x6d, 0x65, 0x64, 0x5f, 0x6d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x5f, 0x69, 0x64,
0x18, 0x03, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x1e, 0x2e, 0x6d, 0x69, 0x6c, 0x76, 0x75, 0x73, 0x2e,
0x70, 0x72, 0x6f, 0x74, 0x6f, 0x2e, 0x63, 0x6f, 0x6d, 0x6d, 0x6f, 0x6e, 0x2e, 0x4d, 0x65, 0x73,
0x73, 0x61, 0x67, 0x65, 0x49, 0x44, 0x52, 0x16, 0x6c, 0x61, 0x73, 0x74, 0x43, 0x6f, 0x6e, 0x66,
0x69, 0x72, 0x6d, 0x65, 0x64, 0x4d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x49, 0x64, 0x12, 0x1b,
0x0a, 0x09, 0x74, 0x69, 0x6d, 0x65, 0x5f, 0x74, 0x69, 0x63, 0x6b, 0x18, 0x04, 0x20, 0x01, 0x28,
0x04, 0x52, 0x08, 0x74, 0x69, 0x6d, 0x65, 0x54, 0x69, 0x63, 0x6b, 0x12, 0x1a, 0x0a, 0x08, 0x76,
0x63, 0x68, 0x61, 0x6e, 0x6e, 0x65, 0x6c, 0x18, 0x05, 0x20, 0x01, 0x28, 0x09, 0x52, 0x08, 0x76,
0x63, 0x68, 0x61, 0x6e, 0x6e, 0x65, 0x6c, 0x22, 0x76, 0x0a, 0x0b, 0x52, 0x65, 0x73, 0x6f, 0x75,
0x72, 0x63, 0x65, 0x4b, 0x65, 0x79, 0x12, 0x3d, 0x0a, 0x06, 0x64, 0x6f, 0x6d, 0x61, 0x69, 0x6e,
0x18, 0x01, 0x20, 0x01, 0x28, 0x0e, 0x32, 0x25, 0x2e, 0x6d, 0x69, 0x6c, 0x76, 0x75, 0x73, 0x2e,
0x70, 0x72, 0x6f, 0x74, 0x6f, 0x2e, 0x6d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x73, 0x2e, 0x52,
0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x52, 0x06, 0x64,
0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x12, 0x10, 0x0a, 0x03, 0x6b, 0x65, 0x79, 0x18, 0x02, 0x20, 0x01,
0x28, 0x09, 0x52, 0x03, 0x6b, 0x65, 0x79, 0x12, 0x16, 0x0a, 0x06, 0x73, 0x68, 0x61, 0x72, 0x65,
0x64, 0x18, 0x03, 0x20, 0x01, 0x28, 0x08, 0x52, 0x06, 0x73, 0x68, 0x61, 0x72, 0x65, 0x64, 0x22,
0x88, 0x01, 0x0a, 0x0c, 0x43, 0x69, 0x70, 0x68, 0x65, 0x72, 0x48, 0x65, 0x61, 0x64, 0x65, 0x72,
0x12, 0x13, 0x0a, 0x05, 0x65, 0x7a, 0x5f, 0x69, 0x64, 0x18, 0x01, 0x20, 0x01, 0x28, 0x03, 0x52,
0x04, 0x65, 0x7a, 0x49, 0x64, 0x12, 0x23, 0x0a, 0x0d, 0x63, 0x6f, 0x6c, 0x6c, 0x65, 0x63, 0x74,
0x69, 0x6f, 0x6e, 0x5f, 0x69, 0x64, 0x18, 0x02, 0x20, 0x01, 0x28, 0x03, 0x52, 0x0c, 0x63, 0x6f,
0x6c, 0x6c, 0x65, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x49, 0x64, 0x12, 0x19, 0x0a, 0x08, 0x73, 0x61,
0x66, 0x65, 0x5f, 0x6b, 0x65, 0x79, 0x18, 0x03, 0x20, 0x01, 0x28, 0x0c, 0x52, 0x07, 0x73, 0x61,
0x66, 0x65, 0x4b, 0x65, 0x79, 0x12, 0x23, 0x0a, 0x0d, 0x70, 0x61, 0x79, 0x6c, 0x6f, 0x61, 0x64,
0x5f, 0x62, 0x79, 0x74, 0x65, 0x73, 0x18, 0x04, 0x20, 0x01, 0x28, 0x03, 0x52, 0x0c, 0x70, 0x61,
0x79, 0x6c, 0x6f, 0x61, 0x64, 0x42, 0x79, 0x74, 0x65, 0x73, 0x22, 0x7c, 0x0a, 0x1f, 0x54, 0x72,
0x75, 0x6e, 0x63, 0x61, 0x74, 0x65, 0x43, 0x6f, 0x6c, 0x6c, 0x65, 0x63, 0x74, 0x69, 0x6f, 0x6e,
0x4d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x48, 0x65, 0x61, 0x64, 0x65, 0x72, 0x12, 0x13, 0x0a,
0x05, 0x64, 0x62, 0x5f, 0x69, 0x64, 0x18, 0x01, 0x20, 0x01, 0x28, 0x03, 0x52, 0x04, 0x64, 0x62,
0x49, 0x64, 0x12, 0x23, 0x0a, 0x0d, 0x63, 0x6f, 0x6c, 0x6c, 0x65, 0x63, 0x74, 0x69, 0x6f, 0x6e,
0x5f, 0x69, 0x64, 0x18, 0x02, 0x20, 0x01, 0x28, 0x03, 0x52, 0x0c, 0x63, 0x6f, 0x6c, 0x6c, 0x65,
0x63, 0x74, 0x69, 0x6f, 0x6e, 0x49, 0x64, 0x12, 0x1f, 0x0a, 0x0b, 0x73, 0x65, 0x67, 0x6d, 0x65,
0x6e, 0x74, 0x5f, 0x69, 0x64, 0x73, 0x18, 0x03, 0x20, 0x03, 0x28, 0x03, 0x52, 0x0a, 0x73, 0x65,
0x67, 0x6d, 0x65, 0x6e, 0x74, 0x49, 0x64, 0x73, 0x22, 0x1f, 0x0a, 0x1d, 0x54, 0x72, 0x75, 0x6e,
0x63, 0x61, 0x74, 0x65, 0x43, 0x6f, 0x6c, 0x6c, 0x65, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x4d, 0x65,
0x73, 0x73, 0x61, 0x67, 0x65, 0x42, 0x6f, 0x64, 0x79, 0x22, 0x47, 0x0a, 0x20, 0x42, 0x61, 0x74,
0x63, 0x68, 0x55, 0x70, 0x64, 0x61, 0x74, 0x65, 0x4d, 0x61, 0x6e, 0x69, 0x66, 0x65, 0x73, 0x74,
0x4d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x48, 0x65, 0x61, 0x64, 0x65, 0x72, 0x12, 0x23, 0x0a,
0x0d, 0x63, 0x6f, 0x6c, 0x6c, 0x65, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x5f, 0x69, 0x64, 0x18, 0x01,
0x20, 0x01, 0x28, 0x03, 0x52, 0x0c, 0x63, 0x6f, 0x6c, 0x6c, 0x65, 0x63, 0x74, 0x69, 0x6f, 0x6e,
0x49, 0x64, 0x22, 0x66, 0x0a, 0x1e, 0x42, 0x61, 0x74, 0x63, 0x68, 0x55, 0x70, 0x64, 0x61, 0x74,
0x65, 0x4d, 0x61, 0x6e, 0x69, 0x66, 0x65, 0x73, 0x74, 0x4d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65,
0x42, 0x6f, 0x64, 0x79, 0x12, 0x44, 0x0a, 0x05, 0x69, 0x74, 0x65, 0x6d, 0x73, 0x18, 0x01, 0x20,
0x03, 0x28, 0x0b, 0x32, 0x2e, 0x2e, 0x6d, 0x69, 0x6c, 0x76, 0x75, 0x73, 0x2e, 0x70, 0x72, 0x6f,
0x74, 0x6f, 0x2e, 0x6d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x73, 0x2e, 0x42, 0x61, 0x74, 0x63,
0x68, 0x55, 0x70, 0x64, 0x61, 0x74, 0x65, 0x4d, 0x61, 0x6e, 0x69, 0x66, 0x65, 0x73, 0x74, 0x49,
0x74, 0x65, 0x6d, 0x52, 0x05, 0x69, 0x74, 0x65, 0x6d, 0x73, 0x22, 0x63, 0x0a, 0x17, 0x42, 0x61,
0x74, 0x63, 0x68, 0x55, 0x70, 0x64, 0x61, 0x74, 0x65, 0x4d, 0x61, 0x6e, 0x69, 0x66, 0x65, 0x73,
0x74, 0x49, 0x74, 0x65, 0x6d, 0x12, 0x1d, 0x0a, 0x0a, 0x73, 0x65, 0x67, 0x6d, 0x65, 0x6e, 0x74,
0x5f, 0x69, 0x64, 0x18, 0x01, 0x20, 0x01, 0x28, 0x03, 0x52, 0x09, 0x73, 0x65, 0x67, 0x6d, 0x65,
0x6e, 0x74, 0x49, 0x64, 0x12, 0x29, 0x0a, 0x10, 0x6d, 0x61, 0x6e, 0x69, 0x66, 0x65, 0x73, 0x74,
0x5f, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x18, 0x02, 0x20, 0x01, 0x28, 0x03, 0x52, 0x0f,
0x6d, 0x61, 0x6e, 0x69, 0x66, 0x65, 0x73, 0x74, 0x56, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x2a,
0x9e, 0x07, 0x0a, 0x0b, 0x4d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x54, 0x79, 0x70, 0x65, 0x12,
0x0b, 0x0a, 0x07, 0x55, 0x6e, 0x6b, 0x6e, 0x6f, 0x77, 0x6e, 0x10, 0x00, 0x12, 0x0c, 0x0a, 0x08,
0x54, 0x69, 0x6d, 0x65, 0x54, 0x69, 0x63, 0x6b, 0x10, 0x01, 0x12, 0x0a, 0x0a, 0x06, 0x49, 0x6e,
0x73, 0x65, 0x72, 0x74, 0x10, 0x02, 0x12, 0x0a, 0x0a, 0x06, 0x44, 0x65, 0x6c, 0x65, 0x74, 0x65,
0x10, 0x03, 0x12, 0x09, 0x0a, 0x05, 0x46, 0x6c, 0x75, 0x73, 0x68, 0x10, 0x04, 0x12, 0x14, 0x0a,
0x10, 0x43, 0x72, 0x65, 0x61, 0x74, 0x65, 0x43, 0x6f, 0x6c, 0x6c, 0x65, 0x63, 0x74, 0x69, 0x6f,
0x6e, 0x10, 0x05, 0x12, 0x12, 0x0a, 0x0e, 0x44, 0x72, 0x6f, 0x70, 0x43, 0x6f, 0x6c, 0x6c, 0x65,
0x63, 0x74, 0x69, 0x6f, 0x6e, 0x10, 0x06, 0x12, 0x13, 0x0a, 0x0f, 0x43, 0x72, 0x65, 0x61, 0x74,
0x65, 0x50, 0x61, 0x72, 0x74, 0x69, 0x74, 0x69, 0x6f, 0x6e, 0x10, 0x07, 0x12, 0x11, 0x0a, 0x0d,
0x44, 0x72, 0x6f, 0x70, 0x50, 0x61, 0x72, 0x74, 0x69, 0x74, 0x69, 0x6f, 0x6e, 0x10, 0x08, 0x12,
0x0f, 0x0a, 0x0b, 0x4d, 0x61, 0x6e, 0x75, 0x61, 0x6c, 0x46, 0x6c, 0x75, 0x73, 0x68, 0x10, 0x09,
0x12, 0x11, 0x0a, 0x0d, 0x43, 0x72, 0x65, 0x61, 0x74, 0x65, 0x53, 0x65, 0x67, 0x6d, 0x65, 0x6e,
0x74, 0x10, 0x0a, 0x12, 0x0a, 0x0a, 0x06, 0x49, 0x6d, 0x70, 0x6f, 0x72, 0x74, 0x10, 0x0b, 0x12,
0x14, 0x0a, 0x0c, 0x53, 0x63, 0x68, 0x65, 0x6d, 0x61, 0x43, 0x68, 0x61, 0x6e, 0x67, 0x65, 0x10,
0x0c, 0x1a, 0x02, 0x08, 0x01, 0x12, 0x13, 0x0a, 0x0f, 0x41, 0x6c, 0x74, 0x65, 0x72, 0x43, 0x6f,
0x6c, 0x6c, 0x65, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x10, 0x0d, 0x12, 0x13, 0x0a, 0x0f, 0x41, 0x6c,
0x74, 0x65, 0x72, 0x4c, 0x6f, 0x61, 0x64, 0x43, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x10, 0x0e, 0x12,
0x12, 0x0a, 0x0e, 0x44, 0x72, 0x6f, 0x70, 0x4c, 0x6f, 0x61, 0x64, 0x43, 0x6f, 0x6e, 0x66, 0x69,
0x67, 0x10, 0x0f, 0x12, 0x12, 0x0a, 0x0e, 0x43, 0x72, 0x65, 0x61, 0x74, 0x65, 0x44, 0x61, 0x74,
0x61, 0x62, 0x61, 0x73, 0x65, 0x10, 0x10, 0x12, 0x11, 0x0a, 0x0d, 0x41, 0x6c, 0x74, 0x65, 0x72,
0x44, 0x61, 0x74, 0x61, 0x62, 0x61, 0x73, 0x65, 0x10, 0x11, 0x12, 0x10, 0x0a, 0x0c, 0x44, 0x72,
0x6f, 0x70, 0x44, 0x61, 0x74, 0x61, 0x62, 0x61, 0x73, 0x65, 0x10, 0x12, 0x12, 0x0e, 0x0a, 0x0a,
0x41, 0x6c, 0x74, 0x65, 0x72, 0x41, 0x6c, 0x69, 0x61, 0x73, 0x10, 0x13, 0x12, 0x0d, 0x0a, 0x09,
0x44, 0x72, 0x6f, 0x70, 0x41, 0x6c, 0x69, 0x61, 0x73, 0x10, 0x14, 0x12, 0x0f, 0x0a, 0x0b, 0x52,
0x65, 0x73, 0x74, 0x6f, 0x72, 0x65, 0x52, 0x42, 0x41, 0x43, 0x10, 0x15, 0x12, 0x0d, 0x0a, 0x09,
0x41, 0x6c, 0x74, 0x65, 0x72, 0x55, 0x73, 0x65, 0x72, 0x10, 0x16, 0x12, 0x0c, 0x0a, 0x08, 0x44,
0x72, 0x6f, 0x70, 0x55, 0x73, 0x65, 0x72, 0x10, 0x17, 0x12, 0x0d, 0x0a, 0x09, 0x41, 0x6c, 0x74,
0x65, 0x72, 0x52, 0x6f, 0x6c, 0x65, 0x10, 0x18, 0x12, 0x0c, 0x0a, 0x08, 0x44, 0x72, 0x6f, 0x70,
0x52, 0x6f, 0x6c, 0x65, 0x10, 0x19, 0x12, 0x11, 0x0a, 0x0d, 0x41, 0x6c, 0x74, 0x65, 0x72, 0x55,
0x73, 0x65, 0x72, 0x52, 0x6f, 0x6c, 0x65, 0x10, 0x1a, 0x12, 0x10, 0x0a, 0x0c, 0x44, 0x72, 0x6f,
0x70, 0x55, 0x73, 0x65, 0x72, 0x52, 0x6f, 0x6c, 0x65, 0x10, 0x1b, 0x12, 0x12, 0x0a, 0x0e, 0x41,
0x6c, 0x74, 0x65, 0x72, 0x50, 0x72, 0x69, 0x76, 0x69, 0x6c, 0x65, 0x67, 0x65, 0x10, 0x1c, 0x12,
0x11, 0x0a, 0x0d, 0x44, 0x72, 0x6f, 0x70, 0x50, 0x72, 0x69, 0x76, 0x69, 0x6c, 0x65, 0x67, 0x65,
0x10, 0x1d, 0x12, 0x17, 0x0a, 0x13, 0x41, 0x6c, 0x74, 0x65, 0x72, 0x50, 0x72, 0x69, 0x76, 0x69,
0x6c, 0x65, 0x67, 0x65, 0x47, 0x72, 0x6f, 0x75, 0x70, 0x10, 0x1e, 0x12, 0x16, 0x0a, 0x12, 0x44,
0x72, 0x6f, 0x70, 0x50, 0x72, 0x69, 0x76, 0x69, 0x6c, 0x65, 0x67, 0x65, 0x47, 0x72, 0x6f, 0x75,
0x70, 0x10, 0x1f, 0x12, 0x16, 0x0a, 0x12, 0x41, 0x6c, 0x74, 0x65, 0x72, 0x52, 0x65, 0x73, 0x6f,
0x75, 0x72, 0x63, 0x65, 0x47, 0x72, 0x6f, 0x75, 0x70, 0x10, 0x20, 0x12, 0x15, 0x0a, 0x11, 0x44,
0x72, 0x6f, 0x70, 0x52, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x47, 0x72, 0x6f, 0x75, 0x70,
0x10, 0x21, 0x12, 0x0f, 0x0a, 0x0b, 0x43, 0x72, 0x65, 0x61, 0x74, 0x65, 0x49, 0x6e, 0x64, 0x65,
0x78, 0x10, 0x22, 0x12, 0x0e, 0x0a, 0x0a, 0x41, 0x6c, 0x74, 0x65, 0x72, 0x49, 0x6e, 0x64, 0x65,
0x78, 0x10, 0x23, 0x12, 0x0d, 0x0a, 0x09, 0x44, 0x72, 0x6f, 0x70, 0x49, 0x6e, 0x64, 0x65, 0x78,
0x10, 0x24, 0x12, 0x0c, 0x0a, 0x08, 0x46, 0x6c, 0x75, 0x73, 0x68, 0x41, 0x6c, 0x6c, 0x10, 0x25,
0x12, 0x16, 0x0a, 0x12, 0x54, 0x72, 0x75, 0x6e, 0x63, 0x61, 0x74, 0x65, 0x43, 0x6f, 0x6c, 0x6c,
0x65, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x10, 0x26, 0x12, 0x13, 0x0a, 0x0f, 0x52, 0x65, 0x73, 0x74,
0x6f, 0x72, 0x65, 0x53, 0x6e, 0x61, 0x70, 0x73, 0x68, 0x6f, 0x74, 0x10, 0x27, 0x12, 0x12, 0x0a,
0x0e, 0x43, 0x72, 0x65, 0x61, 0x74, 0x65, 0x53, 0x6e, 0x61, 0x70, 0x73, 0x68, 0x6f, 0x74, 0x10,
0x28, 0x12, 0x10, 0x0a, 0x0c, 0x44, 0x72, 0x6f, 0x70, 0x53, 0x6e, 0x61, 0x70, 0x73, 0x68, 0x6f,
0x74, 0x10, 0x29, 0x12, 0x17, 0x0a, 0x13, 0x42, 0x61, 0x74, 0x63, 0x68, 0x55, 0x70, 0x64, 0x61,
0x74, 0x65, 0x4d, 0x61, 0x6e, 0x69, 0x66, 0x65, 0x73, 0x74, 0x10, 0x2a, 0x12, 0x1d, 0x0a, 0x19,
0x52, 0x65, 0x66, 0x72, 0x65, 0x73, 0x68, 0x45, 0x78, 0x74, 0x65, 0x72, 0x6e, 0x61, 0x6c, 0x43,
0x6f, 0x6c, 0x6c, 0x65, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x10, 0x2b, 0x12, 0x0d, 0x0a, 0x08, 0x41,
0x6c, 0x74, 0x65, 0x72, 0x57, 0x41, 0x4c, 0x10, 0xbc, 0x05, 0x12, 0x19, 0x0a, 0x14, 0x41, 0x6c,
0x74, 0x65, 0x72, 0x52, 0x65, 0x70, 0x6c, 0x69, 0x63, 0x61, 0x74, 0x65, 0x43, 0x6f, 0x6e, 0x66,
0x69, 0x67, 0x10, 0xa0, 0x06, 0x12, 0x0d, 0x0a, 0x08, 0x42, 0x65, 0x67, 0x69, 0x6e, 0x54, 0x78,
0x6e, 0x10, 0x84, 0x07, 0x12, 0x0e, 0x0a, 0x09, 0x43, 0x6f, 0x6d, 0x6d, 0x69, 0x74, 0x54, 0x78,
0x6e, 0x10, 0x85, 0x07, 0x12, 0x10, 0x0a, 0x0b, 0x52, 0x6f, 0x6c, 0x6c, 0x62, 0x61, 0x63, 0x6b,
0x54, 0x78, 0x6e, 0x10, 0x86, 0x07, 0x12, 0x08, 0x0a, 0x03, 0x54, 0x78, 0x6e, 0x10, 0xe7, 0x07,
0x2a, 0x74, 0x0a, 0x08, 0x54, 0x78, 0x6e, 0x53, 0x74, 0x61, 0x74, 0x65, 0x12, 0x0e, 0x0a, 0x0a,
0x54, 0x78, 0x6e, 0x55, 0x6e, 0x6b, 0x6e, 0x6f, 0x77, 0x6e, 0x10, 0x00, 0x12, 0x0f, 0x0a, 0x0b,
0x54, 0x78, 0x6e, 0x49, 0x6e, 0x46, 0x6c, 0x69, 0x67, 0x68, 0x74, 0x10, 0x01, 0x12, 0x0f, 0x0a,
0x0b, 0x54, 0x78, 0x6e, 0x4f, 0x6e, 0x43, 0x6f, 0x6d, 0x6d, 0x69, 0x74, 0x10, 0x02, 0x12, 0x10,
0x0a, 0x0c, 0x54, 0x78, 0x6e, 0x43, 0x6f, 0x6d, 0x6d, 0x69, 0x74, 0x74, 0x65, 0x64, 0x10, 0x03,
0x12, 0x11, 0x0a, 0x0d, 0x54, 0x78, 0x6e, 0x4f, 0x6e, 0x52, 0x6f, 0x6c, 0x6c, 0x62, 0x61, 0x63,
0x6b, 0x10, 0x04, 0x12, 0x11, 0x0a, 0x0d, 0x54, 0x78, 0x6e, 0x52, 0x6f, 0x6c, 0x6c, 0x62, 0x61,
0x63, 0x6b, 0x65, 0x64, 0x10, 0x05, 0x2a, 0xe2, 0x01, 0x0a, 0x0e, 0x52, 0x65, 0x73, 0x6f, 0x75,
0x72, 0x63, 0x65, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x12, 0x19, 0x0a, 0x15, 0x52, 0x65, 0x73,
0x6f, 0x75, 0x72, 0x63, 0x65, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x55, 0x6e, 0x6b, 0x6e, 0x6f,
0x77, 0x6e, 0x10, 0x00, 0x12, 0x21, 0x0a, 0x19, 0x52, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65,
0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x49, 0x6d, 0x70, 0x6f, 0x72, 0x74, 0x4a, 0x6f, 0x62, 0x49,
0x44, 0x10, 0x01, 0x1a, 0x02, 0x08, 0x01, 0x12, 0x20, 0x0a, 0x1c, 0x52, 0x65, 0x73, 0x6f, 0x75,
0x72, 0x63, 0x65, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x43, 0x6f, 0x6c, 0x6c, 0x65, 0x63, 0x74,
0x69, 0x6f, 0x6e, 0x4e, 0x61, 0x6d, 0x65, 0x10, 0x02, 0x12, 0x18, 0x0a, 0x14, 0x52, 0x65, 0x73,
0x6f, 0x75, 0x72, 0x63, 0x65, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x44, 0x42, 0x4e, 0x61, 0x6d,
0x65, 0x10, 0x03, 0x12, 0x1b, 0x0a, 0x17, 0x52, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x44,
0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x50, 0x72, 0x69, 0x76, 0x69, 0x6c, 0x65, 0x67, 0x65, 0x10, 0x04,
0x12, 0x1e, 0x0a, 0x1a, 0x52, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x44, 0x6f, 0x6d, 0x61,
0x69, 0x6e, 0x53, 0x6e, 0x61, 0x70, 0x73, 0x68, 0x6f, 0x74, 0x4e, 0x61, 0x6d, 0x65, 0x10, 0x05,
0x12, 0x19, 0x0a, 0x15, 0x52, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x44, 0x6f, 0x6d, 0x61,
0x69, 0x6e, 0x43, 0x6c, 0x75, 0x73, 0x74, 0x65, 0x72, 0x10, 0x7f, 0x42, 0x35, 0x5a, 0x33, 0x67,
0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x6d, 0x69, 0x6c, 0x76, 0x75, 0x73,
0x2d, 0x69, 0x6f, 0x2f, 0x6d, 0x69, 0x6c, 0x76, 0x75, 0x73, 0x2f, 0x70, 0x6b, 0x67, 0x2f, 0x76,
0x32, 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x2f, 0x6d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x73,
0x70, 0x62, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33,
0x63, 0x74, 0x69, 0x6f, 0x6e, 0x49, 0x64, 0x22, 0x66, 0x0a, 0x1e, 0x42, 0x61, 0x74, 0x63, 0x68,
0x55, 0x70, 0x64, 0x61, 0x74, 0x65, 0x4d, 0x61, 0x6e, 0x69, 0x66, 0x65, 0x73, 0x74, 0x4d, 0x65,
0x73, 0x73, 0x61, 0x67, 0x65, 0x42, 0x6f, 0x64, 0x79, 0x12, 0x44, 0x0a, 0x05, 0x69, 0x74, 0x65,
0x6d, 0x73, 0x18, 0x01, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x2e, 0x2e, 0x6d, 0x69, 0x6c, 0x76, 0x75,
0x73, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x2e, 0x6d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x73,
0x2e, 0x42, 0x61, 0x74, 0x63, 0x68, 0x55, 0x70, 0x64, 0x61, 0x74, 0x65, 0x4d, 0x61, 0x6e, 0x69,
0x66, 0x65, 0x73, 0x74, 0x49, 0x74, 0x65, 0x6d, 0x52, 0x05, 0x69, 0x74, 0x65, 0x6d, 0x73, 0x22,
0x63, 0x0a, 0x17, 0x42, 0x61, 0x74, 0x63, 0x68, 0x55, 0x70, 0x64, 0x61, 0x74, 0x65, 0x4d, 0x61,
0x6e, 0x69, 0x66, 0x65, 0x73, 0x74, 0x49, 0x74, 0x65, 0x6d, 0x12, 0x1d, 0x0a, 0x0a, 0x73, 0x65,
0x67, 0x6d, 0x65, 0x6e, 0x74, 0x5f, 0x69, 0x64, 0x18, 0x01, 0x20, 0x01, 0x28, 0x03, 0x52, 0x09,
0x73, 0x65, 0x67, 0x6d, 0x65, 0x6e, 0x74, 0x49, 0x64, 0x12, 0x29, 0x0a, 0x10, 0x6d, 0x61, 0x6e,
0x69, 0x66, 0x65, 0x73, 0x74, 0x5f, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x18, 0x02, 0x20,
0x01, 0x28, 0x03, 0x52, 0x0f, 0x6d, 0x61, 0x6e, 0x69, 0x66, 0x65, 0x73, 0x74, 0x56, 0x65, 0x72,
0x73, 0x69, 0x6f, 0x6e, 0x2a, 0x9e, 0x07, 0x0a, 0x0b, 0x4d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65,
0x54, 0x79, 0x70, 0x65, 0x12, 0x0b, 0x0a, 0x07, 0x55, 0x6e, 0x6b, 0x6e, 0x6f, 0x77, 0x6e, 0x10,
0x00, 0x12, 0x0c, 0x0a, 0x08, 0x54, 0x69, 0x6d, 0x65, 0x54, 0x69, 0x63, 0x6b, 0x10, 0x01, 0x12,
0x0a, 0x0a, 0x06, 0x49, 0x6e, 0x73, 0x65, 0x72, 0x74, 0x10, 0x02, 0x12, 0x0a, 0x0a, 0x06, 0x44,
0x65, 0x6c, 0x65, 0x74, 0x65, 0x10, 0x03, 0x12, 0x09, 0x0a, 0x05, 0x46, 0x6c, 0x75, 0x73, 0x68,
0x10, 0x04, 0x12, 0x14, 0x0a, 0x10, 0x43, 0x72, 0x65, 0x61, 0x74, 0x65, 0x43, 0x6f, 0x6c, 0x6c,
0x65, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x10, 0x05, 0x12, 0x12, 0x0a, 0x0e, 0x44, 0x72, 0x6f, 0x70,
0x43, 0x6f, 0x6c, 0x6c, 0x65, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x10, 0x06, 0x12, 0x13, 0x0a, 0x0f,
0x43, 0x72, 0x65, 0x61, 0x74, 0x65, 0x50, 0x61, 0x72, 0x74, 0x69, 0x74, 0x69, 0x6f, 0x6e, 0x10,
0x07, 0x12, 0x11, 0x0a, 0x0d, 0x44, 0x72, 0x6f, 0x70, 0x50, 0x61, 0x72, 0x74, 0x69, 0x74, 0x69,
0x6f, 0x6e, 0x10, 0x08, 0x12, 0x0f, 0x0a, 0x0b, 0x4d, 0x61, 0x6e, 0x75, 0x61, 0x6c, 0x46, 0x6c,
0x75, 0x73, 0x68, 0x10, 0x09, 0x12, 0x11, 0x0a, 0x0d, 0x43, 0x72, 0x65, 0x61, 0x74, 0x65, 0x53,
0x65, 0x67, 0x6d, 0x65, 0x6e, 0x74, 0x10, 0x0a, 0x12, 0x0a, 0x0a, 0x06, 0x49, 0x6d, 0x70, 0x6f,
0x72, 0x74, 0x10, 0x0b, 0x12, 0x14, 0x0a, 0x0c, 0x53, 0x63, 0x68, 0x65, 0x6d, 0x61, 0x43, 0x68,
0x61, 0x6e, 0x67, 0x65, 0x10, 0x0c, 0x1a, 0x02, 0x08, 0x01, 0x12, 0x13, 0x0a, 0x0f, 0x41, 0x6c,
0x74, 0x65, 0x72, 0x43, 0x6f, 0x6c, 0x6c, 0x65, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x10, 0x0d, 0x12,
0x13, 0x0a, 0x0f, 0x41, 0x6c, 0x74, 0x65, 0x72, 0x4c, 0x6f, 0x61, 0x64, 0x43, 0x6f, 0x6e, 0x66,
0x69, 0x67, 0x10, 0x0e, 0x12, 0x12, 0x0a, 0x0e, 0x44, 0x72, 0x6f, 0x70, 0x4c, 0x6f, 0x61, 0x64,
0x43, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x10, 0x0f, 0x12, 0x12, 0x0a, 0x0e, 0x43, 0x72, 0x65, 0x61,
0x74, 0x65, 0x44, 0x61, 0x74, 0x61, 0x62, 0x61, 0x73, 0x65, 0x10, 0x10, 0x12, 0x11, 0x0a, 0x0d,
0x41, 0x6c, 0x74, 0x65, 0x72, 0x44, 0x61, 0x74, 0x61, 0x62, 0x61, 0x73, 0x65, 0x10, 0x11, 0x12,
0x10, 0x0a, 0x0c, 0x44, 0x72, 0x6f, 0x70, 0x44, 0x61, 0x74, 0x61, 0x62, 0x61, 0x73, 0x65, 0x10,
0x12, 0x12, 0x0e, 0x0a, 0x0a, 0x41, 0x6c, 0x74, 0x65, 0x72, 0x41, 0x6c, 0x69, 0x61, 0x73, 0x10,
0x13, 0x12, 0x0d, 0x0a, 0x09, 0x44, 0x72, 0x6f, 0x70, 0x41, 0x6c, 0x69, 0x61, 0x73, 0x10, 0x14,
0x12, 0x0f, 0x0a, 0x0b, 0x52, 0x65, 0x73, 0x74, 0x6f, 0x72, 0x65, 0x52, 0x42, 0x41, 0x43, 0x10,
0x15, 0x12, 0x0d, 0x0a, 0x09, 0x41, 0x6c, 0x74, 0x65, 0x72, 0x55, 0x73, 0x65, 0x72, 0x10, 0x16,
0x12, 0x0c, 0x0a, 0x08, 0x44, 0x72, 0x6f, 0x70, 0x55, 0x73, 0x65, 0x72, 0x10, 0x17, 0x12, 0x0d,
0x0a, 0x09, 0x41, 0x6c, 0x74, 0x65, 0x72, 0x52, 0x6f, 0x6c, 0x65, 0x10, 0x18, 0x12, 0x0c, 0x0a,
0x08, 0x44, 0x72, 0x6f, 0x70, 0x52, 0x6f, 0x6c, 0x65, 0x10, 0x19, 0x12, 0x11, 0x0a, 0x0d, 0x41,
0x6c, 0x74, 0x65, 0x72, 0x55, 0x73, 0x65, 0x72, 0x52, 0x6f, 0x6c, 0x65, 0x10, 0x1a, 0x12, 0x10,
0x0a, 0x0c, 0x44, 0x72, 0x6f, 0x70, 0x55, 0x73, 0x65, 0x72, 0x52, 0x6f, 0x6c, 0x65, 0x10, 0x1b,
0x12, 0x12, 0x0a, 0x0e, 0x41, 0x6c, 0x74, 0x65, 0x72, 0x50, 0x72, 0x69, 0x76, 0x69, 0x6c, 0x65,
0x67, 0x65, 0x10, 0x1c, 0x12, 0x11, 0x0a, 0x0d, 0x44, 0x72, 0x6f, 0x70, 0x50, 0x72, 0x69, 0x76,
0x69, 0x6c, 0x65, 0x67, 0x65, 0x10, 0x1d, 0x12, 0x17, 0x0a, 0x13, 0x41, 0x6c, 0x74, 0x65, 0x72,
0x50, 0x72, 0x69, 0x76, 0x69, 0x6c, 0x65, 0x67, 0x65, 0x47, 0x72, 0x6f, 0x75, 0x70, 0x10, 0x1e,
0x12, 0x16, 0x0a, 0x12, 0x44, 0x72, 0x6f, 0x70, 0x50, 0x72, 0x69, 0x76, 0x69, 0x6c, 0x65, 0x67,
0x65, 0x47, 0x72, 0x6f, 0x75, 0x70, 0x10, 0x1f, 0x12, 0x16, 0x0a, 0x12, 0x41, 0x6c, 0x74, 0x65,
0x72, 0x52, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x47, 0x72, 0x6f, 0x75, 0x70, 0x10, 0x20,
0x12, 0x15, 0x0a, 0x11, 0x44, 0x72, 0x6f, 0x70, 0x52, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65,
0x47, 0x72, 0x6f, 0x75, 0x70, 0x10, 0x21, 0x12, 0x0f, 0x0a, 0x0b, 0x43, 0x72, 0x65, 0x61, 0x74,
0x65, 0x49, 0x6e, 0x64, 0x65, 0x78, 0x10, 0x22, 0x12, 0x0e, 0x0a, 0x0a, 0x41, 0x6c, 0x74, 0x65,
0x72, 0x49, 0x6e, 0x64, 0x65, 0x78, 0x10, 0x23, 0x12, 0x0d, 0x0a, 0x09, 0x44, 0x72, 0x6f, 0x70,
0x49, 0x6e, 0x64, 0x65, 0x78, 0x10, 0x24, 0x12, 0x0c, 0x0a, 0x08, 0x46, 0x6c, 0x75, 0x73, 0x68,
0x41, 0x6c, 0x6c, 0x10, 0x25, 0x12, 0x16, 0x0a, 0x12, 0x54, 0x72, 0x75, 0x6e, 0x63, 0x61, 0x74,
0x65, 0x43, 0x6f, 0x6c, 0x6c, 0x65, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x10, 0x26, 0x12, 0x13, 0x0a,
0x0f, 0x52, 0x65, 0x73, 0x74, 0x6f, 0x72, 0x65, 0x53, 0x6e, 0x61, 0x70, 0x73, 0x68, 0x6f, 0x74,
0x10, 0x27, 0x12, 0x12, 0x0a, 0x0e, 0x43, 0x72, 0x65, 0x61, 0x74, 0x65, 0x53, 0x6e, 0x61, 0x70,
0x73, 0x68, 0x6f, 0x74, 0x10, 0x28, 0x12, 0x10, 0x0a, 0x0c, 0x44, 0x72, 0x6f, 0x70, 0x53, 0x6e,
0x61, 0x70, 0x73, 0x68, 0x6f, 0x74, 0x10, 0x29, 0x12, 0x17, 0x0a, 0x13, 0x42, 0x61, 0x74, 0x63,
0x68, 0x55, 0x70, 0x64, 0x61, 0x74, 0x65, 0x4d, 0x61, 0x6e, 0x69, 0x66, 0x65, 0x73, 0x74, 0x10,
0x2a, 0x12, 0x1d, 0x0a, 0x19, 0x52, 0x65, 0x66, 0x72, 0x65, 0x73, 0x68, 0x45, 0x78, 0x74, 0x65,
0x72, 0x6e, 0x61, 0x6c, 0x43, 0x6f, 0x6c, 0x6c, 0x65, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x10, 0x2b,
0x12, 0x0d, 0x0a, 0x08, 0x41, 0x6c, 0x74, 0x65, 0x72, 0x57, 0x41, 0x4c, 0x10, 0xbc, 0x05, 0x12,
0x19, 0x0a, 0x14, 0x41, 0x6c, 0x74, 0x65, 0x72, 0x52, 0x65, 0x70, 0x6c, 0x69, 0x63, 0x61, 0x74,
0x65, 0x43, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x10, 0xa0, 0x06, 0x12, 0x0d, 0x0a, 0x08, 0x42, 0x65,
0x67, 0x69, 0x6e, 0x54, 0x78, 0x6e, 0x10, 0x84, 0x07, 0x12, 0x0e, 0x0a, 0x09, 0x43, 0x6f, 0x6d,
0x6d, 0x69, 0x74, 0x54, 0x78, 0x6e, 0x10, 0x85, 0x07, 0x12, 0x10, 0x0a, 0x0b, 0x52, 0x6f, 0x6c,
0x6c, 0x62, 0x61, 0x63, 0x6b, 0x54, 0x78, 0x6e, 0x10, 0x86, 0x07, 0x12, 0x08, 0x0a, 0x03, 0x54,
0x78, 0x6e, 0x10, 0xe7, 0x07, 0x2a, 0x74, 0x0a, 0x08, 0x54, 0x78, 0x6e, 0x53, 0x74, 0x61, 0x74,
0x65, 0x12, 0x0e, 0x0a, 0x0a, 0x54, 0x78, 0x6e, 0x55, 0x6e, 0x6b, 0x6e, 0x6f, 0x77, 0x6e, 0x10,
0x00, 0x12, 0x0f, 0x0a, 0x0b, 0x54, 0x78, 0x6e, 0x49, 0x6e, 0x46, 0x6c, 0x69, 0x67, 0x68, 0x74,
0x10, 0x01, 0x12, 0x0f, 0x0a, 0x0b, 0x54, 0x78, 0x6e, 0x4f, 0x6e, 0x43, 0x6f, 0x6d, 0x6d, 0x69,
0x74, 0x10, 0x02, 0x12, 0x10, 0x0a, 0x0c, 0x54, 0x78, 0x6e, 0x43, 0x6f, 0x6d, 0x6d, 0x69, 0x74,
0x74, 0x65, 0x64, 0x10, 0x03, 0x12, 0x11, 0x0a, 0x0d, 0x54, 0x78, 0x6e, 0x4f, 0x6e, 0x52, 0x6f,
0x6c, 0x6c, 0x62, 0x61, 0x63, 0x6b, 0x10, 0x04, 0x12, 0x11, 0x0a, 0x0d, 0x54, 0x78, 0x6e, 0x52,
0x6f, 0x6c, 0x6c, 0x62, 0x61, 0x63, 0x6b, 0x65, 0x64, 0x10, 0x05, 0x2a, 0xe2, 0x01, 0x0a, 0x0e,
0x52, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x12, 0x19,
0x0a, 0x15, 0x52, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e,
0x55, 0x6e, 0x6b, 0x6e, 0x6f, 0x77, 0x6e, 0x10, 0x00, 0x12, 0x21, 0x0a, 0x19, 0x52, 0x65, 0x73,
0x6f, 0x75, 0x72, 0x63, 0x65, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x49, 0x6d, 0x70, 0x6f, 0x72,
0x74, 0x4a, 0x6f, 0x62, 0x49, 0x44, 0x10, 0x01, 0x1a, 0x02, 0x08, 0x01, 0x12, 0x20, 0x0a, 0x1c,
0x52, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x43, 0x6f,
0x6c, 0x6c, 0x65, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x4e, 0x61, 0x6d, 0x65, 0x10, 0x02, 0x12, 0x18,
0x0a, 0x14, 0x52, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e,
0x44, 0x42, 0x4e, 0x61, 0x6d, 0x65, 0x10, 0x03, 0x12, 0x1b, 0x0a, 0x17, 0x52, 0x65, 0x73, 0x6f,
0x75, 0x72, 0x63, 0x65, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x50, 0x72, 0x69, 0x76, 0x69, 0x6c,
0x65, 0x67, 0x65, 0x10, 0x04, 0x12, 0x1e, 0x0a, 0x1a, 0x52, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63,
0x65, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x53, 0x6e, 0x61, 0x70, 0x73, 0x68, 0x6f, 0x74, 0x4e,
0x61, 0x6d, 0x65, 0x10, 0x05, 0x12, 0x19, 0x0a, 0x15, 0x52, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63,
0x65, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x43, 0x6c, 0x75, 0x73, 0x74, 0x65, 0x72, 0x10, 0x7f,
0x42, 0x35, 0x5a, 0x33, 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x6d,
0x69, 0x6c, 0x76, 0x75, 0x73, 0x2d, 0x69, 0x6f, 0x2f, 0x6d, 0x69, 0x6c, 0x76, 0x75, 0x73, 0x2f,
0x70, 0x6b, 0x67, 0x2f, 0x76, 0x32, 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x2f, 0x6d, 0x65, 0x73,
0x73, 0x61, 0x67, 0x65, 0x73, 0x70, 0x62, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33,
}
var (
pkg/util/merr/errors.go
+6
View File
@@ -234,6 +234,12 @@ var (
ErrCompactionResult = newMilvusError("illegal compaction results", 2314, false)
ErrDuplicatedCompactionTask = newMilvusError("duplicated compaction task", 2315, false)
ErrCleanPartitionStatsFail = newMilvusError("fail to clean partition Stats", 2316, true)
// ErrCompactionBlocked is returned when a compaction task is legitimately blocked by
// snapshot protection (pending snapshot creation or a referenced segment must be kept).
// This is a business-level rejection, not an internal fault: the compaction scheduler
// should recognize it and retry with low-frequency backoff instead of triggering alerts.
// Marked retryable so callers do not misinterpret it as a hard failure.
ErrCompactionBlocked = newMilvusError("compaction blocked by snapshot protection", 2317, true)
ErrDataNodeSlotExhausted = newMilvusError("datanode slot exhausted", 2401, false)
pkg/util/merr/utils.go
+11
View File
@@ -1337,6 +1337,17 @@ func WrapErrDuplicatedCompactionTask(msg ...string) error {
return err
}
// WrapErrCompactionBlocked indicates that a compaction task is blocked by snapshot protection.
// This is a business-level rejection (not a service fault) that the scheduler should handle
// with low-frequency backoff, not with P0 alerting.
func WrapErrCompactionBlocked(reason string, msg ...string) error {
err := wrapFieldsWithDesc(ErrCompactionBlocked, reason)
if len(msg) > 0 {
err = errors.Wrap(err, strings.Join(msg, "->"))
}
return err
}
func WrapErrOldSessionExists(msg ...string) error {
err := error(ErrOldSessionExists)
if len(msg) > 0 {
pkg/util/merr/utils_test.go
+19
View File
@@ -95,3 +95,22 @@ func TestIsMilvusError_WrappedChain(t *testing.T) {
assert.False(t, IsMilvusError(errors.New("plain error")))
assert.False(t, IsMilvusError(nil))
}
func TestWrapErrCompactionBlocked(t *testing.T) {
// Without extra message: only the reason is attached to ErrCompactionBlocked.
err := WrapErrCompactionBlocked("collection 100 has pending snapshot")
assert.Error(t, err)
assert.True(t, errors.Is(err, ErrCompactionBlocked))
assert.Contains(t, err.Error(), "collection 100 has pending snapshot")
// Sanity: callers should NOT mistake this for a service-internal failure.
assert.False(t, errors.Is(err, ErrServiceInternal))
// Business-level rejection is marked retryable so compaction scheduler can back off.
assert.False(t, IsNonRetryableErr(err))
// With extra context message: errors.Wrap joins msg slices with "->".
err2 := WrapErrCompactionBlocked("segment 42 protected", "planID=8001", "type=Merge")
assert.Error(t, err2)
assert.True(t, errors.Is(err2, ErrCompactionBlocked))
assert.Contains(t, err2.Error(), "segment 42 protected")
assert.Contains(t, err2.Error(), "planID=8001->type=Merge")
}
pkg/util/paramtable/component_param.go
+33 -10
View File
@@ -4808,16 +4808,18 @@ type dataCoordConfig struct {
LevelZeroCompactionTriggerDeltalogMaxNum ParamItem `refreshable:"true"`
// Garbage Collection
EnableGarbageCollection ParamItem `refreshable:"false"`
GCInterval ParamItem `refreshable:"false"`
GCMissingTolerance ParamItem `refreshable:"false"`
GCDropTolerance ParamItem `refreshable:"false"`
GCRemoveConcurrent ParamItem `refreshable:"false"`
GCScanIntervalInHour ParamItem `refreshable:"false"`
GCSlowDownCPUUsageThreshold ParamItem `refreshable:"false"`
SnapshotPendingTimeout ParamItem `refreshable:"true"`
SnapshotRefIndexLoadInterval ParamItem `refreshable:"true"`
EnableActiveStandby ParamItem `refreshable:"false"`
EnableGarbageCollection ParamItem `refreshable:"false"`
GCInterval ParamItem `refreshable:"false"`
GCMissingTolerance ParamItem `refreshable:"false"`
GCDropTolerance ParamItem `refreshable:"false"`
GCRemoveConcurrent ParamItem `refreshable:"false"`
GCScanIntervalInHour ParamItem `refreshable:"false"`
GCSlowDownCPUUsageThreshold ParamItem `refreshable:"false"`
SnapshotPendingTimeout ParamItem `refreshable:"true"`
SnapshotRefIndexLoadInterval ParamItem `refreshable:"true"`
SnapshotRefIndexLoadTimeout ParamItem `refreshable:"true"`
SnapshotMaxCompactionProtectionSeconds ParamItem `refreshable:"true"`
EnableActiveStandby ParamItem `refreshable:"false"`
BindIndexNodeMode ParamItem `refreshable:"false"`
IndexNodeAddress ParamItem `refreshable:"false"`
@@ -5684,6 +5686,27 @@ During compaction, the size of segment # of rows is able to exceed segment max #
}
p.SnapshotRefIndexLoadInterval.Init(base.mgr)
p.SnapshotRefIndexLoadTimeout = ParamItem{
Key: "dataCoord.snapshot.refIndexLoadTimeout",
Version: "2.6.15",
DefaultValue: "30s",
Doc: "Per-snapshot timeout for loading RefIndex from S3 inside the loader loop. " +
"A single hung S3 read must not block the loader Range, otherwise no other " +
"snapshot's RefIndex can be loaded and GC remains in fail-closed coarse block " +
"for every collection with a snapshot, leaking storage. On timeout the RefIndex " +
"is marked Failed and will be retried on the next loader tick.",
}
p.SnapshotRefIndexLoadTimeout.Init(base.mgr)
p.SnapshotMaxCompactionProtectionSeconds = ParamItem{
Key: "dataCoord.snapshot.maxCompactionProtectionSeconds",
Version: "2.6.10",
DefaultValue: "604800",
Doc: "Maximum allowed compaction protection duration in seconds (default 604800 = 7 days)",
Export: true,
}
p.SnapshotMaxCompactionProtectionSeconds.Init(base.mgr)
p.EnableActiveStandby = ParamItem{
Key: "dataCoord.enableActiveStandby",
Version: "2.0.0",