mirror of
https://github.com/milvus-io/milvus.git
synced 2026-07-21 10:15:43 +00:00
wei liuandGitHub
c703b27ae4
## Summary Part 2/4 of the External Table Part 8 series. End-to-end control plane for external collections. **Depends on [#49061](https://github.com/milvus-io/milvus/pull/49061) (Part 1/4 foundation).** ### RootCoord / Proxy — DDL validation - Validate \`externalSource\` / \`externalSpec\` at create_collection_task (defense-in-depth) - Alter collection properties: detect changes, reject invalid transitions, attach \`FieldMask\` to WAL broadcast - New property keys: \`CollectionExternalSource\`, \`CollectionExternalSpec\`; streaming \`FieldMaskCollectionExternalSpec\` ### DataCoord refresh infrastructure - **refresh manager**: per-collection job state machine - **refresh checker**: periodic tick scanning collections for changed external source - **refresh inspector**: coordinates checker + manager, \`wrapTask\` factory - **refresh task**: encapsulates single refresh job (plan fragments, allocate segments, submit to DataNode, observe) - **schema updater**: broadcasts schema DDL detected by refresh back to RootCoord via WAL ### DataNode refresh task (RefreshExternalCollectionTask) - Reads explore manifest, fetches fragments in assigned file range - Parallelizes segment ID allocation from pre-allocated range ### externalspec — DDL validation helpers - \`ValidateExternalSource\`: scheme allowlist (s3/s3a/aws/minio/oss/gs/gcs/file), rejects userinfo - \`ValidateSourceAndSpec\`: combined Proxy/RootCoord entry point - \`RedactExternalSpec\`: removes sensitive extfs keys before logging - \`BuildFormatProperties\`: per-format reader/writer properties ### Unified job path + mutator meta - Single \`processJob\` routine for both periodic checker + eager task-completion - \`notifiedJobs\` dedup map prevents double-broadcast under concurrent invocations - \`Mutate(fn)\` helper centralizes all meta mutations Related: [#45881](https://github.com/milvus-io/milvus/issues/45881) (External Collection Lakehouse Integration tracking). ## Test plan - [x] Full Go build (\`go build -tags dynamic,test ./...\`) — 0 errors - [x] All test files compile (\`go test -tags dynamic,test -run=^$\` on all internal/...) — 0 build failures - [x] externalspec tests — 19 tests pass, 100% coverage on control-plane helpers - [x] golangci-lint on all changed packages — 0 issues - [ ] CI validation (ut-go / ut-cpp / integration-test / e2e) ## Review order Please review after #49061 (1/4) lands. After this (2/4) merges, Part 3/4 (Iceberg) and 4/4 (load optimization) will follow. Signed-off-by: Wei Liu <wei.liu@zilliz.com>
43 lines
1.3 KiB
Go
43 lines
1.3 KiB
Go
// Licensed to the LF AI & Data foundation under one
|
|
// or more contributor license agreements. See the NOTICE file
|
|
// distributed with this work for additional information
|
|
// regarding copyright ownership. The ASF licenses this file
|
|
// to you under the Apache License, Version 2.0 (the
|
|
// "License"); you may not use this file except in compliance
|
|
// with the License. You may obtain a copy of the License at
|
|
//
|
|
// http://www.apache.org/licenses/LICENSE-2.0
|
|
//
|
|
// Unless required by applicable law or agreed to in writing, software
|
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
// See the License for the specific language governing permissions and
|
|
// limitations under the License.
|
|
|
|
package taskcommon
|
|
|
|
type Type = string
|
|
|
|
const (
|
|
TypeNone Type = "None"
|
|
PreImport Type = "PreImport"
|
|
Import Type = "Import"
|
|
Compaction Type = "Compaction"
|
|
Index Type = "Index"
|
|
Stats Type = "Stats"
|
|
Analyze Type = "Analyze"
|
|
RefreshExternalCollection Type = "RefreshExternalCollection"
|
|
CopySegment Type = "CopySegment"
|
|
)
|
|
|
|
var TypeList = []Type{
|
|
PreImport,
|
|
Import,
|
|
Compaction,
|
|
Index,
|
|
Stats,
|
|
Analyze,
|
|
RefreshExternalCollection,
|
|
CopySegment,
|
|
}
|