mirror of
https://github.com/milvus-io/milvus.git
synced 2026-07-21 10:15:43 +00:00
yihao.daiandGitHub
78a0a53d9e
## Summary Fix QueryNode schema freshness checks by separating two ordering domains: - `CollectionSchema.version` is the logical schema freshness version. QueryNode uses it to prevent old schema payloads from overwriting newer fields/functions. - `schema_barrier_ts` is the DDL/update barrier timestamp. QueryNode uses it to fence stale load results and to refresh same-version schema payloads, such as collection properties (`ttl_field`) that do not bump `schema.version`. Fixes #50364 ## Update rules When QueryNode receives a schema payload: - If incoming `schema.version < current schema.version`, skip the update even if `schema_barrier_ts` is larger. This prevents schema rollback from out-of-order replay/channel delivery. - If incoming `schema.version > current schema.version`, apply the update. - If incoming `schema.version == current schema.version`, apply the update only when `schema_barrier_ts` is newer than the current barrier. This allows properties-only schema snapshots, for example TTL field changes, to refresh the runtime schema. - If the schema payload is absent, fall back to `schema_barrier_ts` for legacy/rolling-upgrade compatibility. Segcore still receives a single schema update token, but QueryNode now derives that token with collection-local monotonic ordering across both logical schema versions and barrier timestamps. This keeps segcore accepting updates when a high-barrier same-version refresh is followed by a higher logical schema version that carries a smaller barrier timestamp. ## Delegator side effects `shardDelegator.UpdateSchema` now runs the same stale/no-op schema check before updating workers, load barriers, function runtime state, IDF oracle state, or local collection metadata. This keeps stale schema messages from producing delegator side effects when the collection manager would skip the schema payload. The delegator load barrier remains timestamp-based and monotonic. A newer logical schema version with a smaller barrier timestamp must not reopen older load results after a previous same-version property refresh advanced the barrier. ## Changes - Rename the legacy QueryCoord fields to `schema_barrier_ts` to reflect their timestamp-barrier semantics. - Track both logical schema version and schema barrier timestamp in QueryNode collection schema snapshots. - Use `CollectionSchema.version` as the logical QueryNode collection schema freshness version for load, sync, pipeline replay, and UpdateSchema paths. - Allow same-version schema payload refresh only when the barrier timestamp advances, so properties-only changes such as `ttl_field` take effect without release/load. - Generate a monotonic segcore schema update token from the current and incoming logical/barrier values. - Skip stale/no-op schema payloads in delegator before worker updates and other schema side effects. - Keep `schema_barrier_ts` in delegator/pipeline load fencing so stale load results are rejected after schema-changing DDL. - Update logs, comments, and tests to distinguish `schemaVersion` from `schemaBarrierTs`. ## Test Plan - [x] `make generated-proto-without-cpp` - [x] `git diff --check upstream/master...HEAD` - [x] `cd pkg && go test -count=1 ./proto/querypb` - [x] `go test -tags dynamic,test -gcflags="all=-N -l" -count=1 ./internal/querycoordv2/task -run 'TestUtils/TestPackLoadMetaSchemaVersions'` - [x] `go test -tags dynamic,test -gcflags="all=-N -l" -count=1 ./internal/querynodev2/segments -run 'TestCollectionManager/(TestUpdateSchema|TestPutOrRefKeepsFreshCollectionInSchemaVersionDomain|TestLoadMetaSchemaVersionCompatibility)'` - [x] `go test -tags dynamic,test -gcflags="all=-N -l" -count=1 ./internal/querynodev2/delegator -run 'TestDelegatorSuite/TestUpdateSchema|TestUpdateSchema'` - [x] `env 'etcd.auth.enabled=false' go test -tags dynamic,test -gcflags="all=-N -l" -count=1 ./internal/querynodev2 -run 'TestQueryNodeService/TestUpdateSchema'` - [x] `make static-check` with Go 1.26.4 and shared C++ build environment before the final upstream rebase Notes: - Full `./internal/querynodev2/delegator` was also attempted locally. It currently fails in unrelated `TestDelegatorDataSuite/TestLoadPartitionStats` because partition stats deserialization reports `json: cannot unmarshal into Go value of type storage.VectorFieldValue`; the schema/update-focused tests above pass. - Per maintainer request, the final post-rebase `make static-check` run was skipped before pushing the latest commit. --------- Signed-off-by: Yihao Dai <yihao.dai@zilliz.com>