mirror of
https://github.com/milvus-io/milvus.git
synced 2026-07-21 02:05:41 +00:00
b16c821b36
### What & why `bump_schema_version`'s full-rewrite path (`runFullSchemaRewrite`, reached only for a **drop-field** schema bump) builds a from-scratch output manifest via `NewBinlogRecordWriter` and did **not** carry the source segment's TEXT LOB references — leaving dangling refs for an out-of-line (`>=64KB`) TEXT column. This was the `TODO(#50021)`. ### What this does Wire **REUSE_ALL** LOB handling into `runFullSchemaRewrite`, mirroring sort compaction. Both are `1->1` with a single output manifest, so REUSE_ALL is always correct: a schema bump never changes the existing TEXT LOB data — only its references are carried. - `internal/compaction/lob_compaction.go`: `GetForcedStrategy` forces REUSE_ALL for `BumpSchemaVersionCompaction`. - `internal/datanode/compactor/bump_schema_version_compactor.go`: collect the source LOB files into a `LOBCompactionContext`, pass `storage.WithTextRefsAsBinary()`, update per-LOB-file `valid_rows` (`SetSegmentRowStats`), and merge the LOB references into the output manifest (`applyLOBCompaction`) **before** building the text-match index — `createTextIndex` reads the TEXT column through this manifest, so the carried LOB files must already be referenced (matches sort/mix ordering). Removed the `TODO(#50021)`. - Unit tests: forced strategy, init/apply LOB wiring, the applyLOBCompaction-before-createTextIndex ordering (`TestFullRewriteMergesLOBRefsBeforeBuildingTextIndex`), and a real add-nullable-TEXT + drop-field full-rewrite regression driving the actual packed writer (`TestFullRewriteFillsMissingNullableTextAsBinary`). ### Also: guard add_function_field behind StorageV3 (#51167) Adding a function to an existing collection must backfill the new output field into **pre-existing** sealed segments; that backfill runs through `bump_schema_version` compaction, which only works on a StorageV3 segment. A pre-existing V2 segment is only backfilled after the storage-version upgrade compaction rewrites it to V3, and that upgrade runs only when **both** `common.storage.useLoonFFI` and `dataCoord.compaction.storageVersion.enabled` are on. The proxy now rejects add-function unless both are enabled (`validateAddFunctionRequiresStorageV3`, wired into `addCollectionFunctionTask` and `alterCollectionSchemaTask`). `create_collection` with a function is unaffected. Full reasoning in #51167. ### Notes - Reuses the LOB machinery from **#50784**. - The `GenerateEmptyArrayFromSchema` binary-TEXT production fix landed independently on master via **#51124**; this PR keeps the regression test guarding the bump path. issue: #50021, #51167 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: MrPresent-Han <chun.han@gmail.com> Co-authored-by: MrPresent-Han <chun.han@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>