relate: #50304
## Summary
Add the local format design doc, use the storage writer format constant,
and introduce local_format metadata propagation with column-group split
policy support.
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
issue: #50182
Storage V3 can now keep writing a segment after
`dataNode.storage.format` changes. The main problem was that an
already-started growing segment may have a committed manifest written
with one format, while the current global config has moved to another
format. In that case, the next append must follow the format already
recorded in the manifest instead of blindly using the new global config.
This keeps writer.format as the current global setting, but makes the
actual split policy format local to the writer path. QueryNode growing
flush resolves writer.split.single.format from the acknowledged manifest
when one exists, and falls back to the global config only for a
brand-new segment. DataNode Storage V3 flush carries column group
formats through SegmentInfo and passes them as
writer.split.schema_based.formats, so schema-based writes can append
with the formats already known for each column group.
The metadata path now records the column group format in FieldBinlog and
preserves it through recovery, merge, duplicate filtering, fake external
segment binlogs, and V3 compaction output. This lets metacache rebuild
the current split with enough information to choose writer formats
without reading the manifest on every flush.
The storage writer wrappers now accept explicit writer format properties
for both regular packed writers and TEXT-aware segment writers. TEXT LOB
files still use their own LOB path and storage behavior; the format
selection here applies to the segment column groups that store normal
data and LOB references, not to changing the LOB payload file layout.
Signed-off-by: jiaqizho <jiaqi.zhou@zilliz.com>
Related to #49398
Bump Go module references from pkg/v2 and milvus-proto/go-api/v2 to
pkg/v3 and milvus-proto/go-api/v3 so the client tracks the Milvus 3.x
release line.
This prepares the repository for the upcoming 3.x.y release by aligning
imports, module dependencies, and proto API references with the new
major-version module paths.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #44257
This PR makes partition key & clustering candidates of system field
group and adds param item controlling the policy
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #44257
This PR:
- Utilize configurable split policy for storage v2, enabling system
field policy
- Store split result in field binlog struct
- Adapt legacy binlog without child fields
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Fix issues in end-to-end tests:
1. **Split column groups based on schema**, rather than estimating by
average chunk row size. **Ensure column group consistency within a
segment**, to avoid errors caused by loading multiple column group
chunks simultaneously.
2. **Use sorted segmentId** when generating the stats binlog path, to
ensure consistent and correct file path resolution.
3. **Determine field IDs as follows**:
For multi-column column groups, retrieve the field ID list from
metadata.
For single-column column groups, use the column group ID directly as the
field ID.
related: #39173fix: #42862
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
