doc: add design docs directory (#49829)

issue: #49831 ## What changed Vendor Milvus design documents into this repository under `docs/design-docs` as regular tracked files. - Keep only the design document content and assets under `docs/design-docs/design_docs/` and `docs/design-docs/assets/`. - Remove standalone repository metadata from the vendored directory, such as `README.md`, `CONTRIBUTING.md`, `MEP-TEMPLATE.md`, `COMMITTERS`, `MAINTAINERS`, `OWNERS`, `OWNERS_ALIASES`, and `.gitignore`. - Document the Milvus design document process in the main `CONTRIBUTING.md`. - Update Mergify and `tools/mgit.py` so feature PRs must provide an in-repo design document path under `docs/design-docs/design_docs/`. ## Why Milvus feature work should have an associated design document. Keeping design docs directly in this repository makes them available from a normal Milvus checkout and lets feature implementations include or link the related design document in the same repository. ## Verification - `git diff --check` - `python3 -m unittest tools/test_mgit_design_doc.py` - `python3 -m py_compile tools/mgit.py tools/test_mgit_design_doc.py` - Parsed `.github/mergify.yml` with Python `yaml.safe_load` - Confirmed `docs/design-docs` only contains `assets/` and `design_docs/` at the top level --------- Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
2026-07-20 09:45:43 +00:00 · 2026-05-15 18:56:24 +08:00
parent ebed483bb4
commit dd6a411e5e
110 changed files with 18777 additions and 31 deletions
@@ -531,8 +531,9 @@ pull_request_rules:
          - 'title~=^feat:'
          - label=kind/feature
      - -title~=\[automated\]
-      # PR body does not contain design doc link
-      - -body~=https://github.com/milvus-io/milvus-design-docs/(blob|tree)/[^/]+/design_docs/
+      # PR body does not reference an in-repo design doc, and the PR does not add one
+      - '-body~=(^|\s)docs/design-docs/design_docs/[^\s]+\.md'
+      - -files~=^docs/design-docs/design_docs/.*\.md$
    actions:
      label:
        add:
@@ -542,18 +543,18 @@ pull_request_rules:
          @{{author}} This is a feature PR (`feat:`). Please provide a design document.

          **How to resolve:**
-          Link a design doc in the PR description:
+          Add a design document under `docs/design-docs/design_docs/` in this PR, or link an existing in-repo design document in the PR description:
          ```
-          design doc: https://github.com/milvus-io/milvus-design-docs/blob/main/design_docs/your_design.md
+          design doc: docs/design-docs/design_docs/YYYYMMDD-your_design.md
          ```

-          Design documents location: https://github.com/milvus-io/milvus-design-docs/tree/main/design_docs
-
  - name: Dismiss block label if design doc is provided
    conditions:
      - or: *BRANCHES
      - label=do-not-merge/missing-design-doc
-      - body~=https://github.com/milvus-io/milvus-design-docs/(blob|tree)/[^/]+/design_docs/
+      - or:
+          - 'body~=(^|\s)docs/design-docs/design_docs/[^\s]+\.md'
+          - files~=^docs/design-docs/design_docs/.*\.md$
    actions:
      label:
        remove:
@@ -42,7 +42,8 @@ scripts/standalone_embed.sh    # embedded standalone (no external deps)
 PR title format: `{type}: {description}`. Valid types: `feat:`, `fix:`, `enhance:`, `test:`, `doc:`, `auto:`, `build(deps):`.
 PR body must be non-empty. Issue/doc linking rules:
 - `fix:` — must link issue (e.g. `issue: #123`)
- `feat:` — must link issue + design doc from milvus-io/milvus-design-docs
+- `feat:` — must link issue + design doc under `docs/design-docs`
+- Every Milvus feature should have a related design doc under `docs/design-docs`; submit the doc in this repository and link it from the Milvus feature PR.
 - `enhance:` — must link issue if size L/XL/XXL
 - `doc:`, `test:` — no issue required
 - 2.x branch PRs must link the corresponding master PR (e.g. `pr: #123`)
@@ -13,6 +13,7 @@ As for everything else in the project, the contributions to Milvus are governed
  - [How can you contribute?](#how-can-you-contribute)
    - [Contributing code](#contributing-code)
    - [GitHub workflow](#github-workflow)
+    - [Design documents](#design-documents)
    - [General guidelines](#general-guidelines)
    - [Developer Certificate of Origin (DCO)](#developer-certificate-of-origin-dco)
  - [Coding Style](#coding-style)
@@ -55,8 +56,8 @@ As for everything else in the project, the contributions to Milvus are governed
 **If you require a new feature or major enhancement, you can**

 - (**Recommended**) File an issue about the feature/enhancement with reasons.
- Provide an MEP for the feature/enhancement.
- Pull a request to implement the MEP.
+- Provide a [design document](#design-documents) for the feature/enhancement.
+- Pull a request to implement the design.

 **If you are a reviewer/approver of Milvus, you can**

@@ -98,6 +99,49 @@ git checkout upstream/master -b my-topic-branch

 ![](docs/developer_guides/figs/local-develop-steps.png)

+### Design documents
+
+Milvus feature pull requests must provide a design document. This applies when the pull request title starts with `feat:` or the pull request is labeled `kind/feature`. Large enhancements that introduce new architecture, storage formats, public behavior, or upgrade impact should also include a design document; reviewers may add `kind/feature` when the design-doc requirement applies.
+
+To satisfy the requirement, do one of the following:
+
+- Add or update the design document in the same pull request as the related implementation.
+- Link an existing in-repo design document in the pull request description.
+
+Use this pull request description format when linking an existing document:
+
+```markdown
+design doc: docs/design-docs/design_docs/YYYYMMDD-short-descriptive-name.md
+```
+
+Design documents must live under `docs/design-docs/design_docs/`. Name each file `YYYYMMDD-short-descriptive-name.md`, keep one design per file, and put images or diagrams under `docs/design-docs/assets/graphs/` or `docs/design-docs/assets/images/`. External design-doc repository links do not satisfy this requirement. Mergify adds the `do-not-merge/missing-design-doc` label to feature PRs until this requirement is met.
+
+Start each design document with a clear title and metadata block:
+
+```markdown
+# MEP: <Title>
+
+- **Created:** YYYY-MM-DD
+- **Author(s):** @github-handle
+- **Status:** Draft | Under Review | Approved | Implemented | Deprecated
+- **Component:** DataNode | QueryNode | Proxy | Coordinator | Storage | Index | SDK | Other
+- **Related Issues:** #xxx
+- **Released:** Milvus release version, if applicable
+```
+
+Every design document should explain the problem, the proposed design, and how the design will be verified. Use these sections:
+
+- **Summary:** Briefly describe the change.
+- **Motivation:** Explain the user problem, operational problem, or architectural limitation being solved.
+- **Public Interfaces:** List API, proto, SDK, config, metrics, or behavior changes that users or other components will observe.
+- **Design Details:** Describe the architecture, data flow, component responsibilities, persistence/metadata changes, and important failure cases.
+- **Compatibility, Deprecation, and Migration Plan:** Call out upgrade, rollback, data-format, API compatibility, and migration impact.
+- **Test Plan:** Describe unit, integration, E2E, upgrade, performance, or failure-injection tests needed to prove the design works.
+- **Rejected Alternatives:** Record meaningful alternatives and why they were not chosen.
+- **References:** Link related issues, pull requests, previous designs, or external references.
+
+Update the design document when review changes the approach, so the merged document matches the implementation.
+
 ### General guidelines

 Before submitting your pull requests for review, make sure that your changes are consistent with the [coding style](CONTRIBUTING.md#coding-style), and run [unit tests](CONTRIBUTING.md#run-unit-test-with-code-coverage) to check your code coverage rate.
@@ -0,0 +1,164 @@
+# DataNode Recovery Design
+
+update: 5.21.2021, by [Goose](https://github.com/XuanYang-cn)  
+update: 6.03.2021, by [Goose](https://github.com/XuanYang-cn)
+update: 6.21.2021, by [Goose](https://github.com/XuanYang-cn)
+
+## What's DataNode?
+
+DataNode processes insert data and persists insert data into storage.
+
+DataNode is based on flowgraph; each flowgraph cares about only one vchannel. There are data definition language (DDL) messages, data manipulation language (DML)
+messages, and timetick messages inside one vchannel, FIFO log stream.
+
+One vchannel only contains DML messages of one collection. A collection consists of many segments, hence
+a vchannel contains DML messages of many segments. **Most importantly, the DML messages of the same segment can appear anywhere in vchannel.**
+
+## What is the real meaning of DataNode recovery?
+
+DataNode is stateless, but vchannel has states. DataNode's statelessness is guaranteed by DataCoord, which
+means the vchannel's state is maintained by DataCoord. So DataNode recovery is no different from starting.
+
+So what's DataNode's starting procedure?
+
+## Objectives
+
+### 1. Service Registration
+
+DataNode registers itself to etcd after grpc server started, in *INITIALIZING* state.
+
+### 2. Service Discovery
+
+DataNode discovers DataCoord and RootCoord, in *HEALTHY* and *IDLE* state.
+
+### 3. Flowgraph Recovery
+
+The detailed design can be found at [datanode flowgraph recovery design](20210604-datanode_flowgraph_recovery_design.md).
+
+After DataNode subscribes to a stateful vchannel, DataNode starts to work, or more specifically, flowgraph starts to work.
+
+Vchannel is stateful because we don't want to process twice what's already processed, as a "processed" message means its
+already persistent. In DataNode's terminology, a message is processed if it's been flushed.
+
+DataCoord tells DataNode stateful vchannel info through RPC `WatchDmChannels` so that DataNode won't process
+the same messages over and over again. So flowgraph needs the ability to consume messages in the middle of a vchannel.
+
+DataNode tells DataCoord vchannel states after each flush through RPC `SaveBinlogPaths`, so that DataCoord
+keeps the vchannel states updated.
+
+
+## Some interface/proto designs below are outdated, will be updated soon
+
+### 1. DataNode no longer interacts with etcd except service registering
+
+#### DataCoord rather than DataNode saves binlog paths into etcd
+
+   ![datanode_design](../assets/graphs/datanode_design_01.jpg)
+
+
+##### DataCoord RPC Design
+
+```proto
+rpc SaveBinlogPaths(SaveBinlogPathsRequest) returns (common.Status){}
+message ID2PathList {
+    int64 ID = 1;
+    repeated string Paths = 2;
+}
+
+message CheckPoint {
+    int64 segmentID = 1;
+    msgpb.MsgPosition position = 2;
+    int64 num_of_rows = 3;
+}
+
+message SaveBinlogPathsRequest {
+    common.MsgBase base = 1;
+    int64 segmentID = 2;
+    int64 collectionID = 3;
+    repeated ID2PathList field2BinlogPaths = 4;
+    repeated CheckPoint checkPoints = 7;
+    repeated SegmentStartPosition start_positions = 6;
+    bool flushed = 7;
+ }
+```
+
+### 4. DataNode with collection with flowgraph with vchannel designs
+
+#### The winner
+  ![datanode_design](../assets/graphs/collection_flowgraph_relation.png)
+
+  ![datanode_design](../assets/graphs/collection_flowgraph_1_n.png)
+
+**O4-1.** DataNode scales flowgraph 2 Day
+
+Change `WatchDmChannelsRequest` proto.
+
+``` proto
+message VchannelInfo {
+  int64 collectionID = 1;
+  string channelName = 2;
+  msgpb.MsgPosition seek_position = 3;
+  repeated SegmentInfo unflushedSegments = 4;
+  repeated int64 flushedSegments = 5;
+}
+
+message WatchDmChannelsRequest {
+  common.MsgBase base = 1;
+  repeated VchannelInfo vchannels = 2;
+}
+```
+
+DataNode consists of multiple DataSyncService, each service controls one flowgraph.
+
+```go
+// DataNode
+type DataNode struct {
+    ...
+    vchan2Sync map[string]*dataSyncService
+    vchan2FlushCh map[string]chan<- *flushMsg
+
+    clearSignal chan UniqueID
+    ...
+}
+
+// DataSyncService
+type dataSyncService struct {
+	ctx          context.Context
+	fg           *flowgraph.TimeTickedFlowGraph
+	flushChan    <-chan *flushMsg
+	replica      Replica
+	idAllocator  allocatorInterface
+	msFactory    msgstream.Factory
+	collectionID UniqueID
+}
+```
+
+DataNode Init -> Register to etcd -> Discovery data service -> Discover master service -> IDLE
+
+WatchDmChannels -> new dataSyncService -> HEALTH
+
+`WatchDmChannels:`
+
+1. If `DataNode.vchan2Sync` is empty, DataNode is in IDLE, `WatchDmChannels` will create new dataSyncService for every unique vchannel, then DataNode is in HEALTH.
+2. If vchannel name of `VchannelPair` is not in `DataNode.vchan2Sync`, create a new dataSyncService.
+3. If vchannel name of `VchannelPair` is in `DataNode.vchan2Sync`, ignore.
+
+```
+
+#### The boring design
+
+• If collection:flowgraph = 1 : 1, datanode must have the ability to scale flowgraph.
+
+![datanode_design](../assets/graphs/collection_flowgraph_1_1.jpg)
+
+•** [Winner]** If collection:flowgraph = 1 : n, flowgraph:vchannel = 1:1
+
+![datanode_design](../assets/graphs/collection_flowgraph_1_n.png)
+
+• If collection:flowgraph = n : 1, in the blue cases, datanode must have the ability to scale flowgraph. In the brown cases, flowgraph must be able to scale channels.
+
+![datanode_design](../assets/graphs/collection_flowgraph_n_1.jpg)
+
+• If collection:flowgraph = n : n  , load balancing on vchannels.
+
+![datanode_design](../assets/graphs/collection_flowgraph_n_n.jpg)
@@ -0,0 +1,86 @@
+# DataNode Flowgraph Recovery Design
+
+update: 6.4.2021, by [Goose](https://github.com/XuanYang-cn)
+update: 6.21.2021, by [Goose](https://github.com/XuanYang-cn)
+
+## 1. Common Sense
+
+A. One message stream to one vchannel, so there are one start and one end position in one message pack.
+
+B. Only when DataNode flushes, DataNode will update every segment's position.
+An optimization: update position of
+
+1. Current flushing segment
+2. StartPosition of segments has never been flushed.
+
+C. DataNode auto-flush is a valid flush.
+
+D. DDL messages are now in DML Vchannels.
+
+## 2. Segments in Flowgraph
+
+![segments](../assets/graphs/segments.png)
+
+## 3. Flowgraph Recovery
+
+### A. Save checkpoints
+
+When a flowgraph flushes a segment, we need to save these things:
+
+- current segment's binlog paths.
+- current segment positions.
+- all other segments' current positions from the replica (If a segment hasn't been flushed, save the position when DataNode first meets it).
+
+Whether save successfully:
+
+- If succeeded, flowgraph updates all segments' positions to the replica.
+- If not
+  - For a grpc failure(this failure will appear after many times retry internally), crash itself.
+  - For a normal failure, retry save 10 times, if still fails, crash itself.
+
+### B. Recovery from a set of checkpoints
+
+1. We need all positions of all segments in this vchannel `p1, p2, ... pn`.
+
+Proto design for WatchDmChannelReq:
+
+```proto
+message VchannelInfo {
+  int64 collectionID = 1;
+  string channelName = 2;
+  msgpb.MsgPosition seek_position = 3;
+  repeated SegmentInfo unflushedSegments = 4;
+  repeated int64 flushedSegments = 5;
+}
+
+message WatchDmChannelsRequest {
+  common.MsgBase base = 1;
+  repeated VchannelInfo vchannels = 2;
+}
+```
+
+2. We want to filter msgPacks based on these positions.
+
+![recovery](../assets/graphs/flowgraph_recovery_design.png)
+
+Supposing we have segments `s1, s2, s3`, corresponding positions `p1, p2, p3`
+
+- Sort positions in reverse order `p3, p2, p1`
+- Get segments dup range time: `s3 ( p3 > mp_px > p1)`, `s2 (p2 > mp_px > p1)`, `s1(zero)`
+- Seek from the earliest, in this example `p1`
+- Then for every msgPack after seeking `p1`, the pseudocode:
+
+```go
+const filter_threshold = recovery_time
+// mp means msgPack
+for mp := seeking(p1) {
+    if mp.position.endtime < filter_threshold {
+        if mp.position < p3 {
+            filter s3
+        }
+        if mp.position < p2 {
+            filter s2
+        }
+    }
+}
+```
@@ -0,0 +1,199 @@
+# 8. IndexCoord Design
+
+update: 7.31.2021, by [Cai.Zhang](https://github.com/xiaocai2333)
+
+## 8.0 Component Description
+
+IndexCoord is a component responsible for scheduling index construction tasks and maintaining index status. IndexCoord accepts requests from rootCoord to build indexes, delete indexes, and query index information. IndexCoord is responsible for assigning IndexBuildID to the request to build the index, and forwarding the request to build the index to IndexNode. IndexCoord records the status of the index, and the index file.
+
+The following figure shows the design of the indexCoord component:
+
+![indexcoord](../assets/graphs/indexcoord_design.png)
+
+## 8.1 Use etcd as a reliable service
+
+IndexCoord, like the other Milvus components, relies on etcd to implement
+service discovery. IndexCoord relies on the lease mechanism of etcd to sense the online and offline news of IndexNode.
+
+In addition to service discovery, Milvus also uses etcd as a reliable meta storage and writes all
+persistent status information to etcd. The purpose is to restore a certain Milvus component to its original
+state after power off and restart.
+
+## 8.2 Receive requests about index from RootCoord
+
+IndexCoord receives requests from RootCoord to build an index, delete an index, and query the status of an index.
+
+In Milvus, index building is performed asynchronously. When IndexCoord receives a request to build an index from
+RootCoord, it will first check whether the same index has been created according to the index parameters. If yes, it would
+return the IndexBuildID of the existing task. Otherwise, it would assign a globally unique IndexBuildID to the task,
+record the task in the MetaTable, write the MetaTable to etcd, and then return the IndexBuildID to RootCoord.
+RootCoord confirms that the index building is generated successfully by the IndexBuildID. At this time, the index construction
+is not completed yet. IndexCoord starts a background process to find all the index tasks that need to be
+allocated periodically and then allocates them to IndexNode for actual execution.
+
+When IndexCoord receives a request to delete an index from RootCoord, IndexCoord traverses the MetaTable,
+marks the corresponding index task as deleted, and returns. It is not deleted from the MetaTable at this time.
+IndexCoord has another background process that periodically queries the index tasks that need to be deleted.
+When the index task is marked as deleted, and the index status is complete, the corresponding index task is actually
+deleted from the MetaTable.
+
+When IndexCoord receives a query index status request from other components, it will first check whether the corresponding
+index task is marked for deletion in the MetaTable. If marked for deletion, it returns that index does not exist, otherwise,
+it returns the index information.
+
+## 8.3 Feature Design
+
+IndexCoord has two main structures, NodeManager and MetaTable. NodeManager is used to manage IndexNode node information,
+and MetaTable is used to maintain index-related information.
+
+IndexCoord mainly has these functions: 
+
+`watchNodeLoop` is mainly responsible for monitoring the changes of IndexNode nodes;
+
+`watchMetaLoop` is mainly responsible for monitoring the changes of Meta;
+
+`assignTaskLoop` is mainly responsible for assigning index building tasks;
+
+`recycleUnusedIndexFiles` is mainly responsible for cleaning up useless index files and deleted index records;
+
+### 8.3.1 The relationship between IndexCoord and IndexNode
+
+IndexCoord is responsible for assigning index construction tasks and maintaining index status. 
+
+IndexNode is a node that executes index building tasks.
+
+### 8.3.2 NodeManager
+
+NodeManager is responsible for managing the node information of IndexNode, and contains a priority queue to save the load information of each IndexNode. 
+The load information of IndexNode is based on the number of tasks executed. When the IndexCoord service starts, it first obtains the node information of all current IndexNodes from etcd, 
+and then adds the node information to the NodeManager. After that, the online and offline information of IndexNode node is obtained from watchNodeLoop. 
+Then it will traverse the entire MetaTable, get the load information corresponding to each IndexNode node, and update the priority queue in the NodeManager. 
+When an index building task needs to be allocated, the IndexNode with the lowest load will be selected according to the priority queue to execute the task.
+
+### 8.3.3 MetaTable
+
+To maintain the status information of the index, we introduced MetaTable to record the status information
+of the index. In order to ensure that the MetaTable information is not lost after IndexCoord is powered off and
+restarted, we write the MetaTable information into etcd. When the IndexCoord service starts, it will first load the
+existing Meta information from etcd, and then monitor the changes of Meta through watchNodeLoop. In order to distinguish
+whether the modification of Meta was initiated by IndexCoord or IndexNode, the revision was introduced in Meta.
+When watchMetaLoop detects that the Meta in etcd is updated, compare the revision in Meta with the Event.Kv.Version
+of the etcd event. If the revision equals to Event.Kv.Version, it means that the update was initiated by IndexCoord.
+If the revision is less than Event.Kv.Version, it means that this Meta update was initiated by IndexNode, and IndexCoord
+needs to update Meta. There will be no situation where revision is greater than Event.Kv.Version.
+
+In order to prevent IndexNode from appearing in a suspended animation state, Version is introduced. When IndexCoord
+finds that IndexNode is offline, it assigns the unfinished tasks that IndexNode is responsible for to other IndexNodes,
+and adds 1 to Version. After the task is completed, it is found that the version corresponding to the task is already
+larger than the version corresponding to the task it is executing, and the Meta is not updated.
+
+### 8.3.4 watchNodeLoop
+
+`watchNodeLoop` is used to monitor IndexNode going online and offline. When IndexNode goes online and offline,
+IndexCoord adds or deletes the corresponding IndexNode information in NodeManager.
+
+### 8.3.5 watchMetaLoop
+
+`watchMetaLoop` is used to monitor whether the Meta in etcd has been changed. When the Meta in etcd is monitored,
+the result of the Meta update is obtained from etcd, and the `Event.Kv.Version` of the update event is compared
+with the `revision` in the MetaTable. If the `Event.Kv.Version` is greater than the `revision` in the MetaTable,
+it means that this update is initiated by IndexNode, and then updates the MetaTable in IndexCoord. Since this update
+is initiated by IndexNode, it indicates that this IndexNode has completed this task, so update the load of this
+IndexNode in NodeManager, and the task amount is reduced by one.
+
+### 8.3.6 assignTaskLoop
+
+`assignTaskLoop` is used to assign index construction tasks. There is a timer here to traverse the MetaTable regularly
+to filter out the tasks that need to be allocated, including unallocated tasks and tasks that have been failed due to
+indexNode crash. Then sort according to the version size of each task, and assign tasks with a smaller
+version first. The purpose is to prevent certain special tasks from occupying resources all the time and always failing
+to execute successfully. When a task is assigned, its corresponding Version is increased by one. Then send the task to
+IndexNode for execution, and update the index status in the MetaTable.
+
+### 8.3.7 recycleUnusedIndexFiles
+
+Delete useless index files, including lower version index files and index files corresponding to the deleted index.
+In order to distinguish whether the low version index file corresponding to the index has been cleaned up, recycled is
+introduced as a mark. Only after the index task is completed, the lower version index files will be cleaned up, and the
+index file corresponding to the lower version index file will be marked as True.
+
+This is also a timer, which periodically traverses the MetaTable to obtain the index corresponding to the index file
+that need to be cleaned up. If the index is marked as deleted, the information corresponding to the index is deleted
+in the MetaTable. Otherwise, only the lower version index file is cleaned up.
+
+## 8.4 IndexNode Create Index
+
+IndexNode is the execution node of index building tasks, and all index building tasks are forwarded to IndexNode by
+IndexCoord for execution. When IndexNode executes an index build request, it first reads IndexMeta information
+from etcd, and checks whether the index task is marked for deletion when IndexCoord is forwarded to IndexNode.
+If it is marked as deleted, then there is no need to actually build the index, just mark the index task status as
+completed, and then write it to etcd. When IndexCoord perceives that the status corresponding to the index is
+complete, it deletes the index task from the MetaTable. If it is checked that the index is not marked for deletion,
+then the index needs to be built. The original data must be loaded first when building the index. The original data
+is stored in MinIO/S3, and the storage path is notified by RootCoord in the index build request. After loading the
+original data, the data is deserialized into data blocks, and then cgo is called to build the index. When the index is
+built, the index data is serialized into data blocks, and then written into the file. The directory organization of the
+index file is "indexBuildID/IndexTaskVersion/partitionID/segmentID/key", where key corresponds to the serialized key
+of index data. After the index is built, record the index file directory in IndexMeta, and then write it to etcd.
+
+## 8.5 API
+
+### 8.5.1 BuildIndex
+
+Index building is asynchronous, so when an index building request comes, an IndexBuildID is assigned to the task, and
+the task is recorded in Meta. The background process assignTaskLoop will find this task and assign it to IndexNode for
+execution.
+
+The following figure shows the state machine of IndexTask during execution:
+
+![IndexState](../assets/graphs/IndexState.png)
+
+### 8.5.2 DropIndex
+
+DropIndex deletes an index based on IndexID. One IndexID corresponds to the index of an entire column. A column is
+divided into many segments, and each segment corresponds to an IndexBuildID. IndexCoord uses IndexBuildID to record
+index tasks. Therefore, when DropIndex, delete all tasks corresponding to IndexBuildID corresponding to IndexID.
+
+## 8.6 Key Term
+
+### 8.6.1 Meta
+
+```go
+type Meta struct {
+        indexMeta *indexpb.IndexMeta
+        revision  int64
+}
+```
+
+Meta is used to record the state of the index.
+
+- Revision: The number of times IndexMeta has been changed in etcd. It's the same as Event.Kv.Version in etcd.
+  When IndexCoord watches the IndexMeta in etcd is changed, can compare `revision` and Event.Kv.Version to determine
+  this modification of IndexMeta is caused by IndexCoord or IndexNode. If it is caused by IndexNode, the Meta in
+  IndexCoord must be updated.
+
+### 8.6.2 IndexMeta
+
+```ProtoBuf
+message IndexMeta {
+  int64 indexBuildID = 1;
+  common.IndexState state = 2;
+  string fail_reason = 3;
+  BuildIndexRequest req = 4;
+  repeated string index_file_paths = 5;
+  bool mark_deleted = 6;
+  int64 nodeID = 7;
+  int64 version = 8;
+  bool recycled = 9;
+}
+```
+
+- indexBuildID: ID of the index task.
+- state: The state of the index.
+- fail_reason: The reason why the index build failed.
+- req: The request for the building index.
+- index_file_paths: The paths of index files.
+- mark_deleted: Mark whether the index has been deleted.
+- nodeID: ID of the IndexNode that built the index.
+- version: Number of retries for the index.
+- recycled: Mark whether the unused files of the index have been cleaned up.
@@ -0,0 +1,163 @@
+# Flush Collection
+The `Flush` operation is used to make sure that inserted data will be written into persistent storage. This document will introduce how the `Flush` operation works in `Milvus 2.0`. The following figure shows the execution flow of `Flush`.
+
+![flush_collections](../assets/graphs/flush_data_coord.png)
+
+1. Firstly, `SDK` sends a `Flush` request to `Proxy` via `Grpc`, the `proto` is defined as follows:
+```proto
+service MilvusService {
+  ...
+  rpc Flush(FlushRequest) returns (FlushResponse) {}
+  ...
+}
+
+message FlushRequest {
+  common.MsgBase base = 1;
+  string db_name = 2;
+  repeated string collection_names = 3;
+}
+
+message FlushResponse{
+  common.Status status = 1;
+  string db_name = 2;
+  map<string, schema.LongArray> coll_segIDs = 3;
+}
+```
+
+
+2. When `Proxy` receives `Flush` request, it would wrap this request into `FlushTask`, and push this task into `DdTaskQueue` queue. After that, `Proxy` would call `WatiToFinish` to wait until the task finished.
+```go
+type task interface {
+	TraceCtx() context.Context
+	ID() UniqueID       // return ReqID
+	SetID(uid UniqueID) // set ReqID
+	Name() string
+	Type() commonpb.MsgType
+	BeginTs() Timestamp
+	EndTs() Timestamp
+	SetTs(ts Timestamp)
+	OnEnqueue() error
+	PreExecute(ctx context.Context) error
+	Execute(ctx context.Context) error
+	PostExecute(ctx context.Context) error
+	WaitToFinish() error
+	Notify(err error)
+}
+
+type FlushTask struct {
+	Condition
+	*milvuspb.FlushRequest
+	ctx       context.Context
+	dataCoord types.DataCoord
+	result    *milvuspb.FlushResponse
+}
+```
+
+3. There is a background service in `Proxy`. This service gets `FlushTask` from `DdTaskQueue`, and executes in three phases:
+    - `PreExecute`
+
+      `FlushTask` does nothing at this phase, and returns directly
+
+    - `Execute`
+
+      `Proxy` sends a `Flush` request to `DataCoord` via `Grpc`, and waits for the response, the `proto` is defined as follows:
+    ```proto
+    service DataCoord {
+      ...
+      rpc Flush(FlushRequest) returns (FlushResponse) {}
+      ...
+    }
+
+    message FlushRequest {
+      common.MsgBase base = 1;
+      int64 dbID = 2;
+      int64 collectionID = 4;
+    }
+
+    message FlushResponse {
+      common.Status status = 1;
+      int64 dbID = 2;
+      int64 collectionID = 3;
+      repeated int64 segmentIDs = 4;
+    }
+    ```
+    - `PostExecute`
+
+      `FlushTask` does nothing at this phase, and returns directly
+
+4. After receiving a `Flush` request from `Proxy`, `DataCoord` would call `SealAllSegments` to seal all the growing segments belonging to this `Collection`, and would not allocate new `ID`s for these segments anymore. After that, `DataCoord` would send a response to `Proxy`, which contains all the sealed segment `ID`s.
+
+5. In `Milvus 2.0`,  `Flush` is an asynchronous operation. So when `SDK` receives the response of `Flush`, it only means that the `DataCoord` has sealed these segments. There are 2 problems that we have to solve.
+    - The sealed segments might still in memory, and have not been written into persistent storage yet.
+    - `DataCoord` would no longer allocate new `ID`s for these sealed segments, but how to make sure all the allocated `ID`s have been consumed by `DataNode`.
+
+
+6. For the first problem, `SDK` should send `GetSegmentInfo` request to `DataCoord` periodically, until all sealed segments are in state of `Flushed`. The `proto` is defined as follows.
+```proto
+service DataCoord {
+  ...
+  rpc GetSegmentInfo(GetSegmentInfoRequest) returns (GetSegmentInfoResponse) {}
+  ...
+}
+
+message GetSegmentInfoRequest {
+  common.MsgBase base = 1;
+  repeated int64 segmentIDs = 2;
+}
+
+message GetSegmentInfoResponse {
+  common.Status status = 1;
+  repeated SegmentInfo infos = 2;
+}
+
+message SegmentInfo {
+  int64 ID = 1;
+  int64 collectionID = 2;
+  int64 partitionID = 3;
+  string insert_channel = 4;
+  int64 num_of_rows = 5;
+  common.SegmentState state = 6;
+  msgpb.MsgPosition dml_position = 7;
+  int64 max_row_num = 8;
+  uint64 last_expire_time = 9;
+  msgpb.MsgPosition start_position = 10;
+}
+
+enum SegmentState {
+    SegmentStateNone = 0;
+    NotExist = 1;
+    Growing = 2;
+    Sealed = 3;
+    Flushed = 4;
+    Flushing = 5;
+}
+
+```
+
+7. For the second problem, `DataNode` would report a timestamp to `DataCoord` every time it consumes a package from `MsgStream`, the `proto` is defined as follows.
+
+ ```proto
+message DataNodeTtMsg {
+  common.MsgBase base = 1;
+  string channel_name = 2;
+  uint64 timestamp = 3;
+}
+ ```
+
+8. There is a background service, `startDataNodeTsLoop`, in `DataCoord` to process the message of `DataNodeTtMsg`.
+    - Firstly, `DataCoord` would extract `channel_name` from `DataNodeTtMsg`, and filter out all sealed segments that are attached on this `channel_name`
+    - Compare the timestamp when the segment enters into state of `Sealed` with the `DataNodeTtMsg.timestamp`, if `DataNodeTtMsg.timestamp` is greater, which means that all `ID`s belonging to that segment have been consumed by `DataNode`, it's safe to notify `DataNode` to write that segment into persistent storage. The `proto` is defined as follows:
+```proto
+service DataNode {
+  ...
+  rpc FlushSegments(FlushSegmentsRequest) returns(common.Status) {}
+  ...
+}
+
+message FlushSegmentsRequest {
+  common.MsgBase base = 1;
+  int64 dbID = 2;
+  int64 collectionID = 3;
+  repeated int64 segmentIDs = 4;
+}
+```
@@ -0,0 +1,206 @@
+# Drop Collection
+
+`Milvus 2.0` uses `Collection` to represent a set of data, like `Table` in traditional database. Users can create or drop `Collection`.
+This article introduces the execution path of `Drop Collection`. At the end of this article, you should know which components are involved in `Drop Collection`.
+
+The execution flow of `Drop Collection` is shown in the following figure:
+
+![drop_collection](../assets/graphs/dml_drop_collection.png)
+
+1. Firstly, `SDK` sends a `DropCollection` request to `Proxy` via `Grpc`, the `proto` is defined as follows:
+
+```proto
+service MilvusService {
+    ...
+
+    rpc DropCollection(DropCollectionRequest) returns (common.Status) {}
+
+    ...
+}
+
+message DropCollectionRequest {
+  // Not useful for now
+  common.MsgBase base = 1;
+  // Not useful for now
+  string db_name = 2;
+  // Required, the collection name in milvus
+  string collection_name = 3;
+}
+```
+
+2. Once the `DropCollection` request is received, the `Proxy` would wrap this request into `DropCollectionTask`, and push this task into `DdTaskQueue` queue. After that, `Proxy` would call `WaitToFinish` method to wait until the task is finished.
+
+```go
+type task interface {
+	TraceCtx() context.Context
+	ID() UniqueID       // return ReqID
+	SetID(uid UniqueID) // set ReqID
+	Name() string
+	Type() commonpb.MsgType
+	BeginTs() Timestamp
+	EndTs() Timestamp
+	SetTs(ts Timestamp)
+	OnEnqueue() error
+	PreExecute(ctx context.Context) error
+	Execute(ctx context.Context) error
+	PostExecute(ctx context.Context) error
+	WaitToFinish() error
+	Notify(err error)
+}
+
+type DropCollectionTask struct {
+	Condition
+	*milvuspb.DropCollectionRequest
+	ctx       context.Context
+	rootCoord types.RootCoord
+	result    *commonpb.Status
+	chMgr     channelsMgr
+	chTicker  channelsTimeTicker
+}
+```
+
+3. There is a background service in `Proxy`, this service would get the `DropCollectionTask` from `DdTaskQueue`, and execute it in three phases:
+
+   - `PreExecute`, do some static checking at this phase, such as check if `Collection Name` is legal etc.
+   - `Execute`, at this phase, `Proxy` would send `DropCollection` request to `RootCoord` via `Grpc`, and wait the response, the `proto` is defined as below:
+
+   ```proto
+       service RootCoord {
+         ...
+
+          rpc DropCollection(milvus.DropCollectionRequest) returns (common.Status) {}
+
+         ...
+       }
+   ```
+
+   - `PostExecute`, `Proxy` would delete `Collection`'s meta from global meta table at this phase.
+
+4. `RootCoord` would wrap the `DropCollection` request into `DropCollectionReqTask`, and then call function `executeTask`. `executeTask` would return until the `context` is done or `DropCollectionReqTask.Execute` is returned.
+
+```go
+type reqTask interface {
+	Ctx() context.Context
+	Type() commonpb.MsgType
+	Execute(ctx context.Context) error
+	Core() *Core
+}
+
+type DropCollectionReqTask struct {
+	baseReqTask
+	Req *milvuspb.DropCollectionRequest
+}
+```
+
+5. Firstly, `RootCoord` would delete `Collection`'s meta from `metaTable`, including `schema`,`partition`, `segment`,`index`. All of these delete operations are committed in one transaction.
+
+6. After `Collection`'s meta has been deleted from `metaTable`, `Milvus` would consider this collection has been deleted successfully.
+
+7. `RootCoord` would alloc a timestamp from `TSO` before deleting `Collection`'s meta from `metaTable`. This timestamp is considered as the point when the collection was deleted.
+
+8. `RootCoord` would send a message of `DropCollectionRequest` into `MsgStream`. Thus other components, who have subscribed to the `MsgStream`, would be notified. The `Proto` of `DropCollectionRequest` is defined as below:
+
+```proto
+message DropCollectionRequest {
+  common.MsgBase base = 1;
+  string db_name = 2;
+  string collectionName = 3;
+  int64 dbID = 4;
+  int64 collectionID = 5;
+}
+
+```
+
+9. After these operations, `RootCoord` would update internal timestamp.
+
+10. Then `RootCoord` would start a `ReleaseCollection` request to `QueryCoord` via `Grpc` , notify `QueryCoord` to release all resources that related to this `Collection`. This `Grpc` request is done in another `goroutine`, so it would not block the main thread. The `proto` is defined as follows:
+
+```proto
+service QueryCoord {
+    ...
+
+    rpc ReleaseCollection(ReleaseCollectionRequest) returns (common.Status) {}
+
+    ...
+}
+
+message ReleaseCollectionRequest {
+  common.MsgBase base = 1;
+  int64 dbID = 2;
+  int64 collectionID = 3;
+  int64 nodeID = 4;
+}
+```
+
+11. At last, `RootCoord` would send `InvalidateCollectionMetaCache` request to each `Proxy`, notify `Proxy` to remove `Collection`'s meta. The `proto` is defined as follows:
+
+```proto
+service Proxy {
+    ...
+
+    rpc InvalidateCollectionMetaCache(InvalidateCollMetaCacheRequest) returns (common.Status) {}
+
+    ...
+}
+
+message InvalidateCollMetaCacheRequest {
+  common.MsgBase base = 1;
+  string db_name = 2;
+  string collection_name = 3;
+}
+```
+
+12. The execution flow of `QueryCoord.ReleaseCollection` is shown in the following figure:
+
+![release_collection](../assets/graphs/dml_release_collection.png)
+
+13. `QueryCoord` would wrap `ReleaseCollection` into `ReleaseCollectionTask`, and push the task into `TaskScheduler`
+
+14. There is a background service in `QueryCoord`. This service would get the `ReleaseCollectionTask` from `TaskScheduler`, and execute it in three phases:
+
+    - `PreExecute`, `ReleaseCollectionTask` would only print debug log at this phase.
+    - `Execute`, there are two jobs at this phase:
+
+      - send a `ReleaseDQLMessageStream` request to `RootCoord` via `Grpc`, `RootCoord` would redirect the `ReleaseDQLMessageStream` request to each `Proxy`, and notify the `Proxy` that stop processing any message of this `Collection` anymore. The `proto` is defined as follows:
+
+      ```proto
+          message ReleaseDQLMessageStreamRequest {
+              common.MsgBase base = 1;
+              int64 dbID = 2;
+              int64 collectionID = 3;
+          }
+      ```
+
+      - send a `ReleaseCollection` request to each `QueryNode` via `Grpc`, and notify the `QueryNode` to release all the resources related to this `Collection`, including `Index`, `Segment`, `FlowGraph`, etc. `QueryNode` would no longer read any message from this `Collection`'s `MsgStream` anymore
+
+      ```proto
+          service QueryNode {
+              ...
+
+              rpc ReleaseCollection(ReleaseCollectionRequest) returns (common.Status) {}
+
+              ...
+          }
+
+          message ReleaseCollectionRequest {
+              common.MsgBase base = 1;
+              int64 dbID = 2;
+              int64 collectionID = 3;
+              int64 nodeID = 4;
+          }
+      ```
+
+    - `PostExecute`, `ReleaseCollectionTask` would only print debug log at this phase.
+
+15. After these operations, `QueryCoord` would send `ReleaseCollection`'s response to `RootCoord`.
+
+16. At `Step 8`, `RootCoord` has sent a message of `DropCollectionRequest` into `MsgStream`. `DataNode` would subscribe this `MsgStream`, so that it would be notified to release related resources. The execution flow is shown in the following figure.
+
+![release_collection](../assets/graphs/dml_release_flow_graph_on_data_node.png)
+
+17. In `DataNode`, each `MsgStream` will have a `FlowGraph`, which processes all messages. When the `DataNode` receives the message of `DropCollectionRequest`, `DataNode` would notify `BackGroundGC`, which is a background service on `DataNode`, to release resources.
+
+_Notes_:
+
+1. Currently, the `DataCoord` doesn't have response to the `DropCollection`. So the `Collection`'s `segment meta` still exists in the `DataCoord`'s `metaTable`, and the `Binlog` files belonging to this `Collection` still exist in the persistent storage.
+2. Currently, the `IndexCoord` doesn't have response to the `DropCollection`. So the `Collection`'s `index file` still exists in the persistent storage.
@@ -0,0 +1,69 @@
+# Hybrid Timestamp in Milvus
+
+In chapter [Milvus TimeSync Mechanism](./milvus_timesync_en.md), we have already known why we need TSO in Milvus. Milvus
+uses the [TiKV's](https://github.com/tikv/tikv) implementation into TSO. So if you are interested in how TSO is
+implemented, you can look into the official documentation of TiKV.
+
+This chapter will only introduce two points:
+
+1. the organization of hybrid TSO in Milvus;
+2. how should we parse the hybrid TSO;
+
+## The Organization of TSO
+
+The type of TSO is `uint64`. As shown in the figure below, TSO was organized by two parts:
+
+1. physical part;
+2. logical part;
+
+The front 46 bits is of physical part, and the last 18 bits is of logical part.
+
+Note, physical part is the UTC time in Milliseconds.
+
+![Timestamp struct](../assets/graphs/time_stamp_struct.jpg)
+
+For some users such as DBAs, they would want to sort the operations and list them in UTC time order.
+
+Actually, we can use the TSO order to sort the `Insert` operations or `Delete` operations.
+
+So the question becomes how we get the UTC time from TSO.
+
+As we have described above, the physical part consists of the front 46 bits of TSO.
+
+So given a TSO which is returned by `Insert` or `Delete`, we can directly shift the left 18 bits to get the UTC time.
+
+For example in Golang:
+
+```go
+const (
+	logicalBits     = 18
+	logicalBitsMask = (1 << logicalBits) - 1
+)
+
+// ParseTS parses the ts to (physical,logical).
+func ParseTS(ts uint64) (time.Time, uint64) {
+	logical := ts & logicalBitsMask
+	physical := ts >> logicalBits
+	physicalTime := time.Unix(int64(physical/1000), int64(physical)%1000*time.Millisecond.Nanoseconds())
+	return physicalTime, logical
+}
+```
+
+In Python:
+
+```python
+>>> import datetime
+>>> LOGICAL_BITS = 18
+>>> LOGICAL_BITS_MASK = (1 << LOGICAL_BITS) - 1
+>>> def parse_ts(ts):
+...     logical = ts & LOGICAL_BITS_MASK
+...     physical = ts >> LOGICAL_BITS
+...     return physical, logical
+... 
+>>> ts = 429164525386203142
+>>> utc_ts_in_milliseconds, _ = parse_ts(ts)
+>>> d = datetime.datetime.fromtimestamp(utc_ts_in_milliseconds / 1000.0)
+>>> d.strftime('%Y-%m-%d %H:%M:%S')
+'2021-11-17 15:05:41'
+>>> 
+```
@@ -0,0 +1,132 @@
+# Timesync -- All The things you should know
+
+`Time Synchronization` is the kernel part of Milvus 2.0; it affects all components of the system. This document describes the detailed design of `Time Synchronization`.
+
+There are 2 kinds of events in Milvus 2.0:
+
+- DDL events
+  - create collection
+  - drop collection
+  - create partition
+  - drop partition
+- DML events
+  - insert
+  - search
+  - etc
+
+All events have a `Timestamp` to indicate when this event occurs.
+
+Suppose there are two users, `u1` and `u2`. They connect to Milvus and do the following operations at the respective timestamps.
+
+| ts  | u1                   | u2           |
+| --- | -------------------- | ------------ |
+| t0  | create Collection C0 | -            |
+| t2  | -                    | search on C0 |
+| t5  | insert A1 into C0    | -            |
+| t7  | -                    | search on C0 |
+| t10 | insert A2            | -            |
+| t12 | -                    | search on C0 |
+| t15 | delete A1 from C0    | -            |
+| t17 | -                    | search on C0 |
+
+Ideally, `u2` expects `C0` to be empty at `t2`, and could only see `A1` at `t7`; while `u2` could see both `A1` and `A2` at `t12`, but only see `A2` at `t17`.
+
+It's easy to achieve this in a `single-node` database. But for a `Distributed System`, such as `Milvus`, it's a little difficult; the following problems need to be solved:
+
+1. If `u1` and `u2` are on different nodes, and their time clock is not synchronized. To give an extreme example, suppose that the time of `u2` is 24 hours later than `u1`, then all the operations of `u1` can't be seen by `u2` until the next day.
+2. Network latency. If `u2` starts the `Search on C0` at `t17`, then how can it be guaranteed that all the `events` before `t17` have been processed? If the events of `delete A1 from C0` have been delayed due to the network latency, then it would lead to an incorrect state: `u2` would see both `A1` and `A2` at `t17`.
+
+`Time synchronization system` is used to solve the above problems.
+
+## Timestamp Oracle(TSO)
+
+Like [TiKV](https://github.com/tikv/tikv), Milvus 2.0 provides `TSO` service. All the events must alloc timestamp from `TSO`, not from the local clock, so the first problem can be solved.
+
+`TSO` is provided by the `RootCoord` component. Clients could alloc one or more timestamp in a single request; the `proto` is defined as follows.
+
+```proto
+service RootCoord {
+    ...
+    rpc AllocTimestamp(AllocTimestampRequest) returns (AllocTimestampResponse) {}
+    ...
+}
+
+message AllocTimestampRequest {
+  common.MsgBase base = 1;
+  uint32 count = 3;
+}
+
+message AllocTimestampResponse {
+    common.Status status = 1;
+    uint64 timestamp = 2;
+    uint32 count = 3;
+}
+```
+
+`Timestamp` is of type `uint64`, and contains physical and logical parts.
+
+This is the format of `Timestamp`
+
+![Timestamp struct](../assets/graphs/time_stamp_struct.jpg)
+
+In an `AllocTimestamp` request, if `AllocTimestampRequest.count` is greater than `1`, `AllocTimestampResponse.timestamp` indicates the first available timestamp in the response.
+
+## Time Synchronization
+
+To better understand `Time Synchronization`, let's introduce the data operation of Milvus 2.0 briefly.
+Take `Insert Operation` as an example.
+
+- User can configure lots of `Proxy` to achieve load balancing, in `Milvus 2.0`
+- User can use `SDK` to connect to any `Proxy`
+- When `Proxy` receives `Insert` Request from `SDK`, it splits `InsertMsg` into different `MsgStream` according to the hash value of `Primary Key`
+- Each `InsertMsg` would be assigned with a `Timestamp` before sending to the `MsgStream`
+
+>*Note: `MsgStream` is the wrapper of message queue, the default message queue in `Milvus 2.0` is `pulsar`*
+
+![proxy insert](../assets/graphs/timesync_proxy_insert_msg.png)
+
+Based on the above information, we know that the `MsgStream` has the following characteristics:
+
+- In `MsgStream`, `InsertMsg` from the same `Proxy` must be incremented in timestamp
+- In `MsgStream`, `InsertMsg` from different `Proxy` have no relationship in timestamp
+
+The following figure shows an example of `InsertMsg` in `MsgStream`. The snippet contains 5 `InsertMsg`, 3 of them from `Proxy1` and others from `Proxy2`.
+
+The 3 `InsertMsg` from `Proxy1` are incremented in timestamp, and the 2 `InsertMsg` from `Proxy2` are also incremented in timestamps, but there is no relationship between `Proxy1` and `Proxy2`.
+
+![msgstream](../assets/graphs/timesync_msgstream.png)
+
+So the second problem has turned into this: After reading a message from `MsgStream`, how to make sure that all the messages with smaller timestamp have been consumed?
+
+For example, when reading a message with timestamp `110` produced by `Proxy2`, but the message with timestamp `80` produced by `Proxy1`, is still in the `MsgStream`. How can this situation be handled?
+
+The following graph shows the core logic of `Time Synchronization System` in `Milvus 2.0`; it should solve the second problem.
+
+- Each `Proxy` will periodically report its latest timestamp of every `MsgStream` to `RootCoord`; the default interval is `200ms`
+- For each `Msgstream`, `Rootcoord` finds the minimum timestamp of all `Proxy` on this `Msgstream`, and inserts this minimum timestamp into the `Msgstream`
+- When the consumer reads the timestamp inserted by the `RootCoord` on the `MsgStream`, it indicates that all messages with smaller timestamp have been consumed, so all actions that depend on this timestamp can be executed safely
+- The message inserted by `RootCoord` into `MsgStream` is of type `TimeTick`
+
+![upload time tick](../assets/graphs/timesync_proxy_upload_time_tick.png)
+
+This is the `Proto` that is used by `Proxy` to report timestamp to `RootCoord`:
+
+```proto
+service RootCoord {
+    ...
+    rpc UpdateChannelTimeTick(internal.ChannelTimeTickMsg) returns (common.Status) {}
+    ... 
+}
+
+message ChannelTimeTickMsg {
+  common.MsgBase base = 1;
+  repeated string channelNames = 2;
+  repeated uint64 timestamps = 3;
+  uint64 default_timestamp = 4;
+}
+```
+
+After inserting `Timetick`, the `Msgstream` should look like this:
+![msgstream time tick](../assets/graphs/timesync_msgtream_timetick.png)
+
+`MsgStream` will process the messages in batches according to `TimeTick`, and ensure that the output messages meet the requirements of timestamp. For more details, please refer to the `MsgStream` design details.
@@ -0,0 +1,140 @@
+# Create Collection
+
+`Milvus 2.0` uses `Collection` to represent a set of data, like `Table` in a traditional database. User can create or drop `Collection`. 
+This article introduces the execution path of `CreateCollection`, at the end of this article, you should know which components are involved in `CreateCollection`.
+
+The execution flow of `CreateCollection` is shown in the following figure:
+
+![create_collection](../assets/graphs/dml_create_collection.png)
+
+1. Firstly, `SDK` starts a `CreateCollection` request to `Proxy` via `Grpc`, the `proto` is defined as follows:
+
+```proto
+service MilvusService {
+    ...
+
+    rpc CreateCollection(CreateCollectionRequest) returns (common.Status) {}
+
+    ...
+}
+
+message CreateCollectionRequest {
+  // Not useful for now
+  common.MsgBase base = 1;
+  // Not useful for now
+  string db_name = 2;
+  // The unique collection name in milvus.(Required)
+  string collection_name = 3;
+  // The serialized `schema.CollectionSchema`(Required)
+  bytes schema = 4;
+  // Once set, no modification is allowed (Optional)
+  // https://github.com/milvus-io/milvus/issues/6690
+  int32 shards_num = 5;
+}
+
+message CollectionSchema {
+  string name = 1;
+  string description = 2;
+  bool autoID = 3; // deprecated later, keep compatible with c++ part now
+  repeated FieldSchema fields = 4;
+}
+
+```
+
+2. When receiving the `CreateCollection` request, `Proxy` would wrap this request into `CreateCollectionTask`, and pushes this task into `DdTaskQueue` queue. After that, `Proxy` would call `WaitToFinish` method to wait until the task is finished.
+
+```go
+type task interface {
+	TraceCtx() context.Context
+	ID() UniqueID // return ReqID
+	SetID(uid UniqueID) // set ReqID
+	Name() string
+	Type() commonpb.MsgType
+	BeginTs() Timestamp
+	EndTs() Timestamp
+	SetTs(ts Timestamp)
+	OnEnqueue() error
+	PreExecute(ctx context.Context) error
+	Execute(ctx context.Context) error
+	PostExecute(ctx context.Context) error
+	WaitToFinish() error
+	Notify(err error)
+}
+
+type createCollectionTask struct {
+	Condition
+	*milvuspb.CreateCollectionRequest
+	ctx       context.Context
+	rootCoord types.RootCoord
+	result    *commonpb.Status
+	schema    *schemapb.CollectionSchema
+}
+```
+
+3. There is a background service in `Proxy`, this service would get the `CreateCollectionTask` from `DdTaskQueue`, and execute it in three phases.
+
+   - `PreExecute`, do some static checking at this phase, such as check if `Collection Name` and `Field Name` are legal, if there are duplicate columns, etc.
+   - `Execute`, at this phase, `Proxy` would send `CreateCollection` request to `RootCoord` via `Grpc`, and wait for response, the `proto` is defined as follows:
+
+   ```proto
+   service RootCoord {
+       ...
+
+       rpc CreateCollection(milvus.CreateCollectionRequest) returns (common.Status){}
+
+       ...
+   }
+   ```
+
+   - `PostExecute`, `CreateCollectionTask` does nothing at this phase, and return directly.
+
+4. `RootCoord` would wrap the `CreateCollection` request into `CreateCollectionReqTask`, and then call function `executeTask`. `executeTask` would return until the `context` is done or `CreateCollectionReqTask.Execute` is returned.
+
+```go
+type reqTask interface {
+	Ctx() context.Context
+	Type() commonpb.MsgType
+	Execute(ctx context.Context) error
+	Core() *Core
+}
+
+type CreateCollectionReqTask struct {
+	baseReqTask
+	Req *milvuspb.CreateCollectionRequest
+}
+```
+
+5. `CreateCollectionReqTask.Execute` would alloc `CollectionID` and default `PartitionID`, and set `Virtual Channel` and `Physical Channel`, which are used by `MsgStream`, then write the `Collection`'s meta into `metaTable`
+
+6. After `Collection`'s meta written into `metaTable`, `Milvus` would consider this collection has been created successfully.
+
+7. `RootCoord` would alloc a timestamp from `TSO` before writing `Collection`'s meta into `metaTable`, and this timestamp is considered as the point when the collection was created
+
+8. At last `RootCoord` will send a message of `CreateCollectionRequest` into `MsgStream`, and other components, who have subscribed to the `MsgStream`, would be notified. The `Proto` of `CreateCollectionRequest` is defined as follows:
+
+```proto
+message CreateCollectionRequest {
+  common.MsgBase base = 1;
+  string db_name = 2;
+  string collectionName = 3;
+  string partitionName = 4;
+  int64 dbID = 5;
+  int64 collectionID = 6;
+  int64 partitionID = 7;
+  // `schema` is the serialized `schema.CollectionSchema`
+  bytes schema = 8;
+  repeated string virtualChannelNames = 9;
+  repeated string physicalChannelNames = 10;
+}
+
+```
+
+9. After the above operations, `RootCoord` would update the internal timestamp and return, so `Proxy` would get the response.
+
+_Notes:_
+
+1. In `Proxy`, all `DDL` requests will be wrapped into `task`, and push the `task` into `DdTaskQueue`. 
+   A background service will read a new `task` from `DdTaskQueue` only when the previous one is finished. 
+   So all the `DDL` requests are executed serially on `Proxy`.
+
+2. In `RootCoord`, all `DDL` requests will be wrapped into `reqTask`, but there is no task queue, so the `DDL` requests will be executed in parallel on `RootCoord`.
@@ -0,0 +1,105 @@
+# Support to retrieve the specified entity from a collection
+
+## Background
+
+In Milvus, a collection has multiple fields, mainly there are two kinds of fields: vector field and scalar field. We call a row an entity, one entity encapsulates multiple vectors and scalar values.
+
+When creating a collection, you can specify using the auto-generated primary key, or using the user-provided primary key.
+If a user sets to use the user-provided primary key, each entity inserted must contain the primary key field. Otherwise, the insertion fails.
+The primary keys will be returned after the insertion request is successful.
+
+Milvus currently only supports primary keys of the int64 type.
+
+QueryNode subscribes to the insert channel and will determine whether to use the data extracted from the insert channel or data processed by DataNode to provide services according to the status of a segment.
+
+## Goals
+
+- Support to retrieve one or more entities from a collection through primary keys
+- Support to retrieve only some fields of an entity
+- Consider backward file format compatibility if a new file is defined
+
+## Non-Goals
+
+- How to deal with duplicate primary keys
+- How to retrieve entity by non-primary key
+
+## Detailed design
+
+When the DataNode processes each inserted entity, it updates the bloomfilter of the Segment to which the entity belongs. If it does not exist, it creates a bloomfilter in memory and updates it.
+
+Once DataNode receives a Flush command from DataCoord, it sorts the data in the segment in ascending order of primary key, records the maximum and minimum values of a primary key, and writes the segment, statistics and bloomfilter to the storage system.
+
+- Key of binlog file: `${tenant}/insert_log/${collection_id}/${partition_id}/${segment_id}/${field_id}/_${log_idx}`
+- Key of statistics file: `${tenant}/insert_log/${collection_id}/${partition_id}/${segment_id}/${field_id}/stats_${log_idx}`
+- Key of bloom filter file: `${tenant}/insert_log/${collection_id}/${partition_id}/${segment_id}/${field_id}/bf_${log_idx}`
+
+QueryNode maintains a mapping from primary key to entities in each segment. This mapping updates every time an insert request is processed.
+
+After receiving the Get request from the client, the Proxy sends the request to the `search` channel and waits for the result returned from the `searchResult` channel.
+
+The processing flow after QueryNode reads the Get request from `search` channel:
+
+1. Search for the existence of the requested primary key in the mapping of the `Growing` status segment, and return directly if found;
+2. If the primary key exists in any mapping of `Growing` segments, return the results;
+3. Load statistics and bloomfilter of all `Sealed` segments;
+4. Convert the statistics into an inverted index from Range to SegmentID for each `Sealed` segment;
+5. Check whether the requested primary key exists in any inverted index of `Sealed` segment, return empty if not found;
+6. [optional] Use the bloomfilter to filter out segments where the primary key does not exist;
+7. Use binary search to find the specified entity in each segment where the primary key may exist;
+
+### APIs
+
+```go
+// pseudo-code
+func get(collection_name string,
+         ids list[string],
+         output_fields list[string],
+         partition_names list[string]) (list[entity], error)
+// Example
+// entities = get("collection1", ["103"], ["_id", "age"], nil)
+```
+
+When the primary key does not exist in specified collection( and partitions), Milvus will return an empty result, which is not considered as an error.
+
+### Storage
+
+Both bloomfilter files and statistical information files belong to Binlog files and follow the Binlog file format.
+
+https://github.com/milvus-io/milvus/blob/master/docs/developer_guides/chap08_binlog.md
+
+Two new types of Binlog are added: BFBinlog and StatsBinlog.
+
+BFBinlog Payload: Refer to https://github.com/milvus-io/milvus/blob/1.1/core/src/segment/SegmentWriter.h for storage methods
+
+StatsBinlog Payload: Json format string, currently only contains the keys `max`, `min`.
+
+## Impact
+
+### API
+
+- A new Get API
+- DropCollection / ReleaseCollection / DropPartition / ReleasePartition requests need to clear the corresponding statistics files and bloomfilter files in the memory
+
+### Storage
+
+- The name of the binlog file has been changed from `${log_idx}` to `_${log_idx}`
+- Each binlog adds a stats file
+- Each binlog adds a bloomfilter file
+
+## Test Plan
+
+### Testcase 1
+
+In the newly created collection, insert an entity with a primary key of 107, call the Get interface to query the entity with a primary key of 107, and each field of the retrieved entity is exactly the same as the inserted entity.
+
+### Testcase 2
+
+In the newly created collection, insert a record with a primary key of 107, call the Get interface to query the record with a primary key of 106, and the retrieved record is empty.
+
+### Testcase 3
+
+In the newly created collection, insert the records with the primary keys of 105, 106, 107, call the Get interface to query the records with the primary keys of 101, 102, 103, 104, 105, 106, 107, the retrieved result only contains the records with primary keys of 105, 106, 107.
+
+### Testcase 4
+
+In the newly created collection, insert a record with a primary key of 107, call the Flush interface, and check whether there are stats and bloomfilter files on MinIO.
@@ -0,0 +1,90 @@
+# What's Knowhere
+
+## Concepts
+
+Vector index is a time-efficient and space-efficient data structure built on vectors through a certain mathematical model. Through the vector index, we can efficiently query several vectors similar to the target vector.
+Since accurate retrieval is usually very time-consuming, most of the vector index types of Milvus use ANNS (Approximate Nearest Neighbors Search). Compared with accurate retrieval, the core idea of ANNS is no longer limited to returning the most accurate result, but only searching for neighbors of the target. ANNS improves retrieval efficiency by sacrificing accuracy within an acceptable range.
+
+## What can Knowhere do
+
+Knowhere is the vector search execution engine of Milvus. It encapsulates many popular vector index algorithm libraries, such as faiss, hnswlib, NGT, annoy, and provides a set of unified interfaces. In addition, Knowhere also supports heterogeneous computing.
+
+## Framework
+
+![Knowhere framework](../assets/graphs/knowhere_framework.png)
+
+For more index types and heterogeneous support, please refer to the vector index document.
+
+## Major Interface
+
+```C++
+/*
+ * Serialize
+ * @return: serialization data
+ */
+BinarySet
+Serialize();
+
+/*
+ * Load from serialization data
+ * @param [in] dataset_ptr: serialization data
+ */
+void
+Load(const BinarySet&);
+
+/*
+ * Create index
+ * @param [in] dataset_ptr: index data (key of the Dataset is "tensor", "rows" and "dim")
+ * @parma [in] config: index param
+ */
+void
+BuildAll(const DatasetPtr& dataset_ptr, const Config& config);
+
+/*
+ * KNN (K-Nearest Neighbors) Query
+ * @param [in] dataset_ptr: query data (key of the Dataset is "tensor" and "rows")
+ * @parma [in] config: query param
+ * @parma [out] blacklist: mark for deletion
+ * @return: query result (key of the Dataset is "ids" and "distance")
+ */
+DatasetPtr
+Query(const DatasetPtr& dataset_ptr, const Config& config, BitsetView blacklist);
+
+/*
+ * Copy the index from GPU to CPU
+ * @return: CPU vector index
+ * @notes: Only valid of the GPU indexes
+ */
+VecIndexPtr
+CopyGpuToCpu();
+
+/*
+ * If the user IDs has been set, they will be returned in the Query interface;
+ * else the range of the returned IDs is [0, row_num-1].
+ * @parma [in] uids: user ids
+ */
+void
+SetUids(std::shared_ptr<std::vector<IDType>> uids);
+
+/*
+ * Get the size of the index in memory.
+ * @return: index memory size
+ */
+int64_t
+Size();
+```
+
+## Data Format
+
+The vector data used for index and query is stored as a one-dimensional array.
+The first `dim * sizeof(data_type)` bytes of the array is the first vector; then `row_num -1` vectors are followed.
+
+## Sequence
+
+### Create index
+
+![create index sequence](../assets/graphs/create_index.png)
+
+### Query
+
+![knn query sequence](../assets/graphs/knn_query.png)
@@ -0,0 +1,89 @@
+# DropCollection release resources
+
+## Before this enhancement
+
+**When dropping a collection**
+
+1. DataNode releases the flowgraph of this collection and drops all the data in a buffer.
+2. DataCoord has no idea whether a collection is dropped or not.
+    - DataCoord will make DataNode watch DmChannels of dropped collections.
+    - Blob files will never be removed even if the collection is dropped.
+
+**For not in used binlogs on blob storage: Why are there such binlogs**
+- A failure flush.
+- A failure compaction.
+- Dropped and out-of timetravel collection binlogs.
+
+This enhancement is focused on solving these 2 problems.
+
+## Object1 DropCollection
+
+DataNode ignites Flush&Drop
+    receive drop collection msg ->
+    cancel compaction ->
+    flush all insert buffer and delete buffer ->
+    release the flowgraph
+
+**Plan 1: Picked**
+
+Add a `dropped` flag in `SaveBinlogPathRequest` proto.
+
+DataNode
+- Flush all segments in this vChannel, When Flush&Drop, set the `dropped` flag true.
+    - If fails, retry at most 10 times and restart.
+
+DataCoord
+- DataCoord marks segmentInfo as `dropped`, doesn't remove segmentInfos from etcd.
+- When recovery, check if the segments in the vchannel are all dropped.
+    - if not, recover before the drop.
+    - if so, no need to recover the vchannel.
+
+Pros:
+    1. The easiest approach in both DataNode and DataCoord.
+    2. DataNode can reuse the current flush manager procedure.
+Cons:
+    1. The No. rpc call is equal to the No. segments in a collection, expensive.
+
+---
+
+**Plan 2: Enhance later**
+
+Add a new rpc `FlushAndDrop`, it's a vchannel scope rpc.
+
+Pros:
+    1. much lesser rpc calls, equal to shard-numbers.
+    2. More clarity of flush procedure in DataNode.
+Cons:
+    1. More efforts in DataNode and DataCoord.
+
+```
+message FlushAndDropRequest {
+  common.MsgBase base = 1;
+  string channelID = 2;
+  int64 collectionID = 3;
+  repeated SegmentBinlogPaths segment_binlog_paths = 6;
+}
+
+message SegmentBinlogPaths {
+  int64 segmentID = 1;
+  CheckPoint checkPoint = 2;
+  repeated FieldBinlog field2BinlogPaths = 2;
+  repeated FieldBinlog field2StatslogPaths = 3;
+  repeated DeltaLogInfo deltalogs = 4;
+}
+```
+
+---
+
+## Object2: DataCoord Garbage Collection (GC) for not in used binlogs
+
+### How to clear unknown binlogs?
+DataCoord runs a background GC goroutine, triggers every 1 day:
+1. Get all minIO/S3 paths(keys).
+2. Filter out keys not in segmentInfo.
+3. According to the meta of blobs from minIO/S3, remove binlogs that exist more than 1 day.
+    - **Why 1 day: **Maybe there are newly uploaded binlogs from flush/compaction
+
+### How to clear dropped-collection's binlogs?
+- DataCoord checks all dropped-segments, removes the binlogs recorded if they've been dropped by 1 day.
+- DataCoord keeps the etcd segmentInfo meta.
@@ -0,0 +1,266 @@
+# Create Index
+
+`Index system` is the core part of `Milvus`, which is used to speed up the searches, this document introduces which components are involved in `Create Index`,and what these components do.
+
+The execution flow of `Create Index` is shown in the following figure:
+
+![create_index](../assets/graphs/milvus_create_index.png)
+
+1. Firstly, `SDK` starts a `CreateIndex` request to `Proxy` via `Grpc`, the `proto` is defined as follows:
+
+```proto
+service MilvusService {
+    ...
+
+    rpc CreateIndex(CreateIndexRequest) returns (common.Status) {}
+
+    ...
+}
+
+message CreateIndexRequest {
+  common.MsgBase base = 1;
+  string db_name = 2;
+  string collection_name = 3;
+  string field_name = 4;
+  int64 dbID = 5;
+  int64 collectionID = 6;
+  int64 fieldID = 7;
+  repeated common.KeyValuePair extra_params = 8;
+}
+```
+
+2. When received the `CreateIndex` request, the `Proxy` would wrap this request into `CreateIndexTask`, and push this task into `DdTaskQueue` queue. After that, `Proxy` would call method of `WatiToFinish` to wait until the task finished.
+
+```go
+type task interface {
+	TraceCtx() context.Context
+	ID() UniqueID       // return ReqID
+	SetID(uid UniqueID) // set ReqID
+	Name() string
+	Type() commonpb.MsgType
+	BeginTs() Timestamp
+	EndTs() Timestamp
+	SetTs(ts Timestamp)
+	OnEnqueue() error
+	PreExecute(ctx context.Context) error
+	Execute(ctx context.Context) error
+	PostExecute(ctx context.Context) error
+	WaitToFinish() error
+	Notify(err error)
+}
+
+type createIndexTask struct {
+	Condition
+	*milvuspb.CreateIndexRequest
+	ctx       context.Context
+	rootCoord types.RootCoord
+	result    *commonpb.Status
+}
+```
+
+3. There is a background service in `Proxy`, this service would get the `CreateIndexTask` from `DdTaskQueue`, and execute it in three phases.
+
+   - `PreExecute`, do some static checking at this phase, such as check if the index param is legal, etc.
+   - `Execute`, at this phase, `Proxy` would send `CreateIndex` request to `RootCoord` via `Grpc`, and wait the response, the `proto` is defined as the following:
+
+   ```proto
+       service RootCoord {
+         ...
+
+          rpc CreateIndex(milvus.CreateIndexRequest) returns (common.Status) {}
+
+         ...
+       }
+   ```
+
+   - `PostExecute`, `CreateIndexTask` does nothing at this phase, and returns directly.
+
+4. `RootCoord` would wrap the `CreateIndex` request into `CreateIndexReqTask`, and then call function `executeTask`. `executeTask` would return until the `context` is done or `CreateIndexReqTask.Execute` returned.
+
+```go
+type reqTask interface {
+	Ctx() context.Context
+	Type() commonpb.MsgType
+	Execute(ctx context.Context) error
+	Core() *Core
+}
+
+type CreateIndexReqTask struct {
+	baseReqTask
+	Req *milvuspb.CreateIndexRequest
+}
+```
+
+5. According to the index type and index parameters, `RootCoord` lists all the `Segments` that need to be indexed on this `Collection`. `RootCoord` would only check those `Segments` which have been flushed at this stage. We will describe how to deal with those newly added segments and growing segments later.
+
+6. For each `Segment`, `RootCoord` would start a `Grpc` request to `DataCoord` to get `Binlog` paths of that `Segment`, the `proto` is defined as following:
+
+```proto
+service DataCoord {
+    ...
+
+    rpc GetInsertBinlogPaths(GetInsertBinlogPathsRequest) returns (GetInsertBinlogPathsResponse) {}
+
+    ...
+
+}
+
+message GetInsertBinlogPathsRequest {
+  common.MsgBase base = 1;
+  int64 segmentID = 2;
+}
+
+message GetInsertBinlogPathsResponse {
+  repeated int64 fieldIDs = 1;
+  repeated internal.StringList paths = 2;
+  common.Status status = 3;
+}
+
+```
+
+7. After getting the `Segment`'s `Binlog` paths, `RootCoord` would send a `Grpc` request to `IndexCoord`, ask `IndexCoord` to build index on this `Segment`, the `proto` is defined as the follow:
+
+```proto
+service IndexCoord {
+  ...
+
+  rpc BuildIndex(BuildIndexRequest) returns (BuildIndexResponse){}
+
+  ...
+}
+
+message BuildIndexRequest {
+  int64 indexBuildID = 1;
+  string index_name = 2;
+  int64 indexID = 3;
+  repeated string data_paths = 5;
+  repeated common.KeyValuePair type_params = 6;
+  repeated common.KeyValuePair index_params = 7;
+}
+
+message BuildIndexResponse {
+  common.Status status = 1;
+  int64 indexBuildID = 2;
+}
+
+```
+
+8. The execution flow of `BuildIndex` on `IndexCoord` is shown in the following figure
+
+![index_coord](../assets/graphs/milvus_create_index_index_coord.png)
+
+9. `IndexCoord` would wrap the `BuildIndex` request into `IndexAddTask`, then alloc a global unique ID as `IndexBuildID`, and write this `Segment`'s `index mate` into `IndexCoord`'s `metaTable`. When finish these operations, `IndexCoord` would send response to `RootCoord`, the response includes the `IndexBuildID`.
+
+10. When `RootCoood` receives the `BuildIndexResponse`, it would extract the `IndexBuildID` from the response, update `RootCoord`'s `metaTable`, then send responses to `Proxy`.
+
+11. There is a background service, `assignTaskLoop`, in `IndexCoord`. `assignTaskLoop` would call `GetUnassignedTask` periodically, the default interval is 3s. `GetUnassignedTask` would list these segments whose `index meta` has been updated, but index has not been created yet.
+
+12. The previous step has listed the segments whose index has not been created, for each those segments, `IndexCoord` would call `PeekClient` to get an available `IndexNode`, and send `CreateIndex` request to this `IndexNode`. The `proto` is defined as follows.
+
+```proto
+service IndexNode {
+  ...
+
+  rpc CreateIndex(CreateIndexRequest) returns (common.Status){}
+
+  ...
+}
+
+message CreateIndexRequest {
+  int64 indexBuildID = 1;
+  string index_name = 2;
+  int64 indexID = 3;
+  int64 version = 4;
+  string meta_path = 5;
+  repeated string data_paths = 6;
+  repeated common.KeyValuePair type_params = 7;
+  repeated common.KeyValuePair index_params = 8;
+}
+```
+
+13. When receiving `CreateIndex` request, `IndexNode` would wrap this request into `IndexBuildTask`, and push this task into `IndexBuildQueue`, then send response to `IndexCoord`.
+
+14. There is a background service, `indexBuildLoop`, in the `IndexNode`. `indexBuildLoop` would call `scheduleIndexBuildTask` to get an `IndexBuildTask` from `IndexBuildQueue`, and then start another `goroutine` to build index and update meta.
+
+_Note_: `IndexNode` will not notify the `QueryCoord` to load the index files, if a user wants to speed up search by these index files, he should call `ReleaseCollection` firstly, then call `LoadCollection` to load these index files.
+
+15. As mentioned earlier, `RootCoord` would only search on these flushed segments on `CreateIndex` request, the following figure shows how to deal with the newly added segments.
+
+![data_coord_flushed](../assets/graphs/milvus_create_index_data_coord_flushed.png)
+
+16. When a segment has been flushed, `DataCoord` would notify `RootCoord` via `SegmentFlushCompleted`, the `proto` is defined as follows:
+
+```proto
+service RootCoord {
+  ...
+
+  rpc SegmentFlushCompleted(data.SegmentFlushCompletedMsg) returns (common.Status) {}
+
+  ...
+}
+
+message SegmentFlushCompletedMsg {
+  common.MsgBase base = 1;
+  SegmentInfo segment = 2;
+}
+
+message SegmentInfo {
+  int64 ID = 1;
+  int64 collectionID = 2;
+  int64 partitionID = 3;
+  string insert_channel = 4;
+  int64 num_of_rows = 5;
+  common.SegmentState state = 6;
+  int64 max_row_num = 7;
+  uint64 last_expire_time = 8;
+  msgpb.MsgPosition start_position = 9;
+  msgpb.MsgPosition dml_position = 10;
+  repeated FieldBinlog binlogs = 11;
+}
+
+```
+
+17. If a user has called `CreateIndex` on this `Collection`, then when `RootCoord` receives `SegmentFlushCompleted` request, it would extract the `SegmentID` from the request, and send a `GetInsertBinlogPaths` request to `DataCoord` to get the `Binlog` paths, finally `RootCoord` would send a `BuildIndex` request to `IndexCoord` to notify `IndexCoord` to build index on this segment.
+
+18. The `Grpc` call of `SegmentFlushCompleted` might be failed due to network problem or some others, so how to create an index if the `Grpc` failed ? The following figure shows the solution.
+
+![data_coord_flushed](../assets/graphs/milvus_create_index_root_coord_check.png)
+
+19. There is a background service, `checkFlushedSegmentLoop`, in `RootCoord`. `checkFlushedSegmentLoop` would periodically check whether there is a segment that needs to be created index but has not been created, the default interval is `10 minutes`, and call `DataCoord` and `IndexCoord`'s service to create index on these segments.
+
+20. In `Milvus 2.0`, `Create Index` is an asynchronous operation, so the `SDK` needs to send `GetIndexStates` request to `IndexCoord` periodically to check if the index has been created, the `proto` is defined as follows.
+
+```proto
+service IndexCoord {
+  ...
+
+  rpc GetIndexStates(GetIndexStatesRequest) returns (GetIndexStatesResponse) {}
+
+  ...
+}
+
+message GetIndexStatesRequest {
+  repeated int64 indexBuildIDs = 1;
+}
+
+message GetIndexStatesResponse {
+  common.Status status = 1;
+  repeated IndexInfo states = 2;
+}
+
+message IndexInfo {
+  common.IndexState state = 1;
+  int64 indexBuildID = 2;
+  int64 indexID = 3;
+  string index_name = 4;
+  string reason = 5;
+}
+
+enum IndexState {
+    IndexStateNone = 0;
+    Unissued = 1;
+    InProgress = 2;
+    Finished = 3;
+    Failed = 4;
+}
+```
@@ -0,0 +1,507 @@
+## 6. Proxy
+
+As the user access layer of Milvus, Proxy mainly plays a role that does some checks and preprocessing for requests from
+clients and then forwards these requests to other components, such as Root Coordinator, Data Coordinator, Query
+Coordinator, Index Coordinator. The below figure shows how Proxy interacts with other components.
+
+<img src="../assets/graphs/proxy.png" width=700>
+
+Proxy divides requests from clients into three classes: `DdRequest`, `DmRequest`, `DqRequest`.
+
+DdRequest is the shorthand of `Data Definition Request`. It means an operation on the meta information of
+collections, including two parts. One part is of the writing operations on collections, such as defining a schema,
+creating or dropping a partition, creating or dropping the index, etc. Another part is of the reading operations on
+collections, such as listing all collections or partitions, checking if a collection or a partition exists.
+
+DmRequest means `Data Manipulation Request`. These requests perform a writing operation on collections, including
+inserting records into a collection, deleting some specific records of a collection.
+
+DqRequest means `Data Query Request`. These requests perform a reading operation on collections, such as searching on a
+collection, querying for specific records of a collection, etc.
+
+For every request, Proxy will first check if it's valid to be executed by Milvus and if the request is invalid then
+Proxy will return the error to clients and won't continue to forward this request to other components. The check
+operation of Proxy includes two parts, one part is static check and another is dynamic check. The static check includes
+parameters check, constraints check, etc. The dynamic check will check some related dependencies of the request, take
+search requests as an example, Proxy should check if the related collection exists in Milvus.
+
+Also, Proxy will do some preprocessing for every request. Proxy will do little things for some requests in the
+preprocessing stage and a lot more for other requests. Every object in Milvus will be assigned with an `ID`, such as
+`CollectionID`, `PartitionID`, `IndexID`, `SegmentID`, etc. Components in Milvus communicate with each other by the
+object IDs, however, users only know the object name. So as a user access layer of Milvus, Proxy should translate
+the object name into object ID. Also taking search request as an example, Proxy should translate the `CollectionName` into
+`CollectionID` and then the Query Node will recognize the request. Proxy holds a cache that translates object name into
+object id and dynamically updates the cache.
+
+#### 6.0 Service Discovery based on etcd
+
+As you know, Proxy depends on some other components. So how Proxy knows the other nodes' information such as host and port,
+it is also called how Milvus implements the service discovery mechanism? As a cloud-native vector database, Milvus uses
+etcd to provide service registration and service discovery. Every node in Milvus registers its node information(including host,
+port, ID etc) into etcd after startup. Nodes should specify the prefix and key of etcd when registration. Nodes with the
+same type have the same prefix in etcd, while nodes with different types have different prefixes. Every key in etcd can be assigned with a lease, and you can specify the `time-to-live(ttl)` of the
+lease. Milvus uses this mechanism to check if a node is online. When a node is healthy, it will continuously renew the
+lease in etcd. Otherwise, if some exceptions occurred in a node, or a node stopped to renew the lease, etcd would delete
+the related node information. By using this mechanism, nodes in Milvus know if there are any other nodes going to be
+online or offline and synchronize the latest node information.
+
+#### 6.1 Interaction with Root Coordinator
+
+Proxy will forward the DdRequest to Root Coordinator. These requests include:
+
+    - CreateCollection
+    - DropCollection
+    - HasCollection
+    - DescribeCollection
+    - ShowCollections
+    - CreatePartition
+    - DropPartition
+    - HasPartition
+    - ShowPartitions
+    - CreateIndex
+    - DropIndex
+    - DescribeIndex
+    - GetIndexBuildProgress
+    - GetIndexState
+
+Proxy handles the DdRequest sequentially. When and only when the earlier entered requests are done, the next request
+would be handled. Proxy forwards these requests to Root Coordinator, waits until getting results from Root Coordinator, and then
+returns to clients with results or errors.
+
+Milvus does not support transactions, but it should guarantee the deterministic execution of every operation. A timestamp
+is tagged on each request. When a request enters Milvus, Proxy tags a timestamp that was assigned by Root Coordinator.
+The component that assigns timestamp in Root Coordinator is called `Timestamp Oracle (TSO)`. TSO ensures that each
+timestamp is globally increasing.
+
+Milvus 2.0 implements the unified Lambda architecture, which integrates the processing of the incremental and historical
+data. Compared with the Kappa architecture, Milvus 2.0 introduces log backfill, which stores log snapshots and indexes
+in the object storage to improve failure recovery efficiency and query performance. To break unbounded (stream) data
+down into bounded windows, Milvus embraces a new watermark mechanism, which slices the stream data into multiple message
+packs according to write time or event time, and maintains a timeline for users to query by time.
+
+To support this watermark mechanism, Proxy should report the timestamp statistics of the physical channel to Root
+Coordinator periodically. When Proxy knows all operations of a specific were done before a `ts`, then Proxy will report
+the `ts` and inform Root Coordinator that updates the timestamp statistics.
+
+Proxy holds a cache about meta information of collections. The meta information includes `CollectionID`, `Schema`,
+`PartitionID`, etc. Components in Milvus communicate with each other using `CollectionID` and `PartitionID`, so the
+object name in a request will be translated to object ID in Proxy. When the meta is not hit in the cache, Proxy will update
+the cache from Root Coordinator. At the same time, in order to keep the consistency of cache, when there are any changes
+of meta information in Root Coordinator, it will inform all Proxies to clear the related meta cache, and any newer
+requests will get the latest meta information.
+
+For inserts to a collection that is auto_id configured in the collection schema, Proxy assigns a primary key for
+every row of insert requests. For now, the only supported data type of auto-generated primary field is `int64`. Proxy 
+applies for a batch of primary keys from Root Coordinator and caches them for local assignments. When the primary keys in the cache
+are not enough, Proxy will continue to apply for another batch of primary keys.
+
+Proxy forwards ReleaseCollection and ReleasePartition to Query Coordinator, Query Coordinator then informs Root
+Coordinator the events. After that, Root Coordinator will inform all Proxies to close the search-related and
+search-result-related message streams.
+
+#### 6.2 Interaction with MsgStream
+
+In Milvus 2.0, the log broker serves as the system 'backbone': All data insert and update operations must go through the
+log broker, and worker nodes execute CRUD operations by subscribing to and consuming logs. This design reduces system
+complexity by moving core functions such as data persistence and flashback down to the storage layer, and log pub-sub
+make the system even more flexible and better positioned for future scaling.
+
+So, we should configure a group of Virtual Channels for every collection. Now every virtual channel has a unique related
+physical channel. Take pulsar as an example, the physical channel mentioned here means the topic of pulsar.
+
+For DmRequest, data will be written to DmChannels, while for DqRequest, requests will be written to DqRequestChannel,
+and the corresponding results of query requests will be written to DqResultChannel.
+
+As the number of tables increases, the number of DmChannels increases on demand, and the number of physical channels
+also increases on demand. In the future, the number of physical channels in the system can also be limited to a fixed
+number, such as 1024. In this case, the same physical channel will be mapped to virtual channels of different
+collections. Shown in the figure below.
+
+![collection_dm_channels](../assets/graphs/collection_dm_channels.png)
+
+When a collection is created, Root Coordinator need to decide the number of its DmChannels and the physical
+channels mapped by each virtual channel, and persist these kinds of information as the meta information of the collection; 
+In addition, when the system finds that the collection receives DmRequest frequently, we can allocate more virtual channels
+to the collection to increase the parallelism and thus increase the system throughput. This function is a key point of
+future work.
+
+For DqRequest, request and result data are written to the stream. The request data will be written to DqRequestChannel,
+and the result data will be written to DqResultChannel. Proxy will write the request of the collection into the
+DqRequestChannel, and the DqRequestChannel will be jointly subscribed by a group of query nodes. When all query nodes
+receive the DqRequest, they will write the query results into the DqResultChannel corresponding to the collection. As
+the consumer of the DqResultChannel, Proxy is responsible for collecting the query results and aggregating them,
+The result is then returned to the client.
+
+The allocation logic of DqRequestChannel and DqResultChannel of a collection is allocated by the Query Coordinator. Proxy needs to ask the Query Coordinator for the names of DqRequestChannel and DqResultChannel of a collection.
+DqRequestChannel and DqResultChannel do not need to be persisted and can be freely allocated by Query Coordinator. In
+the actual implementation, the DqRequestChannel of each collection can be exclusive, and the DqResultChannel can be
+exclusive or shared by all collections on Proxy. When Proxy applies for the DqRequestChannel and DqResultChannel
+information of the collection from the Query Coordinator, it can attach Proxy's own ID: ProxyID.
+
+With DqRequestChannel of the collection, Proxy will create a msgstream object to generate data into
+DqRequestChannel. With the DqResultChannel of the collection, Proxy will create a msgstream object, and Proxy will
+consume the data in the DqResultChannel. When these msgstream objects are closed, messages cannot be written to or
+consumed from them.
+
+![proxy_channels](../assets/graphs/proxy_channels.png)
+
+#### 6.3 Interaction with Data Coordinator
+
+In Milvus, segment is the basic read-write and search unit of data. After consuming the data in DmChannel, the data
+nodes will store the data in the object storage in the unit of segment (in the actual implementation, the segment will
+be divided into multiple small files for writing). In Milvus, segment is uniquely identified by segment ID, and the
+allocation logic of segment ID is the responsibility of the Data Coordinator. The SegmentID to which each row of data is
+written needs to be determined before writing to DmChannel. For a write operation, Proxy will hash each row of data
+again according to the hash value of its primary key, and then determine into which DmChannel each row of data enters. After
+collecting the number of pieces to be written by each DmChannel, it applies to the data coordinator for which SegmentIDs the
+newly written data of these dmchannels belong to. In the specific implementation, Proxy needs to preallocate some
+quotas to the Data Coordinator to avoid frequent direct GRPC communication with the Data Coordinator.
+
+One consideration for uniformly assigning SegmentIDs by Data Coordinator is that Data Coordinator is responsible for
+coordinating the total number of each segment not to be too large, and the location is near a water level, so that the
+size of the segment is limited to a certain range.
+
+Other interactions between Proxy and Data Coordinator are mainly reflected in Proxy querying Data Coordinator for
+the status and statistical information of the segment of the collection. LoadCollection is an example. The
+synchronization semantics of the current LoadCollection needs to know the number of rows currently persisted, so the
+Proxy needs to ask the Data Coordinator for the total number of rows currently persisted.
+
+#### 6.4 Interaction with Query Coordinator
+
+For LoadCollection, LoadPartition, ReleaseCollection, ReleasePartition requests, Proxy directly forwards these
+requests to Query Coordinator for execution after checking and preprocessing these requests. When Proxy receives
+feedback from Query Coordinator, it returns the feedback results to the clients.
+
+The semantics of the Load operation is to load Collection or Partition from persistent storage into the memory of Query
+Nodes, or import streaming data into QueryNode so that it can be queried. If the load operation is not performed, the
+query operation on the Collection or Partition cannot be performed. For the Load operation, Query Coordinator is
+responsible for allocating DmChannels to different queryNodes and subscribing to them, and is responsible for receiving
+those stream data. QueryCoordinator also allocates and loads the segments that have been persisted in storage in
+Query Nodes.
+
+The semantics of the Release operation is the reverse operation of the Load operation, and the function is to unload the
+data of the Collection or Partition from the memory. For Release operations, Query Coordinator is responsible for
+notifying query nodes to unload the corresponding Collection or Partition in memory, and then sending the
+ReleaseDqlMessageStream command to Root Coordinator, and Root Coordinator is responsible for broadcasting the
+ReleaseDqlMessageStream command to all Proxies, so that all related streams used to send search requests and receive
+search result in Proxy will be closed.
+
+The other interaction between Proxy and Query Coordinator is that Proxy needs to query from Query Coordinator for statistics
+about Collection, Partition, and Segment. Taking ShowCollections as an example, if the ShowCollections parameter
+specifies that the query is for Collections that have been loaded into memory, the ShowCollection request will be
+forwarded to QueryCoordinator, and QueryCoordinator will return a list of all the recorded Collections loaded into
+memory. Taking LoadCollection as another example, its synchronization semantics is that the number of rows loaded in
+the memory must be no less than the number of rows that have been persisted. This requires Proxy to ask the Query
+Coordinator for the sum of the number of rows currently loaded into the query nodes in the Collection.
+
+#### 6.5 Decouple Functionality and Communication
+
+![decouple](../assets/graphs/decouple.jpeg)
+
+As shown in the figure above, there are interactions between various types of components in the Milvus2.0 system. The
+implementation of these interactions will vary according to the deployment of Milvus. Milvus Standalone allows the
+various components of Milvus as a whole independent process to be deployed on a single node. Milvus Cluster distributes
+all components on multiple nodes. In Milvus Standalone, the interaction between components can be parameter transfer
+between functions or communication between Grpc. The log system can be either Pulsar or RocksDb. In Milvus Cluster, the
+communication between components is mostly undertaken by grpc, and the message flow is mostly by Pulsar.
+
+Therefore, in the original design, Milvus 2.0 decoupled the core function of the component and the communication between
+components. Taking Proxy as an example, the core function of Proxy component is determined and has nothing to do
+with the deployment form. In the project's internal/proxy directory, it contains the functions of the core components of
+Proxy; and internal/distributed/proxy contain the core functions of Proxy in the deployment of cluster distributed
+which contains the re-encapsulation and communication implementation of Proxy. The following article will mainly
+introduce the functions of Proxy core layer.
+
+#### 6.6 Core Components of Proxy
+
+Proxy is mainly composed of four modules: taskScheduler, channelsMgr, channelsTimeTicker, globalMetaCache. taskScheduler
+is responsible for task scheduling; channelsMgr is responsible for the management of DmChannels, DqRequestChannel,
+DqResultChannel and corresponding MsgStream objects of each Collection; channelsTimeTicker is responsible for collecting
+the timestamp information of all physical Channels regularly; globalMetaCache is responsible for caching the metadata of
+Collection and Partition.
+
+##### 6.6.1 taskScheduler
+
+There are three main functions in taskScheduler:
+
+    - Schedule task
+    - Maintain the snapshot of timestamp statistics
+    - Receive the search results from all streams and then distribute them to related task
+
+taskScheduler maintains three queues: ddQueue, dmQueue and dqQueue correspond to DdRequest, DmRequest, and DqRequest
+respectively. The interface of taskQueue is defined as follows:
+
+```go
+type taskQueue interface {
+  utChan() <-chan int
+  utEmpty() bool
+  utFull() bool
+  addUnissuedTask(t task) error
+  FrontUnissuedTask() task
+  PopUnissuedTask() task
+  AddActiveTask(t task)
+  PopActiveTask(tID UniqueID) task
+  getTaskByReqID(reqID UniqueID) task
+  TaskDoneTest(ts Timestamp) bool
+  Enqueue(t task) error
+	setMaxTaskNum(num int64)
+	getMaxTaskNum() int64
+}
+```
+
+Proxy encapsulates each request with a corresponding task object. Each task object implements the task interface. The
+definition of the task interface is as follows:
+
+```go
+type task interface {
+  TraceCtx() context.Context
+  ID() UniqueID       // return ReqID
+  SetID(uid UniqueID) // set ReqID
+  Name() string
+  Type() commonpb.MsgType
+  BeginTs() Timestamp
+  EndTs() Timestamp
+  SetTs(ts Timestamp)
+  OnEnqueue() error
+  PreExecute(ctx context.Context) error
+  Execute(ctx context.Context) error
+  PostExecute(ctx context.Context) error
+  WaitToFinish() error
+  Notify(err error)
+}
+```
+
+Each specific task object must implement the interface defined by the task.
+
+The key members of taskQueue are unissuedTasks of type List and activateTasks of type maps. Among them, unissuedTasks
+contain all tasks that have not been scheduled, and activateTasks contain all tasks that are being scheduled.
+
+When the external caller of taskScheduler stuffs the task into the corresponding taskQueue, it will call the OnEnqueue
+interface of the task. When OnEnqueue is called, the SetID of the task will be called to assign the taskID to the task.
+The taskID is globally unique and is used to identify the task. OnEnqueue will also call task.SetTs to set the timestamp
+for the task. It can be seen that the timestamp of entering the queue must be greater than the timestamp that already
+exists in the queue, and it will also be greater than the timestamp of the task that exists in activateTask. At the end
+of the task's OnEnqueue, call the taskQueue's addUnissuedTask to add the task to the unissuedTasks. When OnEnqueue is
+executed, the external caller of taskScheduler calls WaitToFinish of the task to synchronously block and wait for the
+execution of the task to be done.
+
+When taskScheduler's background scheduling coroutine decides to schedule a task, it will call the taskQueue's
+PopUnissuedTask to remove a task from unissuedTasks, and then call the taskQueue's AddActivateTask to put the task in
+the activateTasks Map. Then perform operations on the task. In the execution process of the task, its PreExecute,
+Execute, and PostExecute interfaces will be called in sequence. If an exception occurs in a certain step, the error will
+be returned in advance and the subsequent steps will be skipped. Whether it is a successful execution or an error, the
+Notify method will be called, this method will wake up the coroutine that is blocking and waiting for the Task. When the
+task is executed, the PopActivateTask of the taskQueue is called to take the task out of activateTasks.
+
+The following figure is a schematic diagram of taskScheduler's scheduling of DdQueue.
+
+For the task of taskScheduler in DdQueue, it must be scheduled serially, the task that enters the queue first is
+executed, and at most only one task is executing. After the task at the head of the queue is executed, other tasks in
+the queue can be scheduled.
+
+![task_scheduler_1](../assets/graphs/task_scheduler_1.png)
+
+The following figure is a schematic diagram of taskScheduler's scheduling of DmQueue.
+
+The tasks in DmQueue can be scheduled in parallel. In a scheduling process, taskScheduler will execute several tasks
+from each task concurrently.
+
+![task_scheduler_2](../assets/graphs/task_scheduler_2.png)
+
+The following figure is a schematic diagram of taskScheduler's scheduling of DqQueue.
+
+![task_scheduler_1](../assets/graphs/task_scheduler_2.png)
+
+The tasks in DqQueue can be scheduled in parallel. In a scheduling process, taskScheduler will execute several tasks
+concurrently.
+
+In order to facilitate the channelsTimeTicker component to obtain the synchronization point information corresponding to all
+DmChannels, the taskScheduler needs to maintain a copy of the time statistics of the physical channels of all currently
+unexecuted and executing tasks in the DmQueue. The member pChanSatisticsInfos is a map containing the mapping from pChan
+to pChanStatInfo pointers. Among them, pChan is an alias of string, and pChanStatInfo is a custom structure, defined as
+follows:
+
+```go
+type Timestamp = uint64
+type pChan = string
+
+type pChanStatInfo struct {
+   maxTs Timestamp
+   minTs Timestamp
+  tsSet map[Timestamp] struct{}
+}
+```
+
+pChan represents the name of the physical channel corresponding to the DmChannel. The pChanStatInfo structure contains 3
+members minTs, maxTs and tsSet. minTs and maxTs respectively represent the minimum and maximum timestamps of all tasks
+related to the pChan in the current DmQueue. tsSet represents the set of timestamps of all tasks in DmQueue. When the
+task enters the DmQueue, it will call the addPChanStats method of the queue, add the task's own timestamp to the
+collection of pChanStatInfo's tsSet, and use the task's own maxTs to recalculate the pChanStatInfo's maxTs. The
+calculation method is if its own maxTs is greater than pChanStatInfo MaxTs in pChanStatInfo, then update maxTs in
+pChanStatInfo. Since the newly added timestamp is definitely greater than the existing timestamp, there is no need to
+update minTs. When the PopActivateTask interface of the Queue is called to take the task out of the DmTaskQueue, the
+popPChanStats interface of the Queue is called to delete the task timestamp from the timestamp set tsSet of
+pChanStatInfo, and recalculate the minimum timestamp minTs.
+
+DmQueue maintains the timestamp statistics of pChans and provides the method getPChanStatsInfo to the caller.
+pChanStatistics is defined as follows:
+
+```go
+type pChanStatistics struct {
+	minTs Timestamp
+	maxTs Timestamp
+}
+
+func (queue *DmTaskQueue) getPChanStatsInfo() (map[pChan]*pChanStatistics, error)
+```
+
+The return value of this method includes a pChan to pChanStatistics pointer mapping. pChanStatistics includes two
+members minTs and maxTs, which respectively represent the minimum and maximum timestamps of all tasks that have not been
+completed on pChan. The channelsTimeTicker collects timestamp information mainly depends on this method. The awakened
+coroutine reduces the timestamp result and sends the final timestamp result back to the RootCoord.
+
+taskScheduler is also responsible for collecting the results of search requests. For each search request, when Proxy
+writes the request to the DqRequestChannel, it will attach the ReqID, and the query nodes will also bring the ReqID back
+when writing the search result back to the DqResultChannel. taskScheduler will start a background coroutine to consume
+search results from DqResultChannel, and then distribute messages according to the ReqID in it. When several results of
+the same ReqID are collected and the termination condition is reached, these results will be passed to the blocking task
+coroutine which is waiting. The waken task will reduce the search results and then send the final search result to
+clients.
+
+##### 6.6.2 channelsMgr
+
+channelsMgr is responsible for the management of DmChannels, DqRequestChannel, DqResultChannel and corresponding
+MsgStream objects of each Collection. The interface is defined as follows:
+
+```go
+type channelsMgr interface {
+   getChannels(collectionID UniqueID) ([]pChan, error)
+   getVChannels(collectionID UniqueID) ([]vChan, error)
+   createDQLStream(collectionID UniqueID) error
+   getDQLStream(collectionID UniqueID) (msgstream.MsgStream, error)
+   removeDQLStream(collectionID UniqueID) error
+   removeAllDQLStream() error
+   createDMLMsgStream(collectionID UniqueID) error
+   getDMLStream(collectionID UniqueID) (msgstream.MsgStream, error)
+   removeDMLStream(collectionID UniqueID) error
+   removeAllDMLStream() error
+}
+```
+
+- getChannels and getVChannels
+
+  getVChannels returns a list that represents all virtual DmChannels of collection, getChannels returns a list that
+  represents all physical DmChannels of collection. The two lists correspond one-to-one according to a position.
+
+- createDMLStream and getDMLStream
+
+  createDMLStream creates the dml message stream of a collection;
+
+  getDMLStream returns the dml message stream of a collection;
+
+  Proxy uses these dml message streams to write dml data, such as insert requests.
+
+- createDQLStream and getDQLStream
+
+  createDQLStream creates the dql message stream of a collection;
+
+  getDQLStream returns the dql message stream of a collection;
+
+  Proxy uses these dql message streams to send search requests.
+
+The Remove related operation is to delete the corresponding message stream object, but the stream is not immediately
+closed because maybe there are some data that needs to be written into stream currently.
+
+##### 6.6.3 channelsTimeTicker
+
+channelsTimeTicker is responsible for regularly collecting the synchronization timestamp information of all physical channels.
+
+```go
+type channelsTimeTicker interface {
+   start() error
+   close() error
+   addPChan(pchan pChan) error
+   removePChan(pchan pChan) error
+   getLastTick(pchan pChan) (Timestamp, error)
+   getMinTsStatistics() (map[pChan]Timestamp, error)
+}
+```
+
+- addPChan and removePChan
+
+  addPChan adds a physical channel to channelsTimeTicker, channelsTimeTicker only sends the information of existing
+  pChans to Root Coordinator;
+
+- getMinTsStatistics
+
+  getMinTsStatistics returns the timestamp statistics of physical channels, there is a background coroutine in Proxy
+  that call getMinTsStatistics periodically and then send the timestamp statistics to Root Coordinator;
+
+- getLastTick
+
+  getLastTick returns the minimum timestamp which has already been synchronized of a physical channel;
+
+channelsTimeTicker will maintain the map minTsStatistics that can be synchronized and the map currents that will be
+synchronized. They are all mappings from pChan to Timestamp. The channelsTimeTicker itself has a background coroutine,
+which periodically calls getPChanStatsInfo of DmQueue to obtain the minimum and maximum timestamp information pChanStats
+of all unfinished tasks. channelsTimeTicker will also request a timestamp from Root Coordinator as now.
+channelsTimeTicker updates minTsStatistics and currents according to the relationship between now, minTsStatistics,
+currents, and pChanStats. The specific algorithm is as follows:
+
+```go
+now = get a timestamp from RootCoordinator
+For each pChan in currents, do
+	current = currents[pChan]
+	if pChan not exists in pChanStats, then
+		minTsStatistics[pChan] = current
+		currents[pChan] = now
+  else：
+		minTs = pChanStats[pChan].minTs
+		maxTs = pChanStats[pChan].maxTs
+		if minTs > current，then
+			minTsStatistics[pChan] = min(minTs, now)
+      next:= min(now + sendTimeTickMsgInterval, maxTs)
+      currents[pChan] = next
+```
+
+##### 6.6.4 globalMetaCache
+
+globalMetaCache is responsible for caching the meta-information of Collection and Partition. These meta-information
+include CollectionID, PartitionID, CollectionSchema, etc.
+
+The interface of Cache is defined as follows:
+
+```go
+type Cache interface {
+   GetCollectionID(ctx context.Context, collectionName string) (typeutil.UniqueID, error)
+   GetPartitionID(ctx context.Context, collectionName string, partitionName string) (typeutil.UniqueID, error)
+   GetPartitions(ctx context.Context, collectionName string) (map[string]typeutil.UniqueID, error)
+   GetCollectionSchema(ctx context.Context, collectionName string) (*schemapb.CollectionSchema, error)
+   RemoveCollection(ctx context.Context, collectionName string)
+   RemovePartition(ctx context.Context, collectionName string, partitionName string)
+}
+```
+
+- getCollectionID
+
+  GetCollectionID returns the collection ID of collection;
+
+- GetPartitions
+
+  GetPartitions returns a mapping which maps all partition names to partition IDs of collection;
+
+- GetPartition
+
+  GetPartition returns the partition ID of a specific partition and collection;
+
+- GetCollectionSchema
+
+  GetCollectionSchema returns the schema of collection;
+
+- RemoveCollection
+
+  RemoveCollection removes the meta information of collection;
+
+- RemovePartition
+
+  RemovePartition removes the meta information of partition;
@@ -0,0 +1,94 @@
+```haskell
+Expr :=
+    LogicalExpr | NIL
+
+LogicalExpr :=
+    LogicalExpr BinaryLogicalOp LogicalExpr
+  | UnaryLogicalOp LogicalExpr
+  | "(" LogicalExpr ")"
+  | SingleExpr
+
+BinaryLogicalOp :=
+    "&&" | "and"
+  | "||" | "or"
+
+UnaryLogicalOp :=
+    "not"
+
+SingleExpr :=
+    TermExpr
+  | CompareExpr
+
+TermExpr :=
+    IDENTIFIER "in" ConstantArray
+
+ConstantArray :=
+    "[" ConstantExpr { "," ConstantExpr } "]"
+
+ConstantExpr :=
+    Constant
+  | ConstantExpr BinaryArithOp ConstantExpr
+  | UnaryArithOp ConstantExpr
+
+Constant :=
+    INTEGER
+  | FLOAT_NUMBER
+
+UnaryArithOp :=
+    "+"
+  | "-"
+
+BinaryArithOp :=
+    "+"
+  | "-"
+  | "*"
+  | "/"
+  | "%"
+  | "**"
+
+CompareExpr :=
+    IDENTIFIER CmpOp IDENTIFIER
+  | IDENTIFIER CmpOp ConstantExpr
+  | ConstantExpr CmpOp IDENTIFIER
+  | ConstantExpr CmpOpRestricted IDENTIFIER CmpOpRestricted ConstantExpr
+
+CmpOpRestricted :=
+    "<"
+  | "<="
+
+CmpOp :=
+    ">"
+  | ">="
+  | "<"
+  | "<="
+  | "=="
+  | "!="
+
+INTEGER := 整数
+FLOAT_NUM := 浮点数
+IDENTIFIER := 列名
+```
+
+Tips:
+
+1. NIL represents an empty string, which means there is no Predicate for Expr.
+2. Gramma is described by EBNF syntax, expressions that may be omitted or repeated are represented through curly braces `{...}`.
+
+After syntax analysis, the following rules will be applied:
+
+1. Non-vector column must exist in Schema.
+2. CompareExpr/TermExpr requires operand type matching.
+3. CompareExpr between non-vector columns of different types is available.
+4. The modulo operation requires all operands to be integers.
+5. Integer columns can only match integer operands. While float columns can match both integer and float operands.
+6. In BinaryOp, the `and`/`&&` operator has a higher priority than the `or`/`||` operator.
+
+Example：
+
+```python
+A > 3 && A < 4 && (C > 5 || D < 6)
+1 < A <= 2.0 + 3 - 4 * 5 / 6 % 7 ** 8
+A == B
+FloatCol in [1.0, 2, 3.0]
+Int64Col in [1, 2, 3] or C != 6
+```
@@ -0,0 +1,98 @@
+## Root Coordinator recovery on power failure
+
+## 1. Basic idea
+
+1. `RootCoord` (Root Coordinator) reads meta from etcd when it starts.
+2. `RootCoord` needs to store the `position` of the msgstream into etcd every time it consumes the msgstream.
+3. `RootCoord` reads the `position` of msgstream from etcd when it starts up, then it seeks to the specified `position` and re-consumes the msgstream.
+4. Ensure that all messages from the msgstream are processed in an idempotent fashion, so that repeated consumption of the same message does not cause system inconsistencies.
+5. `RootCoord` registers itself in etcd and finds out if the dependent `DataCoord(Data Coordinator)` and `IndexCoord(Index Coordinator)` are online via etcd.
+
+## 2. Specific tasks
+
+### 2.1 Read meta from etcd
+
+1. `RootCoord` needs to load meta from etcd when it starts, this part is already done.
+
+### 2.2 `dd requests` from grpc
+
+1. The `dd requests`, such as create_collection, create_partition, etc., from grpc are marked as done only if the related meta has been written into etcd.
+2. The `dd requests` should be sent to `dd msgstream` when the operation is done.
+3. There may be a fault here, that is, the `dd request` has been written to etcd, but it has not been sent to `dd msgstream` yet, then the `RootCoord` has crashed.
+4. For the scenarios mentioned in item 3, `RootCoord` needs to check if all `dd requests` are sent to `dd msgstream` when it starts up.
+5. `RootCoord`'s built-in scheduler ensures that all grpc requests are executed serially, so it only needs to check whether the most recent `dd requests` are sent to the `dd msgstream`, and resend them if not.
+6. Take `create_collection` as an example to illustrate the process
+   - When `create collection` is written to etcd, 2 additional keys are updated, `dd_msg` and `dd_type`.
+   - `dd_msg` is the serialization of the `dd_msg`.
+   - `dd_type` is the message type of `dd_msg`, such as `create_collection`, `create_partition`, `drop_collection,` etc. It's used to deserialize `dd_msg`.
+   - Update the meta of `create_collection`, `dd_msg` and `dd_type` at the same time in a transactional manner.
+   - When `dd_msg` has been sent to `dd msgstream`, delete `dd_msg` and `dd_type` from etcd.
+   - When the `RootCoord` starts, first check whether there are `dd_msg` and `dd_type` in etcd. If yes, then deserialize `dd_msg` according to `dd_type`, and then send it to the `dd msgstream`. Otherwise, no processing will be done.
+   - There may be a failure here, that is, `dd_msg` has been sent to the `dd msgstream` , but has not been deleted from etcd yet, then the `RootCoord` crashed. In this case, the `dd_msg` would be sent to `dd msgstream` repeatedly, so the receiver needs to count this case.
+
+### 2.3 `create index` requests from grpc
+
+1. In the processing of `create index`, `RootCoord` calls `metaTable`'s `GetNotIndexedSegments` to get all segment ids that are not indexed.
+2. After getting the segment ids, `RootCoord` calls `IndexCoord` to create index on these segment ids.
+3. In the current implementation, the `create index` requests will return after the segment ids are put into a go channel.
+4. The `RootCoord` starts a background task that keeps reading the segment ids from the go channel, and then calls the `IndexCoord` to create the index.
+5. There is a fault here, the segment ids have been put into the go channel in the processing function of the grpc request, and then the grpc returns, but the `RootCoord`'s background task has not yet read them from the go channel, then `RootCoord` crashes. At this time, the client thinks that the index is created, but the `RootCoord` does not call `IndexCoord` to create the index.
+6. The solution for the fault mentioned in item 5:
+   - Remove the go channel and `RootCoord`'s background task.
+   - In the request processing function of `create index`, the call will return only when all segment ids have been sent `IndexCoord`.
+   - Some segment ids may be sent to `IndexCoord` repeatedly, and `IndexCoord` needs to handle such requests.
+
+### 2.4 New segment from `DataCoord`
+
+1. Each time a new segment is created, the `DataCoord` sends the segment id to the `RootCoord` via msgstream.
+2. `RootCoord` needs to update the segment id to the collection meta and record the position of the msgstream in etcd.
+3. Step 2 is transactional and the operation will be successful only if the collection meta in etcd is updated.
+4. So the `RootCoord` only needs to restore the msgstream to the position when recovering from a power failure.
+
+### 2.5 Flushed segment from `data node`
+
+1. Each time the `DataNode` finishes flushing a segment, it sends the segment id to the `RootCoord` via msgstream.
+2. `RootCoord` needs to fetch binlog from `DataCoord` by id and send a request to `IndexCoord` to create an index on this segment.
+3. When the `IndexCoord` is called successfully, it will return a build id, and then `RootCoord` will update the build id to the `collection meta` and record the position of the msgstream in etcd.
+4. Step 3 is transactional and the operation will be successful only if the `collection meta` in etcd is updated.
+5. So the `RootCoord` only needs to restore the msgstream to the position when recovering from a power failure.
+
+### 2.6 Failed to call external grpc service
+
+1. `RootCoord` depends on `DataCoord` and `IndexCoord`, if the grpc call failed, it needs to reconnect.
+2. `RootCoord` does not listen to the status of the `DataCoord` and `IndexCoord` in real time.
+
+### 2.7 Add virtual channel assignment when creating a collection
+
+1. Add a new field, "number of shards" in the `create collection` request. The "num of shards" tells the `RootCoord` to create the number of virtual channels for this collection.
+2. In the current implementation, virtual channels and physical channels have a one-to-one relationship, and the total number of physical channels increases as the number of virtual channels increases; later, the total number of physical channels needs to be fixed, and multiple virtual channels share one physical channel.
+3. The name of the virtual channel is globally unique, and the `collection meta` records the correspondence between the virtual channel and the physical channel.
+
+### Add processing of time synchronization signals from Proxy node
+
+1. A virtual channel can be inserted by multiple proxies, so the timestamp in the virtual channel does not increase monotonically.
+2. All proxies report the timestamp of all the virtual channels to the `RootCoord` periodically.
+3. The `RootCoord` collects the timestamps from the proxies on each virtual channel and gets the minimum one as the timestamp of that virtual channel, and then inserts the timestamp into the virtual channel.
+4. Proxy reports the timestamp to the `RootCoord` via grpc.
+5. Proxy needs to register itself in etcd when it starts, `RootCoord` will listen to the corresponding key to determine how many active proxies there are, and thus determine if all of them have sent timestamps to `RootCoord`.
+6. If a proxy is not registered in etcd but sends a timestamp or any other grpc request to `RootCoord`, `RootCoord` will ignore the grpc request.
+
+### 2.9 Register service in etcd
+
+1. `RootCoord` needs to register itself with etcd when it starts.
+2. The registration should include IP address, port, its own id and global incremental timestamp.
+
+### 2.10 Remove the code related to Proxy service
+
+1. `Proxy service` related code will be removed.
+2. The job of time synchronization which is done by `Proxy service` is partially simplified and handed over to the `RootCoord` (subsection 2.8).
+
+### 2.11 Query collection meta based on timeline
+
+1. Add a new field of `timestamp` to the grpc request of `describe collection`.
+2. `RootCoord` should provide snapshot on the `collection mate`.
+3. Return the `collection meta` at the point of timestamp mentioned in the request.
+
+### 2.12 Timestamp of `dd operations`
+
+1. `RootCoord` response is to set the timestamp of `dd operations`, create collection, create partition, drop collection, drop partition, and send this timestamp into `dml msgstream`.
@@ -0,0 +1,54 @@
+# MEP: Dynamic Configuration
+
+Current state: "Accepted"
+
+ISSUE: https://github.com/milvus-io/milvus/issues/18300
+
+Keywords: config etcd
+
+Released: 2.3.0
+
+## Summary(required)
+
+At present, there are numerous configurations in Milvus that require a restart of Milvus to take effect. This can interrupt service in production environments and is not friendly to operations and maintenance. In this MEP, a solution for dynamically updating configurations will be provided so that users can make configuration changes without restarting the cluster.
+
+## Motivation(required)
+
+Ability to dynamically modify configurations and expose current configuration information through API, simplifying operational complexity.
+
+## Public Interfaces(optional)
+
+No new public interfaces changed.
+
+## Design Details(required)
+
+### Goal
+
+1. Support multiple config sources, including Etcd, environment variables, and configuration files. On this basis, add watch events for changes in the Etcd config path and file changes. When a change event occurs, broadcast it to subscribers through an event handler. Subscribers can decide on subsequent logic based on this to achieve the requirement of dynamically modifying configurations.
+2. Configuration priority: Etcd > Environment > milvus.yaml; higher-priority configurations override lower-priority ones. Even if a higher-priority configuration is deleted, lower-priority configurations can still be used.
+3. To ensure compatibility, ignore case sensitivity and characters such as / . \_ when dealing with configuration item keys.
+
+![flowchart](../assets/graphs/dynamic_config_flowchart.jpg)
+ref: https://github.com/go-chassis/go-archaius/
+
+No-Goal (Not in this release plan)
+
+1. Configuration grading
+1. Node override config
+1. Collection override config
+1. Support for more cloud configs such as Consul, Zookeeper, etc.
+
+## Compatibility, Deprecation, and Migration Plan(optional)
+
+Compatible with old versions.
+
+## Test Plan(required)
+
+- Verify the ability to dynamically modify configurations in etcd. 
+- Verify the ability of helm and operator to deploy milvus.yaml and modify configurations. 
+- Verify the effectiveness of business operations after partially modifying dynamic configurations. 
+- Verify version compatibility under dynamic configuration.
+
+## Rejected Alternatives(optional)
+
+## References(optional)
@@ -0,0 +1,77 @@
+# MEP: Search By Primary Keys
+
+Current state: Under Discussion
+
+ISSUE: [[Feature]: Support to search by primary keys #23184](https://github.com/milvus-io/milvus/issues/23184)
+
+Keywords: Search, ANN
+
+Released: v2.3.0
+
+## Summary
+
+Support to search (ANNS) by the query vectors corresponding to the given primary keys.
+
+## Motivation
+
+For now, Milvus requires passing the query vectors to do anns, we have to fetch the vectors first if they are in the collection, which is complex and slow.
+
+We need a way to do anns directly for the corresponding vectors of given primary keys, which should be more efficient.
+
+## Public Interfaces
+
+Add new field `primary_keys` in `SearchRequest`, all SDKs adds new method:
+```golang
+func SearchByPK(
+    ctx context.Context, 
+    collName string, 
+    partitions []string,
+	expr string, 
+    outputFields []string, 
+    primaryKeys []entity.PrimaryKey, 
+    vectorField string, 
+    metricType entity.MetricType, 
+    topK int, 
+    sp entity.SearchParam, 
+    opts ...SearchQueryOptionFunc,
+) ([]SearchResult, error)
+```
+
+## Design Details
+
+Proxy fetches the vectors by primary keys from QueryNodes first, and then search with these vectors.
+
+For better performance, we will add a new RPC interface for QueryNode:
+```proto
+rpc Fetch(FetchRequest) returns (FetchResponse) {}
+```
+which only supports to fetch by primary keys, this will be more efficient than the present `Query` interface.
+
+
+## Compatibility, Deprecation, and Migration Plan
+
+None
+
+## Test Plan
+
+### Unit Tests
+
+- Test for fetching in segcore
+- Test for fetching in QueryNode
+- Test for searching by primary keys in Proxy
+
+
+### E2E Tests
+|                  Test Cases                  |         Expected Behavior          |
+| :------------------------------------------: | :--------------------------------: |
+|      search by non-existed primary keys      |            report error            |
+|        search by existed primary keys        | return topk results for each query |
+| all present tests cases of search by vectors |              the same              |
+
+## Rejected Alternatives
+
+Implement this without adding the `Fetch` interface, using `Query` to retrieve the vectors and then search by vectors.
+
+## References
+
+None
@@ -0,0 +1,105 @@
+# MEP: Default Value
+
+Current state: Under Discussion
+
+ISSUE: [[Feature]: Support Default Value #23337](https://github.com/milvus-io/milvus/issues/23337)
+
+Keywords: Default, Insert, Upsert 
+
+Released: v2.3.1
+
+## Summary
+
+Support Default Value when input data.
+
+## Motivation
+
+For now, Milvus don't support Default function. If the user pass in the same data under a certain field schema, the data can only be passed in repeatedly, which is not so flexible and user-friendly。
+
+We need a way to support Default function, which is more efficient.
+
+## Public Interfaces
+
+Add new field `default_value` in `FieldSchema`
+```proto
+message FieldSchema {
+  ...
+  ScalarField default_value = 11; // default_value only support scalars for now
+}
+```
+
+## Design Details
+
+1. Add the default_value in the field schema as an optional field.
+
+```proto
+
+message FieldSchema {
+  ...
+  ScalarField default_value = 11; // default_value only support scalars for now
+}
+```
+
+2. Will use the default_value if no data pass(the field get nil when insert and upsert).
+
+```proto
+
+message FieldData {
+  ...
+  oneof field {
+    ScalarField scalars = 3;
+    VectorField vectors = 4;
+  }
+}
+```
+
+```python
+    # create collection
+    nb = 3000
+    fields = [
+        FieldSchema(name="int64", dtype=DataType.INT64, is_primary=True),
+         # restrict at most one value to be passed in as the default value
+        FieldSchema(name="float", dtype=DataType.FLOAT, default_value=1.0)
+    ]
+    schema = CollectionSchema(
+        fields=fields, description="collection")
+
+    collection = Collection(name="hello_milvus", schema=default_schema)
+
+    #  insert data
+    collection.insert(
+        [
+            [i for i in range(nb)],
+            # will use the default_value
+            [],
+        ]
+    )
+```
+## Compatibility, Deprecation, and Migration Plan
+
+|               Test Cases                   |          ExpectedBehavior                 |
+|:-----------------------------------------: | :---------------------------------------: |
+|           schema built in 2.2.x            |  can be used normally in the new version  |
+ 
+## Test Plan
+
+### Unit Tests
+
+- Test for using default value in proxy
+
+
+### E2E Tests
+|                 Test Cases                   |         Expected Behavior                |
+| :------------------------------------------: | :--------------------------------------: |
+|         set illegal default value            |              report error                |
+|          set legal default value             |     use default value as fields data     |
+|           schema built in 2.2.x              |  can be used normally in the new version |
+|          don't set default value             |                the same                  |
+
+## Rejected Alternatives
+
+Default value is set by column, and the writing method of [1,2,3, {default}, {default}, 4, 5] is not supported.
+
+## References
+
+None
@@ -0,0 +1,271 @@
+# MEP: Refactor QueryNode v2
+
+Current state: Merged
+
+ISSUE: [[Enhancement]: Refactor QueryNode #21624](https://github.com/milvus-io/milvus/issues/21624)
+
+Keywords: Search, ANN
+
+Released: v2.3.0
+
+## Summary
+
+By refactoring querynode, we plan to achieve:
+
+- Separate "Delegator" and "Worker" 
+- Remove delta channel for deletion forwarding
+- Maintain growing segments in distribution
+- Improve the readability of the code
+
+## Delegator and Worker
+
+`Delegator`, aka `ShardLeader` in querynode v1, handles the segment distribution and consumes data from the dml channel. All the distribution changes(load&release) shall be forwarded by delegators so that they shall always have the latest workable segment distribution information for the shard.
+
+On the other hand, `Worker` serves as pure computing labor and provides search/query services on the segments on it.
+
+One querynode could be `Delegator` and `Worker` at the same time for now. After separating them into two sub packages, we could easily rearrange them into different components in the future if needed.
+
+### Interface Definition
+
+```Go
+// ShardDelegator is the interface definition.
+type ShardDelegator interface {
+        // Search & Query APIs
+        Search(ctx context.Context, req *querypb.SearchRequest) ([]*internalpb.SearchResults, error)
+        Query(ctx context.Context, req *querypb.QueryRequest) ([]*internalpb.RetrieveResults, error)
+        GetStatistics(ctx context.Context, req *querypb.GetStatisticsRequest) ([]*internalpb.GetStatisticsResponse, error)
+
+
+        // Distribution & dml related APIs
+        ProcessInsert(insertRecords map[int64]*InsertData)
+        ProcessDelete(deleteData []*DeleteData, ts uint64)
+        LoadGrowing(ctx context.Context, infos []*querypb.SegmentLoadInfo, version int64) error
+        LoadSegments(ctx context.Context, req *querypb.LoadSegmentsRequest) error
+        ReleaseSegments(ctx context.Context, req *querypb.ReleaseSegmentsRequest, force bool) error
+        SyncDistribution(ctx context.Context, entries ...SegmentEntry)
+}
+```
+```Go
+// Worker is the interface definition for querynode worker role.
+type Worker interface {
+        LoadSegments(context.Context, *querypb.LoadSegmentsRequest) error
+        ReleaseSegments(context.Context, *querypb.ReleaseSegmentsRequest) error
+        Delete(ctx context.Context, req *querypb.DeleteRequest) error
+        Search(ctx context.Context, req *querypb.SearchRequest) (*internalpb.SearchResults, error)
+        Query(ctx context.Context, req *querypb.QueryRequest) (*internalpb.RetrieveResults, error)
+        GetStatistics(ctx context.Context, req *querypb.GetStatisticsRequest) (*internalpb.GetStatisticsResponse, error)
+
+
+        IsHealthy() bool
+        Stop()
+}
+```
+
+## Remove delta channel
+
+After supporting `Delete` operation in Milvus 2.0.x, delta channels are needed for forwarding delete operation to the querynodes on which there are no related DML channels. 
+
+This mechanism makes the system require double the message queue topic compared to earlier Milvus version. Also, it couples the querynode search&query functionality with the forwarder of the delete records. Unfortunately, datanodes took this role, which may lead to search/query unavailability when some datanodes go down for some period of time.
+
+Naturally, the Delegator shall become the forwarder since it could consume all the dml data(including delete) from the message queue. There are some critical points that need to be designed carefully:
+
+- How to determine which segment/querynode shall be the target when forwarding the delete operations
+- How to guarantee that all the segments have the whole picture of the deletion data
+
+### Primary Key Oracle(PKOracle)
+
+We need a component which could determine or estimate which segments might have the data with the provided pk value. Naming after PKOracle, it could be implemented in the following ways:
+
+- Delegator has all the PK column data
+- Delegator has all the statslog(Bloom filter) files
+- A third party component stores the PK value-segment ID mapping
+
+Since we implemented delete using BF before, it's the first choice to have option 2.
+
+<img alt="PK Oracle" src="../assets/graphs/pk_oracle.png" width="600" />
+
+Delete grpc def:
+
+```Go
+Delete(context.Context, *querypb.DeleteRequest) (*commonpb.Status, error)
+```
+``` Protobuf
+message DeleteRequest {
+  common.MsgBase base = 1;
+  int64 collection_id = 2;
+  int64 partition_id = 3;
+  string vchannel_name = 4;
+  int64 segment_id = 5;
+  schema.IDs primary_keys = 6;
+  repeated uint64 timestamps = 7; 
+}
+```
+
+### Delete Forwarding Policy
+
+Delegators need to forward delete operation via grpc before any search/query operation can be executed. There are still several ways to forward the deletion data.
+
+- Forward  the delete ASAP and blocks the consuming workflow if forwarding fails
+- Forward the delete lazily. Which means deleting data could be forwarded in search or query request when needed with extra periodically "flush" jobs
+- Forward the processed bitset only
+
+Policy 3 can not be done without delegators having all the primary key data. So with pre-determined BF PKOracle implementation, we need to choose between policy 1 & 2.
+
+After some investigation, it turned out to be that all the deletion records need to be applied strictly in the sequence order by its timestamp. Otherwise, the internal binary search may return wrong bitset for deletion. So  Policy 1 became the only choice before we changed the segment inner implementation.
+
+### Data Integrity Guarantee
+
+Since Milvus2.x could be deployed as a distributed system, there are several cases that may damage data integrity
+
+- Load Asynchronizely 
+
+In current design, there is no guarantee that all segments will be ready when delegators forward the deletion records while the collection is being loaded. 
+
+- Load a new Segment
+
+A new segment might be loaded after the collection loaded due to some compaction might happen. If the consumed position is after the safe point (all delete operations before is synced to delta log), some delete entries might be missing during this procedure.
+
+- Balance, Node down or Rolling upgrade
+
+Similar to the previous case, when balancing segments, some deletion records might be missing as well. The same logic could apply to node down recovery and rolling upgrade.
+
+- Solution: Delete buffer with failure re-consume
+
+To solve the cases in which delete data might be lost, delegators will have a delete buffer to store "recent" delete data. So anytime a segment is loaded, the delegator will try to patch all the "needed" delete data from this buffer.
+By "recent", it means a limited double buffer with configurable size. 
+And "needed" delete data means the delete records after the segment checkpoint.
+If the segment checkpoint is beyond the delete buffer,the delegator will re-consume the delete data from the checkpoint as a last resort.
+
+## Other changes
+
+### Use pipeline instead of flowgraph
+
+Pipeline was a simplified flowgraph that every node could have one in-degree and one out-degree at most.
+
+Like an assembly line, pipeline splits a work that needs repeating over a period of time into many different parts. Every node was a single go routine work for one part of these. To improve  running speed by improving parallelism.  
+
+At querynode, pipeline was used to deal with msg from MsgStream, FilterNode filterates the invalid part in msg, Insert Node Insert rows to segment from msg,Delete Node Insert delete rows to segment from Msg and update TSafe.
+
+### Search/Query tsafe
+
+Since the only consumer is the delegator, the waiting tsafer logic is moved to delegator for now.
+
+## Interfaces
+
+### Manager
+```Go
+type CollectionManager interface {
+    // Get returns collection within a LRU cache,
+    // it will pull the collection from QueryCoord if it's not in the cache,
+    // returns error if failed to pull
+    Get(collectionID int64) (*Collection, error)
+}
+
+type SegmentManager interface {
+    // Put puts the given segments in,
+    // and increases the ref count of the corresponding collection,
+    // dup segments will not increase the ref count
+    Put(segmentType SegmentType, segments ...*Segment)
+    Get(segmentID UniqueID) *Segment
+    GetSealed(segmentID UniqueID) *Segment 
+    GetGrowing(segmentID UniqueID) *Segment
+    // Remove removes the given segment,
+    // and decreases the ref count of the corresponding collection,
+    // will not decrease the ref count if the given segment not exists
+    Remove(segmentID UniqueID, scope querypb.DataScope)
+}
+```
+
+### Loader
+```Go
+type Loader interface {
+    // Load loads binlogs, and spawn segments,
+    // NOTE: make sure the ref count of the corresponding collection will never go down to 0 during this
+    Load(ctx context.Context, collectionID int64, segmentType SegmentType, version int64, infos ...*querypb.SegmentLoadInfo) ([]Segment, error)
+}
+```
+
+### Segment
+```Go
+type Segment interface {
+    // Properties
+    ID() int64
+    Collection() int64
+    Partition() int64
+    Channel() string
+    Version() int64
+    StartPosition() *internalpb.MsgPosition
+    Type() SegmentType
+
+    // Index related
+    AddIndex(fieldID int64, index *IndexedFieldInfo)
+    GetIndex(fieldID int64) *IndexedFieldInfo
+    HaveIndex(fieldID int64) bool
+
+    // Insert related
+    Insert(entityIDs []int64, timestamps []Timestamp, record *segcorepb.InsertRecord) error
+    Delete(entityIDs []storage.PrimaryKey, timestamps []typeutil.Timestamp) error
+
+    // Query related
+    Search(searchReq *searchRequest) (*SearchResult, error)
+    Retrieve(plan *RetrievePlan) (*segcorepb.RetrieveResults, error)
+}
+
+func NewSegment(collection *Collection,
+    segmentID int64,
+    partitionID int64,
+    collectionID int64,
+    channel string,
+    segmentType SegmentType,
+    version int64,
+    startPosition *internalpb.MsgPosition) (*Segment, error)
+func DeleteSegment(segment *Segment)
+```
+
+## Collection
+```Go
+type Collection struct {
+}
+
+func (c *Collection) ID() UniqueID
+func (c *Collection) Schema() *schemapb.CollectionSchema
+func (c *Collection) GetPartitions() []int64
+func (c *Collection) HasPartition(partitionID int64) bool
+func (c *Collection) AddPartition(partitionIDs ...int64)
+func (c *Collection) RemovePartition(partitionID int64)
+func (c *Collection) GetLoadType() querypb.LoadType
+func NewCollection(collectionID int64, schema *schemapb.CollectionSchema, loadType querypb.LoadType) *Collection
+func DeleteCollection(collection *Collection)
+```
+
+## PipelineManager
+```Go
+type PipelineManager struct {
+}
+
+func (m *PipelineManager) Num() int
+func (m *PipelineManager) Add(collectionID UniqueID, dmlChannels []string) error
+func (m *PipelineManager) Get(collectionID UniqueID, channel Channel) (*Pipeline, error)
+func (m *PipelineManager) Remove(channels []Channel)
+func (m *PipelineManager) Close()
+```
+
+## Test Plan
+
+### Unit tests
+
+All packages in querynode v2 coverage about 80%
+
+### E2E Tests
+
+All existing load/release/search/query test cases passes.
+
+### Integration Tests
+
+- Worker delete failed test cases
+- Worker offline test cases
+
+## References
+
+None
+
@@ -0,0 +1,52 @@
+# MEP: Add collection level auto compaction config 
+
+Current state: In Progress
+
+ISSUE: [[Enhancement]: Support collection level config to disable auto-compaction #23993](https://github.com/milvus-io/milvus/issues/23993)
+
+Keywords: Collection, Compaction, Config
+
+Released: N/A
+
+## Summary
+
+Compaction has a config item to control whether auto-compaction is enabled or not. This configuration is global and impacts all collections in system.
+
+In some scenarios, we might want to control the granularity of auto-compaction switch so that it could be achieved that:
+
+- Disable auto-compaction during importing data to prevent rebuilt indexes
+- Disable auto-compaction during some test cases to make system behavior stable
+
+## Design
+
+Add collection level attribute, attribute key is "collection.autocompaction.enabled"(see also pkg/common/common.go).
+
+While handling all compaction signal, check collection level configuration:
+
+- If not set, use global auto-compaction setting
+- If config is valid, use collection level setting
+- If config value is invalid, fallback to global setting
+
+
+## How to change this setting
+
+All collection-level attribute could be changed by `AlterCollection` API
+
+## Test Plan
+
+### Unit tests
+
+Add unit tests for collection level auto compaction switch.
+
+### E2E Tests
+
+Change some case to disable collection auto compaction to rectify test case behavior.
+
+### Integration Tests
+
+- Add test to check auto compaction disabled
+ 
+## References
+
+None
+
@@ -0,0 +1,121 @@
+# MEP: Datanode remove dependency of `Datacoord`
+
+Current state: "Accepted" 
+
+ISSUE: https://github.com/milvus-io/milvus/issues/26758
+
+Keywords: datacoord, datanode, flush, dependency, roll-upgrade
+
+## Summary
+
+Remove the dependency of `Datacoord` for `Datanodes`.
+
+## Motivation
+
+1. Datanodes shall be always be running even when the data coordinator is not alive
+
+If datanodes performs `sync` during rolling upgrade, it needs datacoord to change the related meta in metastore. If datacoord happens to be offline or it is during some period of rolling-upgrade, datanode has to panic to ensure there is no data lost.
+
+2. Flush operation is complex and error-prone due since the whole procedure involves datacoord, datanodes and grpc
+
+This proposal means to remove the dependency of datacoord ensuring:
+
+- the data is integrate and no duplicate data is kept in records
+- no compatibility issue during or after rolling upgrade
+- `Datacoord` shall be able to detect the segment meta updates and provides recent targets for `QueryCoord`
+
+## Design Details
+
+The most brief description if this proposal is to:
+
+- Make `Datanode` operating the segment meta directly
+- Make `Datacoord` refresh the latest segment change periodically
+
+
+### Preventing multiple writers
+
+There is a major concern that if multiple `Datanodes` are handling the same dml channel, there shall be only one `DataNode` could update segment meta successfully.
+
+This guarantee is previously implemented by singleton writer in `Datacoord`: it checks the valid watcher id before update the segment meta when receiving the `SaveBinlogPaths` grpc call.
+
+In this proposal, `DataNodes` update segment meta on its own, so we need to introduce a new mechanism to prevent this error from happening:
+
+{% note %}
+
+**Note:** Like the "etcd lease for key", the ownership of each dml channel is bound to a lease id. This lease id shall be recorded in metastore (etcd/tikv or any other implementation).
+When a `DataNode` start to watch a dml channel, it shall read this lease id (via etcd or grpc call). ANY operations on this dml channel shall under a transaction with the lease id is equal to previously read value.
+If a `datanode` finds the lease id is revoke or updated, it shall close the flowgraph/pipeline and cancel all pending operations instead of panicking.
+
+{% endnote %}
+
+- [] Add lease id field in etcd channel watch info/ grpc watch request
+- [] Add `TransactionIf` like APIs in `TxnKV` interface
+
+### Updating channel checkpoint
+
+Likewise, all channel checkpoints update operations are performed by `Datacoord` invoking by grpc calls from `DataNodes`. So it has the same problem in previously stated scenarios.
+
+So, "updating channel checkpoint" shall also be processed in `DataNodes` while removing the dependency of `DataCoord`.
+
+The rules system shall follow is:
+
+{% note %}
+
+**Note:** Segments meta shall be updated *BEFORE* changing the channel checkpoint in case of datanode crashing during the prodedure. Under this premise, reconsuming from the old checkpoint shall recover all the data and duplidated entries will be discarded by segment checkpoints.
+
+{% endnote %}
+
+### Updating segment status in `DataCoord`
+
+As previous described, `DataCoord` shall refresh the segment meta and channel checkpoint periodically to provide recent target for `QueryCoord`.
+
+The `watching via Etcd` strategy is ruled out first since `Watch` operation shall avoided in the future design: currently Milvus system tends to not use `Watch` operation and try to remove it from metastore.
+Also `Watch` is heavy and has caused lots of issue before.
+
+The winning option is to:
+
+{% note %}
+
+**Note:** `Datacoord` reloads from metastore periodically.
+Optimization 1: reload channel checkpoint first, then reload segment meta if newly read revision is greater than in-memory one.
+Optimization 2: After `L0 segment` is implemented, datacoord shall refresh growing segments only.
+
+{% endnote %}
+
+
+## Compatibility, Deprecation, and Migration Plan
+
+This change shall guarantee that:
+
+- When new `Datacoord` starts, it shall be able to upgrade the old watch info and add lease id into it
+    - For watch info, release then watch
+    - For grpc, `release then watch` is the second choice, try call watch with lease id
+- Older `DataNodes` could invoking `SaveBinlogPaths` and other legacy grpc calls without panicking
+- The new `DataNodes` receiving old watch request(without lease id) shall fallback to older strategy, which is to update meta via grpc
+- `SaveBinlogPaths`, `UpdateChannelCheckpoints` APIs shall be kept until next break change
+
+## Test Plan 
+
+### Unit test
+Coverage over 90%
+
+### Integration Test
+
+#### Datacoord offline
+
+1. Insert data without datanodes online
+2. Start datanodes
+3. Make datacoord go offline after channel assignment
+4. Assert no datanode panicking and all data shall be intact
+5. Bring back datacoord and test `GetRecoveryInfo`, which shall returns latest target
+
+
+#### Compatibility
+
+1. Start mock datacoord
+2. construct a watch info (without lease)
+3. Datanode start to watch dml channel and all meta update shall be performed via grpc
+
+## Rejected Alternatives
+
+DataCoord refresh meta via Etcd watch
@@ -0,0 +1,116 @@
+# JSON Storage Design Document
+
+## 1. Data Model Design
+
+### 1.1 Data Layering
+
+#### Dense Part
+A set of "core fields" (such as primary keys and commonly used metadata) that are present in most records.
+
+#### Sparse Part
+Additional attributes that appear only in some records, potentially involving unstructured or dynamically extended information.
+
+### 1.2 JSON Splitting and Mapping
+
+#### Dense Field Extraction
+When parsing JSON, predefined dense fields are extracted and mapped to independent columns in Parquet. A method similar to Parquet Variant Shredding is used to flatten nested data.
+
+#### Sparse Data Preservation
+Fields not included in the dense part are stored in a sparse data field. They are serialized using BSON (Binary JSON) format, leveraging its efficient binary representation and rich data type support, with the result stored in a Parquet BINARY type field.
+
+## 2. Storage Strategy
+
+### 2.1 Columnar Storage for Dense Data
+- **Schema Definition**: Create independent columns in Parquet for each dense field, explicitly specifying data types (such as numeric, string, list, etc.).
+- **Query Performance**: Columnar format is suitable for large data scanning and aggregation operations, improving query efficiency, especially for vectors, indexes, and frequently queried fields.
+
+### 2.2 Row Storage for Sparse Data
+- **BSON Storage**:
+  - Serialize sparse data as BSON binary format and store it in a single binary column of the Parquet file.
+  - BSON format not only compresses more efficiently but also preserves complete data type information of the original data, avoiding numerous null values and file fragmentation issues.
+
+## 3. Parquet Schema Construction
+- **Columnar Part**: Build a fixed schema based on dense fields, with each field having a clear data type definition.
+- **Row Part**: Define a dedicated field (e.g., `sparse_data`) for storing sparse data, with type set to BINARY, directly storing BSON data.
+- **Hybrid Mode**: When writing, dense data is filled into respective columns, and remaining sparse data is serialized as BSON and written to the `sparse_data` field, achieving a balance between query efficiency and storage flexibility.
+
+## 4. Integration and Implementation Considerations
+
+### 4.1 Data Classification Strategy
+- **Density Classification**:
+  - Classify fields as dense or sparse based on their frequency of occurrence in records (e.g., greater than 30% for dense), while considering data type consistency. If a field has multiple data types, we should treat data types that appear in more than 30% of records as dense fields, with the remaining types stored as sparse fields.
+- **Dynamic Extension**:
+  - For dynamically extended fields, regardless of frequency, store them in the BSON-formatted sparse part to simplify schema evolution.
+
+### 4.2 Indexing for Sparse Data Access
+
+#### Sparse Column Key Indexing
+To accelerate BSON parsing, an inverted index stores BSON keys along with their offsets and sizes or values if they are of numeric type.
+
+##### Value Data Structure Diagram
+| Valid | Type  | Row ID | Offset/Value |
+|:-----:|:-----:|:------:|:------------:|
+| 1bit  | 4bit  | 27bit  | 16 offset, 16bit size |
+
+- **64-bit Structure Breakdown**:
+  - **Bit 1 (Valid)**: 1 bit indicating data validity (1 = valid, 0 = invalid).
+  - **Bits 2-5 (Type)**: 4 bits representing the data type.
+  - **Bits 5-31 (Row ID)**: 27 bits for the row ID, uniquely identifying the data row.
+  - **Bits 32-64 (Last 32 bits)**:
+    - If **Valid = 1**: Last 32 bits store the actual data value.
+    - If **Valid = 0**: Last 32 bits are split into:
+      - **First 16 bits (Offset)**: Indicates the data offset position.
+      - **Last 16 bits (Size)**: Indicates the data size.
+
+The column key index is optional, and can be configured at table creation time or modified later through field properties.
+
+## 5. Example Data
+
+### 5.1 Example JSON Records
+
+```json
+[
+    {"id": 1, "attr1": "value1", "attr2": 100},
+    {"id": 2, "attr1": "value2", "attr3": true},
+    {"id": 3, "attr1": "value3", "attr4": "extra", "attr5": 3.14}
+]
+```
+
+- **Dense Data:**
+  - The field `id` is considered dense.
+- **Sparse Data:**
+  - Record 1: `attr1`, `attr2`
+  - Record 2: `attr1`, `attr3`
+  - Record 3: `attr1`, `attr4`, `attr5`
+
+### 5.2 Parquet File Storage
+
+#### Schema Representation
+
+| Column Name   | Data Type | Description |
+|--------------|-----------|-------------|
+| **id**       | int64     | Dense column storing the integer identifier. |
+| **sparse_data** | binary  | Sparse column storing BSON-serialized data of all remaining fields. |
+| **sparse_index** | binary  | Index column storing key offsets for efficient parsing. |
+
+#### Stored Data Breakdown
+
+- **Dense Column (`id`)**:
+  - Row 1: `1`
+  - Row 2: `2`
+  - Row 3: `3`
+
+- **Sparse Column (`sparse_data`)**:
+  - **Row 1:** BSON representation of `{"attr1": "value1", "attr2": 100}`
+  - **Row 2:** BSON representation of `{"attr1": "value2", "attr3": true}`
+  - **Row 3:** BSON representation of `{"attr1": "value3", "attr4": "extra", "attr5": 3.14}`
+
+- **Sparse Index (`sparse_index`)**:
+  - **Row 1:** Index entries mapping `attr1` and `attr2` to their respective positions in `sparse_data`.
+  - **Row 2:** Index entries mapping `attr1` and `attr3`.
+  - **Row 3:** Index entries mapping `attr1`, `attr4`, and `attr5`.
+
+In an actual system, the sparse data would be serialized using a BSON library (e.g., bsoncxx) for a compact binary format. The example above demonstrates the logical mapping of JSON data to the Parquet storage format.
+
+---
+
@@ -0,0 +1,142 @@
+# Primary Key Index Design Document
+
+## 1. Introduction
+
+This document outlines the design of Milvus' primary key indexing system, which enables fast lookups of string or integer primary keys across multiple segments. The index will be loaded in the Delegator and persisted in S3 storage.
+
+## 2. Objectives and Benefits
+
+1. **Deduplication**: Identify duplicate data during write operations, automatically converting them to Insert + Delete operations
+2. **Accelerate Partial Updates**: Improve performance of partial upsert and point query operations
+3. **Optimize Delete Forwarding**: Reduce Bloom Filter check overhead in the Delegator during Delete operations
+
+## 3. Design Overview
+
+### 3.1 Core Components
+
+1. **BBhash**: A space-efficient hash structure that maps keys to a continuous range of integers without collisions. The master branch works with Plain Old Data types (POD), while the "alltypes" branch supports other types including strings.
+2. **Value Array**: A memory-mapped array storing segment position information for each primary key.
+
+### 3.2 Architecture Details
+
+1. **BBhash**:  
+   BBhash is a minimal perfect hash library for static key collections, capable of mapping each key to a unique, compact integer index. For example:
+
+   - "user123" → 0  
+   - "user456" → 1  
+   - "user789" → 2
+
+   For string primary keys, BBhash processes the raw byte sequence directly without type conversion and supports variable-length strings. Key features include:
+   - No need to store original strings
+   - Full content hashing reduces collision probability
+   - Extremely low memory usage
+
+2. **Value Array**:  
+   This array stores segment metadata for each primary key. It can be accessed directly using the BBhash mapping result, providing **O(1)** query efficiency:
+
+3. **example code**
+   ```cpp
+   // Example code for building and using the primary key index
+   
+   // Building the index
+   void buildPrimaryKeyIndex(const std::vector<std::string>& keys, const std::vector<SegmentInfo>& segmentInfos) {
+       // Initialize BBhash with the keys
+       bbhash::PerfectHasher<std::string> hasher(keys);
+       
+       // Initialize value array with appropriate size
+       std::vector<SegmentInfo> valueArray(keys.size());
+       
+       // Populate value array with segment information
+       for (size_t i = 0; i < keys.size(); i++) {
+           size_t index = hasher.lookup(keys[i]);
+           valueArray[index] = segmentInfos[i];
+       }
+       
+       // Persist the index to storage
+       hasher.save("bbhash.idx");
+       saveValueArray(valueArray, "value_array.bin");
+   }
+   
+   // Reading from the index
+   SegmentInfo lookupPrimaryKey(const std::string& key) {
+       // Load BBhash from storage (or use cached instance)
+       bbhash::PerfectHasher<std::string> hasher;
+       hasher.load("bbhash.idx");
+       
+       // Load value array (or use memory-mapped instance)
+       std::vector<SegmentInfo> valueArray = loadValueArray("value_array.bin");
+       
+       // Lookup the key
+       size_t index = hasher.lookup(key);
+       if (index != bbhash::NOT_FOUND) {
+           return valueArray[index];
+       }
+       
+       return SegmentInfo(); // Return empty segment info if not found
+   }
+   ```
+
+## 4. Index Structure Illustration
+
+### 4.1 BBhash Workflow
+
+BBhash (Bin Bloom Hash) maps keys to unique indices through multi-level hash functions:
+
+1. The first level hash attempts to map all keys to non-conflicting positions
+2. For keys with conflicts, a next-level hash function is used for remapping
+3. This process iterates until all keys are mapped without conflicts
+
+### 4.2 Value Array Storage Structure
+
+Each entry in the value array contains:
+- Segment ID (pointing to the segment containing the primary key)
+
+![Primary Key Index Structure](../assets/images/bucket.png)
+
+For L1 Segments, we don't need primary key indexing and can use Bloom Filters for approximate filtering with false positives. For L2 Segments, we build PK → Segment mappings for data under each bucket. Note that false positives still exist here due to: 1. Data that has been deleted, and 2. BBhash's small probability of false positives (approximately 1/2³² ≈ 2.3×10⁻¹⁰).
+
+3. **Memory Efficiency**:  
+   - BBhash: 2–4 bits/key (1B keys ≈ 250–500MB)
+   - Value Array: ~4 bytes/key (Segment ID)
+   - Total: ~4.5 bytes/key → 1B keys ≈ 4.5GB
+   - mmap implementation allows the operating system to load and reclaim memory as needed, supporting billion-scale datasets
+
+### 3.3 Performance Analysis
+
+#### 3.3.1 Index Building Performance
+
+- **Single-thread Performance**: BBhash constructs a minimal perfect hash function (MPHF) for 100 million keys in about 10 seconds on a single thread, processing approximately 10 million keys/second
+- **Multi-thread Scalability**: Using 8 threads, building an MPHF for 1 billion keys takes about 35 seconds, averaging approximately 28.57 million keys/second
+- **Billion-scale Construction Feasibility**:
+  - On a 32-core server, theoretical time to build a 1 billion key index is about 10-15 seconds
+  - In actual testing, end-to-end time including data reading and index construction reaches 1 minute
+  - Peak memory usage does not exceed 16GB
+
+#### 3.3.2 Query Performance Comparison
+
+- **Single Primary Key Index vs. Multiple Bloom Filters**:
+  - **Query Latency**:
+    - Primary Key Index: ~200 nanoseconds per query
+    - 10,000 Bloom Filters: Sequential querying required, average latency ~10,000 × 10 nanoseconds = 0.1 milliseconds
+    - **Performance Gap**: Primary key index query speed is approximately 500 times faster than the Bloom filter approach
+  
+  - **Throughput**:
+    - Primary Key Index: ~10-20 million queries per second per node
+    - Bloom Filter Approach: ~1,000 queries per second per node
+    - **Advantage**: Primary key index supports higher query loads in high-concurrency scenarios
+
+#### 3.3.3 Precision Comparison
+
+- **BBhash Precision**:
+  - Actual implementation may have an extremely small probability of hash collisions, but far lower than Bloom filters
+
+- **Bloom Filter Precision**:
+  - Single Bloom filter false positive rate is typically set to 0.1%
+  - Cumulative false positive rate when querying 10,000 Bloom filters approaches 100%
+
+## 4. Additional Considerations
+
+1. Performance validation, including index construction and querying, comparing BBhash and other libraries such as CMPH 
+2. Whether BBHash can also replace Bloom filters for individual Segments
+3. How to handle false positives - ignore? verify in each segment
+4. Value index redundancy fields for point query optimization, such as recording additional offset information or even fields
@@ -0,0 +1,297 @@
+# Milvus Row Level Security (RLS) Design Document
+
+## ✨ Overview
+
+Row Level Security (RLS) provides fine-grained access control at the row level for collections in Milvus. By enabling RLS and defining policies based on user identity, roles, or dynamic tags, administrators can enforce data access restrictions without modifying application logic or data structures.
+
+---
+
+## ⚙️ Core Capabilities
+
+| Feature               | Description                                                  |
+|-----------------------|--------------------------------------------------------------|
+| Enable/Disable RLS    | Toggle RLS at the collection level with runtime control      |
+| Enforce RLS           | Enforce RLS even for superusers and administrators          |
+| Policy Definition     | Define policies based on user ID, roles, field values, or tags |
+| Multi-policy Support  | Support for multiple policies per action/role combination    |
+| User Tag Mechanism    | Use dynamic user metadata for flexible access filtering      |
+| Expression Language   | Rich expression syntax for complex access control rules      |
+
+---
+
+## 🔖 User Tag Mechanism
+
+RLS leverages runtime user context including `$current_user_name` and `$current_user_tags` to evaluate access policies dynamically.
+
+### ✅ Setting User Tags
+
+```python
+client.set_user_tags(
+    user="user_abc",
+    tags={
+        "department": "engineering",
+        "region": "us-west-1",
+        "tenant": "customer_a",
+        "security_level": "confidential"
+    }
+)
+```
+
+### ✅ Tag Management APIs
+
+| API                          | Description                                    |
+| ---------------------------- | ---------------------------------------------- |
+| `set_user_tags(user, tags)`  | Set or update user tags (overwrites existing) |
+| `delete_user_tag(user, key)` | Delete a specific tag key for a user          |
+| `get_user_tags(user)`        | Fetch all user tag information                 |
+| `list_users_with_tag(key, value)` | Find users with specific tag values           |
+
+Tags can be referenced in policy expressions using the following syntax:
+
+```python
+using_expr="region == $current_user_tags['region']"
+check_expr="security_level >= $current_user_tags['clearance']"
+```
+
+---
+
+## 🛠️ API Design
+
+### 1. Enable or Disable RLS
+
+```python
+# Enable RLS for a collection
+client.alter_collection_properties(
+    collection="my_collection",
+    properties={"rls.enabled": True}
+)
+
+# Disable RLS for a collection
+client.alter_collection_properties(
+    collection="my_collection", 
+    properties={"rls.enabled": False}
+)
+```
+
+### 2. Enforce RLS (even for superusers)
+
+```python
+client.alter_collection_properties(
+    collection="my_collection",
+    properties={
+        "rls.enabled": True, 
+        "rls.force": True  # Applies to all users including superusers
+    }
+)
+```
+
+### 3. Create an RLS Policy
+
+```python
+client.create_row_policy(
+    collection="user_documents",
+    policy_name="limit_to_user",
+    actions=["query", "insert", "delete", "update"],
+    roles=["$current_user", "user_role"],
+    using_expr="user_id == $current_user_name",
+    check_expr="user_id == $current_user_name",
+    description="Restrict users to their own documents"
+)
+```
+
+**Policy Parameters:**
+- `collection`: Target collection name
+- `policy_name`: Unique identifier for the policy
+- `actions`: List of operations this policy applies to (`query`, `insert`, `delete`, `update`)
+- `roles`: List of roles this policy applies to (`$current_user`, `admin`, custom roles)
+- `using_expr`: Expression for filtering data during queries
+- `check_expr`: Expression for validating data during mutations
+- `description`: Optional human-readable description
+
+### 4. Delete an RLS Policy
+
+```python
+client.drop_row_policy(
+    collection="user_documents",
+    policy_name="limit_to_user"
+)
+```
+
+### 5. List All RLS Policies
+
+```python
+policies = client.list_row_policies(collection="user_documents")
+# Example response:
+# [
+#   {
+#     "policy_name": "limit_to_user",
+#     "using_expr": "user_id == $current_user_name",
+#     "check_expr": "user_id == $current_user_name", 
+#     "roles": ["$current_user"],
+#     "actions": ["query", "insert", "delete"],
+#     "description": "Restrict users to their own documents",
+#     "created_at": "2024-01-15T10:30:00Z"
+#   }
+# ]
+```
+
+### 6. Get Collection RLS Status
+
+```python
+status = client.get_collection_properties(
+    collection="user_documents",
+    properties=["rls.enabled", "rls.force"]
+)
+# Returns: {"rls.enabled": True, "rls.force": False}
+```
+
+---
+
+## ✅ Usage Examples
+
+### Example 1: Users Can Only Access Their Own Data
+
+**Scenario:** A document management system where users should only see and modify their own documents.
+
+**Collection Schema:**
+```python
+# Collection includes a user_id field
+{
+    "user_id": "string",
+    "document_name": "string", 
+    "content": "string",
+    "created_at": "timestamp"
+}
+```
+
+**RLS Policy:**
+```python
+client.create_row_policy(
+    collection="user_documents",
+    policy_name="user_own_data",
+    actions=["query", "insert", "delete", "update"],
+    roles=["$current_user"],
+    using_expr="user_id == $current_user_name",
+    check_expr="user_id == $current_user_name",
+    description="Users can only access their own documents"
+)
+```
+
+---
+
+### Example 2: Role-Based Access Control
+
+**Scenario:** Admins have full access, managers see department data, users see only their own data.
+
+**User Policy (restricted):**
+```python
+client.create_row_policy(
+    collection="employee_records",
+    policy_name="user_scope",
+    actions=["query", "insert", "delete", "update"],
+    roles=["$current_user"],
+    using_expr="employee_id == $current_user_name",
+    check_expr="employee_id == $current_user_name"
+)
+```
+
+**Manager Policy (department scope):**
+```python
+client.create_row_policy(
+    collection="employee_records", 
+    policy_name="manager_scope",
+    actions=["query", "insert", "update"],
+    roles=["manager"],
+    using_expr="department == $current_user_tags['department']",
+    check_expr="department == $current_user_tags['department']"
+)
+```
+
+**Admin Policy (full access):**
+```python
+client.create_row_policy(
+    collection="employee_records",
+    policy_name="admin_full_access",
+    actions=["query", "insert", "delete", "update"],
+    roles=["admin"],
+    using_expr="true",
+    check_expr="true"
+)
+```
+
+---
+
+### Example 3: Multi-Tenant Data Isolation
+
+**Scenario:** SaaS application with tenant-based data isolation using user tags.
+
+**Policy:**
+```python
+client.create_row_policy(
+    collection="customer_data",
+    policy_name="tenant_isolation",
+    actions=["query", "insert", "delete", "update"],
+    roles=["$current_user"],
+    using_expr="tenant_id == $current_user_tags['tenant']",
+    check_expr="tenant_id == $current_user_tags['tenant']"
+)
+```
+
+**User Tag Setup:**
+```python
+client.set_user_tags(
+    user="user_123",
+    tags={"tenant": "acme_corp", "role": "analyst"}
+)
+```
+
+---
+
+### Example 4: Time-Based Access Control
+
+**Scenario:** Documents are only accessible during business hours for non-admin users.
+
+**Policy:**
+```python
+client.create_row_policy(
+    collection="sensitive_documents",
+    policy_name="business_hours_access",
+    actions=["query"],
+    roles=["$current_user"],
+    using_expr="(hour(now()) >= 9 AND hour(now()) <= 17) OR $current_user_tags['role'] == 'admin'",
+    check_expr="true"
+)
+```
+
+---
+
+## 🔒 Security Model Notes
+
+### Policy Evaluation
+- **OR Logic**: All policies for a user are OR-combined - if any policy grants access, the operation is allowed
+- **Action-Specific**: Policies are evaluated based on the specific action being performed
+- **Role Matching**: Users must have at least one role that matches the policy's role list
+
+### Access Control Levels
+- **Default Behavior**: RLS applies only to non-superusers
+- **Force Mode**: With `rls.force=True`, RLS applies to everyone including superusers and administrators
+- **Bypass Options**: Superusers can temporarily bypass RLS for maintenance operations
+
+### Expression Language
+- **Field References**: Use field names directly in expressions
+- **Variables**: `$current_user_name`, `$current_user_tags`, `$current_roles`
+- **Functions**: Support for common functions like `now()`, `hour()`, `date()`
+- **Operators**: Standard comparison and logical operators
+
+### Performance Considerations
+- **Index Usage**: RLS expressions should leverage indexed fields for optimal performance
+- **Expression Complexity**: Complex expressions may impact query performance
+- **Policy Count**: Large numbers of policies per collection may affect evaluation speed
+
+### Best Practices
+- **Principle of Least Privilege**: Start with restrictive policies and gradually expand access
+- **Regular Auditing**: Periodically review and test RLS policies
+- **Documentation**: Maintain clear documentation of policy purposes and effects
+- **Testing**: Test policies with various user roles and scenarios before production deployment
+
+
@@ -0,0 +1,498 @@
+# Background
+
+Milvus is a high-performance vector database widely used in fields such as image retrieval, recommender systems, and semantic search. With the increasing integration of scenarios such as LBS (location-based service), MultiModal Machine Learning retrieval, and driverless technology **spatial awareness** and **semantic retrieval** requirements, users increasingly need to be able to perform vector searches in "geographical context".
+
+Currently, Milvus only supports numerical, string, and vector type fields, **lacking native understanding and index support for geographical information**. This limits its capabilities in emerging spatial computing applications.
+
+# Requirement
+
+The following is a typical use case that combines geographic information with vector retrieval:
+
+_Perform vector search under spatial constraints_
+
+> I would like to search for products, stores, and images that are within 1 kilometer of me and semantically most similar.
+
+Example Industries:
+
+- Local services (such as Ele.me, Uber)
+    
+- Map content recommendation (e.g., AutoNavi Maps recommendation)
+    
+- E-commerce scenarios (LBS advertising, nearby same-item retrieval)
+    
+- Security surveillance (tracking similar faces near a given location)
+    
+
+In addition to being combined with vector retrieval, supporting Geographic Information System (GIS) can also meet many common requirements for geographic information-based analysis. For example, heat map analysis, planning transportation routes, and planning market locations through statistical analysis, etc.
+
+Sometimes, users may also choose different coordinate systems to analyze geographic data from different perspectives based on different usage scenarios. And since geographic data usually comes from third-party platforms (maps, remote sensing, testing, etc.), with diverse data formats, it is necessary to support at least WKT and WKB, and GeoJson, GeoHash and other data formats can be added as needed later.
+
+Based on the above requirements, it is necessary to provide support for the geospatial data structure in Milvus, including the definition of its core data types, various query operators, index optimization, etc. The following details the feasible implementation plan from several aspects.
+
+# Implementation Plan
+
+## Data Type
+
+Geo Spatial DataType is a data structure used to describe geospatial information. In the SFA (Simple Feature Access) standard developed by the OGC, the following common geometric types are defined:
+
+1. Point: Represents a two-dimensional coordinate, usually representing different objects depending on the scale
+    
+2. LineString: An ordered collection composed of two or more points, often used to represent rivers, roads, etc.
+    
+3. Polygon: Represents a planar region that can have "holes".
+    
+4. MultiPoint: A collection of multiple points
+    
+5. MultiLineString: A collection of multiple line strings
+    
+6. MultiPolygon: A collection of multiple polygons
+    
+7. GeometryCollection: A collection composed of all the above geometries
+    
+
+The input and output of these data types have two representation methods in the SFA standard: WKT (Well Known Text) and WKB (Well Known Binary). The former is a human-readable format, such as: `POINT (0 0)` , `MULTILINESTRING ((0 0, 1 1), (2 2, 3 3))` , etc., while the latter is a binary format for efficient storage.
+
+In Milvus, we do not need to manually define the above data types. Instead, we can introduce third-party libraries, wrap the geometric classes provided by third-party libraries in Milvus, and then perform unified processing on geometric data and provide a unified processing interface externally.
+
+Common third-party libraries include GEOS, GDAL, etc. Among them, GEOS (Geometry Engine - Open Source) is an open-source geometric computation library that adheres to the OGC SFA standard. OGR, on the other hand, is a wrapper library for the geos library and some functions such as coordinate transformation and other geographic coordinate types. It has now been integrated into GDAL, which contains many contents unrelated to geographic information, such as grating processing. For the scalability of the geographic information system, we choose to use the OGR library encapsulated by GDAL. Meanwhile, for the sake of simplicity, we need to remove the extra parts we don't need (such as grating processing) during compilation.
+
+## Coordinate System
+
+Commonly used coordinate systems include WGS 84, the international standard for latitude and longitude; Web Mercator, the standard for web maps; etc.
+
+The underlying layer of GEOS has no coordinate system and only performs operations on geometric shapes in the Cartesian set (plane coordinate system). However, the WKT information input by the user may be based on latitude and longitude information, and when the scale is large enough, there will be a problem of excessive error caused by direct conversion between spherical coordinates and plane coordinates.
+
+Therefore, when high-precision calculations are required, it is necessary to first use the coordinate transformation function provided by GDAL for conversion before performing the calculations.
+
+## Query Operator
+
+Common query operators include:
+
+### 1. Topological Relationships
+
+These functions are used to determine whether there is a certain topological relationship (such as intersection, inclusion, coverage, etc.) between two geometric objects, based on the logical relationship between the shape and position of spatial objects.
+
+| Function Name         | Description                                                                                                                                                                                          |
+| --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `ST_Contains(A, B)`   | Determine **whether A completely contains B (excluding the boundary)** , and there is at least one common point in the interiors of A and B. For example: whether a city contains a park.            |
+| `ST_Covers(A, B)`     | Determine **whether A covers B**, i.e., all points of B are inside or on the boundary of A.                                                                                                          |
+| `ST_Equals(A, B)`     | Determine **whether A and B represent the same geometric object** , i.e., have the same coordinate sequence and type.                                                                                |
+| `ST_Intersects(A, B)` | Determine **whether A and B have at least one common point** , which is the most commonly used spatial relationship judgment function.                                                               |
+| `ST_Overlaps(A, B)`   | Determine **whether A and B overlap**, i.e., they have the same dimension (e.g., both are faces or both are lines), and their regions partially coincide, but neither completely contains the other. |
+| `ST_Touches(A, B)`    | Determine **whether A and B only touch at the boundary**, but do not intersect internally. For example: the boundary of adjacent plots.                                                              |
+| `ST_Within(A, B)`     | Determine **whether A is completely inside B** and that A and B have at least one common point in their interiors. It is `an alias for ST_Contains(B, A)`.                                           |
+## Index Optimization
+
+To speed up querying, it is necessary to create an index for geographic information data. In a standard database, whether scalar or vector, indexes are built based on the values of the indexed columns, but it is different for spatial indexes because the database cannot directly index the values of geometric fields, that is, the geometric objects themselves. Therefore, it is necessary to index the **bounding box** of geometric objects. The following example is from the PostGIS documentation:
+
+![_images/bbox.png](https://postgis.net/workshops/zh_Hans/postgis-intro/_images/bbox.png)
+
+In the figure above, the number of lines intersecting the yellow star is **1** , which is the red line. However, the range boxes intersecting the yellow box include the red and blue ones, a total of 2.
+
+The database solves the problem of "what line intersects with the yellow star" by first using a spatial index to solve the problem of "what bounding box intersects with the yellow bounding box" (which is very fast), and then "what line intersects with the yellow star". The above process only applies to the spatial features of the first test.
+
+For large data tables, this "two-pass method" of indexing first and then performing local precise calculations can fundamentally reduce the amount of query computation.
+
+Common indexing methods include: R-Tree, QuadTree, GeoHash, S2, H3, etc.
+
+> R-Tree：
+> 
+> For each spatial object, a Minimum Bounding Rectangle (MBR) is established, and these MBRs are recursively organized into a tree structure. The internal nodes of the tree contain the MBRs of multiple sub-nodes, which are used to quickly filter out irrelevant regions; leaf nodes store the actual spatial objects. When performing a spatial query, the R-Tree first uses the MBRs for quick filtering to find the set of objects that may meet the conditions, and then conducts precise spatial relationship judgments, thereby significantly improving query efficiency.
+
+
+> QuadTree：
+> 
+> Its core idea is to divide the entire space into four quadrants, with each quadrant continuing to be recursively divided until the number of objects contained within each region is less than the set threshold. Each node represents a spatial region, and leaf nodes store actual data objects. When performing a query, the QuadTree starts from the root node, sequentially checks which sub-nodes intersect with the query range, and recursively enters these sub-nodes, ultimately finding matching objects in the leaf nodes. This structure is simple to implement and suitable for scenarios such as image processing and map tile systems, but it may encounter imbalance issues when dealing with high-density or unevenly distributed data, affecting performance.
+
+
+> geohash：
+> 
+> The basic principle of geohash is: recursively divide the Earth's surface using a quadtree, where each division assigns a binary bit to the resulting sub-region, and finally a hash string for a specific region is given through base32/64 encoding. When querying, input the latitude and longitude, calculate its geohash, then specify the desired prefix length to match (e.g., 6 digits), and the algorithm will return regions with matching prefixes.
+> 
+> This indexing method is suitable for nearest neighbor search. And the longer the prefix, the more precise the match. However, it performs poorly at the "boundary", i.e., there may be cases where neighbors at the junction of rectangular regions do not have the same prefix.
+
+
+> s2：
+> 
+> Its core idea is to project the Earth's surface onto the six faces of a cube, with each face further recursively divided into small cells (referred to as Cells) in a quadtree structure. Each cell has a unique 64-bit integer ID (CellID) and supports up to 30 levels of resolution. The division at each level maintains the Hilbert curve order to ensure spatial locality. When performing queries, a set of spatial cells covering these geometries can be generated based on points, lines, or polygons, enabling efficient operations such as range queries and intersection judgments. This structure has a rigorous mathematical foundation, supports global seamless stitching, avoids boundary issues in traditional planar divisions, and is particularly suitable for complex scenarios such as high-precision spatial analysis, map tile systems, polygon coverage, and spatial aggregation. Although its implementation is relatively complex, it provides rich API support and is suitable for application systems requiring precise spatial operations.
+
+
+> h3：
+> 
+> The core idea is to divide the Earth's surface into a series of regular hexagonal grids (hexagons), with most regions being regular hexagons except for the polar regions. The entire division uses the Icosahedron unfolding method to form a hierarchical hexagonal grid system, with each layer having a different resolution (a total of 15 layers). Each hexagonal cell is assigned a unique 64-bit integer ID, which contains information such as the cell's layer information, parent cell path, and position offset. When performing queries, data in the surrounding area can be quickly obtained by finding the neighbors of a certain hex (up to 6), the K-ring range, etc. This structure is naturally suitable for neighborhood analysis and heat map display, with good spatial uniformity and aggregation capabilities. H3 performs particularly well in scenarios such as spatial aggregation, spatial connection, and path planning.
+
+The geos library provides support for R-Tree and Quad-Tree. s2 and h3 each have their own officially provided libraries. GeoHash, on the other hand, requires additional third-party library support.
+
+First, focus on the R-Tree index and S2 index:
+
+The R-Tree index is the index type used by PostGIS to support spatial relationship query functions such as ST_XXX defined in the OGC standard. In actual development, you can directly create an R-Tree object, insert data, serialize it, and write it to disk.
+
+The S2 index library does not support standard geographic information SQL query statements, but these functions can be simulated through its provided APIs. S2 provides `S2ShapeIndex` as the core index structure, which is used to manage geometric objects such as points, lines, and polygons, and supports various spatial query operations such as efficient inclusion judgment, nearest neighbor search, and intersection detection. At the same time, S2 provides serialization and deserialization functions. It should be noted that the S2 library uses its own defined spatial data types rather than the OGC standard WKT/WKB, and manual conversion is required.
+
+> **Note: Not all spatial functions will use indexes!**
+> Functions that support the use of spatial indexes in PostGIS include:
+>ST_Within,ST_DWithin,ST_Intersects,ST_Contains,ST_Covers,ST_Overlaps,ST_Crosses,ST_Equals
+
+# User Use Case: pymilvus + restful
+
+## pymilvus
+
+```Python
+from pymilvus import MilvusClient,DataType
+collection_name = "geo_point_collection"
+milvus_client = MilvusClient("http://localhost:19530")
+
+schema = MilvusClient.create_schema(
+        auto_id = False,
+        enable_dynamic_field = False,
+)
+
+# create geo fields
+schema.add_field(name = "id",datatype = DataType.INT64,is_primary = True)
+schema.add_field(name = "name",datatype = DataType.VARCHAR,max_length = 255)
+schema.add_field(name = "location",datatype = DataType.GEOMETRY)
+
+# create collection
+milvus_client.create_collection(collection_name,schema)
+
+# insert 
+data =[
+        {"id": 1001,"name": "Shop A","location": "POINT(116.4 39.9)"},
+        {"id": 1002,"name": "Shop B","location": "POINT(116.5 39.8)"},
+        {"id": 1003,"name": "Shop C","location": "POINT(116.6 39.7)"}
+]
+
+milvus_client.insert(collection_name,data)
+# query
+# 1. spatial relationship
+  
+# The usage of spatial relationship querys are like this:
+# ST_XXX({field_name}, {wkt_string})
+# where {field_name} is the name of the field that you want to query,
+# and {wkt_string} is the wkt string of the geometry.
+# So the results of the query are the specific geometry objects in the field that 
+# meet the spatial relationship to a given geometry
+# Including:
+# ST_Contains,ST_Within,ST_Covers,ST_Intersects
+# ST_Disjoint,ST_Equals,ST_Crosses,ST_Overlaps,ST_Touches
+
+# ST_Within
+within_wkt = "POLYGON((116.4 39.9,116.5 39.9,116.6 39.9,116.4 39.9))"
+results = milvus_client.query(
+        collection_name=collection_name,
+        filter=f"ST_Within(location, '{within_wkt}')",
+        output_fields=["name","location"]
+)
+print(results)
+
+  
+# ST_Covers
+covers_wkt = "POLYGON((116.4 39.9,116.5 39.9,116.6 39.9,116.4 39.9))"
+results = milvus_client.query(
+        collection_name=collection_name,
+        filter=f"ST_Covers(location, '{covers_wkt}')",
+        output_fields=["name","location"]
+)
+print(results)
+
+# 2. distance relationship
+# The usage of distance relationship querys are like this:
+# ST_DWithin
+point_wkt = "POINT(116.5 39.9)"
+distance = 10000 # meters
+results = milvus_client.query(
+        collection_name=collection_name,
+        filter=f"ST_DWithin(location, '{point_wkt}', {distance})",
+        output_fields=["name","location"]
+)
+print(results)
+
+# ST_Distance
+target_point = "POINT(116.5 39.9)"
+results = milvus_client.query(
+        collection_name=collection_name,
+        filter=f"ST_Distance(location, '{target_point}') < 10000",
+        output_fields=["name","ST_Distance(location, '{target_point}')"] # Here we may need to support the calculation of the result
+)
+print(results)
+
+# 3. Others
+# These queries need to do some calcutions in filter or output_fields like ST_Distance above,so we need to support this function.
+# But it may be not easy,and confilict with the current implementation of the filter and output_fields.
+
+# 4. hybrid query
+# We may sometimes use a specific condition to query and get some caculation results,consider the following case:
+# Hybrid query example: filter by coverage condition and calculate area for qualifying polygons
+
+coverage_wkt = "POLYGON((116.3 39.8,116.7 39.8,116.7 40.0,116.3 40.0,116.3 39.8))"
+results = milvus_client.query(
+collection_name=collection_name,
+        filter=f"ST_Covers(location, '{coverage_wkt}')", # Filter polygons that cover the specified area,here the location may not be a point,but can be a polygon
+        output_fields=["name", "location", "ST_Area(location)"] # Calculate area for qualifying polygons
+)
+
+print("Polygons covering the specified area and their areas:")
+for result in results:
+        print(f"Name: {result['name']}, Area: {result['ST_Area(location)']} square meters")
+```
+
+## restful
+
+The usage method can be roughly inferred from the pymilvus example.
+
+```Plain
+curl --request POST \
+    --url "${CLUSTER_ENDPOINT}/v2/vectordb/entities/query" \
+    --header 'accept: application/json' \
+    --header 'content-type: application/json' \
+    -d '{
+        "dbName": "_default",
+        "collectionName": "geo_point_collection",
+        "filter": "ST_Within(location, ''POLYGON((116.4 39.9,116.5 39.9,116.6 39.9,116.4 39.9))'')",
+        "outputFields": ["name", "location"]
+    }'
+```
+
+Calculate the search results:
+
+```Plain
+curl --request POST \
+    --url "${CLUSTER_ENDPOINT}/v2/vectordb/entities/query" \
+    --header 'accept: application/json' \
+    --header 'content-type: application/json' \
+    -d '{
+        "dbName": "_default",
+        "collectionName": "geo_point_collection",
+        "filter": "ST_Distance(location, ''POINT(116.5 39.9)'') < 10000",
+        "outputFields": ["name", "ST_Distance(location, ''POINT(116.5 39.9)'')"]
+    }'
+```
+
+Combining the above possible usage scenarios, in the query and deletion scenarios, for the filter and output_field fields, it is necessary to support calculation during query and return the calculation results.
+
+# Implementation Plan
+
+## Dependency Installation
+
+We currently require the following third-party libraries as basic dependencies:
+
+| Dependency Name | Function Description                                                                |
+| --------------- | ----------------------------------------------------------------------------------- |
+| gdal            | Provides encapsulation of geometric objects and support for spatial query operators |
+| libspatialindex | Provides spatial index support                                                      |
+
+## Data Insertion
+
+### Overview of Insertion Mechanism
+
+- Shard mechanism: Users can configure the number of shards for each collection.
+    
+- Mapping relationship:
+    
+    - Each shard → one virtual channel (vchannel)
+        
+    - A vchannel is assigned to a physical channel (pchannel), and multiple vchannels can share a pchannel
+        
+    - pchannel → StreamingNode (SN)
+        
+- Data Flow:
+    
+    - After the Proxy layer verifies the data, it splits it into multiple packages;
+        
+    - Distributed to the corresponding shard's pchannel according to the rules;
+        
+    - StreamingNode receives and processes data.
+        
+
+### Data writing process
+
+1. SN timestamps each package to establish the operation sequence;
+    
+2. Data is first written on the WAL (Write Ahead Log) and divided into segments;
+    
+3. When a WAL segment is processed, a refresh operation is triggered;
+    
+4. Data is ultimately written to object storage;
+    
+5. The above steps are completed by StreamingNode.
+    
+
+### Development sequence: Add support for the geo field from top to bottom
+
+| Level          | File/Module                | Modify Target                                                    |
+| -------------- | -------------------------- | ---------------------------------------------------------------- |
+| Proxy Layer    | internal/proxy/validate.go | Add validation logic for the geometry field                      |
+| Data Layer     | Multiple files             | Responsible for memory structure, serialization, packaging, etc. |
+| C++ Core Layer | Multiple C++ files         | Added Geometry class, supporting Segment-related logic           |
+See the appendix for the specific plan
+
+## Query
+
+- **segment type**
+
+| Type    | Status Description                          | Load Node     |
+| ------- | ------------------------------------------- | ------------- |
+| Growing | can write data and is in an active state    | StreamingNode |
+| Sealed  | Read-only state, has been persisted to disk | QueryNode     |
+
+> Trigger mechanism: When the Growing Segment reaches a certain size or after a period of time, it will be written as a Sealed Segment.
+
+- **Brief Description of** **Query** **Execution Process**
+    
+    - The user initiates a query request;
+        
+    - Proxy distributes queries in parallel to all StreamingNodes that hold the shard;
+        
+    - Each SN generates its own query plan:
+        
+        1. Query data in the local Growing Segment;
+            
+        2. Simultaneously communicate with QueryNode to query data in Sealed Segment;
+            
+    - All results are merged and then returned to the user.
+        
+- Modification Point
+    
+    - Phase 1: Support for regular query expressions (Expr)
+
+| File Path/Module                                | Modify Target                                                           |
+| ----------------------------------------------- | ----------------------------------------------------------------------- |
+| plan.g4                                         | Add syntax definition for GIS query operators                           |
+| internal/core/src/expr/exec/expression/Expr.cpp | Add expression evaluation logic for the geo field                       |
+| Add Expression Node                             | Implement AST nodes for GIS operators such as ST_Contains and ST_Covers |
+> This stage only supports the use of GIS operators in the filter and does not involve aggregate calculations in output_fields.
+
+- Phase 2: Support aggregation functions (such as ST_Area) in output_fields
+
+  Functional Objectives
+  Supports queries in the following forms:
+```Python
+results = milvus_client.query(
+    collection_name=collection_name,
+    filter=f"ST_Covers(location, '{coverage_wkt}')",
+    output_fields=["name", "location", "ST_Area(location)"]
+)
+```
+
+> Requires directly returning`the calculation result of ST_Area(location)`, rather than the original WKB/WKT data.
+
+## Index
+
+- Index building trigger process
+    
+    - Client requests to create an index:
+        
+        1. SDK sends`create_index`request;
+            
+    - Server level processing:
+        
+        1. Milvus does not immediately execute index building;
+            
+        2. The request information is written to the log and sent to Datacoord via the channel;
+            
+    - Task Scheduling Phase:
+        
+        1. Datacoord listens to this channel;
+            
+        2. After receiving the request, create an indexing task and add it to the scheduling queue;
+            
+    - Execution Phase:
+        
+        1. The scheduler distributes the task to a DataNode;
+            
+        2. DataNode loads the target segment data from the object storage;
+            
+        3. Build index;
+            
+        4. Write the indexing results back to the object storage.
+            
+    
+    > ⚠️ Note: Each flushed segment will build an index independently.
+    
+- Modification Point
+
+| Level          | Module/File                                 | Modify Target                                                    |
+| -------------- | ------------------------------------------- | ---------------------------------------------------------------- |
+| Go layer       | pkg/util/typeutil/schema.go                 | Add IsGeometryType() function                                    |
+|                | internal/proxy/task_index.go                | Update parseIndexParams() to support parsing Geo type parameters |
+|                | pkg/util/paramtable/autoindex_param.go      | Add Geo index configuration item in AutoIndexConfig              |
+| C++ Core layer | internal/core/src/index/IndexFactory.cpp    | Supports the creation of R-Tree or Geo indexes                   |
+| tools          | internal/util/indexparamcheck/index_type.go | Update index type validation logic                               |
+| Query Engine   | internal/cgo/src/query/ScalarIndex.h        | Add Scalar Index helper function                                 |
+| Segment Core   | internal/core/src/segcore/FieldIndexing.cpp | Update the field index construction logic to support Geo types   |
+# Appendix
+
+## insert modification point
+
+### Proxy Layer
+
+- File:`validate_util.go`
+    
+- Modified content: Added logic for parsing and validating the geometry field.
+### Storage Layer
+
+| File Name              | Function Description                                          |
+| ---------------------- | ------------------------------------------------------------- |
+| data_codec.go          | Implement serialization and deserialization of geo data       |
+| data_sorter.go         | Supports sorting of geo types                                 |
+| insert_data.go         | Add support for geo data type and factory function            |
+| payload.go             | Implement binary read and write support for geo data          |
+| payload_writer/reader  | Provides read and write interfaces                            |
+| serde.go               | Mutual conversion between Geo data and Arrow, Parquet formats |
+| util,print_binlog,test | Supplement of utility classes and test cases                  |
+### C++ Core Layer
+
+#### Common Module
+
+| File Name                   | Modified Content                                                                                                      |
+| --------------------------- | --------------------------------------------------------------------------------------------------------------------- |
+| Types.h                     | New GEOMETRY type enumeration added                                                                                   |
+| Geometry.h/.cpp             | Define the Geometry class, encapsulate WKB, and implement construction, serialization, basic spatial operations, etc. |
+| fieldData.h/cpp/interface.h | Supports data management, population, and access for geo fields                                                       |
+| array.h                     | Add support for geo array type                                                                                        |
+| chunk.h/writer              | Implement binary block writing of geo data                                                                            |
+| vectorTrait.h               | Add judgment logic for the geo type, with the same status as JSON                                                     |
+#### Segcore Module
+
+| File Name                  | Modified Content                                          |
+| -------------------------- | --------------------------------------------------------- |
+| ChunkSegmentSealedImpl.cpp | Add geo type processing during loading and index building |
+| ConcurrentVector.cpp       | Store inserted data and add support for geo type          |
+| InsertRecord.h             | Add geo support in append_field_meta                      |
+| SegmentGrowingImpl.cpp/h   | Add geo processing logic to the Insert method             |
+| SegmentSealedImpl.cpp      | Add geo support to the data loading logic                 |
+| Utils.cpp                  | Add geo-related support to the utility function           |
+#### Storage Module
+
+| File Name | Modified Content                             |
+| --------- | -------------------------------------------- |
+| Event.cpp | Add geo support in the serialization section |
+| Util      | Add geo type handling in relevant cases      |
+### Explanation of Design Key Points
+
+| Member variable/function | Description                                                             |
+| ------------------------ | ----------------------------------------------------------------------- |
+| context_                 | GEOS context handle, thread-safe                                        |
+| geometry_                | Geometry object in GEOS internal representation                         |
+| wkb_ / wkb_size_         | Original data source in WKB format and its size                         |
+| srid_                    | Spatial Reference Identifier, default is 4326 (WGS84)                   |
+| ConstructFromWKT/WKB     | Supports constructing Geometry objects using WKT/WKB                    |
+| to_wkt_string()          | Returns a WKT string representation                                     |
+| to_wkb_vector()          | Return WKB binary data                                                  |
+| GetArea(), GetLength()   | Get area and length                                                     |
+| SpatialOps               | Supports common spatial operations: equality, intersection, union, etc. |
+| TransformToSRID()        | Coordinate System Transformation                                        |
+| isValid(), repair()      | Check validity and attempt to repair invalid geometry                   |
+## query modification point
+
+#### Overview of Modifications
+
+| File Path/Module                   | Modify Target                                                      |
+| ---------------------------------- | ------------------------------------------------------------------ |
+| parser_visitor.go                  | Expand the SQL parsing layer to recognize aggregate functions      |
+| plan.proto                         | Define a new proto structure to represent the aggregation function |
+| internal/proxy/util.go             | Extend the processing logic of output_fields                       |
+| task_query.go                      | Add aggregate function support in createPlan                       |
+| internal/core/src/exec/aggregation | New aggregate function execution engine added to C++ layer         |
+| default_limit_reducer.go           | Process calculated fields during the result merging phase          |
+| Custom Operator Implementation     | Implement calculation logic such as ST_Area, ST_Length             |
@@ -0,0 +1,990 @@
+# Milvus Snapshot Design Document
+
+## Implementation Overview
+
+Milvus Snapshot mechanism provides complete collection-level data snapshot capabilities, implementing point-in-time backup and restore functionality. The implementation includes the following core components:
+
+### Architecture Components
+
+**1. Snapshot Storage Layer (S3/Object Storage)**
+- Stores complete snapshot data using Iceberg-like manifest format
+- Metadata files (JSON): snapshot basic information, schema, index definitions, and manifest file paths array
+- Manifest files (Avro): detailed segment descriptions and file paths
+
+**2. Snapshot Metadata Management (Etcd)**
+- Stores basic SnapshotInfo metadata
+- Maintains references from snapshots to segments/indexes
+- Provides fast query and list operations
+
+**3. Collection Meta Restoration**
+- Restores collection schema and index definitions from snapshot
+- Maintains Field ID consistency: ensures field IDs remain unchanged via PreserveFieldId
+- Maintains Index ID consistency: ensures index IDs remain unchanged via PreserveIndexId
+- ID consistency guarantees full compatibility between snapshot data files and new collection
+
+**4. Data Restore Mechanism (Copy Segment)**
+- Implements fast recovery through direct segment file copying
+- Manages copy operations using CopySegmentJob and CopySegmentTask
+- Supports parallel copying of multiple segments to improve restore speed
+
+**5. Garbage Collection Integration**
+- Protects segments referenced by snapshots from accidental deletion
+- Protects indexes referenced by snapshots to ensure restore availability
+- Automatically cleans up associated storage files when snapshot is deleted
+
+## Snapshot Creation Process
+
+### Creation Workflow
+
+**Execution Steps**:
+
+1. **Acquire Snapshot Timestamp**: Obtain the minimum timestamp from channel seek positions as snapshotTs
+2. **Filter Segments**: Select all non-dropped segments where StartPosition < snapshotTs
+3. **Collect Metadata**: Retrieve collection schema, index definitions, and segment details (binlog/deltalog/indexfile paths)
+4. **Write to S3**: Write complete metadata, schema, index, and segment information to S3 in manifest format
+5. **Write to Etcd**: Save basic SnapshotInfo to Etcd and establish segment/index references
+
+**Important Notes**:
+
+- **CreateSnapshot does not actively flush data**: Only collects existing sealed segments
+- **SnapshotTs Source**: Obtained from the minimum value of current consumption positions (seek positions) across all channels
+- **Data Coverage**: All segments where StartPosition < snapshotTs
+- **Best Practice (Strongly Recommended)**: Call Flush API before creating snapshot to ensure latest data is persisted. Flush is not mandatory but highly recommended to avoid missing data in growing segments
+
+**Data Point-in-Time**:
+
+- Snapshot contains all sealed segment data before snapshotTs
+- To include latest data, it is strongly recommended to call Flush API before creating snapshot (not mandatory)
+- Data in growing segments will not be included in the snapshot
+
+## Snapshot Storage Implementation
+
+### Storage Architecture
+
+Adopts layered storage architecture with separation of responsibilities:
+
+**Etcd Storage Layer** - Fast query and management
+- SnapshotInfo basic information (name, description, collection_id, create_ts, s3_location)
+- Used for fast list and query operations
+- Does not contain complete schema/segment/index detailed information
+
+**S3 Storage Layer** - Complete data persistence
+- CollectionDescription: complete schema definition
+- IndexInfo list: all index definitions and parameters
+- SegmentDescription list: all segment file paths (binlog/deltalog/statslog/indexfile)
+
+### S3 Storage Path Structure
+
+Data is organized using Iceberg-like manifest format with the following directory structure:
+
+```
+snapshots/{collection_id}/
+├── metadata/
+│   └── {snapshot_id}.json         # Snapshot metadata (JSON format)
+│
+└── manifests/
+    └── {snapshot_id}/             # Directory for each snapshot
+        ├── {segment_id_1}.avro    # Individual segment manifest (Avro format)
+        ├── {segment_id_2}.avro
+        └── ...
+```
+
+**Design Note**: Each segment has its own manifest file to enable:
+- Parallel reading of segment data
+- Incremental updates without rewriting large files
+- Better fault isolation (corrupted file affects only one segment)
+
+### File Format Details
+
+**1. metadata/{snapshot_id}.json (JSON format)**
+- SnapshotInfo: snapshot basic information
+- CollectionDescription: complete collection schema
+- IndexInfo list: all index definitions
+- ManifestList: string array containing paths to all manifest files
+- SegmentIDs/IndexIDs: pre-computed ID lists for fast reload
+- StorageV2ManifestList: segment-to-manifest mappings for StorageV2 format (optional)
+
+**Structure of metadata JSON:**
+```go
+type SnapshotMetadata struct {
+    SnapshotInfo          *datapb.SnapshotInfo          // Snapshot basic info
+    Collection            *datapb.CollectionDescription // Complete schema
+    Indexes               []*indexpb.IndexInfo          // Index definitions
+    ManifestList          []string                      // Array of manifest file paths
+    SegmentIDs            []int64                       // Pre-computed segment IDs for fast reload
+    IndexIDs              []int64                       // Pre-computed index IDs for fast reload
+    StorageV2ManifestList []*StorageV2SegmentManifest   // StorageV2 (Lance/Arrow) manifest mappings
+}
+
+type StorageV2SegmentManifest struct {
+    SegmentID int64  // Segment ID
+    Manifest  string // Path to StorageV2 manifest file
+}
+```
+
+**Fast Reload Optimization**: The `SegmentIDs` and `IndexIDs` fields enable DataCoord to quickly reload snapshot metadata during startup without parsing heavy Avro manifest files. Full segment data is loaded on-demand.
+
+**2. manifests/{snapshot_id}/{segment_id}.avro (Avro format)**
+- One file per segment for parallel I/O and fault isolation
+- Each file contains a single ManifestEntry with complete SegmentDescription:
+  - segment_id, partition_id, segment_level
+  - binlog_files, deltalog_files, statslog_files, bm25_statslog_files
+  - index_files, text_index_files, json_key_index_files
+  - num_of_rows, channel_name, storage_version
+  - start_position, dml_position
+  - is_sorted (whether segment data is sorted by primary key)
+- Uses Avro format to provide schema evolution support
+
+**StorageV2 Support**: For segments using Milvus's newer storage format, additional manifest paths are stored in `StorageV2ManifestList`. This enables seamless snapshot/restore for both legacy and new storage formats.
+
+### Write Process (SnapshotWriter)
+
+```
+1. Save() - Main entry point for writing snapshot
+   a. Create manifest directory: manifests/{snapshot_id}/
+   b. For each segment, call writeSegmentManifest():
+      - Convert SegmentDescription to Avro ManifestEntry
+      - Serialize to Avro binary format
+      - Write to S3: manifests/{snapshot_id}/{segment_id}.avro
+   c. Collect StorageV2 manifest paths (if applicable)
+   d. Call writeMetadataFile()
+
+2. writeSegmentManifest() - Write individual segment manifest
+   - Convert SegmentDescription to ManifestEntry (Avro struct)
+   - Include all binlog, deltalog, statslog, index files
+   - Include position info (start_position, dml_position)
+   - Serialize and write to: manifests/{snapshot_id}/{segment_id}.avro
+
+3. writeMetadataFile() - Write main metadata file
+   - Create SnapshotMetadata containing:
+     * SnapshotInfo
+     * Collection schema
+     * Index definitions
+     * ManifestList array (paths from step 1)
+     * SegmentIDs/IndexIDs (pre-computed for fast reload)
+     * StorageV2ManifestList (if applicable)
+   - Serialize to JSON format
+   - Write to S3: metadata/{snapshot_id}.json
+```
+
+### Read Process (SnapshotReader)
+
+```
+1. ReadSnapshot(metadataFilePath, includeSegments) - Read snapshot data
+   a. Read and parse metadata JSON file
+   b. If includeSegments=false:
+      - Return only SnapshotInfo, Collection, Indexes
+      - Use pre-computed SegmentIDs/IndexIDs for fast reload
+   c. If includeSegments=true:
+      - For each path in ManifestList array:
+        * Read Avro binary data
+        * Parse ManifestEntry records
+        * Convert to SegmentDescription
+      - Populate StorageV2 manifest paths if present
+   d. Return complete SnapshotData
+
+2. ListSnapshots(collectionID) - List all snapshots for a collection
+   - List all files in metadata/ directory
+   - Parse each metadata file to extract SnapshotInfo
+   - Return list of SnapshotInfo
+```
+
+**Performance Optimization**: The `includeSegments` parameter allows DataCoord to defer loading heavy segment data. During startup, only metadata is loaded (includeSegments=false), and full segment data is loaded on-demand when needed for restore operations.
+
+## Snapshot Data Management
+
+### SnapshotManager
+
+**SnapshotManager** centralizes all snapshot-related business logic, providing a unified interface for snapshot lifecycle management and restore operations.
+
+**Design Principles**:
+- Encapsulates business logic from RPC handlers
+- Manages dependencies through constructor injection
+- Eliminates code duplication (state conversion, progress calculation)
+- Maintains separation from background services (Checker/Inspector)
+
+**Core Interface**:
+```go
+type SnapshotManager interface {
+    // Snapshot lifecycle management
+    CreateSnapshot(ctx context.Context, collectionID int64, name, description string) (int64, error)
+    DropSnapshot(ctx context.Context, name string) error
+    DescribeSnapshot(ctx context.Context, name string) (*SnapshotData, error)
+    ListSnapshots(ctx context.Context, collectionID, partitionID int64) ([]string, error)
+
+    // Restore operations
+    RestoreSnapshot(ctx context.Context, snapshotName string, targetCollectionID int64) (int64, error)
+    GetRestoreState(ctx context.Context, jobID int64) (*datapb.RestoreSnapshotInfo, error)
+    ListRestoreJobs(ctx context.Context, collectionID int64) ([]*datapb.RestoreSnapshotInfo, error)
+}
+```
+
+
+### Snapshot Metadata Management
+
+**SnapshotMeta** manages snapshot reference relationships and lifecycle:
+
+```go
+type SnapshotDataInfo struct {
+    snapshotInfo *datapb.SnapshotInfo
+    SegmentIDs   typeutil.UniqueSet   // List of segments referenced by this snapshot
+    IndexIDs     typeutil.UniqueSet    // List of indexes referenced by this snapshot
+}
+
+type snapshotMeta struct {
+    catalog metastore.DataCoordCatalog
+    snapshotID2DataInfo *typeutil.ConcurrentMap[UniqueID, *SnapshotDataInfo]
+    reader *SnapshotReader
+    writer *SnapshotWriter
+}
+```
+
+**Core Functions**:
+
+**1. SaveSnapshot() - Two-Phase Commit (2PC)**
+
+SaveSnapshot uses a two-phase commit approach to ensure atomic creation and enable GC cleanup of orphan files:
+
+```
+Phase 1 (Prepare):
+  - Save snapshot with PENDING state to catalog (etcd)
+  - Snapshot ID is allocated and stored
+  - PendingStartTime is recorded for timeout tracking
+
+Write S3 Data:
+  - Write segment manifest files to S3
+  - Write metadata.json to S3
+  - Update S3Location in snapshot info
+
+Phase 2 (Commit):
+  - Update snapshot state to COMMITTED in catalog
+  - Insert into in-memory cache with precomputed ID sets
+```
+
+**Failure Handling**:
+| Failure Point | State | Recovery Action |
+|--------------|-------|-----------------|
+| Phase 1 fails | No changes | Return error, no cleanup needed |
+| S3 write fails | PENDING in catalog | GC will cleanup S3 files using snapshot ID |
+| Phase 2 fails | PENDING in catalog, S3 data exists | GC will cleanup S3 files and catalog record |
+
+**Key Design Points**:
+- Snapshot ID is used to compute S3 paths, no S3 list operations needed for cleanup
+- PENDING snapshots are filtered out during DataCoord reload
+- GC uses `PendingStartTime` + timeout to identify orphaned snapshots
+
+**2. DropSnapshot()**
+- Removes snapshot metadata from memory and Etcd
+- Calls SnapshotWriter.Drop() to clean up files on S3
+- Deletes metadata file and all manifest files referenced in ManifestList
+
+**3. GetSnapshotBySegment()/GetSnapshotByIndex()**
+- Queries which snapshots reference a specific segment or index
+- Used during GC to determine if resources can be deleted
+- Uses precomputed SegmentIDs/IndexIDs sets for O(1) lookup per snapshot
+
+**4. GetPendingSnapshots()**
+- Returns all PENDING snapshots that have exceeded the timeout
+- Used by GC to find orphaned snapshots for cleanup
+
+**5. CleanupPendingSnapshot()**
+- Removes a pending snapshot record from catalog
+- Called by GC after S3 files have been cleaned up
+
+### Garbage Collection Integration
+
+**Pending Snapshot GC (recyclePendingSnapshots)**:
+
+Cleans up orphaned snapshot files from failed 2PC commits:
+
+```
+Process flow:
+1. Get all PENDING snapshots from catalog that have exceeded timeout
+2. For each pending snapshot:
+   a. Compute manifest directory: snapshots/{collection_id}/manifests/{snapshot_id}/
+   b. Compute metadata file: snapshots/{collection_id}/metadata/{snapshot_id}.json
+   c. Delete manifest directory using RemoveWithPrefix (no S3 list needed)
+   d. Delete metadata file
+   e. Delete etcd record
+```
+
+**Key Design Points**:
+- NO S3 list operations: Uses RemoveWithPrefix for directory cleanup
+- File paths computed from collection_id + snapshot_id stored in etcd
+- Timeout mechanism (configurable via `SnapshotPendingTimeout`) prevents cleanup of snapshots still being created
+- Runs as part of DataCoord's regular GC cycle
+
+**Configuration**:
+```yaml
+dataCoord:
+  gc:
+    snapshotPendingTimeout: 10m  # Time before pending snapshot is considered orphaned
+```
+
+**Segment GC Protection**:
+
+Snapshot protection mechanism is integrated in DataCoord's garbage_collector:
+
+```go
+func (gc *garbageCollector) isSnapshotSegment(collectionID, segmentID int64) bool {
+    snapshotIDs := gc.meta.GetSnapshotMeta().GetSnapshotBySegment(ctx, collectionID, segmentID)
+    return len(snapshotIDs) > 0
+}
+```
+
+- In clearEtcd(), skips dropped segments referenced by snapshots
+- In clearS3(), skips segment files referenced by snapshots
+- GC is only allowed when segment is not referenced by any snapshot
+
+**Index GC Protection**:
+
+Index protection mechanism is integrated in IndexCoord:
+
+```go
+func (gc *garbageCollector) recycleUnusedIndexes() {
+    // Check if index is referenced by snapshots
+    snapshotIDs := gc.meta.GetSnapshotMeta().GetSnapshotByIndex(ctx, collectionID, indexID)
+    if len(snapshotIDs) > 0 {
+        // Skip index files that are still referenced by snapshots
+        continue
+    }
+}
+```
+
+- Prevents drop index from deleting index files referenced by snapshots
+- Ensures index data is available when restoring snapshots
+
+### Snapshot Lifecycle
+
+```
+Create:
+  User Request -> Proxy CreateSnapshot -> DataCoord
+    -> GetSnapshotTs() (obtain minimum timestamp from channels)
+    -> SelectSegments() (filter segments where StartPosition < snapshotTs)
+    -> Collect Schema/Indexes/Segment detailed information
+    -> SnapshotWriter.Save() (write to S3)
+    -> SnapshotMeta.SaveSnapshot() (write to Etcd + memory)
+
+Drop:
+  User Request -> Proxy DropSnapshot -> DataCoord
+    -> SnapshotMeta.DropSnapshot()
+    -> Remove from Etcd
+    -> SnapshotWriter.Drop() (delete S3 files)
+    -> Remove from memory
+
+List:
+  User Request -> Proxy ListSnapshots -> DataCoord
+    -> SnapshotMeta.ListSnapshots()
+    -> Filter by collection/partition
+
+Describe:
+  User Request -> Proxy DescribeSnapshot -> DataCoord
+    -> SnapshotMeta.GetSnapshot()
+    -> Return SnapshotInfo
+
+Restore:
+  User Request -> Proxy RestoreSnapshot -> RootCoord
+    -> DescribeSnapshot() from DataCoord (get schema, partitions, indexes)
+    -> CreateCollection() with PreserveFieldId=true
+    -> CreatePartition() for each user partition
+    -> RestoreSnapshotDataAndIndex() to DataCoord
+       -> Create indexes with PreserveIndexId=true
+       -> Create CopySegmentJob
+    -> Return job_id for async tracking
+```
+
+## Restore Snapshot Implementation
+
+### Overview
+
+Restore is implemented using the **Copy Segment mechanism**, which significantly improves recovery speed by directly copying segment files, avoiding the overhead of rewriting data and rebuilding indexes.
+
+### Architecture Overview
+
+The restore operation follows a layered architecture with clear separation of responsibilities:
+
+```
+User Request
+    │
+    ▼
+┌─────────┐
+│  Proxy  │  ← Entry point, request validation
+└────┬────┘
+     │
+     ▼
+┌───────────┐
+│ RootCoord │  ← Orchestration: CreateCollection, CreatePartition, coordinate DataCoord
+└─────┬─────┘
+      │
+      ▼
+┌───────────┐
+│ DataCoord │  ← Data restoration: CopySegmentJob creation, index creation
+└─────┬─────┘
+      │
+      ▼
+┌──────────┐
+│ DataNode │  ← Execution: Copy segment files from S3
+└──────────┘
+```
+
+### Restore Process
+
+#### Phase 1: Request Entry (Proxy Layer)
+
+```
+1. Proxy RestoreSnapshotTask.Execute():
+   - Validates request parameters
+   - Delegates to RootCoord.RestoreSnapshot()
+   - RootCoord orchestrates the entire restore process
+```
+
+**Design Notes**:
+- Proxy layer is simplified to just delegate to RootCoord
+- All orchestration logic is centralized in RootCoord for better transactional control
+
+#### Phase 2: Collection and Schema Recreation (RootCoord Layer)
+
+```
+2. RootCoord restoreSnapshotTask.Execute():
+   a. Get snapshot info from DataCoord
+      - DescribeSnapshot() retrieves complete snapshot information (schema, partitions, indexes)
+
+   b. Create collection with preserved field IDs
+      - CreateCollection() with PreserveFieldId=true
+      - Schema properties are completely copied (consistency level, num_shards, etc.)
+
+   c. Create user partitions
+      - CreatePartition() for each user-created partition in snapshot
+
+   d. Restore data and indexes via DataCoord
+      - RestoreSnapshotDataAndIndex() handles data copying and index creation
+```
+
+**Key Design Points**:
+- `PreserveFieldId=true`: Ensures new collection field IDs match those in snapshot
+- RootCoord handles rollback on failure (drops collection/partitions if restore fails)
+- Schema properties are completely copied, including consistency level, num_shards, etc.
+
+#### Phase 3: Data and Index Restoration (DataCoord Layer)
+
+```
+3. DataCoord RestoreSnapshotDataAndIndex():
+   a. Read snapshot data
+      - SnapshotMeta.ReadSnapshotData() reads complete segment information from S3
+
+   b. Generate mappings
+      - Channel Mapping: Map snapshot channels to new collection channels
+      - Partition Mapping: Map snapshot partition IDs to new partition IDs
+
+   c. Create indexes (if requested)
+      - Create indexes with PreserveIndexId=true
+      - Ensures index IDs match, allowing direct use of index files from snapshot
+
+   d. Pre-register Target Segments
+      - Allocate new segment IDs
+      - Create SegmentInfo (with correct PartitionID and InsertChannel)
+      - Pre-register to meta via meta.AddSegment(), State set to Importing
+
+   e. Create Copy Segment Job
+      - Create lightweight ID mappings (source_segment_id -> target_segment_id)
+      - Generate CopySegmentJob and save to meta
+      - CopySegmentChecker automatically creates CopySegmentTasks
+```
+
+**ID Mapping Structure**:
+```protobuf
+message CopySegmentIDMapping {
+    int64 source_segment_id = 1;  // Segment ID in snapshot
+    int64 target_segment_id = 2;  // Segment ID in new collection
+    int64 partition_id = 3;       // Target partition ID (cached for grouping)
+}
+```
+
+**Design Notes**:
+
+- **Lightweight Design**: IDMapping only stores segment ID mappings (~48 bytes/segment), used to guide subsequent file copy operations
+- **Actual Mapping Storage**: Real channel and partition mapping relationships are stored in SegmentInfo in meta
+  - When creating job, target segments are pre-registered via `meta.AddSegment()`
+  - Each target segment contains complete metadata: `PartitionID`, `InsertChannel`, `State`, etc.
+  - Copy task reads segment information from meta during execution, no need to duplicate storage in IDMapping
+- **Mapping Application Flow**: Calculate channel/partition mappings during job creation → Create target segments and write to meta → IDMapping only retains ID references
+
+#### Phase 4: Copy Segment Execution (DataNode Layer)
+
+```
+4. CopySegmentChecker monitors job execution:
+   
+   Pending -> Executing:
+   - Group segments by maxSegmentsPerCopyTask (currently default 10/group)
+   - Create one CopySegmentTask for each group
+   - Distribute tasks to DataNodes for execution
+   
+   Executing:
+   - DataNode CopySegmentTask executes:
+     a. Read all file paths of source segment
+     b. Copy files to new paths (update segment_id/partition_id/channel_name)
+     c. Update segment's binlog/deltalog/indexfile information in meta
+   - CopySegmentChecker monitors progress of all tasks
+   
+   Executing -> Completed:
+   - After all tasks complete
+   - Update all target segments status to Flushed
+   - Job marked as Completed
+```
+
+**Task Granularity Concurrency Design**:
+
+- **Current Implementation**: Supports task-level concurrency based on segment grouping
+  - Create tasks grouped by maxSegmentsPerCopyTask (default 10 segments)
+  - Each group generates independent CopySegmentTask, which can be distributed to different DataNodes for parallel execution
+  
+  - **Long-term Optimization**: If copy performance is insufficient, implement concurrency acceleration at task level
+    - **Adjust Grouping Granularity**: Reduce maxSegmentsPerCopyTask value to increase task count
+      - Example: Reduce from 10 segments/task to 5 or fewer
+      - More tasks can be scheduled in parallel to multiple DataNodes
+    - **File-level Concurrency**: Parallelize copying of multiple files within a single task
+      - Parallel copying of binlog/deltalog/indexfile
+      - Need to coordinate concurrency count to avoid object storage throttling
+
+**Copy Segment Details**:
+
+File copy operations on DataNode:
+```go
+// For each segment in the task
+1. Copy Binlog files
+   - Read all binlog files of source segment
+   - Update segment_id/partition_id/channel in file paths
+   - Copy to new paths
+
+2. Copy Deltalog files
+   - Process delete logs
+   - Update paths and copy
+
+3. Copy Statslog files
+   - Process statistics files
+   - Update paths and copy
+
+4. Copy Index files
+   - Copy vector index, scalar index
+   - Keep index ID unchanged (because PreserveIndexId=true)
+
+5. Update Segment metadata
+   - Create new SegmentInfo with target_segment_id
+   - Preserve num_of_rows, storage_version, and other information
+   - State set to Importing (will be updated to Flushed later)
+```
+
+### Copy Segment Task Management
+
+**CopySegmentJob**:
+```protobuf
+message CopySegmentJob {
+    int64 job_id = 1;
+    int64 db_id = 2;
+    int64 collection_id = 3;
+    string collection_name = 4;
+    CopySegmentJobState state = 5;
+    string reason = 6;
+    repeated CopySegmentIDMapping id_mappings = 7;  // Lightweight ID mapping (~48 bytes per segment)
+    uint64 timeout_ts = 8;
+    uint64 cleanup_ts = 9;
+    string start_time = 10;
+    string complete_time = 11;
+    repeated common.KeyValuePair options = 12;       // Option configuration (e.g. copy_index)
+    int64 total_segments = 13;
+    int64 copied_segments = 14;
+    int64 total_rows = 15;
+    string snapshot_name = 16;                       // For restore snapshot scenario
+}
+```
+
+**CopySegmentTask**:
+```protobuf
+message CopySegmentTask {
+    int64 task_id = 1;
+    int64 job_id = 2;
+    int64 collection_id = 3;
+    int64 node_id = 4;                              // DataNode ID executing the task
+    int64 task_version = 5;
+    int64 task_slot = 6;
+    ImportTaskStateV2 state = 7;
+    string reason = 8;
+    repeated CopySegmentIDMapping id_mappings = 9;  // Segments this task is responsible for
+    string created_time = 10;
+    string complete_time = 11;
+}
+```
+
+**Task Scheduling**:
+- CopySegmentInspector: Assigns pending tasks to DataNodes
+- CopySegmentChecker: Monitors job and task status, handles failures
+- Supports task-level retry (max 3 retries)
+- Supports parallel execution of multiple tasks
+
+### Task Retry and Failure Cleanup Mechanisms
+
+#### Retry Mechanism
+
+**DataCoord Layer Automatic Retry**:
+- **Query Failure Retry**: When querying task status on DataNode fails, automatically reset task to `Pending` state and wait for rescheduling
+- **DataNode Return Retry**: When DataNode explicitly returns `Retry` status, reset task to `Pending` state
+
+**Retry Configuration**:
+- `max_retries`: Default 3 times
+- `retry_count`: Records current retry count
+
+**Retry Flow**:
+1. When task encounters error, status is reset to `Pending`
+2. `CopySegmentInspector` periodically scans tasks in `Pending` state
+3. Re-enqueue task to scheduler for scheduling
+4. Implements implicit retry: through state transition rather than explicit retry counting
+
+**DataNode Layer Failure Strategy**:
+- Adopts fail-fast strategy, does not implement local retry
+- Any file copy error immediately reports `Failed` status
+- DataCoord layer makes unified retry decision
+
+#### Failure Cleanup Mechanism
+
+**1. Task-level Cleanup (Inspector)**:
+- When task fails, `CopySegmentInspector` automatically iterates through all ID mappings of the task
+- Finds corresponding target segments in meta and marks status as `Dropped`
+- Prevents invalid segment data from remaining in meta
+
+**2. Job-level Cleanup (Checker)**:
+- When job fails, `CopySegmentChecker` finds all tasks in `Pending` and `InProgress` states for that job
+- Cascades update of all related tasks to `Failed` status and synchronizes failure reason
+- Ensures Job and Task state consistency
+
+**3. Timeout Mechanism**:
+- `CopySegmentChecker` periodically checks job's `timeout_ts`
+- When current time exceeds timeout, automatically marks job as `Failed`
+- Prevents job from being stuck for long periods, triggers subsequent cleanup process
+
+**4. GC Mechanism**:
+- `CopySegmentChecker` regularly checks jobs in `Completed` or `Failed` state
+- When `cleanup_ts` time is reached, sequentially deletes tasks and job metadata
+- Cleanup order: Remove all tasks first → then remove job
+
+**GC Protection Strategy**:
+- Only cleans up jobs in `Completed` or `Failed` state
+- GC is executed only after reaching `CleanupTs` (based on `ImportTaskRetention` configuration)
+- **Protection Mechanisms**:
+  - If target segments of failed job still exist in meta → Delay cleanup (requires manual intervention)
+  - If task is still running on DataNode → Delay cleanup (wait for node release)
+- **Cleanup Order**: Delete tasks first → then delete job
+
+#### Error Recovery Summary
+
+| Mechanism | Trigger Condition | Action | Purpose |
+|------|---------|---------|------|
+| **Auto Retry** | Query failure, DataNode returns Retry | Reset to Pending, reschedule | Automatically recover temporary errors |
+| **Task Cleanup** | Task marked as Failed | Delete target segments | Clean up invalid meta data |
+| **Job Cleanup** | Job marked as Failed | Cascade mark all tasks | Ensure state consistency |
+| **Timeout Protection** | Exceeds TimeoutTs | Mark job as Failed | Prevent long-term resource occupation |
+| **GC Collection** | Reaches CleanupTs | Delete job/task metadata | Prevent meta bloat |
+
+### Restore State Tracking
+
+**State Machine**:
+```
+RestoreSnapshotPending
+  ↓
+RestoreSnapshotExecuting (copying segments)
+  ↓
+RestoreSnapshotCompleted / RestoreSnapshotFailed
+```
+
+**Progress Calculation**:
+```
+Progress = (copied_segments / total_segments) * 100
+```
+
+**Job Monitoring APIs**:
+- GetRestoreSnapshotState(job_id): Query single restore job status
+- ListRestoreSnapshotJobs(): List all restore jobs
+
+### Performance Characteristics
+
+**Advantages**:
+1. **Fast Recovery**: Direct file copying avoids data rewriting and index rebuilding
+2. **Parallel Execution**: Multiple segments copied in parallel, fully utilizing I/O bandwidth
+3. **Incremental Progress**: Supports real-time progress queries, good user experience
+4. **Resume from Breakpoint**: Task-level retry mechanism, partial failure does not affect overall progress
+
+**Comparison with Bulk Insert**:
+- Bulk Insert: Requires data rewriting and index building, time-consuming
+- Copy Segment: Direct file copying, 10-100x faster (depending on data volume)
+
+### Error Handling
+
+**1. Collection Creation Failure**:
+- Automatic rollback, delete created collection
+
+**2. Copy Task Failure**:
+- Automatic retry mechanism: Temporary errors automatically reset to Pending state, rescheduled
+- Failure cleanup mechanism: Automatically clean up target segments of failed task (marked as Dropped)
+- Detailed mechanism see [Task Retry and Failure Cleanup Mechanisms](#task-retry-and-failure-cleanup-mechanisms)
+
+**3. Job Timeout and GC**:
+- Timeout protection: Automatically mark job as Failed when exceeding timeout_ts
+- GC collection: Automatically clean up job/task metadata after reaching cleanup_ts
+- Protection strategy: Delay GC before segments of failed job are cleaned up, prevent data loss
+
+### Limitations and Considerations
+
+**1. Channel/Partition Count Must Match**:
+- New collection's shard count must match snapshot
+- Partition count must match (auto-created partitions + user-created partitions)
+
+**2. Field ID and Index ID Preservation**:
+- Ensure compatibility through PreserveFieldId and PreserveIndexId
+- Must be correctly set during CreateCollection
+
+**3. TTL Handling**:
+- Current implementation does not handle collection TTL
+- Restored historical data may conflict with TTL mechanism
+- Recommend disabling TTL or adjusting TTL time during snapshot restore
+
+## Data Access Patterns
+
+### Read on Milvus
+
+Reading snapshot data through Milvus service:
+
+1. Restore data from snapshot to new collection using RestoreSnapshot feature
+2. Execute all Milvus-supported query operations on restored collection
+3. New collection is no different from regular collection, supports CRUD operations
+
+### Read without Milvus
+
+Offline access through snapshot data on S3:
+
+**Core Design Goals**:
+- Snapshot data on S3 achieves self-describing state
+- Third-party tools can directly read snapshot data
+- Independent of Milvus service operation
+
+**Technical Implementation**:
+- Uses Iceberg-like manifest format
+- Metadata.json contains complete schema definition
+- Manifest files contain all data file paths
+- Third-party tools can parse Avro format manifests to obtain data locations
+
+## API Reference
+
+### CreateSnapshot
+
+**Function**: Create snapshot for specified collection, automatically export data to object storage
+
+**Request**:
+```protobuf
+message CreateSnapshotRequest {
+  common.MsgBase base = 1;
+  string db_name = 2;               // database name
+  string collection_name = 3;       // collection name
+  string name = 4;                  // user-defined snapshot name (unique)
+  string description = 5;           // user-defined snapshot description
+}
+```
+
+**Response**:
+```protobuf
+message CreateSnapshotResponse {
+  common.Status status = 1;
+}
+```
+
+**Implementation Details**:
+1. Obtain minimum timestamp from current channels as snapshotTs
+2. Filter sealed segments where StartPosition < snapshotTs (exclude dropped/importing segments)
+3. Collect collection schema, index definitions, segment detailed information
+4. Write complete data to S3 in manifest format
+5. Save SnapshotInfo in Etcd and establish reference relationships
+
+**Notes**:
+- Does not actively flush data, only collects existing sealed segments
+- **Best Practice (Strongly Recommended)**: Call Flush API before creating snapshot to ensure latest data is persisted. Flush is not mandatory but highly recommended
+- Creation will fail if collection has no sealed segments
+
+### DropSnapshot
+
+**Function**: Delete specified snapshot and all its files on S3
+
+**Request**:
+```protobuf
+message DropSnapshotRequest {
+  common.MsgBase base = 1;
+  string name = 2;                  // snapshot name to drop
+}
+```
+
+**Response**:
+```protobuf
+message DropSnapshotResponse {
+  common.Status status = 1;
+}
+```
+
+**Implementation Details**:
+1. Delete SnapshotInfo from Etcd
+2. Delete metadata file and all manifest files referenced in ManifestList from S3
+3. Remove reference relationships from memory
+
+### ListSnapshots
+
+**Function**: List all snapshots for specified collection
+
+**Request**:
+```protobuf
+message ListSnapshotsRequest {
+  common.MsgBase base = 1;
+  string db_name = 2;               // database name
+  string collection_name = 3;       // collection name
+}
+```
+
+**Response**:
+```protobuf
+message ListSnapshotsResponse {
+  common.Status status = 1;
+  repeated string snapshots = 2;    // list of snapshot names
+}
+```
+
+### DescribeSnapshot
+
+**Function**: Get detailed information about snapshot
+
+**Request**:
+```protobuf
+message DescribeSnapshotRequest {
+  common.MsgBase base = 1;
+  string name = 2;                  // snapshot name
+}
+```
+
+**Response**:
+```protobuf
+message DescribeSnapshotResponse {
+  common.Status status = 1;
+  string name = 2;
+  string description = 3;
+  int64 create_ts = 4;
+  string collection_name = 5;
+  repeated string partition_names = 6;
+}
+```
+
+**Return Information**:
+- Snapshot basic information (name, description, create_ts)
+- Collection and partition names
+- S3 storage location
+
+### RestoreSnapshot
+
+**Function**: Restore data from snapshot to new collection
+
+**Request** (milvuspb - User-facing API):
+```protobuf
+message RestoreSnapshotRequest {
+  common.MsgBase base = 1;
+  string name = 2;                  // snapshot name to restore
+  string db_name = 3;               // target database name
+  string collection_name = 4;       // target collection name (must not exist)
+}
+```
+
+**Response**:
+```protobuf
+message RestoreSnapshotResponse {
+  common.Status status = 1;
+  int64 job_id = 2;                 // restore job ID for progress tracking
+}
+```
+
+**Internal API** (datapb - DataCoord API):
+```protobuf
+// RestoreSnapshotDataAndIndex - Called by RootCoord after collection/partition creation
+message RestoreSnapshotRequest {
+  common.MsgBase base = 1;
+  string name = 2;                  // snapshot name
+  int64 collection_id = 3;          // target collection ID (already created by RootCoord)
+  bool create_indexes = 4;          // whether to create indexes during restore
+}
+```
+
+**Implementation Flow**:
+1. **Proxy**: Validates request and delegates to RootCoord
+2. **RootCoord** (orchestration):
+   - Get snapshot info from DataCoord (schema, partitions, indexes)
+   - Create new collection with PreserveFieldId=true
+   - Create user partitions
+   - Call DataCoord.RestoreSnapshotDataAndIndex()
+   - Handle rollback on failure
+3. **DataCoord** (data restoration):
+   - Create indexes with PreserveIndexId=true (if requested)
+   - Create CopySegmentJob for data recovery
+4. Return job_id for progress tracking
+
+**Limitations**:
+- Target collection must not exist
+- New collection's shard/partition count will match snapshot
+- Does not automatically handle TTL-related issues
+
+### GetRestoreSnapshotState
+
+**Function**: Query restore job execution status
+
+**Request**:
+```protobuf
+message GetRestoreSnapshotStateRequest {
+  common.MsgBase base = 1;
+  int64 job_id = 2;                 // restore job ID
+}
+```
+
+**Response**:
+```protobuf
+message GetRestoreSnapshotStateResponse {
+  common.Status status = 1;
+  RestoreSnapshotInfo info = 2;
+}
+
+message RestoreSnapshotInfo {
+  int64 job_id = 1;
+  string snapshot_name = 2;
+  string db_name = 3;
+  string collection_name = 4;
+  RestoreSnapshotState state = 5;   // Pending/Executing/Completed/Failed
+  int32 progress = 6;               // 0-100
+  string reason = 7;                // error reason if failed
+  uint64 time_cost = 8;             // milliseconds
+}
+```
+
+**State Values**:
+- RestoreSnapshotPending: Waiting for execution
+- RestoreSnapshotExecuting: Currently copying segments
+- RestoreSnapshotCompleted: Restore successful
+- RestoreSnapshotFailed: Restore failed
+
+### ListRestoreSnapshotJobs
+
+**Function**: List all restore jobs (optionally filter by collection)
+
+**Request**:
+```protobuf
+message ListRestoreSnapshotJobsRequest {
+  common.MsgBase base = 1;
+  string collection_name = 2;       // optional: filter by collection
+}
+```
+
+**Response**:
+```protobuf
+message ListRestoreSnapshotJobsResponse {
+  common.Status status = 1;
+  repeated RestoreSnapshotInfo jobs = 2;
+}
+```
@@ -0,0 +1,297 @@
+# Entity-level TTL Design
+
+## Background
+
+Currently, Milvus supports **collection-level TTL** for data expiration, but does not support defining an independent expiration time for individual entities (rows). As application scenarios become more diverse, for example:
+
+* Data from different tenants or businesses stored in the same collection but with different lifecycles;
+* Hot and cold data mixed together, where short-lived data should be cleaned automatically while long-term data is retained;
+* IoT / logging / MLOps data that requires record-level retention policies;
+
+Relying solely on collection-level TTL can no longer satisfy these requirements. If users want to retain only part of the data, they must periodically perform **upsert** operations to refresh the timestamps of those entities. This approach is unintuitive and increases operational and maintenance costs.
+
+Therefore, **Entity-level TTL** becomes a necessary feature.
+
+Related issues:
+
+* [milvus-io/milvus#45917](https://github.com/milvus-io/milvus/issues/45917)
+* [milvus-io/milvus#45923](https://github.com/milvus-io/milvus/issues/45923)
+
+---
+
+## Design Principles
+
+* Fully compatible with existing collection-level TTL behavior.
+* Allow users to choose whether to enable entity-level TTL.
+* User-controllable: support explicit declaration in schema or transparent system management.
+* Minimize changes to compaction and query logic; expiration is determined only by the TTL column and write timestamp.
+* Support dynamic upgrade for existing collections.
+
+---
+
+## Basic Approach
+
+Milvus already supports the `TIMESTAMPTZ` data type. Entity TTL information will therefore be stored in a field of this type.
+
+---
+
+## Design Details
+
+Entity-level TTL is implemented by allowing users to explicitly add a `TIMESTAMPTZ` column in the schema and mark it in collection properties:
+
+```text
+"ttl_field": "ttl"
+```
+
+Here, `ttl` is the name of the column that stores TTL information. This mechanism is **mutually exclusive** with collection-level TTL.
+
+---
+
+### Terminology and Conventions
+
+* **ExpireAt** : The value stored in the TTL field, representing the absolute expiration timestamp of an entity (UTC by default if no timezone is specified).
+* **Collection-level TTL** : The existing mechanism where retention duration is defined at the collection level (e.g., retain 30 days).
+* **insert_ts / mvcc_ts** : Existing Milvus write or MVCC timestamps, used as fallback when needed.
+* **expirQuantiles** : A time point corresponding to a certain percentile of expired data within a segment, used to quickly determine whether compaction should be triggered.
+
+Example:
+
+* 20% of data expires at time `t1`
+* 40% of data expires at time `t2`
+
+---
+
+### 1. Collection Properties and Constraints
+
+* Only fields with `DataType == TIMESTAMPTZ` can be configured as a TTL field.
+* Mutually exclusive with collection-level TTL:
+  * If collection-level TTL is enabled, specifying a TTL field is not allowed.
+  * Collection-level TTL must be disabled first.
+* One TTL field per collection:
+  * A collection may contain multiple `TIMESTAMPTZ` fields, but only one can be designated as the TTL field.
+
+---
+
+### 2. Storage Semantics
+
+* Unified convention: the TTL field stores an **absolute expiration time** (`ExpireAt`).
+* Duration-based TTL is not supported.
+* `NULL` value semantics:
+  * A `NULL` TTL value means the entity never expires.
+
+---
+
+### 3. Compatibility Rules
+
+#### Existing Collections
+
+For an existing collection to enable entity-level TTL:
+
+1. Disable collection-level TTL using `AlterCollection`.
+2. Add a new `TIMESTAMPTZ` field using `AddField`.
+3. Update collection properties via `AlterCollection` to mark the new field as the TTL field.
+
+If historical data should also have expiration times, users must perform an **upsert** operation to backfill the TTL field.
+
+---
+
+### 4. SegmentInfo Extension and Compaction Trigger
+
+#### 4.1 SegmentInfo Metadata Extension
+
+A new field `expirQuantiles` is added to the segment metadata:
+
+```proto
+message SegmentInfo {
+  int64 ID = 1;
+  int64 collectionID = 2;
+  int64 partitionID = 3;
+  string insert_channel = 4;
+  int64 num_of_rows = 5;
+  common.SegmentState state = 6;
+  int64 max_row_num = 7 [deprecated = true]; // deprecated, we use the binary size to control the segment size but not a estimate rows.
+  uint64 last_expire_time = 8;
+  msg.MsgPosition start_position = 9;
+  msg.MsgPosition dml_position = 10;
+  // binlogs consist of insert binlogs
+  repeated FieldBinlog binlogs = 11;
+  repeated FieldBinlog statslogs = 12;
+  // deltalogs consists of delete binlogs. FieldID is not used yet since delete is always applied on primary key
+  repeated FieldBinlog deltalogs = 13;
+  bool createdByCompaction = 14;
+  repeated int64 compactionFrom = 15;
+  uint64 dropped_at = 16; // timestamp when segment marked drop
+  // A flag indicating if:
+  // (1) this segment is created by bulk insert, and
+  // (2) the bulk insert task that creates this segment has not yet reached `ImportCompleted` state.
+  bool is_importing = 17;
+  bool is_fake = 18;
+
+  // denote if this segment is compacted to other segment.
+  // For compatibility reasons, this flag of an old compacted segment may still be False.
+  // As for new fields added in the message, they will be populated with their respective field types' default values.
+  bool compacted = 19;
+
+  // Segment level, indicating compaction segment level
+  // Available value: Legacy, L0, L1, L2
+  // For legacy level, it represent old segment before segment level introduced
+  // so segments with Legacy level shall be treated as L1 segment
+  SegmentLevel level = 20;
+  int64 storage_version = 21;
+
+  int64 partition_stats_version = 22;
+  // use in major compaction, if compaction fail, should revert segment level to last value
+  SegmentLevel last_level = 23;
+  // use in major compaction, if compaction fail, should revert partition stats version to last value 
+  int64 last_partition_stats_version = 24;
+
+  // used to indicate whether the segment is sorted by primary key.
+  bool is_sorted = 25;
+
+  // textStatsLogs is used to record tokenization index for fields.
+  map<int64, TextIndexStats> textStatsLogs = 26;
+  repeated FieldBinlog bm25statslogs = 27;
+
+  // This field is used to indicate that some intermediate state segments should not be loaded.
+  // For example, segments that have been clustered but haven't undergone stats yet.
+  bool is_invisible = 28;
+
+
+  // jsonKeyStats is used to record json key index for fields.
+  map<int64, JsonKeyStats> jsonKeyStats = 29;
+    // This field is used to indicate that the segment is created by streaming service.
+  // This field is meaningful only when the segment state is growing.
+  // If the segment is created by streaming service, it will be a true.
+  // A segment generated by datacoord of old arch, will be false.
+  // After the growing segment is full managed by streamingnode, the true value can never be seen at coordinator.
+  bool is_created_by_streaming = 30;
+  bool is_partition_key_sorted = 31;
+
+  // manifest_path stores the fullpath of LOON manifest file of segemnt data files.
+  // we could keep the fullpath since one segment shall only have one active manifest
+  // and we could keep the possiblity that manifest stores out side of collection/partition/segment path
+  string manifest_path = 32;
+
+  // expirQuantiles records the expiration timestamps of the segment
+  // at the 20%, 40%, 60%, 80%, and 100% data distribution levels
+  repeated int64 expirQuantiles = 33;
+}
+```
+
+Meaning:
+
+* `expirQuantiles`: The expiration timestamps corresponding to the 20%, 40%, 60%, 80%, and 100% percentiles of data within the segment.
+
+---
+
+#### 4.2 Metadata Writing
+
+* Statistics are collected  **only during compaction** .
+* `expirQuantiles` is computed during sort or mix compaction tasks.
+* For streaming segments, sort compaction is required as the first step, making this approach sufficient.
+
+---
+
+#### 4.3 Compaction Trigger Strategy
+
+* Based on a configured expired-data ratio, select the corresponding percentile from `expirQuantiles` (rounded down).
+* Compare the selected expiration time with the current time.
+* If the expiration condition is met, trigger a compaction task.
+
+Special cases:
+
+* If `expirQuantiles` is `NULL`, the segment is treated as non-expiring.
+* For old segments without a TTL field, expiration logic is skipped.
+* Subsequent upsert operations will trigger the corresponding L0 compaction.
+
+---
+
+### 5. Query / Search Logic
+
+* Each query is executed with an MVCC timestamp assigned by Milvus.
+* When loading a collection, the system records which field is configured as the TTL field.
+
+During query execution, expired data is filtered by comparing the TTL field value with the MVCC timestamp inside `mask_with_timestamps`.
+
+---
+
+## PyMilvus Example
+
+### 1. Create Collection
+
+Specify the TTL field in the schema:
+
+```python
+schema = client.create_schema(auto_id=False, description="test entity ttl")
+schema.add_field("id", DataType.INT64, is_primary=True)
+schema.add_field("ttl", DataType.TIMESTAMPTZ, nullable=True)
+schema.add_field("vector", DataType.FLOAT_VECTOR, dim=dim)
+
+prop = {"ttl_field": "ttl"}
+client.create_collection(
+    collection_name,
+    schema=schema,
+    enable_dynamic_field=True,
+    properties=prop,
+)
+```
+
+---
+
+### 2. Insert Data
+
+Insert data the same way as a normal `TIMESTAMPTZ` field:
+
+```python
+rows = [
+    {"id": 0, "vector": [random.random() for _ in range(dim)], "ttl": None},
+    {"id": 1, "vector": [random.random() for _ in range(dim)], "ttl": None},
+    {"id": 2, "vector": [random.random() for _ in range(dim)], "ttl": None},
+    {"id": 3, "vector": [random.random() for _ in range(dim)], "ttl": "2025-12-31T00:00:00Z"},
+    {"id": 4, "vector": [random.random() for _ in range(dim)], "ttl": "2025-12-31T01:00:00Z"},
+    {"id": 5, "vector": [random.random() for _ in range(dim)], "ttl": "2025-12-31T02:00:00Z"},
+    {"id": 6, "vector": [random.random() for _ in range(dim)], "ttl": "2025-12-31T03:00:00Z"},
+    {"id": 7, "vector": [random.random() for _ in range(dim)], "ttl": "2025-12-31T04:00:00Z"},
+    {"id": 8, "vector": [random.random() for _ in range(dim)], "ttl": "2025-12-31T23:59:59Z"},
+]
+
+insert_result = client.insert(collection_name, rows)
+client.flush(collection_name)
+```
+
+---
+
+### 3. Index and Load
+
+Index creation and loading are unaffected. Indexes can still be built on the TTL field if needed.
+
+```python-repl
+index_params = client.prepare_index_params()
+index_params.add_index(
+    field_name="vector",
+    index_type="IVF_FLAT",
+    index_name="vector",
+    metric_type="L2",
+    params={"nlist": 128},
+)
+client.create_index(collection_name, index_params=index_params)
+
+client.load_collection(collection_name)
+```
+
+---
+
+### 4. Search / Query
+
+Use queries with different timestamps to validate expiration behavior:
+
+```python
+query_expr = "id > 0"
+res = client.query(
+    collection_name,
+    query_expr,
+    output_fields=["id", "ttl"],
+    limit=100,
+)
+print(res)
+```
@@ -0,0 +1,385 @@
+# Milvus Search Order By Feature Design Document
+
+## Overview
+
+The Order By feature enables users to sort vector search results by scalar fields instead of (or in addition to) the default distance/similarity score ordering. This is useful for scenarios where users want to prioritize results based on business-relevant attributes like price, timestamp, rating, etc.
+
+## Motivation
+
+Vector search results are typically sorted by similarity score (distance). However, users often need to:
+1. Sort results by a business-relevant field (e.g., "sort by price ascending")
+2. Apply secondary sorting criteria after the primary vector similarity ranking
+3. Re-order grouped results (from group_by searches) based on scalar attributes
+
+## Feature Scope
+
+### Supported Operations
+- **Common Search**: Single vector search with `order_by_fields`
+- **Hybrid Search**: Multi-vector search with `order_by_fields` in main search params
+- **Search with Group By**: Order by applies to groups (sorts groups by first row's value)
+
+**Cross-link**: Query ORDER BY is documented separately in:
+- `design_docs/20260203-query-orderby.md`
+
+### Unsupported Combinations
+- **Search Iterator**: `order_by` is not supported when using search iterators
+- **Function Score + Order By**: Cannot use both simultaneously (conflicting sort criteria)
+
+## API Design
+
+### User Interface
+
+Users specify order by fields via the `order_by_fields` search parameter:
+
+```python
+# PyMilvus Example
+collection.search(
+    data=[[0.1, 0.2, 0.3, ...]],
+    anns_field="embedding",
+    param={"metric_type": "L2", "params": {"nprobe": 10}},
+    limit=10,
+    order_by_fields=[
+        {"field": "price", "order": "asc"}
+    ]
+)
+```
+
+### Syntax
+
+```python
+order_by_fields = [
+    {
+        "field": "field_name",           # Required: field name or JSON path
+        "order": "asc" | "desc"          # Optional: default is "asc"
+    },
+    # ... additional fields for multi-field sorting
+]
+```
+
+**Field specification formats:**
+- Simple field: `{"field": "price", "order": "asc"}`
+- Dynamic field: `{"field": "age", "order": "desc"}` (automatically resolved from dynamic field storage)
+- Dynamic field with path: `{"field": "metadata[\"price\"]", "order": "asc"}` (nested access in dynamic field)
+- Nested dynamic field path: `{"field": "user[\"profile\"][\"score\"]", "order": "desc"}`
+
+**Note**: JSON path is only supported for dynamic fields. Regular JSON fields (defined in schema) cannot be used for ordering.
+
+**Order options:**
+- `"asc"` or `"ascending"`: Sort in ascending order (default)
+- `"desc"` or `"descending"`: Sort in descending order
+
+### Supported Field Types
+
+| Field Type | Supported | Notes |
+|------------|-----------|-------|
+| Bool | Yes | false < true |
+| Int8/Int16/Int32/Int64 | Yes | Numeric comparison |
+| Float/Double | Yes | Numeric comparison (NaN rejected at insert) |
+| String/VarChar | Yes | Lexicographic comparison |
+| JSON | No | Comparing JSON values on bytes is meaningless |
+| Dynamic field subpath | Yes | Access via JSON path (e.g., `age` or `metadata["price"]`) |
+| Array | No | Not sortable |
+| Vector types | No | Not sortable |
+
+### Dynamic Field Support
+
+Dynamic fields are fully supported. Access them directly by field name:
+
+```python
+# Access dynamic field directly
+order_by_fields=[
+    {"field": "age", "order": "desc"}  # age is a dynamic field
+]
+
+# Multiple dynamic fields work the same as scalar fields
+order_by_fields=[
+    {"field": "rating", "order": "desc"},   # dynamic field
+    {"field": "timestamp", "order": "asc"}  # mixing with schema-defined scalar fields
+]
+```
+
+**Note**: Dynamic fields are automatically resolved. If a field name is not found in the schema, Milvus will look for it in the dynamic field storage.
+
+### JSON Path Support (Dynamic Fields Only)
+
+JSON path is only supported for dynamic fields, not for regular JSON fields. This is because Milvus currently only supports JSON path expressions for dynamic field storage. Ordering by a regular JSON field is not supported as comparing raw JSON bytes is meaningless.
+
+Dynamic field subpaths can be accessed using bracket notation:
+
+```python
+# Access dynamic field subpath
+order_by_fields=[
+    {"field": "metadata[\"price\"]", "order": "asc"}  # metadata is a dynamic field key
+]
+
+# Nested path in dynamic field
+order_by_fields=[
+    {"field": "user[\"profile\"][\"score\"]", "order": "desc"}  # user is a dynamic field key
+]
+
+# Multiple order by fields (mixing simple and nested dynamic fields)
+order_by_fields=[
+    {"field": "price", "order": "asc"},              # simple dynamic field
+    {"field": "timestamp", "order": "desc"}          # another dynamic field
+]
+```
+
+**Note**: If you have a regular JSON field (defined in schema with JSON type), you cannot use it for ordering. Only dynamic fields support JSON path access for ordering.
+
+## Architecture
+
+### Data Flow
+
+```
+┌─────────────────────────────────────────────────────────────────────────────┐
+│                              Proxy (Search Task)                             │
+├─────────────────────────────────────────────────────────────────────────────┤
+│  1. Parse order_by_fields from search params                                 │
+│  2. Validate field names against schema                                      │
+│  3. Add order_by fields to output_fields for requery                        │
+│  4. Execute search on QueryNodes                                            │
+│  5. Run search pipeline with order_by operator                              │
+└─────────────────────────────────────────────────────────────────────────────┘
+                                      │
+                                      ▼
+┌─────────────────────────────────────────────────────────────────────────────┐
+│                           Search Pipeline                                    │
+├─────────────────────────────────────────────────────────────────────────────┤
+│                                                                              │
+│  ┌──────────┐   ┌─────────┐   ┌───────────┐   ┌──────────┐   ┌──────────┐  │
+│  │  Reduce  │ → │ Requery │ → │ Organize  │ → │ Order By │ → │   End    │  │
+│  └──────────┘   └─────────┘   └───────────┘   └──────────┘   └──────────┘  │
+│                                                                              │
+│  For Hybrid Search:                                                          │
+│  ┌──────────┐   ┌────────┐   ┌─────────┐   ┌───────────┐   ┌──────────┐    │
+│  │  Reduce  │ → │ Rerank │ → │ Requery │ → │ Order By  │ → │   End    │    │
+│  └──────────┘   └────────┘   └─────────┘   └───────────┘   └──────────┘    │
+│                                                                              │
+└─────────────────────────────────────────────────────────────────────────────┘
+```
+
+### Key Components
+
+#### 1. OrderByField Structure
+
+```go
+// internal/proxy/search_util.go
+type OrderByField struct {
+    FieldName       string // Top-level field name for result lookup
+    FieldID         int64  // Field ID for validation
+    JSONPath        string // JSON Pointer format: "/price" or "/user/age"
+    Ascending       bool   // true for ASC, false for DESC
+    OutputFieldName string // Field name to request in requery
+    IsDynamicField  bool   // true if field is resolved from dynamic field storage
+}
+```
+
+#### 2. Search Info Extension
+
+```go
+// internal/proxy/search_util.go
+type SearchInfo struct {
+    planInfo      *planpb.QueryInfo
+    offset        int64
+    isIterator    bool
+    collectionID  int64
+    orderByFields []OrderByField  // NEW: Order by specifications
+}
+```
+
+#### 3. Order By Operator
+
+The `orderByOperator` is a pipeline operator that sorts search results:
+
+```go
+// internal/proxy/search_pipeline.go
+type orderByOperator struct {
+    orderByFields  []OrderByField
+    groupByFieldId int64
+    groupSize      int64
+}
+```
+
+## Implementation Details
+
+### Parsing Order By Fields
+
+Location: `internal/proxy/search_util.go:parseOrderByFields()`
+
+1. Extract `order_by_fields` parameter from search params (list of dictionaries)
+2. For each dictionary in the list:
+   - Extract `"field"` key as field name (required)
+   - Extract `"order"` key as direction (optional, default: "asc")
+   - Handle JSON bracket notation in field name (e.g., `metadata["price"]`)
+   - Resolve dynamic field references (fields not in schema are automatically looked up in dynamic field storage)
+   - Validate field against schema
+   - Build `OrderByField` struct with parsed values
+
+### Pipeline Integration
+
+Location: `internal/proxy/search_pipeline.go`
+
+Two new pipeline definitions are added:
+
+**searchWithOrderByPipe** (for common search):
+```
+reduce → merge_ids → requery → gen_ids → organize → pick → order_by
+```
+
+**hybridSearchWithOrderByPipe** (for hybrid search):
+```
+reduce → rerank → pick_ids → requery → organize → result → order_by
+```
+
+### Order By Operator Execution
+
+Location: `internal/proxy/search_pipeline.go:orderByOperator.run()`
+
+1. **Validation**: Verify all order_by fields exist in result
+2. **Per-Query Sorting**: For nq > 1, sort each query's results independently
+3. **Group-By Handling**: If group_by is active, sort groups by first row's value
+4. **JSON Value Caching**: Pre-extract JSON values to avoid repeated parsing
+5. **Stable Sort**: Use `sort.SliceStable` to maintain original order for equal values
+6. **Result Reordering**: Reorder all result arrays (IDs, scores, fields) based on sorted indices
+
+### Comparison Logic
+
+#### Regular Fields
+- Numeric types: Standard numeric comparison
+- String types: Lexicographic comparison
+- Bool: false < true
+
+#### Dynamic Field Subpaths
+- Extract value using JSON path from dynamic field storage
+- Compare based on JSON value type (Number, String, Bool)
+- Mixed types: Fallback to raw JSON string comparison
+- NULLS FIRST: Missing or null values sort before non-null
+
+**Note**: Regular JSON fields (defined in schema) are not supported for ordering because comparing JSON values on raw bytes is meaningless.
+
+### Null Handling
+
+The implementation uses NULLS FIRST semantics:
+- Null values (from nullable fields) sort before non-null values
+- Non-existent JSON paths are treated as null
+- Explicit JSON null values are treated as null
+
+## Error Handling
+
+| Error Condition | Error Message |
+|-----------------|---------------|
+| Missing "field" key | "missing 'field' key in order_by_fields entry" |
+| Empty field name | "empty field name in order_by_fields" |
+| Invalid direction | "invalid order direction 'X' for field 'Y' (must be 'asc', 'desc', 'ascending', or 'descending')" |
+| Field not found | "order_by field 'X' does not exist in collection schema" |
+| Unsortable type | "order_by field 'X' has unsortable type Y" |
+| Regular JSON field | "order_by is not supported for JSON field 'X', use dynamic field with JSON path instead" |
+| Iterator conflict | "order_by is not supported when using search iterator" |
+| Function score conflict | "order_by and function_score cannot be used together" |
+| Invalid JSON path | "invalid JSON path in order_by field 'X'" |
+
+## Performance Considerations
+
+### Optimizations Implemented
+
+1. **JSON Value Caching**: Pre-extract JSON values before sorting to avoid O(n log n) extractions
+2. **Sparse Cache for nq > 1**: Only cache values for indices being sorted
+3. **Field Data Map**: Build O(1) lookup map for field data
+4. **Stable Sort**: Maintain original order stability without additional comparison overhead
+
+### Complexity Analysis
+
+| Operation | Time Complexity | Space Complexity |
+|-----------|-----------------|------------------|
+| Parsing | O(f * k) | O(f) |
+| JSON Extraction | O(n * f) | O(n * f) |
+| Sorting | O(n log n * f) | O(n) |
+| Reordering | O(n * d) | O(n * d) |
+
+Where:
+- n = number of results
+- f = number of order_by fields
+- k = average JSON path depth
+- d = number of output fields
+
+### Limitations
+
+1. **Requery Required**: Order by always triggers a requery to fetch field values
+2. **Memory Overhead**: All order_by field data must be loaded into memory
+3. **Single Proxy Sorting**: Sorting happens entirely at the Proxy level
+4. **No Regular JSON Field Support**: Regular JSON fields (defined in schema) cannot be used for ordering because Milvus only supports JSON path for dynamic fields, and comparing raw JSON bytes is meaningless
+
+## Testing
+
+### Unit Tests
+
+Location: `internal/proxy/search_pipeline_test.go`
+
+Coverage includes:
+- Basic field ordering (ascending/descending)
+- Multiple order_by fields
+- Dynamic field subpath extraction
+- Dynamic field support
+- Group-by integration
+- Null value handling
+- nq > 1 scenarios
+- Edge cases (empty results, single result)
+- Regular JSON field rejection
+
+### Test Categories
+
+1. **Parsing Tests**: Validate order_by_fields parameter parsing
+2. **Comparison Tests**: Verify comparison logic for all data types
+3. **Pipeline Tests**: End-to-end pipeline execution with order_by
+4. **Error Tests**: Verify error conditions are properly handled
+
+## Files Modified
+
+| File | Changes |
+|------|---------|
+| `internal/proxy/search_util.go` | OrderByField struct, parseOrderByFields() for parsing dictionary entries |
+| `internal/proxy/search_pipeline.go` | orderByOperator, pipeline definitions, comparison functions |
+| `internal/proxy/task_search.go` | orderByFields field, pipeline integration |
+| `internal/proxy/task.go` | OrderByFieldsKey constant |
+| `internal/proxy/search_pipeline_test.go` | Unit tests for dictionary-based order_by_fields |
+
+## Query ORDER BY (Moved)
+
+The Query ORDER BY design has been separated into:
+- `design_docs/20260203-query-orderby.md`
+
+## Future Improvements
+
+1. **Push-down Optimization**: Push order_by to QueryNode for early sorting
+2. **Index-based Sorting**: Leverage scalar indexes for efficient sorting
+3. **Pagination Support**: Combine with offset/limit for efficient pagination
+4. **Expression-based Ordering**: Support computed expressions (e.g., `price * quantity`)
+
+## Appendix
+
+### Dynamic Field JSON Path Conversion
+
+For dynamic fields, JSON Pointer (RFC 6901) is converted to gjson path format:
+
+JSON Pointer uses `/` as separator:
+- `/user/name` → `user.name`
+- `~0` → `~` (escaped tilde)
+- `~1` → `/` (escaped slash)
+
+gjson uses `.` as separator:
+- Dots in keys escaped with `\.`
+
+Example: `/key~1with~1slash` → `key\/with\/slash` (single key with slashes)
+
+**Note**: This conversion only applies to dynamic fields. Regular JSON fields (defined in schema) are not supported for ordering.
+
+### Requery Field Selection
+
+The `requeryOperator` unions order_by fields with output fields:
+```go
+for _, orderByField := range t.orderByFields {
+    outputFieldNames.Insert(orderByField.OutputFieldName)
+}
+```
+
+This ensures order_by field data is fetched even if not explicitly requested.
@@ -0,0 +1,247 @@
+# Truncate Collection Design Document
+
+## Overview
+
+This document describes the design and implementation of the `TruncateCollection` feature in Milvus.
+
+**Feature**: Clear all data within a collection without dropping the collection itself.
+
+**GitHub Issue**: https://github.com/milvus-io/milvus/issues/26280
+
+## Goals
+
+- Add `TruncateCollection` API to efficiently remove all data from a collection
+- Maintain collection schema, indexes, and configuration
+- Ensure data consistency across distributed components
+
+## Basic Approach
+
+`TruncateCollection` is treated as a DDL (Data Definition Language) operation with three main steps:
+
+1. **Streaming Node**: Flush all growing segments
+2. **Data Coordinator**: Drop all segments with update timestamp earlier than the truncate message
+3. **Query Coordinator**: Trigger view update on `currentTarget` to ensure query invisibility
+
+## Constraints & Requirements
+
+- Must not block Root Coordinator or other components
+- Must not block read/write operations on other collections or the current collection
+- During truncate, DDL operations on the target collection are not allowed
+
+## Architecture
+
+### Component Interaction Flow
+
+```
+┌──────┐    ┌───────┐    ┌────────────┐    ┌───────────┐    ┌────────────┐    ┌─────────────┐
+│ User │    │ Proxy │    │ Root Coord │    │ Streaming │    │ Data Coord │    │ Query Coord │
+└──┬───┘    └───┬───┘    └─────┬──────┘    │   Node    │    └─────┬──────┘    └──────┬──────┘
+   │            │              │           └─────┬─────┘          │                  │
+   │ truncate   │              │                 │                │                  │
+   │ collection │              │                 │                │                  │
+   │───────────>│              │                 │                │                  │
+   │            │  truncate    │                 │                │                  │
+   │            │  collection  │                 │                │                  │
+   │            │─────────────>│                 │                │                  │
+   │            │              │   append wal    │                │                  │
+   │            │              │────────────────>│                │                  │
+   │            │              │                 │ flush segments │                  │
+   │            │              │                 │───────────────>│                  │
+   │            │              │    fast ack     │                │                  │
+   │            │              │<────────────────│                │                  │
+   │            │              │                 │                │                  │
+   │            │              │   drop segment  │                │                  │
+   │            │              │─────────────────────────────────>│                  │
+   │            │              │                 │                │ filter segment   │
+   │            │              │                 │                │ to drop          │
+   │            │              │                 │                │                  │
+   │            │              │  wait for view to update         │                  │
+   │            │              │─────────────────────────────────────────────────────>│
+   │            │              │                 │                │                  │
+   │            │              │                 │                │  update          │
+   │            │              │                 │                │  currentTarget   │
+   │            │              │                 │                │                  │
+```
+
+## Protocol Buffer Definitions
+
+### New TruncateCollection Message
+
+```protobuf
+// TruncateCollectionMessageHeader is the header of truncate collection message.
+message TruncateCollectionMessageHeader {
+    int64 db_id = 1;
+    string db_name = 2;
+    int64 collection_id = 3;
+    string collection_name = 4;
+    repeated int64 segment_ids = 5;
+}
+
+// TruncateCollectionMessageBody is the body of truncate collection message.
+message TruncateCollectionMessageBody {
+}
+```
+
+## Component Implementation
+
+### Proxy
+
+New interface:
+
+```go
+func (node *Proxy) TruncateCollection(ctx context.Context, request *milvuspb.TruncateCollectionRequest) (*commonpb.Status, error)
+```
+
+### Streaming Node
+
+New interfaces:
+
+```go
+// Called when writing to WAL, invokes FlushAndFenceSegmentAllocUntil to mark all growing segments
+// as flush and return segmentIDs to update the message header
+func (impl *shardInterceptor) handleTruncateCollectionMessage(ctx context.Context, msg message.MutableMessage, appendOp interceptors.Append) (message.MessageID, error)
+
+// Called when WAL flusher consumes messages, executes sealSegments and flushChannel to flush growing segments
+func (impl *msgHandlerImpl) HandleTruncateCollection(flushMsg message.ImmutableTruncateCollectionMessageV2) error
+```
+
+### Root Coordinator
+
+New interfaces:
+
+```go
+// Entry point
+func (c *Core) TruncateCollection(ctx context.Context, in *milvuspb.TruncateCollectionRequest) (*commonpb.Status, error)
+
+// Construct message and write to WAL
+func (c *Core) broadcastTruncateCollection(ctx context.Context, req *milvuspb.TruncateCollectionRequest) error
+
+// Callback after WAL write, notify DC to drop segments, wait for QC to update query view
+func (c *DDLCallback) truncateCollectionV2AckCallback(ctx context.Context, result message.BroadcastResultTruncateCollectionMessageV2) error
+```
+
+### Data Coordinator
+
+New interface:
+
+```go
+// Wait for flush to complete, filter segments with update time less than truncate request timestamp, then drop them
+func (s *Server) DropSegmentsByTime(ctx context.Context, req *datapb.DropSegmentsByTimeRequest) (*commonpb.Status, error)
+```
+
+### Query Coordinator
+
+New interface:
+
+```go
+// Call WaitCurrentTargetUpdated to trigger view update
+func (s *Server) ManualUpdateCurrentTarget(ctx context.Context, req *querypb.ManualUpdateCurrentTargetRequest) (*commonpb.Status, error)
+```
+
+## Design Details
+
+### DDL Request Isolation
+
+Under the existing DDL framework, DDL requests acquire resource locks before writing to WAL. Locks are categorized by level (database, collection) and type (shared, exclusive). Different DDL operations are isolated at this step:
+
+```go
+func (*Core) startBroadcastWithCollectionLock(ctx context.Context, dbName string, collectionName string) (broadcaster.BroadcastAPI, error) {
+    broadcaster, err := broadcast.StartBroadcastWithResourceKeys(ctx,
+        message.NewSharedDBNameResourceKey(dbName),
+        message.NewExclusiveCollectionNameResourceKey(dbName, collectionName),
+    )
+    if err != nil {
+        return nil, errors.Wrap(err, "failed to start broadcast with collection lock")
+    }
+    return broadcaster, nil
+}
+```
+
+### Segment Metadata Update Steps
+
+1. **Streaming Node**: Update in-memory segment state, set flush timestamp
+
+```go
+type SegmentInfo struct {
+    segmentID        int64
+    partitionID      int64
+    state            commonpb.SegmentState
+    startPosition    *msgpb.MsgPosition
+    checkpoint       *msgpb.MsgPosition
+    startPosRecorded bool
+    flushedRows      int64
+    bufferRows       int64
+    syncingRows      int64
+    bfs              pkoracle.PkStat
+    bm25stats        *SegmentBM25Stats
+    level            datapb.SegmentLevel
+    syncingTasks     int32
+    storageVersion   int64
+    binlogs          []*datapb.FieldBinlog
+    statslogs        []*datapb.FieldBinlog
+    deltalogs        []*datapb.FieldBinlog
+    bm25logs         []*datapb.FieldBinlog
+    currentSplit     []storagecommon.ColumnGroup
+    manifestPath     string
+}
+```
+
+2. **Streaming Node**: WriteNode performs flush, notifies DC to update state after completion
+
+```go
+type SegmentInfo struct {
+    *datapb.SegmentInfo
+    allocations     []*Allocation
+    lastFlushTime   time.Time
+    isCompacting    bool
+    // a cache to avoid calculate twice
+    size            atomic.Int64
+    deltaRowcount   atomic.Int64
+    earliestTs      atomic.Uint64
+    lastWrittenTime time.Time
+}
+```
+
+3. **Data Coordinator**: Drop relevant segments
+
+4. **Query Coordinator**: Trigger view update, pull latest segment information from DC
+
+### Determining Which Segments to Drop
+
+- When the truncate message is written to WAL, it is assigned a timestamp
+- When data is inserted into a segment, the `DmlPosition` is updated, which contains a timestamp representing the latest data position consumed by that segment
+
+```go
+type MsgPosition struct {
+    state         protoimpl.MessageState
+    sizeCache     protoimpl.SizeCache
+    unknownFields protoimpl.UnknownFields
+
+    ChannelName string
+    MsgID       []byte
+    MsgGroup    string
+    Timestamp   uint64
+}
+```
+
+- During compaction, the new segment's `DmlPosition` takes the minimum value from the old segments
+
+**Drop Condition**: Segment is non-empty AND `DmlPosition.Timestamp < msg.Timestamp`
+
+### Preventing Compaction Impact on Old and New Data
+
+A new property `CollectionOnTruncatingKey` is added to collection properties to indicate whether the collection is being truncated:
+
+- Set to `1` after the truncate message is written to WAL
+- Set back to `0` after truncate completes
+
+In the compaction validation flow, add validation for `CollectionOnTruncatingKey`. If truncation is in progress, reject the compaction request.
+
+## Summary
+
+The `TruncateCollection` operation provides an efficient way to clear all data from a collection while preserving its structure. By leveraging the existing DDL framework and coordinating across multiple components (Root Coordinator, Streaming Node, Data Coordinator, Query Coordinator), the implementation ensures:
+
+- **Consistency**: All segments are properly flushed before being dropped
+- **Isolation**: Other operations are not blocked during truncation
+- **Visibility**: Query results are updated to reflect the truncated state
+- **Safety**: Compaction is blocked during truncation to prevent data inconsistencies
@@ -0,0 +1,347 @@
+# Milvus Embedded Group By - Design Document
+
+**Version**: 1.0
+**Date**: 2026-01-30
+**Author**: Milvus Development Team
+**Status**: Draft
+
+---
+
+## 1. Overview
+
+### 1.1 Background
+
+Current Milvus Search Group By only supports single-field grouping. This document describes the design for **Embedded Group By**, which introduces:
+
+1. **Nested Grouping**: Multi-level hierarchical grouping (category → brand → ...)
+2. **Per-Group Metrics**: Aggregate statistics (count/max/min/avg/sum) at each group level
+3. **Structured Results**: Tree-shaped JSON response avoiding data duplication
+
+### 1.2 Design Principles
+
+1. **Segcore**: Only extend for **multi-field flat group by** (no nesting, no metrics awareness)
+2. **Reduce**: Reuse existing reducers (flat composite key merge)
+3. **EmbeddedGroupOperator**: New proxy-side operator handles nesting and metrics
+
+---
+
+## 2. Architecture
+
+### 2.1 Data Flow
+
+```
+┌──────────────────────────────────────────────────────────────────────────┐
+│                         Proxy (Pre-Processing)                            │
+│  1. Parse embedded_group_by                                               │
+│  2. Flatten to multi-field group_by_field_ids                            │
+│  3. Ensure metric fields in output_fields                                 │
+└──────────────────────────────────────────────────────────────────────────┘
+                                    ↓
+┌──────────────────────────────────────────────────────────────────────────┐
+│                              QueryNode                                    │
+│  Segcore: PhySearchGroupByNode (extended for multi-field)                │
+│    - Group by [category, brand] as flat composite key                    │
+│  Reduce: Existing flat reduce by composite key                           │
+└──────────────────────────────────────────────────────────────────────────┘
+                                    ↓
+┌──────────────────────────────────────────────────────────────────────────┐
+│                         Proxy (Reduce)                                    │
+│  MilvusAggReducer: Merge results from shards by composite key (flat)     │
+└──────────────────────────────────────────────────────────────────────────┘
+                                    ↓
+┌──────────────────────────────────────────────────────────────────────────┐
+│                    Proxy (EmbeddedGroupOperator)                          │
+│  1. Build nested tree from flat composite keys                           │
+│  2. Compute metrics at each level (bottom-up)                            │
+│  3. Apply size limits (prune tree)                                       │
+│  4. Format nested JSON response                                          │
+└──────────────────────────────────────────────────────────────────────────┘
+                                    ↓
+                                 Client
+```
+
+### 2.2 Layer Responsibilities
+
+| Layer | Responsibility | Nesting Aware? | Metrics Aware? |
+|-------|---------------|----------------|----------------|
+| Segcore | Multi-field flat group by | No | No |
+| QueryNode Reduce | Merge by composite key | No | No |
+| Proxy Reduce | Merge by composite key | No | No |
+| **EmbeddedGroupOperator** | Tree + metrics + pruning | **Yes** | **Yes** |
+
+---
+
+## 3. API Design
+
+### 3.1 Request
+
+```python
+results = client.search(
+    collection_name="products",
+    data=[query_vector],
+    limit=5,  # documents per leaf group
+    embedded_group_by={
+        "field": "category",
+        "size": 10,
+        "metrics": [{"type": "count"}, {"type": "avg", "field": "price"}],
+        "sub_group_by": {
+            "field": "brand",
+            "size": 5,
+            "metrics": [{"type": "count"}, {"type": "max", "field": "rating"}]
+        }
+    }
+)
+```
+
+### 3.2 Response
+
+```json
+{
+    "groups": [
+        {
+            "key": "electronics",
+            "doc_count": 100,
+            "metrics": {"count": 100, "avg_price": 549.99},
+            "sub_groups": [
+                {
+                    "key": "Apple",
+                    "doc_count": 45,
+                    "metrics": {"count": 45, "max_rating": 4.9},
+                    "documents": [{"id": 123, "distance": 0.15}, ...]
+                }
+            ]
+        }
+    ]
+}
+```
+
+### 3.3 Constraints
+
+- Maximum nesting depth: 3 levels
+- Maximum size per level: 1000
+- Supported metrics: count, sum, avg, min, max
+
+---
+
+## 4. Segcore: Multi-Field Group By
+
+### 4.1 Key Change
+
+Extend `PhySearchGroupByNode` to support grouping by multiple fields using a **CompositeGroupKey**.
+
+### 4.2 CompositeGroupKey
+
+| Property | Description |
+|----------|-------------|
+| Structure | `vector<GroupByValueType>` - one value per field |
+| Hash | FNV-1a combining hash of each value |
+| Equality | Element-wise comparison |
+
+### 4.3 Grouping Logic
+
+1. Iterate vector search results via iterator
+2. For each result, read values for all group-by fields → build `CompositeGroupKey`
+3. Use `unordered_map<CompositeGroupKey, entries>` for grouping
+4. Each composite group keeps at most `group_size` results
+5. Early termination when all groups are full
+
+### 4.4 SearchResult Extension
+
+Add `composite_group_by_values_` field to store one `CompositeGroupKey` per result.
+
+---
+
+## 5. Reduce: Composite Key Merge
+
+### 5.1 Logic
+
+Reduce operates on **flat composite keys** - no tree awareness.
+
+1. Use priority queue (min-heap by distance)
+2. Track count per composite key: `map<CompositeGroupKey, count>`
+3. Accept result if `count[key] < group_size`
+4. Merge results from multiple segments/shards
+
+### 5.2 Reuse
+
+- QueryNode: Extend existing `SearchGroupByReduce` for composite keys
+- Proxy: Extend existing `MilvusAggReducer` for composite keys
+
+---
+
+## 6. EmbeddedGroupOperator
+
+### 6.1 Purpose
+
+Post-reduction operator that transforms flat results into nested structure with metrics.
+
+### 6.2 Execution Steps
+
+| Step | Input | Output |
+|------|-------|--------|
+| 1. Build Tree | Flat composite keys | Nested GroupNode tree |
+| 2. Compute Metrics | Document field values | Metrics at each node |
+| 3. Prune Tree | Size limits per level | Trimmed tree |
+| 4. Format Response | GroupNode tree | Nested JSON |
+
+### 6.3 Tree Building
+
+Transform flat `(category, brand)` composite keys into nested structure:
+
+```
+Flat: [("electronics", "Apple"), ("electronics", "Samsung"), ("books", "Penguin")]
+         ↓
+Tree:
+  ├─ electronics
+  │    ├─ Apple → [documents]
+  │    └─ Samsung → [documents]
+  └─ books
+       └─ Penguin → [documents]
+```
+
+### 6.4 Metrics Computation
+
+**Strategy**: Bottom-up computation from leaf to root.
+
+| Node Type | Computation |
+|-----------|-------------|
+| Leaf | Compute from documents (e.g., avg = sum of prices / count) |
+| Non-leaf | Roll up from children (e.g., count = sum of child counts) |
+
+**Roll-up rules**:
+
+| Metric | Roll-Up |
+|--------|---------|
+| count | Sum of child counts |
+| sum | Sum of child sums |
+| avg | (Sum of child sums) / (Sum of child counts) |
+| min | Min of child mins |
+| max | Max of child maxes |
+
+### 6.5 Tree Pruning
+
+At each level, keep only top `size` groups (sorted by doc_count descending).
+
+---
+
+## 7. Proto Changes
+
+### 7.1 SearchInfo (Segcore)
+
+```protobuf
+message SearchInfo {
+    // Existing single-field (backward compatible)
+    optional int64 group_by_field_id = 8;
+
+    // New: multi-field flat group by
+    repeated int64 group_by_field_ids = 15;
+}
+```
+
+### 7.2 SearchResultData
+
+```protobuf
+message CompositeGroupByValue {
+    repeated GenericValue values = 1;
+}
+
+message SearchResultData {
+    // New: composite keys for multi-field group by
+    repeated CompositeGroupByValue composite_group_by_values = 20;
+}
+```
+
+---
+
+## 8. Key Design Decisions
+
+### 8.1 Why Proxy-Side Metrics?
+
+| Option | Pros | Cons |
+|--------|------|------|
+| Segcore metrics | Less data transfer | More segcore complexity |
+| **Proxy metrics** | Simple segcore, reuse agg infra | Transfer field values |
+
+**Decision**: Proxy-side. Result set is limited, transfer overhead acceptable.
+
+### 8.2 Why Flat Reduce + Post Transform?
+
+| Option | Pros | Cons |
+|--------|------|------|
+| Tree reduce at each layer | Incremental | Complex, new reduce logic |
+| **Flat reduce + transform** | Reuse existing reduce | Proxy does more work |
+
+**Decision**: Flat reduce. Reuses existing infrastructure, complexity isolated in one operator.
+
+### 8.3 Backward Compatibility
+
+- Single-field `group_by_field` API unchanged
+- `embedded_group_by` is new parameter, mutually exclusive with `group_by_field`
+
+---
+
+## 9. File Changes
+
+### New Files
+
+| File | Purpose |
+|------|---------|
+| `internal/proxy/embedded_group_operator.go` | Tree building, metrics, pruning |
+
+### Modified Files
+
+| File | Change |
+|------|--------|
+| `internal/core/src/common/Types.h` | Add `CompositeGroupKey` |
+| `internal/core/src/common/QueryResult.h` | Add `composite_group_by_values_` |
+| `internal/core/src/exec/operator/SearchGroupByNode.cpp` | Multi-field support |
+| `internal/core/src/segcore/reduce/GroupReduce.cpp` | Composite key reduce |
+| `internal/proxy/search_util.go` | Parse `embedded_group_by` |
+| `internal/proxy/task_search.go` | Integrate EmbeddedGroupOperator |
+| `pkg/proto/plan.proto` | Add `group_by_field_ids`, `CompositeGroupByValue` |
+
+---
+
+## 10. Implementation Phases
+
+### Phase 1: Multi-Field Group By (Segcore)
+
+- CompositeGroupKey type and hash
+- Extend PhySearchGroupByNode
+- Extend reduce for composite keys
+- Proto changes
+
+### Phase 2: EmbeddedGroupOperator (Proxy)
+
+- Parse embedded_group_by parameter
+- Tree building from flat results
+- Metrics computation (bottom-up)
+- Tree pruning
+
+### Phase 3: Integration & Testing
+
+- End-to-end integration
+- PyMilvus client support
+- Unit and integration tests
+
+---
+
+## 11. Summary
+
+```
+┌─────────────┐     ┌──────────────────┐     ┌─────────────────────────┐
+│   Segcore   │ --> │  Existing Reduce │ --> │  EmbeddedGroupOperator  │
+│ Multi-field │     │  (flat composite │     │  - Build tree           │
+│ flat group  │     │   key merge)     │     │  - Compute metrics      │
+│ by          │     │                  │     │  - Prune & format       │
+└─────────────┘     └──────────────────┘     └─────────────────────────┘
+```
+
+**Key Points**:
+1. Segcore only handles flat multi-field grouping
+2. Reduce reuses existing infrastructure with composite keys
+3. EmbeddedGroupOperator handles all nesting and metrics logic at proxy
+
+---
+
+**Document End**
@@ -0,0 +1,349 @@
+# MEP: Client-Side Telemetry with Heartbeat and Server Command Support
+
+- **Created:** 2026-01-31
+- **Author(s):** @xiaofanluan
+- **Status:** Draft
+- **Component:** SDK | Proxy | Coordinator
+- **Related Issues:** #46934
+- **Released:** [TBD]
+
+## Summary
+
+This MEP introduces a client-side telemetry system for the Go SDK that collects operational metrics, sends periodic heartbeats to the server, and supports bidirectional communication through server-pushed commands. The system provides visibility into client behavior, enables real-time monitoring through a WebUI dashboard, and allows server-initiated configuration changes.
+
+## Motivation
+
+Currently, Milvus lacks visibility into client-side operations and behavior. Operators cannot:
+- Monitor which clients are connected to the cluster
+- Understand client-side performance characteristics (latency, error rates)
+- Identify problematic clients or usage patterns
+- Push configuration changes to connected clients dynamically
+
+This feature addresses these gaps by implementing a comprehensive client telemetry system that:
+1. Collects client-side metrics (request counts, latencies, errors)
+2. Reports telemetry data to the server via heartbeats
+3. Enables server-to-client command channels for dynamic configuration
+4. Provides a WebUI for operators to monitor and manage clients
+
+## Public Interfaces
+
+### Go SDK APIs
+
+```go
+// TelemetryConfig holds configurable settings for client telemetry
+type TelemetryConfig struct {
+    Enabled           bool          // Enable/disable telemetry collection
+    HeartbeatInterval time.Duration // Heartbeat frequency (default: 30s)
+    SamplingRate      float64       // Sampling rate 0.0-1.0 (default: 1.0)
+    ErrorMaxCount     int           // Max errors to track (default: 100)
+}
+
+// ClientConfig gains a new field
+type ClientConfig struct {
+    // ... existing fields ...
+    TelemetryConfig *TelemetryConfig
+}
+```
+
+### HTTP REST APIs (Proxy)
+
+```
+GET  /api/v1/telemetry/clients          - List connected clients with optional filtering
+POST /api/v1/telemetry/commands         - Push commands to clients
+DELETE /api/v1/telemetry/commands/{id}  - Delete a command
+```
+
+### gRPC APIs (RootCoord)
+
+```protobuf
+service RootCoord {
+    // Client heartbeat with metrics
+    rpc ClientHeartbeat(ClientHeartbeatRequest) returns (ClientHeartbeatResponse);
+
+    // Query connected clients
+    rpc GetClientTelemetry(GetClientTelemetryRequest) returns (GetClientTelemetryResponse);
+
+    // Push commands to clients
+    rpc PushClientCommand(PushClientCommandRequest) returns (PushClientCommandResponse);
+
+    // Delete commands
+    rpc DeleteClientCommand(DeleteClientCommandRequest) returns (DeleteClientCommandResponse);
+}
+```
+
+## Design Details
+
+### Architecture Overview
+
+```
+┌─────────────────────────────────────────────────────────────────────────────┐
+│                              Client (Go SDK)                                 │
+│  ┌───────────────────────────────────────────────────────────────────────┐  │
+│  │                    ClientTelemetryManager                              │  │
+│  │  ┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐       │  │
+│  │  │ Operation       │  │ Error           │  │ Command         │       │  │
+│  │  │ Metrics         │  │ Collector       │  │ Handler         │       │  │
+│  │  │ Collector       │  │ (Ring Buffer)   │  │ Router          │       │  │
+│  │  └────────┬────────┘  └────────┬────────┘  └────────┬────────┘       │  │
+│  │           │                    │                    │                 │  │
+│  │           └────────────────────┼────────────────────┘                 │  │
+│  │                                ▼                                       │  │
+│  │                    ┌───────────────────────┐                          │  │
+│  │                    │   Heartbeat Loop      │───────── 30s interval    │  │
+│  │                    │   (Background)        │                          │  │
+│  │                    └───────────┬───────────┘                          │  │
+│  └────────────────────────────────┼──────────────────────────────────────┘  │
+└───────────────────────────────────┼─────────────────────────────────────────┘
+                                    │
+                          ClientHeartbeat RPC
+                                    │
+                                    ▼
+┌─────────────────────────────────────────────────────────────────────────────┐
+│                              Server (Milvus)                                 │
+│  ┌────────────────┐          ┌───────────────────────────────────────────┐  │
+│  │     Proxy      │◄────────►│              RootCoord                     │  │
+│  │                │          │  ┌─────────────────────────────────────┐  │  │
+│  │  HTTP API      │          │  │      Telemetry Manager              │  │  │
+│  │  /telemetry/*  │          │  │  ┌───────────┐  ┌───────────────┐   │  │  │
+│  │                │          │  │  │ Client    │  │ Command       │   │  │  │
+│  │  WebUI         │          │  │  │ Store     │  │ Store         │   │  │  │
+│  │  telemetry.html│          │  │  └───────────┘  └───────────────┘   │  │  │
+│  └────────────────┘          │  └─────────────────────────────────────┘  │  │
+│                              └───────────────────────────────────────────┘  │
+└─────────────────────────────────────────────────────────────────────────────┘
+```
+
+### Client-Side Components
+
+#### 1. ClientTelemetryManager
+
+The central component managing telemetry collection and heartbeat communication.
+
+```go
+type ClientTelemetryManager struct {
+    config         *TelemetryConfig
+    client         *Client
+    clientID       string                              // Stable UUID
+    collectors     map[string]*OperationMetricsCollector
+    errorCollector *ErrorCollectorImpl
+    commandHandlers map[string]CommandHandler
+
+    // Heartbeat management
+    stopCh         chan struct{}
+    wg             sync.WaitGroup
+}
+```
+
+**Key behaviors:**
+- Generates stable client UUID on creation
+- Starts background heartbeat loop on `Start()`
+- Sends first heartbeat immediately, then every `HeartbeatInterval`
+- Collects and resets metrics atomically during snapshot creation
+
+#### 2. OperationMetricsCollector
+
+Per-operation metrics collection with global and per-collection breakdown.
+
+```go
+type OperationMetricsCollector struct {
+    // Global metrics
+    requestCount int64
+    successCount int64
+    errorCount   int64
+    totalLatency int64  // microseconds
+    maxLatency   int64
+
+    // P99 calculation (ring buffer of 1000 samples)
+    latencySamples []int64
+    totalSamples   int64
+
+    // Per-collection metrics
+    collectionMetrics map[string]*CollectionMetrics
+}
+```
+
+**Metrics tracked:**
+- Request count
+- Success/error counts
+- Average latency (ms)
+- P99 latency (ms) - calculated from 1000 sample buffer
+- Max latency (ms)
+
+#### 3. ErrorCollectorImpl
+
+Ring buffer implementation for tracking recent errors.
+
+```go
+type ErrorCollectorImpl struct {
+    errors   []*ErrorInfo
+    maxCount int
+    index    int  // Ring buffer index
+}
+
+type ErrorInfo struct {
+    Timestamp  int64
+    Operation  string
+    ErrorMsg   string
+    Collection string
+    RequestID  string
+}
+```
+
+#### 4. Command Handler System
+
+Supports server-pushed commands with extensible handler registration.
+
+```go
+type CommandHandler func(cmd *ClientCommand) string  // Returns error message or ""
+
+// Built-in commands:
+const (
+    CmdSetSamplingRate   = "set_sampling_rate"        // Adjust sampling rate
+    CmdEnableCollections = "enable_collections"       // Enable specific collections
+    CmdUpdateConfig      = "update_config"            // Update telemetry config
+)
+```
+
+### Server-Side Components
+
+#### 1. Telemetry Manager (RootCoord)
+
+Central server-side storage for client telemetry data.
+
+```go
+type TelemetryManager struct {
+    clients      map[string]*ClientTelemetry  // clientID -> telemetry
+    commandStore *CommandStore
+}
+
+type ClientTelemetry struct {
+    ClientInfo        *commonpb.ClientInfo
+    LastHeartbeatTime int64
+    Status            string  // "active" or "inactive"
+    Databases         []string
+    Metrics           []*commonpb.OperationMetrics
+}
+```
+
+#### 2. Command Store
+
+Manages pending commands for clients with support for persistent and one-time commands.
+
+```go
+type CommandStore struct {
+    commands     []*commonpb.ClientCommand  // One-time commands
+    persistent   []*commonpb.ClientCommand  // Persistent commands
+}
+```
+
+**Command targeting:**
+- `TargetScope = "*"` - All clients
+- `TargetScope = "client_id:xxx"` - Specific client
+- `TargetScope = "db:database_name"` - Clients using specific database
+
+#### 3. HTTP Handlers (Proxy)
+
+REST API endpoints for WebUI and external integrations.
+
+| Endpoint | Method | Description |
+|----------|--------|-------------|
+| `/api/v1/telemetry/clients` | GET | List connected clients |
+| `/api/v1/telemetry/clients?database=X` | GET | Filter by database |
+| `/api/v1/telemetry/clients?include_metrics=true` | GET | Include operation metrics |
+| `/api/v1/telemetry/commands` | POST | Push command to clients |
+| `/api/v1/telemetry/commands/{id}` | DELETE | Remove a command |
+
+**Authentication:** Uses existing Milvus Basic Auth when authorization is enabled.
+
+### Heartbeat Protocol
+
+```
+Client                                      Server
+   │                                           │
+   │──── ClientHeartbeatRequest ──────────────►│
+   │     - ClientInfo (ID, SDK version, host)  │
+   │     - Metrics (per-operation, per-coll)   │
+   │     - CommandReplies                      │
+   │     - ConfigHash (for change detection)   │
+   │                                           │
+   │◄─── ClientHeartbeatResponse ─────────────│
+   │     - Commands (pending for this client)  │
+   │     - NewConfigHash (if config changed)   │
+   │                                           │
+```
+
+**Heartbeat interval:** 30 seconds (configurable, server can override)
+
+**Config hash:** SHA-256 of client configuration, used to detect when server pushes new config.
+
+### Data Flow
+
+1. **Metrics Collection:**
+   - Each SDK operation (Search, Insert, Query, etc.) records metrics
+   - Sampling rate determines if operation is tracked (default: 100%)
+   - Metrics stored in per-operation collectors
+
+2. **Heartbeat Cycle:**
+   - Background goroutine wakes every 30 seconds
+   - Creates atomic snapshot of all metrics (resets counters)
+   - Sends snapshot to RootCoord via ClientHeartbeat RPC
+   - Receives and processes any pending commands
+
+3. **Command Processing:**
+   - Server pushes commands via heartbeat response
+   - Client executes command handler
+   - Reply sent in next heartbeat request
+   - Persistent commands re-sent until explicitly deleted
+
+### WebUI Dashboard
+
+A new telemetry dashboard (`/webui/telemetry.html`) provides:
+
+- **Client List:** Active/inactive clients with connection details
+- **Metrics View:** Per-client operation metrics with latency charts
+- **Command Interface:** Send commands to specific clients or broadcast
+- **Filtering:** By database, client ID, or status
+
+## Compatibility, Deprecation, and Migration Plan
+
+- **Backward Compatible:** Telemetry is opt-in and disabled by default can be enabled
+- **No Breaking Changes:** Existing SDK usage remains unchanged
+- **Server Compatibility:** Old clients work with new servers (no heartbeats sent)
+- **Client Compatibility:** New clients work with old servers (heartbeats fail silently)
+
+## Test Plan
+
+### Unit Tests
+- `telemetry_test.go`: Metrics collection, P99 calculation, ring buffer
+- `telemetry_http_handler_test.go`: HTTP API handlers
+- `command_router_test.go`: Command routing and handling
+
+### Integration Tests
+- `telemetry_integration_test.go`: End-to-end heartbeat flow
+- Multi-client scenarios with different configurations
+- Command push and execution verification
+
+### Manual Testing
+- WebUI functionality verification
+- Performance impact measurement under load
+- Network failure and recovery scenarios
+
+## Rejected Alternatives
+
+### 1. Streaming Telemetry (Rejected)
+Using streaming RPC instead of periodic heartbeats.
+- **Rejected because:** Higher resource usage, more complex failure handling
+
+### 2. External Metrics System (Rejected)
+Push metrics to external systems (Prometheus, etc.) from client.
+- **Rejected because:** Adds external dependencies, complicates deployment
+
+### 3. Pull-Based Model (Rejected)
+Server polls clients for metrics.
+- **Rejected because:** Doesn't scale, requires client to expose endpoints
+
+## References
+
+- Milvus existing telemetry: `internal/proxy/impl.go`
+- gRPC health checking: https://github.com/grpc/grpc/blob/master/doc/health-checking.md
+- OpenTelemetry SDK patterns: https://opentelemetry.io/docs/instrumentation/go/
@@ -0,0 +1,439 @@
+# Query ORDER BY
+
+- **Created:** 2026-02-03
+- **Author(s):** @MrPresent-Han
+- **Status:** Under Review
+- **Component:** QueryNode / Proxy
+- **Related Issues:** N/A
+- **Released:** N/A
+
+## Summary
+
+Query ORDER BY pushes sorting into the segcore execution pipeline. Each segment filters, projects, sorts locally, and returns top-K results. The QueryNode merges results from multiple segments, and the proxy performs global deduplication, merge-sort, pagination, and field re-mapping before returning to the client.
+
+## Motivation
+
+Query ORDER BY is fundamentally different from Search ORDER BY and needs a dedicated design document to define its plan format, execution path, merge semantics, and edge-case behavior.
+
+## Design Details
+
+Detailed design is documented in Sections 2-13 below, including API/proto contract, segcore plan and pipeline, proxy/querynode merge behavior, GROUP BY interaction, edge cases, and known limitations.
+
+## Test Plan
+
+- Unit tests for parse/validate/remap/order operators in proxy.
+- Integration tests for end-to-end ORDER BY correctness across multi-segment and multi-querynode paths.
+- Edge-case tests for schema evolution, null ordering, tie-order behavior, and limit/offset boundaries.
+- Stress tests for memory behavior on high-cardinality ORDER BY queries, including known limitation coverage for missing segcore `max_sort_rows` guard.
+
+## 1. Overview
+
+Query ORDER BY pushes sorting into the segcore execution pipeline. Each segment filters, projects, sorts locally, and returns top-K results. The QueryNode merges results from multiple segments, and the proxy performs global deduplication, merge-sort, pagination, and field re-mapping before returning to the client.
+
+This differs fundamentally from Search ORDER BY, which operates at the proxy level after vector search requery. Query ORDER BY has no vector search step — it is a pure filter-project-sort pipeline.
+
+### End-to-End Data Flow
+
+```
+Client SDK
+  │  order_by=["price:asc", "rating:desc"]
+  ▼
+Proxy (task_query.go)
+  │  Parse order_by strings → OrderByField protos
+  │  Validate field types, build QueryPlanNode
+  ▼
+QueryNode (segment-level)
+  │  Segcore pipeline: Filter → Project → OrderBy → output
+  │  FillOrderByResult: set field_id, fetch deferred fields, populate system fields
+  │  MergeSegcoreRetrieveResults: merge results from multiple segments
+  ▼
+Proxy Pipeline (query_pipeline.go)
+  │  ReduceByPK → OrderOperator → SliceOperator → RemapOperator
+  │  Populate FieldName from schema, filter to user's output_fields
+  ▼
+Client SDK
+  │  Results with correct field names and user-requested field order
+```
+
+### Supported Data Types
+
+| Sortable | Not Sortable |
+|----------|-------------|
+| Bool, Int8, Int16, Int32, Int64 | FloatVector, BFloat16Vector, ... |
+| Float, Double | Array, JSON (without path) |
+| String, VarChar | Timestamptz (not yet supported) |
+
+## 2. API & Proto
+
+### Client API (PyMilvus)
+
+Format: `"field_name:direction"` where direction is `asc` (default) or `desc`. Multiple fields specify multi-key sort priority.
+
+### Proto Definition (plan.proto)
+
+`OrderByField` message carries field_id, ascending (bool, default true), and nulls_first (bool, proto default false, but proxy sets it per SQL convention: ASC → false/NULLS LAST, DESC → true/NULLS FIRST). `QueryPlanNode` includes a repeated `OrderByField order_by_fields` field alongside predicates, limit, group_by, and aggregates.
+
+## 3. Proxy: Request Parsing & Validation
+
+**File**: `internal/proxy/task_query.go`
+
+### 3.1 Parsing order_by Strings
+
+`translateOrderByFields` parses user-provided strings like `"price:desc"` into OrderByField protos:
+
+- Splits on `:` or space → field name + direction
+- Looks up field_id from collection schema
+- Validates the field type is sortable via `isSortableFieldType`
+- Default direction: ascending. Default null ordering follows SQL/PostgreSQL convention: **ASC → NULLS LAST, DESC → NULLS FIRST** (NULL is treated as larger than any non-null value). Users can override with explicit `:nulls_first` or `:nulls_last` suffix (e.g., `"price:desc:nulls_last"`).
+
+### 3.2 Validation Rules
+
+1. **Type check**: Only sortable types are allowed (see table above). Vectors, arrays, JSON (without path) are rejected.
+2. **GROUP BY compatibility**: When GROUP BY is present, ORDER BY can only reference columns in the GROUP BY clause or aggregate function results (e.g., `sum(price)`).
+3. **limit required**: ORDER BY always requires an explicit limit, regardless of whether a predicate is present. Unlike plain queries (which can stream results without limit, protected by `MaxOutputSize` at the proxy reduce stage), ORDER BY must load all matching rows into the segcore SortBuffer for in-memory sorting — a stage where `MaxOutputSize` provides no protection. Requests with ORDER BY but no limit are rejected with `"ORDER BY requires explicit limit"`.
+
+### 3.3 Plan Assembly
+
+The parsed OrderByField list is set on the QueryPlanNode and sent to the QueryNode via gRPC.
+
+## 4. Segcore: Plan Creation & Pipeline
+
+**File**: `internal/core/src/query/PlanProto.cpp`
+
+### 4.1 Pipeline Structure
+
+For `output_fields=["A","B","C"], order_by=["B:asc","C:desc"]`:
+
+**Single-project mode** (all non-sort output fields are fixed-width):
+
+```
+FilterBitsNode → MvccNode → ProjectNode[pk, B, C, A, SegOffset] → OrderByNode(B, C, limit)
+```
+
+**Two-project mode** (any non-sort output field is variable-width, e.g., VARCHAR):
+
+```
+FilterBitsNode → MvccNode → ProjectNode[pk, B, C, SegOffset] → OrderByNode(B, C, limit)
+                                                                       │
+                                        deferred fields [A] fetched later by FillOrderByResult
+```
+
+### 4.2 Two-Project Mode: Why It Matters
+
+VARCHAR values require heap allocation per row during RowContainer storage. Consider a segment with 10,000 matching rows and `limit=10`:
+
+- **Single-project**: Heap-allocates 10,000 strings, sorts, discards 9,990. Wasteful.
+- **Two-project**: Only projects fixed-width columns into the sort buffer. After TopK selects 10 winners, bulk-fetches the 10 VARCHAR values via segment offsets. 1000x fewer allocations.
+
+### 4.3 Mode Decision Logic
+
+For each non-sort output column: fixed-width types (bool, int8..int64, float, double) are included in the ProjectNode; variable-width types (VARCHAR, STRING) trigger two-project mode.
+
+Any variable-width column causes ALL non-sort output columns to be deferred. This "all or nothing" deferral simplifies the positional layout: pipeline always produces `[pk, orderby_fields, SegOffset]` in two-project mode, regardless of how many fixed-width fields exist.
+
+### 4.4 Positional Layout Contract
+
+The pipeline returns columns in a fixed positional order. SegmentOffsetFieldID is always appended as the last column.
+
+```
+Pipeline output: [pk, B, C, (A if single-project), SegOffset]
+                  ^   ^----^   ^                      ^
+                  |   |        |                      +-- segment offset for deferred fetch
+                  |   |        +-- non-sort output (single-project only)
+                  |   +---------- ORDER BY fields (sort key order)
+                  +-------------- PK (for proxy reduce/dedup)
+```
+
+If a field appears in both ORDER BY and output_fields, it occupies only the ORDER BY position and is not repeated.
+
+### 4.5 OrderByNode & SortBuffer
+
+The OrderByNode wraps a SortBuffer:
+
+- **Storage**: All projected columns stored in RowContainer. Fixed-width values are inline; VARCHAR values are heap-allocated pointers.
+- **Sorting**: Sorts row pointers (8 bytes each), not row data. Uses sort_keys_ to compare — non-sort columns are completely ignored during comparison.
+- **TopK optimization**: When `limit < n * threshold`, uses `std::partial_sort` O(n log k) instead of `std::sort` O(n log n). Current threshold is 1/2 (`SortBuffer.cpp:144`). **TODO: benchmark needed** — `partial_sort` is heap-based with poor cache locality; the crossover point where it beats introsort may be much lower than n/2 (likely around n/4 or n/8 depending on row width and comparison cost). A micro-benchmark varying k/n ratio with realistic row sizes should determine the optimal threshold.
+- **Output**: Extracts columns from sorted row pointers via copy into new ColumnVectors.
+
+## 5. Segcore: Result Assembly (FillOrderByResult)
+
+**File**: `internal/core/src/segcore/SegmentInterface.cpp`
+
+After the pipeline produces sorted top-K rows, FillOrderByResult assembles the final RetrieveResults proto in five steps:
+
+### Step 1: Move Pipeline Columns
+
+Move all columns except the last (SegmentOffsetFieldID) into results.fields_data. Set field_id on each DataArray using pipeline_field_ids_ — this is critical because pipeline-produced DataArrays have field_id=0 by default, and the QN-side merge matches by field_id.
+
+### Step 2: Extract Segment Offsets
+
+The last pipeline column contains segment-level row offsets. These are stored in results.offset() for deferred field fetching (Step 3) and QN-side merge validation (MergeSegcoreRetrieveResults checks offset length > 0).
+
+### Step 3: Bulk-Fetch Deferred Fields (Two-Project Mode)
+
+For each deferred field, bulk_subscript is called with the segment offsets to fetch exactly the top-K rows. Special handling exists for:
+- **Dynamic fields**: JSON subfield projection via target_dynamic_fields_
+- **Schema evolution**: Fields absent in this segment get default values via bulk_subscript_not_exist_field. Both deferred fields (this path) and eagerly projected ORDER BY fields (ProjectNode) are covered — see Section 12.3.
+- **Array type**: element_type is set on the output DataArray
+
+### Step 4: Populate System Fields
+
+Fetch system fields (e.g., TimestampField) via segment offsets. The timestamp field is required by QN-side pk+ts deduplication logic.
+
+### Step 5: Set IDs from PK Column
+
+Extract PK values from position 0 and populate results.ids (either int_id or str_id). This is required by the proxy's ReduceByPK operator.
+
+## 6. ShouldIgnoreNonPk Bypass
+
+**File**: `internal/core/src/segcore/plan_c.cpp`
+
+Normal queries use a two-phase retrieval optimization: first fetch only PKs, then re-fetch full fields via offsets. This is controlled by ShouldIgnoreNonPk.
+
+ORDER BY queries **must bypass** this optimization because:
+- The pipeline returns data in a positional layout [pk, orderby, remaining]
+- The proxy's Remap operator depends on this exact positional order
+- Two-phase retrieval would re-fetch via FillTargetEntry in field_ids_ order, breaking the positional contract
+
+When has_order_by_ is true, ShouldIgnoreNonPk returns false so the pipeline output is used directly.
+
+## 7. QueryNode: Segment Merge
+
+**File**: `internal/querynodev2/segments/result.go`
+
+MergeSegcoreRetrieveResults merges results from multiple segments on the same QueryNode:
+
+- Uses PrepareResultFieldData to create output containers, copying field schema from the first non-empty segment result (field_id, field_name, field_type)
+- AppendFieldData matches fields by field_id (not position) using a map keyed by FieldId
+- Preserves the positional layout from segcore — the merged result has the same column order as individual segment results
+
+**Critical invariant**: Since AppendFieldData matches by field_id, FillOrderByResult must set correct field_id values on pipeline-produced DataArrays (Step 1 above). Without this, all columns would have field_id=0 and data would be corrupted during merge.
+
+## 8. Proxy Pipeline
+
+**File**: `internal/proxy/query_pipeline.go`
+
+The proxy-side pipeline processes the merged results from all QueryNodes:
+
+```
+input → ReduceByPK → OrderOperator → SliceOperator → RemapOperator → output
+```
+
+### 8.1 ReduceByPK
+
+Deduplicates rows across QueryNodes using the PK column (position 0). Rows with the same PK but older timestamps are discarded.
+
+### 8.2 OrderOperator
+
+**File**: `internal/util/queryutil/order_op.go`
+
+Global merge-sort of results from all QueryNodes.
+
+- Uses **positional comparison**: ORDER BY field i is at FieldsData[i+1] (position 0 is PK)
+- When offset + limit < rowCount, uses **heap-based partial sort** O(N log K) instead of full sort O(N log N)
+- Supports null handling with configurable NULLS FIRST / NULLS LAST
+- **Tie-order is non-deterministic**: when all ORDER BY keys compare equal, relative row order is not guaranteed across runs.
+
+Tie-order notes:
+- The merge input order depends on segment/querynode execution and merge path, so equal-key rows may appear in different relative order.
+- To get deterministic order today, users should add an explicit final key (typically primary key), for example: `ORDER BY price DESC, id ASC`.
+- Future option (not implemented): append PK as an implicit final tiebreaker so internal ordering becomes equivalent to `ORDER BY <user_keys...>, pk`.
+
+### 8.3 SliceOperator
+
+Applies user-specified offset and limit to the sorted result. For pagination: page 2 with limit=10 uses offset=10, limit=10.
+
+### 8.4 RemapOperator
+
+**File**: `internal/util/queryutil/remap_op.go`
+
+Reorders FieldsData from segcore's positional layout to the user's requested output field order.
+
+BuildRemapIndices reconstructs the segcore positional layout (mirroring PlanProto.cpp logic), then computes an index mapping for each user output field:
+
+```
+Segcore layout: [pk(pos=0), price(pos=1), rating(pos=2), name(pos=3), category(pos=4)]
+User wants:     [name, category, price]
+Remap indices:  [3, 4, 1]
+```
+
+Fields that are only in ORDER BY (not in user's output_fields) and the implicit PK are stripped by this step.
+
+## 9. Proxy: Response Assembly
+
+**File**: `internal/proxy/task_query.go`
+
+After the pipeline completes, two final steps prepare the response:
+
+### 9.1 FieldName Population
+
+Segcore does not set field_name on DataArray protos (only field_id). PyMilvus uses field_data.field_name (not response.output_fields) to build result dict keys. Without this step, all fields would have field_name="".
+
+The proxy iterates over all FieldsData entries, looks up each field_id in the collection schema, and sets FieldName, Type, and IsDynamic accordingly. This mirrors the search pipeline's endOperator behavior.
+
+### 9.2 Output Field Filtering
+
+Filter FieldsData to only include fields the user requested in output_fields. Removes any internal fields (PK if not requested, sort-only fields, system fields) that leaked through the pipeline.
+
+## 10. GROUP BY + ORDER BY Interaction
+
+### 10.1 Segcore Pipeline (C++)
+
+When GROUP BY is present, the segcore pipeline becomes:
+
+```
+FilterBitsNode → MvccNode → ProjectNode → AggregationNode → OrderByNode
+```
+
+**AggregationNode** groups matching rows via a hash table (`GroupingSet`) and computes aggregate accumulators per group. Its output layout is a fixed schema:
+
+```
+[groupKey_0, groupKey_1, ..., aggResult_0, aggResult_1, ...]
+```
+
+To avoid ambiguity between `alias`, aggregate expression text, and physical fields, the pipeline uses a unified identifier contract:
+
+- **GROUP BY key columns**: identified by **`field_id`** (schema-stable physical ID).
+- **Aggregate result columns**: identified by **`expression_id`** (unique per aggregate expression in the plan).
+- **Alias**: user-facing label only; resolved to `expression_id` during parsing/validation, never used as the cross-layer canonical key.
+
+`OrderByNode` resolves sorting keys using this canonical identity (`field_id` / `expression_id`), not raw function names such as `"sum"` which are not unique.
+
+### 10.2 ORDER BY Field Resolution
+
+ORDER BY in GROUP BY context can only reference:
+- **GROUP BY columns**: resolved to `field_id`
+- **Aggregate function results**: resolved to `expression_id`
+
+Resolution rules:
+1. If token matches a GROUP BY column, bind to its `field_id`.
+2. Else if token matches an aggregate alias, bind alias -> `expression_id`.
+3. Else if token matches a normalized aggregate expression (e.g., `sum(price)`, `count(*)`), bind to `expression_id`.
+4. Otherwise reject as invalid ORDER BY field.
+
+Validation constraints:
+- Aggregate aliases must be unique within one query.
+- Bare function names (for example `ORDER BY sum`) are rejected in GROUP BY mode because they are ambiguous across multiple aggregate expressions.
+
+This validation is enforced by `validateOrderByFieldsWithGroupBy` at the proxy level before plan creation.
+
+### 10.3 Proxy Pipeline (Go)
+
+The proxy pipeline for GROUP BY + ORDER BY differs significantly from plain ORDER BY:
+
+```
+input → ReduceByGroups → OrderOperator → SliceOperator → output
+```
+
+Key differences from plain ORDER BY pipeline:
+
+| Aspect | Plain ORDER BY | GROUP BY + ORDER BY |
+|--------|---------------|---------------------|
+| Reduce | ReduceByPK (dedup by primary key) | ReduceByGroups (hash-merge groups, accumulate aggregates) |
+| Order | Positional comparison (field at index i+1) | Canonical ID-based comparison (`field_id` / `expression_id`, with alias resolved before sort) |
+| Remap | RemapOperator reorders from segcore positional layout | Not needed — ReduceByGroups output is already in user-visible column order |
+| PK | Always at position 0 | Not projected |
+
+**ReduceByGroups** (`GroupAggReducer`): Merges grouped results from multiple QueryNodes using a hash map keyed on grouping columns. For matching groups, aggregate accumulators are combined (e.g., sums are added, counts are added). Special case: `avg` is internally expanded to `sum + count`, merged separately, and divided at the final assembly step.
+
+**OrderOperator** in GROUP BY context: Sorts merged groups using canonical keys (`field_id` / `expression_id`), not positional indices; aliases are already translated to canonical IDs before this stage.
+
+### 10.4 Why No RemapOperator
+
+Plain ORDER BY uses RemapOperator because segcore produces a positional layout `[pk, orderby, output, SegOffset]` that must be reordered to match the user's requested field order. GROUP BY output goes through `GroupAggReducer` which assembles `FieldData` arrays directly in user-visible column order via `AggregationFieldMap`. No positional-to-named remapping is needed.
+
+## 11. Key Implementation Files
+
+| Layer | File | Purpose |
+|-------|------|---------|
+| Proto | `pkg/proto/plan.proto` | OrderByField, QueryPlanNode definition |
+| Proxy | `internal/proxy/task_query.go` | Parse, validate, plan creation, FieldName population |
+| Proxy | `internal/proxy/query_pipeline.go` | Pipeline construction (reduce → order → slice → remap) |
+| Proxy | `internal/util/queryutil/order_op.go` | OrderOperator: global merge-sort with partial sort optimization |
+| Proxy | `internal/util/queryutil/remap_op.go` | RemapOperator: positional → user field order mapping |
+| Proxy | `internal/util/reduce/orderby/types.go` | OrderByField type definition, sortable type check |
+| C++ | `internal/core/src/query/PlanProto.cpp` | BuildOrderByProjectNode, BuildOrderByNode |
+| C++ | `internal/core/src/query/PlanNode.h` | RetrievePlanNode: deferred_field_ids_, pipeline_field_ids_ |
+| C++ | `internal/core/src/segcore/SegmentInterface.cpp` | FillOrderByResult: deferred fetch, system fields, IDs |
+| C++ | `internal/core/src/segcore/plan_c.cpp` | ShouldIgnoreNonPk bypass for ORDER BY |
+| C++ | `internal/core/src/exec/SortBuffer.h` | RowContainer sort with TopK optimization |
+| QN | `internal/querynodev2/segments/result.go` | MergeSegcoreRetrieveResults: field_id-based merge |
+
+## 12. Error Handling & Edge Cases
+
+### 12.1 limit=0 and Missing limit
+
+| Condition | Behavior |
+|-----------|----------|
+| `limit=0` (explicit) | Rejected by `validateMaxQueryResultWindow`: `"limit [0] is invalid, should be greater than 0"` |
+| No limit + ORDER BY (with or without predicate) | Rejected at PreExecute: `"ORDER BY requires explicit limit"`. Unlike plain queries which can stream results protected by `MaxOutputSize`, ORDER BY loads all matching rows into the segcore SortBuffer for in-memory sorting — `MaxOutputSize` only applies at the proxy reduce stage and cannot prevent OOM during segment-level sorting. |
+
+### 12.2 Empty Result Sets
+
+Both OrderOperator and RemapOperator handle empty results gracefully:
+
+- **OrderOperator** (`order_op.go`): If result is nil, FieldsData is empty, or rowCount <= 1, the operator passes through the input unchanged — no sort logic executes.
+- **RemapOperator** (`remap_op.go`): Triple nil-guard on result, FieldsData, and outputIndices. Additionally, index bounds are checked per-field to prevent out-of-range panics.
+- **MergeSegcoreRetrieveResults** (`result.go`): Segments with zero rows are filtered out before merge. If all segments are empty, an empty result is returned without error.
+
+### 12.3 ORDER BY Fields Missing Due to Schema Evolution
+
+When a collection schema evolves (e.g., a new field is added), older segments may not contain the ORDER BY field. There are two distinct code paths with **different behaviors**:
+
+#### Deferred fields (two-project mode) — handled correctly
+
+In `FillOrderByResult` (Step 3), deferred fields are fetched after sorting. This path checks `is_field_exist(field_id)` before calling `bulk_subscript`, and falls back to `bulk_subscript_not_exist_field` for missing fields:
+- If the field has a **default value**: fills all rows with the default value, `valid_data = true`.
+- If the field has **no default value**: fills all rows with placeholder data, `valid_data = false` (NULL).
+
+#### Eagerly projected fields (ORDER BY columns in ProjectNode)
+
+ORDER BY fields are eagerly projected in the ProjectNode (Section 4.1).
+
+**Previous behavior (without the fix)**: `PhyProjectNode::GetOutput()` called `bulk_subscript_field_data` → `segment->bulk_subscript` without checking field existence. When an ORDER BY field was absent in an older segment, `bulk_subscript` hit `AssertInfo(column != nullptr, ...)` (`ChunkedSegmentSealedImpl.cpp:1590`). Note: `AssertInfo` in Milvus throws a C++ exception (not `abort()`/`SIGABRT`), so this would fail the individual request with an error rather than crash the process — but it is still incorrect behavior for a legitimate user query on an evolved schema.
+
+**Current fix**: `PhyProjectNode::GetOutput()` (`ProjectNode.cpp:86`) now checks `is_field_exist(field_id)` before accessing each field. When a field is absent in an older segment (schema evolution), it creates an all-NULL ColumnVector (`valid_data` all false) instead of calling `bulk_subscript`. This ensures:
+1. The SortBuffer receives valid NULL markers for the missing ORDER BY column.
+2. `SortBuffer::CompareNulls` applies the configured NULLS FIRST/LAST semantics.
+3. The end-to-end behavior is: **missing ORDER BY fields due to schema evolution are treated as NULL and sorted according to nulls_first setting** (ASC → NULLS LAST, DESC → NULLS FIRST).
+
+#### Go layer consistency
+
+No additional alignment is needed on the Go side — the C++ layer guarantees that every segment result contains the same set of fields (with NULLs/defaults for missing ones). The OrderOperator then applies standard null comparison (NULLS LAST by default) to these synthesized values.
+
+**Consistency contract**: Both C++ (`SortBuffer::CompareNulls`) and Go (`OrderOperator.compareFieldValuesAt`) implement identical null-handling semantics — configurable per-field via `nulls_first`. The default follows SQL/PostgreSQL convention: ASC → NULLS LAST, DESC → NULLS FIRST (set by `ParseOrderByFields` in `types.go:141`). This ensures that segment-level sort order (C++) is compatible with the proxy-level global merge-sort (Go).
+
+### 12.4 Known Limitation: Missing Segcore Sort-Row Guard
+
+**WARNING (current behavior)**: There is currently no enforced segcore-side `max_sort_rows` guard in the ORDER BY path. SortBuffer may still attempt to materialize and sort a very large match set in memory before proxy-side controls can take effect.
+
+Implications:
+- `limit` is required and reduces output size, but does not cap the number of rows that must be considered during segment-local sorting.
+- Proxy-side `MaxOutputSize` cannot protect memory consumed inside segcore `SortBuffer`.
+- Queries with broad predicates (or no predicate) can trigger high memory usage or OOM risk at query execution time.
+
+Current mitigation:
+- Use selective predicates and conservative limits.
+- Avoid running unconstrained ORDER BY on high-cardinality collections during peak load.
+
+## 13. Follow-up Items
+
+### 13.1 Cross-Language Positional Layout: Design Trade-off and Testing
+
+The positional layout contract (Section 4.4) is implemented independently in two languages:
+- **C++**: `PlanProto.cpp` (`BuildOrderByProjectNode`) determines the column order `[pk, orderby_fields, (remaining if single-project), SegOffset]`.
+- **Go**: `BuildRemapIndices` in `remap_op.go` reconstructs the same layout to compute remap indices.
+
+**Risk**: If either side changes without updating the other, fields with the same data type silently map to wrong data — no error is raised, just incorrect values returned under correct field names. For example, two VARCHAR columns (name, category) could swap values without any type error or exception.
+
+**This is a deliberate design trade-off for simplicity.** The alternative — passing explicit field_id metadata per column through the pipeline — would add complexity to the segcore exec pipeline (RowContainer, SortBuffer, ColumnVector all operate on positional indices, not named columns). The positional contract keeps the C++ pipeline simple and allocation-free for metadata. The cost is that the Go remap logic must mirror the C++ layout logic exactly.
+
+**Mitigation**: End-to-end conformance tests should cover multiple field type combinations (fixed-width only, mixed fixed/variable-width, ORDER BY field overlapping output_fields, ORDER BY field not in output_fields) and verify final output by **value** — not just field names and types. These tests serve as the contract enforcement between the two languages.
+
+### 13.2 Other Non-Blocking Follow-ups
+
+- **Deterministic tie-order option**: Evaluate appending PK as an implicit final sort key for ORDER BY, and align behavior across segcore local sort and proxy global merge-sort.
+- **GROUP BY + ORDER BY detailed design**: Full specification of aggregation pipeline interaction (Section 10 is overview only).
+- **partial_sort threshold benchmark**: Determine optimal crossover point for TopK optimization (Section 4.5).
+- **Implement segcore `max_sort_rows` guard (fix for known limitation in Section 12.4)**: SortBuffer should enforce a configurable upper bound on sortable rows per segment and reject requests that exceed it, with a clear user-facing error.
+- **Deferred fetch optimization**: Explore lazy evaluation for two-project mode to avoid bulk-fetching all deferred fields when only a subset is needed.
@@ -0,0 +1,135 @@
+# MEP: Pinyin Filter for Text Analyzer
+
+- **Created:** 2026-02-09
+- **Author(s):** @aoiasd
+- **Status:** Implemented
+- **Component:** QueryNode/DataNode
+- **Related Issues:** [#45811](https://github.com/milvus-io/milvus/issues/45811)
+- **Released:** [TBD]
+
+## Summary
+
+Add a Pinyin filter to Milvus's tantivy-based text analyzer pipeline, enabling Chinese character tokens to be converted into their Pinyin (romanized) representations. This allows users to search Chinese text using Pinyin input, supporting common scenarios such as name lookup, autocomplete, and search-as-you-type for Chinese content.
+
+## Motivation
+
+Chinese text search in Milvus currently relies on tokenizers like Jieba for word segmentation, but there is no built-in way to search Chinese content using Pinyin input. This is a fundamental requirement for many Chinese-language applications:
+
+- **Name search:** Users frequently search for people or places by typing Pinyin instead of Chinese characters (e.g., typing "zhangsan" to find "张三").
+- **Autocomplete / search-as-you-type:** Input methods on most devices convert Pinyin keystrokes to Chinese characters, so supporting Pinyin matching enables faster, more natural search experiences.
+- **Cross-input-method search:** Users may not have a Chinese input method available and need to search using Latin characters.
+
+Without a Pinyin filter, users would need to maintain a separate Pinyin-mapped field or implement application-level conversion, adding complexity and overhead.
+
+## Public Interfaces
+
+The Pinyin filter is exposed as a new filter type `"pinyin"` in the analyzer configuration JSON. It can be used in any analyzer's filter pipeline.
+
+**Analyzer configuration example:**
+
+```json
+{
+  "tokenizer": "jieba",
+  "filter": [
+    {
+      "type": "pinyin",
+      "keep_original": true,
+      "keep_full_pinyin": true,
+      "keep_joined_full_pinyin": false,
+      "keep_separate_first_letter": false
+    }
+  ]
+}
+```
+
+**Filter parameters:**
+
+| Parameter | Type | Default | Description |
+|---|---|---|---|
+| `keep_original` | bool | `true` | Retain the original Chinese token in output |
+| `keep_full_pinyin` | bool | `true` | Emit individual Pinyin for each Chinese character (e.g., "中文" → "zhong", "wen") |
+| `keep_joined_full_pinyin` | bool | `false` | Emit concatenated Pinyin for the entire token (e.g., "中文" → "zhongwen") |
+| `keep_separate_first_letter` | bool | `false` | Emit concatenated first letters of each character's Pinyin (e.g., "中文" → "zw") |
+
+When used with no parameters (i.e., `"pinyin"` as a plain string filter), the default options apply.
+
+## Design Details
+
+### Implementation Location
+
+The filter is implemented in the tantivy-binding Rust crate, which provides the text analysis infrastructure for Milvus's full-text search:
+
+- `internal/core/thirdparty/tantivy/tantivy-binding/src/analyzer/filter/pinyin_filter.rs` — core filter implementation
+- `internal/core/thirdparty/tantivy/tantivy-binding/src/analyzer/filter/filter.rs` — registration in the filter dispatch system
+
+### Architecture
+
+The Pinyin filter follows the same pattern as existing tantivy-binding filters (RegexFilter, SynonymFilter, etc.):
+
+1. **`PinyinFilter`** — implements `tantivy::tokenizer::TokenFilter` trait, holds `PinyinOptions` configuration.
+2. **`PinyinFilterWrapper<T>`** — wraps an inner tokenizer, produced by `TokenFilter::transform()`.
+3. **`PinyinFilterStream<T>`** — the token stream that performs the actual conversion. Uses a cache-based approach to expand a single input token into multiple output tokens.
+
+### Token Expansion Logic
+
+For each incoming token from the upstream tokenizer:
+
+1. If `keep_original` is true, the original token is preserved in output.
+2. Each Chinese character in the token is converted to Pinyin using the `pinyin` crate (`ToPinyin` trait). Non-Chinese characters are skipped.
+3. Based on the configured options:
+   - `keep_full_pinyin`: Each character's Pinyin is emitted as a separate token.
+   - `keep_joined_full_pinyin`: All characters' Pinyin are concatenated into a single token.
+   - `keep_separate_first_letter`: The first letter of each character's Pinyin is concatenated into a single token.
+4. All generated tokens share the same `offset_from`, `offset_to`, and `position` as the original token.
+
+### Dependency
+
+The filter uses the [`pinyin`](https://crates.io/crates/pinyin) Rust crate (version 0.10) for Chinese-to-Pinyin conversion. This crate provides accurate conversion including tone-less plain Pinyin and first-letter extraction.
+
+### Filter Registration
+
+The `"pinyin"` filter type is registered in the `SystemFilter` enum alongside existing filters. It supports both:
+- **String shorthand:** `"pinyin"` (uses default options)
+- **JSON object:** `{"type": "pinyin", ...}` (with custom parameters)
+
+### Example Token Output
+
+Input text: "中文测试" with Jieba tokenizer (segments into "中文" and "测试"):
+
+| Configuration | Output tokens |
+|---|---|
+| `keep_original=true, keep_full_pinyin=true` | "中文", "zhong", "wen", "测试", "ce", "shi" |
+| `keep_original=true, keep_joined_full_pinyin=true` | "中文", "zhongwen", "测试", "ceshi" |
+| `keep_original=true, keep_separate_first_letter=true` | "中文", "zw", "测试", "cs" |
+| All options enabled | "中文", "zhong", "wen", "zhongwen", "zw", "测试", "ce", "shi", "ceshi", "cs" |
+
+## Compatibility, Deprecation, and Migration Plan
+
+- **Backward compatible:** This is a purely additive change. No existing analyzer configurations are affected.
+- **No migration needed:** Users opt in by adding the `"pinyin"` filter to their analyzer configuration.
+- **Tantivy binding dependency:** Adds `pinyin = "0.10"` to the tantivy-binding Cargo.toml. This increases the compiled binary size marginally.
+
+## Test Plan
+
+Unit tests are included in `pinyin_filter.rs` covering three scenarios:
+
+1. **Joined full Pinyin:** Verifies that `keep_joined_full_pinyin=true` produces concatenated Pinyin tokens ("zhongwen", "ceshi") for Jieba-segmented Chinese input.
+2. **Full Pinyin per character:** Verifies that `keep_full_pinyin=true` produces individual Pinyin tokens ("zhong", "wen", "ce", "shi").
+3. **First letter extraction:** Verifies that `keep_separate_first_letter=true` produces first-letter tokens ("zw", "cs").
+
+All tests use the Jieba tokenizer as the upstream tokenizer and verify output using subset matching.
+
+Integration/E2E tests should cover:
+- Creating a collection with a VARCHAR field using an analyzer configured with the Pinyin filter.
+- Inserting Chinese text and searching with Pinyin queries.
+- Verifying that both original Chinese and Pinyin queries return the expected results.
+
+## Rejected Alternatives
+
+- **Application-level Pinyin conversion:** Requires users to maintain separate Pinyin fields or pre-process data, adding complexity and storage overhead. An in-pipeline filter is more ergonomic and efficient.
+- **Standalone Pinyin tokenizer:** A filter-based approach is more composable — it can be combined with any tokenizer (Jieba, standard, etc.) and stacked with other filters (stop words, lowercase, etc.).
+
+## References
+
+- [`pinyin` Rust crate](https://crates.io/crates/pinyin) — Chinese-to-Pinyin conversion library
+- Tantivy tokenizer pipeline: `internal/core/thirdparty/tantivy/tantivy-binding/src/analyzer/`
@@ -0,0 +1,575 @@
+# Scalar Index V3 Format
+
+## 1. Goals
+
+| Goal | Description |
+|------|-------------|
+| Single file | All entries packed into one remote object |
+| Concurrent upload | Multipart Upload in parallel |
+| Concurrent encrypt/decrypt | Slice-level parallelism, same granularity as V2 |
+| Concurrent download | `ReadAt` multi-threaded parallel |
+| Low memory peak | `O(W × slice_size)`, does not grow with entry size |
+| Encryption transparent | Index classes unaware of encryption |
+| Reduce small IO | O(1) meta packed into a single entry |
+| Compatible with V2 | Go control plane routes by `SCALAR_INDEX_ENGINE_VERSION`; data lake by filename format |
+
+---
+
+## 2. File Format
+
+### 2.1 Layout
+
+```
+┌─────────────────────────────────────────┐
+│ Magic Number (8 bytes): "MVSIDXV3"       │
+├──────────── Data Region ────────────────┤
+│ Entry 0 data                             │
+│ ...                                      │
+│ Entry N data                             │
+│ Meta Entry data (always last entry)      │
+├──────────── Directory Table (JSON) ─────┤
+│ { "entries": [...] }                     │
+├──────────── Footer (32 bytes) ──────────┤
+│ version       (uint16)  — format version │
+│ reserved      (22 bytes) — zero-filled   │
+│ meta_entry_size   (uint32)               │
+│ directory_table_size (uint32)            │
+└─────────────────────────────────────────┘
+```
+
+Tail three sections can be located from the end of file:
+- Footer: last 32 bytes.
+- Directory Table: `directory_table_size` bytes before Footer.
+- Meta Entry: `meta_entry_size` bytes before Directory Table.
+
+Data is written first, Meta Entry written last, then Directory Table and Footer appended.
+
+### 2.2 Slice — Encryption Only
+
+Slice is the **encryption boundary**, not a general format concept:
+
+- **Encrypted**: Each entry split into fixed-size slices (default 16MB). Each slice independently encrypted via `IEncryptor::Encrypt()`. Slice boundaries recorded in Directory Table.
+- **Unencrypted**: Entry is contiguous plaintext. Directory Table records `offset` + `size`. EntryReader self-splits by range for concurrent `ReadAt`.
+
+### 2.3 Directory Table
+
+**Unencrypted**:
+
+```json
+{
+  "entries": [
+    {"name": "SORT_INDEX_META", "offset": 0, "size": 48, "crc32": "A1B2C3D4"},
+    {"name": "index_data", "offset": 48, "size": 100000000, "crc32": "E5F6A7B8"},
+    {"name": "__meta__", "offset": 100000048, "size": 64, "crc32": "C3D4E5F6"}
+  ]
+}
+```
+
+**Encrypted**:
+
+```json
+{
+  "slice_size": 16777216,
+  "entries": [
+    {
+      "name": "SORT_INDEX_META",
+      "original_size": 48,
+      "crc32": "A1B2C3D4",
+      "slices": [
+        {"offset": 0, "size": 76}
+      ]
+    },
+    {
+      "name": "index_data",
+      "original_size": 100000000,
+      "crc32": "E5F6A7B8",
+      "slices": [
+        {"offset": 76, "size": 16777244},
+        {"offset": 16777320, "size": 16777244}
+      ]
+    },
+    {
+      "name": "__meta__",
+      "original_size": 64,
+      "crc32": "C3D4E5F6",
+      "slices": [
+        {"offset": 33554564, "size": 92}
+      ]
+    }
+  ],
+  "__edek__": "base64_encoded_encrypted_dek",
+  "__ez_id__": "12345"
+}
+```
+
+Field semantics:
+- `version` is in the Footer (uint16), not in the Directory Table.
+- Unencrypted: `offset` + `size` are position within Data Region (relative to Magic end).
+- Encrypted: all entries use `original_size` + `slices[]` format uniformly.
+  - `original_size`: plaintext size, for pre-allocating buffer.
+  - `slices[]`: each slice's position and ciphertext size (including IV/Tag overhead) in Data Region.
+  - Even single-slice entries use `slices[]`.
+- `crc32`: CRC-32C (Castagnoli) checksum of the entry's **plaintext** data, stored as 8-char uppercase hex string. Present on every entry in both encrypted and unencrypted modes. Computed over the original data before encryption; verified after decryption (encrypted) or after read (unencrypted).
+- `__edek__` / `__ez_id__`: encryption metadata, present only when encrypted. `__ez_id__` stored as string.
+- `slice_size`: plaintext slice size, present only when encrypted.
+
+**Detection**: `__edek__` present → encrypted; absent → unencrypted.
+
+### 2.4 Meta Entry
+
+The Meta Entry is the **last entry** in the Data Region, always named `__meta__`. Physically it is a regular entry — same write, read, encryption treatment as all other entries. It is listed in the Directory Table like any other entry.
+
+Its position can also be located directly from the Footer (`meta_entry_size` bytes before Directory Table), enabling a single tail read to fetch Footer + Directory Table + Meta Entry without scanning the full directory.
+
+Content is a JSON object. Initial fields:
+
+```json
+{
+  "index_type": "stlsort",
+  "build_id": 12345
+}
+```
+
+The schema is extensible — new fields can be added without format version change.
+
+### 2.5 Footer
+
+Fixed 32 bytes, always the last 32 bytes of the file:
+
+```
+┌──────────────────────────────────────────┐
+│ Offset  Size   Field                      │
+├──────────────────────────────────────────┤
+│  0      2B     version (uint16, LE)       │
+│  2      22B    reserved (zero-filled)     │
+│  24     4B     meta_entry_size (uint32, LE)│
+│  28     4B     directory_table_size (uint32, LE)│
+└──────────────────────────────────────────┘
+```
+
+- `version`: format version, currently `3`.
+- `reserved`: zero-filled, available for future use.
+- `meta_entry_size`: on-disk byte size of the Meta Entry (ciphertext size if encrypted, plaintext size if not).
+- `directory_table_size`: byte size of the Directory Table JSON.
+
+**Read sequence from tail**:
+1. Read last 32 bytes → parse Footer → get `version`, `meta_entry_size`, `directory_table_size`.
+2. Compute `tail_size = 32 + directory_table_size + meta_entry_size`.
+3. If `tail_size ≤ initial_tail_read_size`, a single initial tail read already covers all three sections. `initial_tail_read_size` is a Milvus runtime config (default 64KB), not part of the file format.
+
+---
+
+## 3. Upload Path
+
+### 3.1 IndexEntryWriter Interface
+
+Index classes only see this interface, unaware of encryption and transport:
+
+```cpp
+// storage/IndexEntryWriter.h
+namespace milvus::storage {
+
+constexpr char MILVUS_V3_MAGIC[] = "MVSIDXV3";
+constexpr size_t MILVUS_V3_MAGIC_SIZE = 8;
+
+class IndexEntryWriter {
+public:
+    virtual ~IndexEntryWriter() = default;
+    virtual void WriteEntry(const std::string& name, const void* data, size_t size) = 0;
+    virtual void WriteEntry(const std::string& name, int fd, size_t size) = 0;
+    // Set file-level metadata (written as the __meta__ entry during Finish).
+    virtual void SetMeta(const std::string& json) = 0;
+    virtual void Finish() = 0;
+    virtual size_t GetTotalBytesWritten() const = 0;
+};
+}
+```
+
+**Entry name uniqueness**: `CheckDuplicateName` is in the base class, inherited by both subclasses. Duplicate names throw immediately.
+
+### 3.2 Two Implementations
+
+```
+CreateIndexEntryWriterV3(filename, is_index_file)
+    ├─ No encryption → IndexEntryDirectStreamWriter → direct write RemoteOutputStream
+    └─ Encryption    → IndexEntryEncryptedLocalWriter → local temp file → upload
+```
+
+**Why two paths**:
+- Unencrypted: data size predictable, write directly to remote stream. `RemoteOutputStream` internal `background_writes` handles parallel upload.
+- Encrypted: ciphertext size unpredictable. Write to local file (>1GB/s, no network backpressure blocking encryption pipeline). Local file also provides task-level retry capability.
+
+### 3.3 IndexEntryDirectStreamWriter (Unencrypted)
+
+```
+Index → WriteEntry → RemoteOutputStream → background_writes → S3 Multipart
+```
+
+```cpp
+// storage/IndexEntryDirectStreamWriter.h
+class IndexEntryDirectStreamWriter : public IndexEntryWriter {
+
+    explicit IndexEntryDirectStreamWriter(
+        std::shared_ptr<milvus::OutputStream> output,
+        size_t read_buf_size = 16 * 1024 * 1024);
+
+    void WriteEntry(const std::string& name, const void* data, size_t size) override;
+    void WriteEntry(const std::string& name, int fd, size_t size) override;
+    void Finish() override;
+    size_t GetTotalBytesWritten() const override;
+};
+```
+
+Constructor writes Magic. `WriteEntry()` computes CRC-32C incrementally (`crc32c_update` per chunk as data streams through) and records the final checksum in the directory entry. For fd-based entries, CRC is updated per 16MB read chunk — no extra memory. `Finish()` writes the Meta Entry (as the last regular entry), then Directory Table JSON, then 32-byte Footer (`version` + `meta_entry_size` + `directory_table_size`), computes `total_bytes_written_`, closes stream.
+
+### 3.4 IndexEntryEncryptedLocalWriter (Encrypted)
+
+```
+Index → WriteEntry → encrypt thread pool (parallel) → ordered queue → sequential write local file
+                                                            ↓ Finish()
+                                                 upload local file → S3
+```
+
+Encryption pipeline uses sliding window: W slices encrypt in parallel, dequeued in submission order, written sequentially to local file. Offset from actual `encrypted.size()` increment, always correct. CRC-32C is computed incrementally by the main thread: each slice's plaintext is fed to `crc32c_update` before submitting to the encryption thread pool. Sequential read order guarantees correctness, no extra memory.
+
+**Thread pool**: Reuses V2 global priority thread pool `ThreadPools::GetThreadPool(MIDDLE)`.
+
+**Thread safety**: Each slice encryption task creates its own `IEncryptor` inside the lambda, destroyed after use. No sharing, no caching — global thread pool threads outlive Writer, caching would leave stale edek.
+
+```cpp
+// storage/IndexEntryEncryptedLocalWriter.h
+class IndexEntryEncryptedLocalWriter : public IndexEntryWriter {
+public:
+    IndexEntryEncryptedLocalWriter(const std::string& remote_path,
+                         milvus_storage::ArrowFileSystemPtr fs,
+                         std::shared_ptr<plugin::ICipherPlugin> cipher_plugin,
+                         int64_t ez_id, int64_t collection_id,
+                         const std::string& temp_dir = "",
+                         size_t slice_size = 16 * 1024 * 1024);
+    ~IndexEntryEncryptedLocalWriter();
+
+    void WriteEntry(const std::string& name, const void* data, size_t size) override;
+    void WriteEntry(const std::string& name, int fd, size_t size) override;
+    void Finish() override;
+    size_t GetTotalBytesWritten() const override;
+
+private:
+    void EncryptAndWriteSlices(const std::string& name, uint64_t original_size,
+                               const uint8_t* data, size_t size);
+    void UploadLocalFile();
+};
+```
+
+Constructor obtains edek, creates temp file at `<temp_dir>/milvus_enc_<uuid>`, writes Magic. `temp_dir` defaults to the Milvus config value `localStorage.path` (typically `/var/lib/milvus/data`); if that is empty, falls back to `/tmp`. Destructor cleans up fd and temp file. `Finish()` writes Meta Entry (encrypted, as last entry), then Directory Table (with `__edek__`, `__ez_id__`, `slice_size`), then 32-byte Footer, closes fd, uploads local file via `UploadLocalFile()`, unlinks temp file.
+
+### 3.5 Factory Method
+
+```cpp
+// In FileManagerImpl (storage/FileManager.h)
+std::unique_ptr<IndexEntryWriter>
+FileManagerImpl::CreateIndexEntryWriterV3(const std::string& filename,
+                                          bool is_index_file = true) {
+    if (plugin_context_) {
+        auto cipher_plugin = PluginLoader::GetInstance().getCipherPlugin();
+        if (cipher_plugin) {
+            auto remote_path = is_index_file ? GetRemoteIndexObjectPrefixV2()
+                                             : GetRemoteTextLogPrefixV2();
+            remote_path += "/" + GetFileName(filename);
+            return std::make_unique<IndexEntryEncryptedLocalWriter>(
+                remote_path, fs_, cipher_plugin,
+                plugin_context_->ez_id, plugin_context_->collection_id,
+                GetLocalTempDir());
+        }
+    }
+    return std::make_unique<IndexEntryDirectStreamWriter>(
+        OpenOutputStream(filename, is_index_file));
+}
+```
+
+---
+
+## 4. Load Path
+
+### 4.1 IndexEntryReader Interface
+
+```cpp
+// storage/IndexEntryReader.h
+struct Entry {
+    std::vector<uint8_t> data;
+};
+
+class IndexEntryReader {
+public:
+    static std::unique_ptr<IndexEntryReader> Open(
+        std::shared_ptr<milvus::InputStream> input,
+        int64_t file_size,
+        int64_t collection_id = 0,
+        ThreadPoolPriority priority = ThreadPoolPriority::HIGH);
+
+    std::vector<std::string> GetEntryNames() const;
+    Entry ReadEntry(const std::string& name);
+    void ReadEntryToFile(const std::string& name, const std::string& local_path);
+    void ReadEntriesToFiles(
+        const std::vector<std::pair<std::string, std::string>>& name_path_pairs);
+
+private:
+    // Small entries (≤ 1MB) are cached after first read
+    static constexpr size_t kSmallEntryCacheThreshold = 1 * 1024 * 1024;
+    std::unordered_map<std::string, Entry> small_entry_cache_;
+};
+```
+
+**Encryption plugin**: `Open()` does not take `ICipherPlugin` explicitly. If Directory Table contains `__edek__`, `IndexEntryReader` obtains the global cipher plugin via `PluginLoader::GetInstance().getCipherPlugin()`. Decryption uses `collection_id`, `ez_id` (from Directory Table), and `edek` to create `IDecryptor`.
+
+**Thread pool**: Reuses V2 global priority pool `ThreadPools::GetThreadPool(priority)`. Default `HIGH`.
+
+### 4.2 Load Flow
+
+1. **Open()**: Validates Magic (1 ReadAt). Reads file tail (up to `initial_tail_read_size` in one ReadAt, default 64KB, configurable) to get Footer + Directory Table + Meta Entry. Parses 32-byte Footer to get `version`, `meta_entry_size`, `directory_table_size`. If `meta_entry_size + directory_table_size + 32` exceeds the initial read, does one additional ReadAt. Parses Directory Table JSON, builds `name → EntryMeta` index. If `__edek__` present, obtains cipher plugin. Meta Entry (`__meta__`) is accessible via normal `ReadEntry("__meta__")`.
+2. **ReadEntry(name)** (small entry): Checks `small_entry_cache_` first. One `ReadAt`; if encrypted, creates `IDecryptor` to decrypt. Verifies CRC-32C against the `crc32` field in Directory Table; throws on mismatch. Result cached if ≤ 1MB.
+3. **ReadEntry(name)** (large entry, to memory):
+   - Encrypted: Thread pool concurrent `ReadAt` for each slice, each task creates own `IDecryptor`, decrypts, copies to target buffer. After all slices assembled, verifies CRC-32C over the full plaintext buffer; throws on mismatch.
+   - Unencrypted: Self-splits by 16MB ranges, concurrent `ReadAt`, copies to target buffer. After all ranges assembled, verifies CRC-32C over the full buffer; throws on mismatch.
+   - CRC verification is a single sequential pass over the already-assembled buffer. No extra memory.
+4. **ReadEntryToFile(name, path)**: Same concurrent read but output via `pwrite` to local file. `pwrite` is lock-free (each range writes different offset). Encrypted path does `ftruncate` first. CRC verified via per-range `crc32c_combine` (see §6.1); no need to re-read the file.
+5. **ReadEntriesToFiles(pairs)**: Submits all entries to the thread pool concurrently — each entry's ranges/slices are read in parallel, and multiple entries are also processed in parallel. Cross-entry concurrency shares the same thread pool; total in-flight tasks bounded by pool size.
+
+**Thread safety**: Each decryption task creates own `IDecryptor` in lambda, destroyed after use.
+
+### 4.3 IO Counts
+
+| Step | IO Count |
+|------|----------|
+| Read Footer + Directory Table + Meta Entry | 1–2 (tail read) |
+| Read meta entry | 1 (all O(1) meta packed) |
+| Read large data entry | Concurrent, by slice/range count |
+
+---
+
+## 5. Index Class Interface
+
+### 5.1 Base Class
+
+```cpp
+// index/ScalarIndex.h
+template <typename T>
+class ScalarIndex : public IndexBase {
+public:
+    // V3 streaming upload — calls WriteEntries(), returns IndexStats
+    IndexStatsPtr UploadV3(const Config& config) override;
+
+    // V3 streaming load — opens packed file, calls LoadEntries()
+    void LoadV3(const Config& config) override;
+
+    virtual void WriteEntries(storage::IndexEntryWriter* writer) {
+        ThrowInfo(Unsupported, "WriteEntries is not implemented");
+    }
+    virtual void LoadEntries(storage::IndexEntryReader& reader, const Config& config) {
+        ThrowInfo(Unsupported, "LoadEntries is not implemented");
+    }
+
+protected:
+    storage::MemFileManagerImplPtr file_manager_;
+
+    // Controls remote path prefix for V3 upload/load:
+    //   true  → index_files/...  (default, normal scalar indexes)
+    //   false → text_log/...     (TextMatchIndex)
+    bool is_index_file_ = true;
+};
+```
+
+**UploadV3 flow**:
+1. Build filename: `milvus_packed_<type>_index.v3` (type from `GetIndexType()`, lowercased).
+2. Create writer via `file_manager_->CreateIndexEntryWriterV3(filename, is_index_file_)`.
+3. Call `WriteEntries(writer.get())`.
+4. Call `writer->SetMeta(json)` with file-level metadata (index type, build id, etc.).
+5. Call `writer->Finish()` — writes `__meta__` entry, Directory Table, Footer.
+6. Return `IndexStats` with the single packed file path and `writer->GetTotalBytesWritten()`.
+
+**LoadV3 flow**:
+1. Read `INDEX_FILES` from config — must contain exactly 1 packed file path.
+2. Open `InputStream` via `file_manager_->OpenInputStream(packed_file, is_index_file_)`.
+3. Create `IndexEntryReader::Open(input, file_size, collection_id)`.
+4. Call `LoadEntries(*reader, config)`.
+
+### 5.2 Meta Packing
+
+Each index packs all O(1) metadata into a **single meta entry**, eliminating multiple small IOs by design.
+
+| Index | Meta Entry | Data Entries |
+|-------|-----------|-------------|
+| ScalarIndexSort | `SORT_INDEX_META` | `index_data` |
+| BitmapIndex | `BITMAP_INDEX_META` | `BITMAP_INDEX_DATA` |
+| StringIndexMarisa | (none) | `MARISA_TRIE_INDEX`, `MARISA_STR_IDS` |
+| StringIndexSort | `STRING_SORT_META` | `index_data`, `valid_bitset` |
+| InvertedIndexTantivy | `TANTIVY_META` | N tantivy files, `index_null_offset` |
+| JsonInvertedIndex | `TANTIVY_META` (with `has_non_exist`) | N tantivy files, `index_null_offset`, `json_index_non_exist_offsets` |
+| NgramInvertedIndex | `TANTIVY_META` | N tantivy files, `index_null_offset`, `ngram_avg_row_size` |
+| RTreeIndex | `RTREE_META` | N rtree files, `index_null_offset` |
+| HybridScalarIndex | `HYBRID_INDEX_META` | (delegated to internal index) |
+
+Meta structure definitions:
+
+```cpp
+// ScalarIndexSort:  {"index_length": N, "num_rows": N, "is_nested": bool}
+// BitmapIndex:      {"bitmap_index_length": N, "bitmap_index_num_rows": N}
+// StringIndexSort:  {"version": V, "num_rows": N, "is_nested": bool}
+// Tantivy:          {"file_names": [...], "has_null": bool}
+// Tantivy (Json):   {"file_names": [...], "has_null": bool, "has_non_exist": bool}
+// RTree:            {"file_names": [...], "has_null": bool}
+// Hybrid:           {"index_type": uint8}
+```
+
+### 5.3 Inheritance Patterns
+
+**InvertedIndexTantivy** provides `BuildTantivyMeta()` as a virtual hook. Subclasses extend by:
+
+| Subclass | Pattern |
+|----------|---------|
+| JsonInvertedIndex | Overrides `BuildTantivyMeta` to add `has_non_exist`. Overrides `WriteEntries`/`LoadEntries`, calling parent first then adding extra entries. |
+| NgramInvertedIndex | Overrides `WriteEntries`/`LoadEntries`, calling parent first then adding `ngram_avg_row_size`. |
+
+This keeps parent meta fields (`file_names`, `has_null`) in the single `TANTIVY_META` entry, one IO. Subclass-specific entries are separate.
+
+---
+
+## 6. Concurrency Model
+
+All thread pools reuse V2 `ThreadPools::GetThreadPool(priority)`, no custom pools.
+
+| Path | Thread Pool | V2 Equivalent |
+|------|------------|---------------|
+| Encrypted upload | `MIDDLE` (`MIDD_SEGC_POOL`) | V2 `PutIndexData` |
+| Download/decrypt | Caller-specified, default `HIGH` (`HIGH_SEGC_POOL`) | V2 `GetObjectData` |
+| Unencrypted upload | None (RemoteOutputStream internal `background_writes`) | — |
+
+### Upload — Unencrypted
+
+```
+Main thread: [read chunk → crc32c_update → Write to RemoteOutputStream] ──→ sequential fill
+RemoteOutputStream:  background_writes parallel upload Part 0, 1, 2...  ──→ S3
+```
+
+### Upload — Encrypted
+
+```
+Main thread:  read slice_i → crc32c_update(crc, slice_i) → submit encrypt(slice_i)
+MIDD_SEGC_POOL (W threads):  parallel Encrypt slice 0, 1, 2...
+Main thread:  in-order .get() → sequential write to local file (>1GB/s)
+    ↓ Finish()
+Upload local file via RemoteOutputStream → S3
+```
+
+CRC-32C is computed in the main thread's sequential read loop, before encryption submission. No impact on encryption parallelism.
+
+### Download/Load (Unified)
+
+```
+HIGH_SEGC_POOL (N threads, or caller-specified priority):
+  Thread i: ReadAt(range_i) → [Decrypt*] → output to memory or pwrite to file
+  * Decrypt only when encrypted
+  "range" = encrypted: slice boundaries; unencrypted: 16MB self-split
+```
+
+### 6.1 CRC-32C Verification in Parallel Download
+
+`crc32c_update(crc, data, len)` is chain-dependent — each call requires the previous CRC value, so it cannot run out of order. For **ReadEntry to memory**, the full plaintext buffer is already assembled after parallel download; CRC is verified via a single sequential pass.
+
+For **ReadEntryToFile**, data is written to file via `pwrite` without passing through a single buffer. CRC is verified using `crc32c_combine`:
+
+```
+Thread 0: ReadAt(range_0) → [Decrypt] → pwrite → crc_0 = crc32c(0, range_0_data, len_0)
+Thread 1: ReadAt(range_1) → [Decrypt] → pwrite → crc_1 = crc32c(0, range_1_data, len_1)
+Thread 2: ReadAt(range_2) → [Decrypt] → pwrite → crc_2 = crc32c(0, range_2_data, len_2)
+  (all threads run in parallel, compute per-range CRC independently)
+
+Main thread (after all tasks complete, combine in sequential order):
+  crc = crc32c_combine(crc_0, crc_1, len_1)
+  crc = crc32c_combine(crc,   crc_2, len_2)
+  verify crc == expected
+```
+
+**Key properties**:
+- Each thread computes its range CRC independently over data already in memory (the read buffer before `pwrite`). No extra memory.
+- `crc32c_combine(crc_a, crc_b, len_b)` merges two CRCs algebraically (GF(2) matrix exponentiation, `O(log(len_b))`), without accessing original data.
+- Per-range CRC **computation is parallel and order-independent**; only the final **combine must be in data order**.
+- Hardware-accelerated CRC-32C (SSE4.2 / ARMv8) runs at ~30 GB/s, negligible vs encryption (~5-10 GB/s) and network IO.
+
+---
+
+## 7. Memory Peak
+
+| Scenario | Peak |
+|----------|------|
+| Upload, unencrypted, memory entry | entry itself |
+| Upload, unencrypted, disk entry | 1 × `read_buf_size` (16MB) |
+| Upload, encrypted, memory entry | entry + W × `slice_size` |
+| Upload, encrypted, disk entry | W × `slice_size` × 2 |
+| Download, to memory | N × `range_size` + `original_size` |
+| Download, to file | N × `range_size` (reusable) |
+
+Consistent with V2: peak determined by concurrency × slice size, does not grow with entry size.
+
+CRC-32C adds no extra memory: upload path uses incremental `crc32c_update` on data already being read; download-to-memory verifies over the output buffer; download-to-file computes per-range CRC on the existing read buffer and combines via `crc32c_combine`.
+
+---
+
+## 8. Compatibility
+
+### Version Routing
+
+V3 does **not** detect version by reading file Magic Number at load time. Version routing is entirely caller-side:
+
+1. **Build side**: `ScalarIndexCreator::Upload()` checks `SCALAR_INDEX_ENGINE_VERSION` from config. If `>= 3`, calls `index->UploadV3(config)`; otherwise calls `index->Upload(config)`.
+2. **Load side**: `SealedIndexTranslator` checks `scalar_index_engine_version` from config. If `>= 3` and not vector type, calls `index->LoadV3(config)`; otherwise calls `index->Load(ctx, config)`.
+3. **Go control plane**: `CurrentScalarIndexEngineVersion` in `pkg/common/common.go` (currently `2`). Bump to `3` to enable V3 for new index builds.
+4. **UT routing**: `kScalarIndexUseV3` global bool (default `false`). When set to `true`, individual index `Upload()` and `Load()` methods internally redirect to `UploadV3()`/`LoadV3()`. This flag is for unit testing only; production routing uses the caller-side config.
+
+### File Naming
+
+V3 packed file name format: `milvus_packed_<type>_index.v3`, where `<type>` is the lowercased `ScalarIndexType` string (e.g., `stlsort`, `bitmap`, `marisa`, `inverted`, `hybrid`, `rtree`, `ngram`).
+
+### Remote Paths
+
+V3 keeps V2 remote root directory classification unchanged:
+
+| Root | Index Type | V3 Path |
+|------|-----------|---------|
+| `index_files/` | Normal scalar indexes, InvertedIndexTantivy, etc. | `.../milvus_packed_<type>_index.v3` |
+| `text_log/` | TextMatchIndex | `.../milvus_packed_<type>_index.v3` |
+
+`FileManagerImpl` has `OpenOutputStream(filename, bool is_index_file)` and `CreateIndexEntryWriterV3(filename, bool is_index_file)`. Default `is_index_file=true`. TextMatchIndex sets `is_index_file_ = false` on its `ScalarIndex` base.
+
+### Backward Compatibility
+
+- V2 `Upload()` and `Load()` methods retained, not deleted.
+- Rollback: set `CurrentScalarIndexEngineVersion` back to `2`. No code change needed.
+
+---
+
+## 9. Open Questions
+
+### 9.1 Why not encode the index into existing file format, like Lance?
+
+Lance is a columnar storage format that stores data in a columnar format. Any single Lance file has its own schema, and multiple Lance files form a table format.
+
+In the end of the day, Lance still follows a tabular format design, while using a columnar storage.
+
+#### How does Lance store an index?
+
+In most cases, indexes would have their own specialized data structure, algorithm, etc. Those are highly optimized for the index type, and may not fit easily into a tabular format.
+
+Lance addresses this by storing indexes in **separate Lance files** (typically `index.idx` + `auxiliary.idx`) that are still columnar under the hood. These files use Lance's own Arrow-based schema tailored to the index type:
+
+- For vector indexes (e.g. IVF_PQ), the index file stores graph/structure (e.g. IVF partitions, HNSW neighbors), while the auxiliary file stores quantized vector codes, both organized as columns with `_rowid`, codes, distances, etc.
+- Even scalar indexes (bitmap, BTree-like) are encoded into columnar pages with metadata in Protobuf.
+
+This approach works well for Lance's ecosystem — it reuses the same file format, zero-copy reads, and manifest versioning. However, it comes with trade-offs:
+
+- **Forced columnar mapping** — arbitrary graph structures, tree nodes, hash tables, or custom layouts must be “flattened” into columns (e.g. lists/arrays), which can introduce overhead in encoding, padding, or indirection.
+- **Schema rigidity** — every index type needs a custom Arrow schema + Protobuf metadata, limiting flexibility if you want to support diverse or evolving index algorithms.
+- **Not truly generic** — third-party indexes (e.g. modified Faiss, custom implementations) are hard to import directly without re-implementation in Lance's layout.
+
+For a **generic scalar index format** that aims to support many different scalar index types (bitmap, trie, inverted list, RTree, etc.) with minimal adaptation cost, a **pure KV style** (key → serialized index blob + metadata) avoids these constraints entirely.
+
+It trades some of Lance's columnar optimizations for maximum flexibility and easier integration with existing index libraries.
@@ -0,0 +1,380 @@
+# Milvus Storage Manifest Format
+
+- **Created:** 2026-02-26
+- **Author(s):** @tedxu
+- **Status:** Implemented
+- **Component:** Storage
+- **Related Issues:** milvus-io/milvus-storage#406
+
+## Summary
+
+The manifest is the core metadata format in milvus-storage that tracks the state of a dataset. Each manifest version is an immutable snapshot describing which column groups, delta logs, statistics, and indexes comprise the dataset at a given point in time. Manifests are serialized using Apache Avro binary encoding and stored as versioned files on the filesystem.
+
+## Motivation
+
+Milvus Storage is a multi-format columnar storage engine that supports concurrent reads and writes across multiple storage backends (local, S3, GCS, Azure, etc.). The system needs a lightweight, versioned metadata layer that:
+
+1. **Tracks dataset composition**: which data files, column groups, and formats are in use
+2. **Supports transactional updates**: enables atomic commits with conflict resolution
+3. **Records auxiliary metadata**: delta logs for deletes, statistics for query optimization, and index references for accelerated search
+4. **Works across storage backends**: serializes compactly and uses relative paths for portability
+5. **Evolves without breaking readers**: older manifest versions remain readable
+
+---
+
+## 1. Directory Layout
+
+All files for a dataset are organized under a base directory:
+
+```
+base_dir/
+├── _metadata/                         # Manifest metadata directory
+│   └── manifest-{version}.avro       # Manifest files (e.g., manifest-1.avro)
+├── _data/                             # Column group data files
+│   └── {group_id}_{uuid}.{format}    # Data files (Parquet, Vortex, etc.)
+├── _delta/                            # Delta log files
+│   └── {delta_log_files}             # Delete operation logs
+├── _stats/                            # Statistics files
+│   └── {stats_files}                 # Bloom filters, BM25, etc.
+└── _index/                            # Index files
+    └── {index_files}                  # HNSW, IVF, bitmap, etc.
+```
+
+Manifest file naming follows the pattern `manifest-{version}.avro` where version is an integer starting from 1 and incrementing with each commit. The latest version is determined by scanning the `_metadata/` directory for the highest numbered file.
+
+---
+
+## 2. Binary Format
+
+Manifests use Apache Avro binary encoding. The on-disk byte layout is:
+
+```
+-------------------+-------------------+
+| MAGIC (int32)     | VERSION (int32)   |
+| 0x4D494C56        | 1 or 2            |
+-------------------+-------------------+
+| COLUMN_GROUPS     | DELTA_LOGS        |
+| (avro array)      | (avro array)      |
+-------------------+-------------------+
+| STATS             | INDEXES           |
+| (avro map)        | (avro array,      |
+|                   |  version 2+ only) |
+-------------------+-------------------+
+```
+
+### 2.1 Header
+
+| Field | Type | Value | Description |
+|-------|------|-------|-------------|
+| Magic | int32 | `0x4D494C56` | ASCII "MILV", identifies valid manifest files |
+| Version | int32 | 1 or 2 | Manifest format version |
+
+### 2.2 Version History
+
+| Version | Description |
+|---------|-------------|
+| 1 | Initial format: column_groups, delta_logs, stats |
+| 2 | Added indexes field for index metadata (current) |
+
+Backward compatibility: version 1 manifests are readable by version 2 code; the indexes field is treated as empty. Forward compatibility is not supported; versions greater than `MANIFEST_VERSION` fail with an error.
+
+---
+
+## 3. Data Structures
+
+### 3.1 ColumnGroupFile
+
+Represents a single physical file within a column group.
+
+```cpp
+struct ColumnGroupFile {
+  std::string path;               // File path (relative within _data/)
+  int64_t start_index;            // Start row index in the file (inclusive)
+  int64_t end_index;              // End row index in the file (exclusive)
+  std::vector<uint8_t> metadata;  // Optional metadata (e.g., external table info)
+};
+```
+
+Avro encoding order: `path`, `start_index`, `end_index`, `metadata`.
+
+### 3.2 ColumnGroup
+
+A set of related columns stored together in the same physical file(s).
+
+```cpp
+struct ColumnGroup {
+  std::vector<std::string> columns;    // Column names in this group
+  std::string format;                  // Storage format
+  std::vector<ColumnGroupFile> files;  // Physical file references
+};
+```
+
+Avro encoding order: `columns`, `files`, `format`.
+
+Supported formats:
+
+| Format | Description |
+|--------|-------------|
+| `parquet` | Apache Parquet columnar format |
+| `vortex` | Vortex compressed format |
+| `lance` | Lance format (read-only) |
+| `binary` | Raw binary format |
+
+### 3.3 DeltaLog
+
+Records delete operations against the dataset.
+
+```cpp
+struct DeltaLog {
+  std::string path;     // File path (relative within _delta/)
+  DeltaLogType type;    // Delete type (encoded as int32)
+  int64_t num_entries;  // Number of delete entries
+};
+```
+
+Avro encoding order: `path`, `type` (as int32), `num_entries`.
+
+| DeltaLogType | Value | Description |
+|--------------|-------|-------------|
+| PRIMARY_KEY | 0 | Delete by primary key |
+| POSITIONAL | 1 | Delete by row position |
+| EQUALITY | 2 | Delete by equality predicate |
+
+### 3.4 Stats
+
+A map from stat name to a list of file paths. Each key identifies a type of statistic (e.g., bloom filter, BM25) and the values are relative paths within `_stats/`.
+
+```
+std::map<std::string, std::vector<std::string>> stats;
+```
+
+### 3.5 Index (Version 2+)
+
+Metadata for a column index.
+
+```cpp
+struct Index {
+  std::string column_name;                         // Column this index is on
+  std::string index_type;                          // Index algorithm
+  std::string path;                                // File path (relative within _index/)
+  std::map<std::string, std::string> properties;   // Index-specific parameters
+};
+```
+
+Avro encoding order: `column_name`, `index_type`, `path`, `properties`.
+
+Supported index types:
+
+| Index Type | Description |
+|------------|-------------|
+| `hnsw` | Hierarchical Navigable Small World graph |
+| `ivf-sq` | IVF with scalar quantization |
+| `ivf-pq` | IVF with product quantization |
+| `inverted` | Inverted index for scalar fields |
+| `bitmap` | Bitmap index for low-cardinality fields |
+| `ordered` | Ordered index for range queries |
+
+The `properties` map is intentionally flexible. Common keys include:
+
+| Key | Example | Description |
+|-----|---------|-------------|
+| `index_id` | `"42"` | Unique index identifier |
+| `version` | `"1"` | Index build version |
+| `M` | `"16"` | HNSW max connections per layer |
+| `efConstruction` | `"128"` | HNSW construction quality parameter |
+| `metric_type` | `"L2"` | Distance metric |
+| `num_rows` | `"1000000"` | Number of indexed rows |
+| `index_size` | `"134217728"` | Index file size in bytes |
+
+---
+
+## 4. Path Handling
+
+All paths in the manifest are stored as **relative paths** within their respective subdirectories. During serialization, absolute paths are stripped of the base directory prefix. During deserialization, relative paths are reconstructed as absolute paths.
+
+```
+Serialization:   /data/base/_data/0_abc.parquet  →  0_abc.parquet
+Deserialization:  0_abc.parquet  →  /data/base/_data/0_abc.parquet
+```
+
+**External table exception**: paths that contain a URI scheme (e.g., `s3://bucket/file.parquet`, `gs://bucket/file.parquet`) are kept as absolute paths. This allows external table column groups to reference files in external storage systems.
+
+---
+
+## 5. Transaction System
+
+Manifests are updated through a transaction system that provides atomic commits and conflict resolution.
+
+### 5.1 Transaction Lifecycle
+
+```
+Open(fs, base_path, version)
+        |
+        v
+  Read manifest at version
+  (or empty manifest if version=0)
+        |
+        v
+  Record updates via fluent API:
+    .AddColumnGroup(cg)
+    .AppendFiles(cgs)
+    .AddDeltaLog(delta)
+    .UpdateStat(key, files)
+    .AddIndex(idx)
+    .DropIndex(col, type)
+        |
+        v
+  Commit()
+    1. Read latest manifest (seen_manifest)
+    2. Invoke resolver(read, read_ver, seen, seen_ver, updates)
+    3. Write resolved manifest as manifest-{seen_ver+1}.avro
+    4. On conflict, retry up to retry_limit times
+        |
+        v
+  Return committed version
+```
+
+### 5.2 Updates
+
+The `Updates` class tracks all changes made during a transaction:
+
+| Operation | Description |
+|-----------|-------------|
+| `AddColumnGroup` | Add a new column group |
+| `AppendFiles` | Append files to existing column groups |
+| `AddDeltaLog` | Add a delta log entry |
+| `UpdateStat` | Add or replace a stat entry |
+| `AddIndex` | Add or replace an index (keyed by column_name + index_type) |
+| `DropIndex` | Remove an index by column_name + index_type |
+
+### 5.3 Conflict Resolution
+
+Three built-in resolvers handle concurrent commits:
+
+| Resolver | Behavior |
+|----------|----------|
+| `FailResolver` | Fails if `read_version != seen_version` (strict serialization) |
+| `MergeResolver` | Applies updates to `seen_manifest` (merges concurrent changes) |
+| `OverwriteResolver` | Applies updates to `read_manifest` (ignores concurrent changes) |
+
+### 5.4 Index Deprecation
+
+When files are appended to existing column groups via `AppendFiles`, indexes on affected columns are **automatically dropped**. This is because appending new data invalidates existing index structures. Other operations (delta logs, stats updates) do not affect indexes.
+
+```
+Transaction: AppendFiles to column group containing "embedding"
+  → Any index with column_name="embedding" is silently removed
+```
+
+---
+
+## 6. Manifest API
+
+### 6.1 Manifest Class
+
+```cpp
+class Manifest final {
+  // Construction
+  explicit Manifest(ColumnGroups column_groups = {},
+                    const std::vector<DeltaLog>& delta_logs = {},
+                    const std::map<std::string, std::vector<std::string>>& stats = {},
+                    const std::vector<Index>& indexes = {},
+                    uint32_t version = MANIFEST_VERSION);
+
+  // Serialization
+  arrow::Status serialize(std::ostream& output,
+                          const std::optional<std::string>& base_path = std::nullopt) const;
+  arrow::Status deserialize(std::istream& input,
+                            const std::optional<std::string>& base_path = std::nullopt);
+
+  // Accessors
+  ColumnGroups& columnGroups();
+  std::shared_ptr<ColumnGroup> getColumnGroup(const std::string& column_name) const;
+  std::vector<DeltaLog>& deltaLogs();
+  std::map<std::string, std::vector<std::string>>& stats();
+  std::vector<Index>& indexes();
+  const Index* getIndex(const std::string& column_name,
+                        const std::string& index_type) const;
+  int32_t version() const;
+};
+```
+
+### 6.2 Transaction Class
+
+```cpp
+class Transaction {
+  static arrow::Result<std::unique_ptr<Transaction>> Open(
+      const ArrowFileSystemPtr& fs,
+      const std::string& base_path,
+      int64_t version = LATEST,
+      const Resolver& resolver = FailResolver,
+      uint32_t retry_limit = 1);
+
+  arrow::Result<int64_t> Commit();
+  arrow::Result<std::shared_ptr<Manifest>> GetManifest();
+  int64_t GetReadVersion() const;
+
+  // Fluent builder
+  Transaction& AddColumnGroup(const std::shared_ptr<ColumnGroup>& cg);
+  Transaction& AppendFiles(const std::vector<std::shared_ptr<ColumnGroup>>& cgs);
+  Transaction& AddDeltaLog(const DeltaLog& delta_log);
+  Transaction& UpdateStat(const std::string& key,
+                           const std::vector<std::string>& files);
+  Transaction& AddIndex(const Index& index);
+  Transaction& DropIndex(const std::string& column_name,
+                          const std::string& index_type);
+};
+```
+
+---
+
+## 7. Example
+
+A typical manifest for a dataset with two column groups, one delta log, and one HNSW index:
+
+```
+manifest-3.avro
+├── Magic: 0x4D494C56
+├── Version: 2
+├── ColumnGroups:
+│   ├── [0] columns=["id", "text"], format="parquet"
+│   │   └── files:
+│   │       ├── path="0_a1b2c3.parquet", start=0, end=50000
+│   │       └── path="0_d4e5f6.parquet", start=0, end=30000
+│   └── [1] columns=["embedding"], format="parquet"
+│       └── files:
+│           ├── path="1_g7h8i9.parquet", start=0, end=50000
+│           └── path="1_j0k1l2.parquet", start=0, end=30000
+├── DeltaLogs:
+│   └── path="delete_001.del", type=PRIMARY_KEY, num_entries=128
+├── Stats:
+│   └── "bloomfilter" → ["bf_001.bin"]
+└── Indexes:
+    └── column_name="embedding", index_type="hnsw",
+        path="embedding_hnsw.idx",
+        properties={"M": "16", "efConstruction": "128", "metric_type": "L2"}
+```
+
+---
+
+## 8. Source Files
+
+| File | Description |
+|------|-------------|
+| `cpp/include/milvus-storage/manifest.h` | Manifest, DeltaLog, Index struct definitions |
+| `cpp/include/milvus-storage/column_groups.h` | ColumnGroup and ColumnGroupFile definitions |
+| `cpp/include/milvus-storage/common/layout.h` | Directory layout constants and path utilities |
+| `cpp/include/milvus-storage/transaction/transaction.h` | Transaction API and resolver definitions |
+| `cpp/src/manifest.cpp` | Avro serialization/deserialization implementation |
+| `cpp/src/transaction/transaction.cpp` | Transaction, Updates, and resolver logic |
+| `cpp/test/column_groups_test.cpp` | Manifest serialization round-trip tests |
+| `cpp/test/api_transaction_test.cpp` | Transaction and index operation tests |
+
+---
+
+## References
+
+- [Apache Avro Specification](https://avro.apache.org/docs/current/specification/)
+- [Apache Parquet Format](https://parquet.apache.org/)
+- [Milvus Storage Repository](https://github.com/milvus-io/milvus-storage)
+- [Apache Iceberg Manifest Design](https://iceberg.apache.org/spec/) (inspiration for versioned manifest approach)
@@ -0,0 +1,274 @@
+# MEP: Add Yandex Cloud Text Embedding Provider (`yc`)
+
+- **Created:** 2026-02-27
+- **Author(s):** @edddoubled
+- **Status:** Draft
+- **Component:** Proxy | QueryNode | DataNode | Function
+- **Related Issues:** TBD
+- **Released:** [TBD]
+
+## Summary
+
+This proposal introduces a new text embedding provider `yc` for Milvus `TextEmbedding` function.  
+The provider integrates with Yandex Cloud AI Studio text embedding API and enables users to generate embeddings during insert/search pipelines in the same way as existing providers (`openai`, `cohere`, `tei`, etc.).
+
+The feature includes:
+
+1. New provider implementation in `internal/util/function/embedding`.
+2. Provider selection integration in `TextEmbeddingFunction`.
+3. Provider config and credentials support in `paramtable`/`milvus.yaml`.
+4. Unit and integration tests with existing function test patterns.
+
+## Motivation
+
+Milvus currently supports multiple external embedding providers but does not provide a built-in Yandex Cloud provider.
+Users on Yandex Cloud currently need custom middleware or external embedding jobs, which creates:
+
+- additional latency and operational complexity,
+- duplicated auth/retry/error handling logic,
+- weaker parity with first-class Milvus function providers.
+
+Adding `yc` keeps user experience consistent across cloud providers and reduces integration friction.
+
+## Public Interfaces
+
+### Function schema parameters
+
+No new function type is introduced. Existing `FunctionType_TextEmbedding` is reused with:
+
+- `provider=yc`
+- `model_name=<yandex modelUri>`
+- `dim=<optional, must match output field dim>`
+- `credential=<optional, preferred>`
+
+### Config interfaces
+
+New config group keys under:
+
+- `function.textEmbedding.providers.yc.enable`
+- `function.textEmbedding.providers.yc.credential`
+- `function.textEmbedding.providers.yc.url`
+
+New environment variable:
+
+- `MILVUS_YC_API_KEY`
+
+## Design Details
+
+### Architecture placement
+
+The provider follows existing `textEmbeddingProvider` interface:
+
+- `MaxBatch() int`
+- `FieldDim() int64`
+- `CallEmbedding(ctx, texts, mode) (any, error)`
+
+The `yc` provider is selected in `NewTextEmbeddingFunction(...)` switch by `provider=yc`.
+
+### Request/Response mapping
+
+Milvus provider parameters map to Yandex API fields:
+
+- `model_name` -> `modelUri`
+- input text(s) -> request text payload
+- API key -> `Authorization` header
+
+Provider output type:
+
+- `[][]float32` only
+
+Validation rules:
+
+1. Returned embedding count must equal input text count.
+2. Returned embedding dimension must equal output field dimension.
+3. If `dim` param is provided, it must match output field dimension (existing Milvus rule).
+
+### Batching and timeout
+
+Batch behavior follows existing providers:
+
+- internal chunking by `maxBatch`
+- external cap by `extraInfo.BatchFactor`
+
+Default values:
+
+- `maxBatch = 128`
+- `timeoutSec = 30`
+
+These defaults align with existing provider implementations and can be tuned later by follow-up changes if needed.
+
+### Credential resolution order
+
+Credential parsing uses existing utility `models.ParseAKAndURL(...)` with standard precedence:
+
+1. Function param (`credential`)
+2. `milvus.yaml` provider config
+3. Environment variable (`MILVUS_YC_API_KEY`)
+
+This keeps behavior consistent with other providers and avoids introducing a provider-specific credential flow.
+
+### Error handling
+
+Provider reuses existing HTTP utility `models.PostRequest(...)` for:
+
+- HTTP error propagation (status/body),
+- timeout handling,
+- retry with exponential backoff and jitter.
+
+Provider-level errors are normalized to existing embedding provider style:
+
+- missing credential,
+- embedding count mismatch,
+- embedding dim mismatch.
+
+### API compatibility strategy
+
+Yandex documentation may evolve request/response schema over time.
+To reduce tight coupling risk, the provider supports response adaptation for both:
+
+- single-embedding response shape,
+- batched embeddings response shape.
+
+If API contract changes in future, the adaptation layer can be extended without changing function runtime interfaces.
+
+## Compatibility, Deprecation, and Migration Plan
+
+### Compatibility
+
+- Fully backward compatible for existing users.
+- No behavior change for existing providers.
+- No schema migration required.
+
+### Deprecation / migration
+
+- No deprecations in this MEP.
+- Existing function definitions continue to work unchanged.
+
+## Security Considerations
+
+1. API keys must be configured via credential config/env; avoid hard-coding in function params.
+2. Credentials should be redacted in logs (existing Milvus credential handling path).
+3. Requests must use HTTPS endpoints.
+4. Future IAM-token support should follow same secure storage guidance.
+
+## Observability
+
+Initial version relies on existing error surfaces from function execution path.
+Follow-up (optional) improvements:
+
+- provider-specific request latency metrics,
+- response code counters by provider.
+
+## Test Plan
+
+### Unit tests (`yc_embedding_provider_test.go`)
+
+1. Happy path with 1 text.
+2. Batch path with multiple texts, order preserved.
+3. Embedding count mismatch should return error.
+4. Embedding dim mismatch should return error.
+5. Missing credential should return error.
+6. Custom URL and default URL behavior.
+
+### Integration tests (`text_embedding_function_test.go`)
+
+1. `provider=yc` function creation and insert path.
+2. Provider disabled path (`yc.enable=false`).
+3. Unsupported provider behavior remains unchanged.
+
+### Regression checks
+
+Run existing embedding package test suites with required Milvus flags:
+
+```bash
+go test -tags dynamic,test -gcflags="all=-N -l" -count=1 ./internal/util/function/embedding/...
+go test -tags dynamic,test -gcflags="all=-N -l" -count=1 ./pkg/util/paramtable/...
+```
+
+## Rejected Alternatives
+
+### 1) Implement provider outside TextEmbedding framework
+
+Rejected because it duplicates runtime logic and creates inconsistent UX.
+
+### 2) Add Yandex-specific function type
+
+Rejected because provider extension is sufficient and aligns with existing architecture.
+
+### 3) Introduce new HTTP client dependency
+
+Rejected because existing `models.PostRequest` already provides retries, timeout, and standardized behavior.
+
+## Open Questions
+
+1. Should first release support IAM token in addition to API key, or API key-only with IAM in follow-up?
+2. What is the final supported request schema for batch mode in Yandex endpoint used by Milvus deployment target?
+3. Are there provider-specific token/input limits that should be surfaced in user-facing docs?
+
+## User Documentation Draft (milvus.io style)
+
+This section is a draft outline for the user-facing documentation page (similar in structure to existing provider pages such as OpenAI).
+
+### Title and scope
+
+- Page title: `Yandex Cloud`
+- Feature scope: `TextEmbedding` provider `yc`
+- Audience: users configuring function-based embedding in Milvus
+
+### Prerequisites
+
+1. Milvus instance with function feature enabled.
+2. Yandex Cloud account and AI Studio embeddings access.
+3. Valid credential (API key in phase 1).
+4. A valid `modelUri` compatible with Yandex text embedding API.
+
+### Configuration example
+
+```yaml
+function:
+  textEmbedding:
+    providers:
+      yc:
+        credential: yandex_cred
+        enable: true
+        url: https://llm.api.cloud.yandex.net/foundationModels/v1/textEmbedding
+```
+
+```yaml
+credential:
+  yandex_cred:
+    apikey: <YOUR_YC_API_KEY>
+```
+
+### Function parameter table
+
+- `provider` (required): must be `yc`
+- `model_name` (required): mapped to Yandex `modelUri`
+- `dim` (optional): must match output field dimension if specified
+- `credential` (recommended): credential name from Milvus credential config
+
+### End-to-end usage
+
+1. Create collection with source text field and float vector output field.
+2. Add `TextEmbedding` function with `provider=yc`.
+3. Insert plain text data and verify vector output generated automatically.
+4. Run text query path and verify embedding + search pipeline.
+
+### Troubleshooting section
+
+- 401/403: invalid or missing API key, insufficient Yandex IAM permission.
+- 429: request rate exceeded; reduce batch size and retry with backoff.
+- Dim mismatch: output field dim is not equal to model output dim.
+- Provider disabled: `function.textEmbedding.providers.yc.enable` is false.
+
+### Notes and limitations
+
+- Initial release supports float embeddings only.
+- Batch mode request/response shape must be confirmed against final API contract.
+- IAM token auth is planned as a follow-up if not included in phase 1.
+
+## References
+
+- Milvus embedding provider architecture: `internal/util/function/embedding`
+- Milvus provider config path: `pkg/util/paramtable/function_param.go`
+- Yandex text embedding API docs: https://yandex.cloud/ru/docs/ai-studio/embeddings/api-ref/Embeddings/textEmbedding
@@ -0,0 +1,138 @@
+# Struct Data Type
+
+## Summary
+
+Introduce a new `Struct` data type in Milvus as the element type of `DataType.ARRAY`, enabling an **Array of Struct** composite data model — logically binding multiple scalar and vector fields together within each array element.
+
+## Motivation
+
+Users frequently need to store multiple related embeddings and metadata per row. For example, a video may contain multiple clips, each with its own embedding and tags. Currently this requires either splitting data across multiple collections or using unstructured JSON fields, sacrificing query efficiency and schema enforcement.
+
+The Struct type allows vectors to exist inside array elements. Prior to Struct, `DataType.ARRAY` could not contain vector fields. With Struct, Milvus natively supports:
+
+- **Embedding List** — each row contains multiple vectors, with multi-to-multi vector similarity search using metrics such as MaxSim.
+- **Element Filter & Element-Level Search** — filtering and searching at the granularity of individual array elements. Element Filter evaluates filter conditions independently on each array element; Element-Level Search performs vector search at the granularity of each embedding in the Embedding List. The two can be used together.
+- **Match Family** — a set of array-level quantified filtering operators (ANY / ALL / LEAST / MOST / EXACT), supporting same-element multi-field matching for Struct arrays (nested semantics).
+
+Additionally, Struct enforces strong typing on sub-fields (types, dimensions, etc.) via `StructFieldSchema`, making it safer and more efficient than JSON-based approaches.
+
+## Design
+
+### Schema
+
+Struct can only be used as the `element_type` of a `DataType.ARRAY` field. Each Struct field is defined by a `StructFieldSchema` that specifies its sub-fields.
+
+**Allowed sub-field types:** scalar types (INT64, VARCHAR, FLOAT, etc.), scalar ARRAY, and vector types (FLOAT_VECTOR, etc.).
+
+**Not allowed:** nested Struct, JSON, primary key, or auto-id.
+
+```python
+struct_schema = client.create_struct_field_schema()
+struct_schema.add_field("clip_embedding", DataType.FLOAT_VECTOR, dim=128)
+struct_schema.add_field("clip_id", DataType.INT64)
+struct_schema.add_field("clip_tags", DataType.ARRAY,
+                        element_type=DataType.VARCHAR, max_capacity=10)
+
+schema.add_field("clips", datatype=DataType.ARRAY,
+                 element_type=DataType.STRUCT,
+                 struct_schema=struct_schema, max_capacity=1000)
+```
+
+**Proto changes:**
+
+```protobuf
+message CollectionSchema {
+    ...
+    repeated StructFieldSchema struct_fields = 9;
+}
+
+message StructFieldSchema {
+  int64 fieldID = 1;
+  string name = 2;
+  string description = 3;
+  repeated FieldSchema fields = 4;
+  repeated common.KeyValuePair type_params = 5;
+}
+```
+
+### Storage
+
+Each sub-field of a Struct is physically stored as an independent column, typed as `ARRAY of <field_type>`. For example, a `clip_id` (INT64) sub-field under a Struct array is stored identically to a regular `ARRAY<INT64>` column.
+
+This design avoids defining a new composite physical type and reuses the existing columnar storage and serialization infrastructure. The system maintains a metadata mapping from the Struct field name to its set of physical columns (e.g., `"clips" -> {clip_embedding, clip_id, clip_tags}`).
+
+### Vector Index (Embedding List Index)
+
+All vectors across array elements within a row are flattened and passed to **knowhere** along with per-row offset information to build the index:
+
+- `float*` — all vectors concatenated
+- `offsets` — cumulative element counts per row, e.g., row sizes [3, 2] yield offsets `[0, 3, 5]`
+
+| Aspect | Supported |
+|--------|-----------|
+| **Index types** | HNSW, IVF_FLAT, DISKANN |
+| **Metric types** | MAX_SIM_COSINE, MAX_SIM_IP |
+| **Vector types** | FLOAT_VECTOR, BinaryVector, Float16, BFloat16, Int8 |
+
+### Scalar Index (Nested Index)
+
+Scalar indexes on Struct sub-fields use **nested** semantics: each array element is indexed as an independent document, preserving positional information. This is the foundation for both `element_filter` and `Match Family` operations.
+
+```
+==== milvus format ====
+row0:
+  element[0]: { "color": "Red",  "size": "L" }
+  element[1]: { "color": "Blue", "size": "M" }
+
+row1:
+  element[0]: { "color": "Blue",   "size": "L" }
+  element[1]: { "color": "Yellow", "size": "XS" }
+  element[2]: { "color": "Blue",   "size": "LL" }
+
+==== nested index ====
+doc 0: {"color": "Red",    "size": "L"}    <- row0, element[0]
+doc 1: {"color": "Blue",   "size": "M"}    <- row0, element[1]
+doc 2: {"color": "Blue",   "size": "L"}    <- row1, element[0]
+doc 3: {"color": "Yellow", "size": "XS"}   <- row1, element[1]
+doc 4: {"color": "Blue",   "size": "LL"}   <- row1, element[2]
+
+offsets: [0, 2, 5]  (row0 has 2 elements, row1 has 3 elements)
+```
+
+The offset array maps element IDs back to row IDs at query time.
+
+### Element Filter
+
+`element_filter` filters Struct arrays at the granularity of individual array elements. `$[subFieldName]` references a sub-field of the current element. It can be combined with row-level filter conditions:
+
+```python
+filter = 'price > 100 && element_filter(clips, $[tag] == "sports" && $[score] > 0.8)'
+```
+
+**Constraint:** `element_filter` may appear at most once per filter expression and must be the last operand.
+
+### Match Family
+
+A set of operators that apply a predicate to each element of an array and quantify the number of matches to determine whether the row passes. For Struct arrays, all sub-field conditions in the predicate must be satisfied by **the same element** (nested semantics).
+
+| Operator | Semantics |
+|----------|-----------|
+| `MATCH_ANY(field, pred)` | At least one element matches |
+| `MATCH_ALL(field, pred)` | All elements match |
+| `MATCH_LEAST(field, pred, count=N)` | At least N elements match |
+| `MATCH_MOST(field, pred, count=N)` | At most N elements match |
+| `MATCH_EXACT(field, pred, count=N)` | Exactly N elements match |
+
+```python
+# Row 0: [{Red, L}, {Blue, M}] -> match (Red and L on the same element)
+# Row 1: [{Blue, L}, {Red, LL}] -> no match (Red and L on different elements)
+MATCH_ANY(structA, ($[color] == "Red") && $[size] == "L")
+```
+
+Implementation reuses the element-level filtering mechanism: evaluate the predicate at element-level, then aggregate per-doc bit patterns to apply the quantifier semantics (any/all/least/most/exact), producing a final doc-level result.
+
+## Limitations
+
+- Struct can only be used as the element type of `ARRAY`, not as a standalone field
+- Sub-fields cannot be Struct, Array, or JSON
+- Sub-fields do not support nullable
@@ -0,0 +1,213 @@
+# MEP: Scalar Index Version Management
+
+- **Created:** 2026-03-13
+- **Author(s):** @zhengbuqian
+- **Status:** Proposal
+- **Component:** Coordinator, QueryNode, DataNode
+- **Released:** TBD
+
+## Summary
+
+Introduce scalar index version management capabilities on par with vector indexes, including target version override, force rebuild, and MaximumVersion propagation and clamping for both vector and scalar indexes.
+
+## Motivation
+
+Milvus already has a complete **vector index version management system**:
+
+- Each QueryNode registers its `[MinimalIndexVersion, CurrentIndexVersion]` from knowhere at startup via etcd Session.
+- DataCoord's `IndexEngineVersionManager` aggregates all QN versions to determine cluster-wide safe build versions.
+- `autoUpgradeSegmentIndex` triggers compaction to rebuild indexes when versions lag behind.
+- `targetVecIndexVersion` provides temporary flexibility to override the default build version.
+- `forceRebuildSegmentIndex` forces all vector indexes to rebuild to a specified version.
+
+However, **scalar indexes** lack equivalent version management. Currently, scalar index versions only have hardcoded `MinimalScalarIndexEngineVersion` and `CurrentScalarIndexEngineVersion` constants, with no target override, force rebuild, or MaxVersion validation.
+
+Additionally, the **vector index** side's `MaximumIndexVersion` (the upper bound knowhere supports) exists in C++ but was never propagated to the Go layer, and `targetVecIndexVersion` was not validated against the upper bound.
+
+### Knowhere Version Semantics
+
+| Version | Meaning |
+|---------|---------|
+| `MinimalIndexVersion` | Lowest version knowhere can **build and search** |
+| `CurrentIndexVersion` | Hardcoded default build version |
+| `MaximumIndexVersion` | Highest version knowhere can **build and search** (non-beta) |
+
+Knowhere guarantees it can build and search any index version in `[MinimalIndexVersion, MaximumIndexVersion]`.
+
+### Configuration Parameter Design Intent
+
+| Parameter | Intent |
+|-----------|--------|
+| `autoUpgradeSegmentIndex` | After rolling upgrade, automatically upgrade old indexes to the cluster's current version |
+| `targetVecIndexVersion` | Temporary flexibility: override default build version without code changes (default -1, inactive) |
+| `forceRebuildSegmentIndex` | Emergency measure: force all indexes to rebuild to the target version (rebuilds already-built indexes too) |
+
+## Public Interfaces
+
+### New Configuration Parameters
+
+| Parameter | Key | Default | Description |
+|-----------|-----|---------|-------------|
+| `TargetScalarIndexVersion` | `dataCoord.targetScalarIndexVersion` | `-1` | Target scalar index version, -1 means unspecified |
+| `ForceRebuildScalarSegmentIndex` | `dataCoord.forceRebuildScalarSegmentIndex` | `false` | Force rebuild scalar index toggle |
+
+Semantics symmetric with the vector side:
+- `forceRebuildScalarSegmentIndex=true` + `targetScalarIndexVersion=N`: Force all scalar indexes to rebuild to version N
+- `forceRebuildScalarSegmentIndex=false` + `targetScalarIndexVersion=N`: New scalar indexes use `max(cluster aggregated value, N)`
+- `targetScalarIndexVersion=-1`: Inactive, follows cluster aggregated value
+
+### New Constants
+
+```go
+const MaximumScalarIndexEngineVersion = int32(2)
+```
+
+Currently Maximum equals Current for scalar indexes (unlike knowhere vector side, scalar has no reserved beta versions). In the future, Maximum will be bumped first, then Current once stable.
+
+### IndexEngineVersionManager Interface Additions
+
+```go
+GetMaximumIndexEngineVersion() int32
+GetMaximumScalarIndexEngineVersion() int32
+ResolveVecIndexVersion() int32
+ResolveScalarIndexVersion() int32
+```
+
+### Session IndexEngineVersion Struct
+
+```go
+type IndexEngineVersion struct {
+    MinimalIndexVersion int32 `json:"MinimalIndexVersion,omitempty"`
+    CurrentIndexVersion int32 `json:"CurrentIndexVersion,omitempty"`
+    MaximumIndexVersion int32 `json:"MaximumIndexVersion,omitempty"`  // new
+}
+```
+
+## Design Details
+
+### Overall Architecture
+
+Data flow after changes:
+
+```
+┌──────────────────────────────────────────────────────┐
+│                   QueryNode Startup                   │
+│                                                       │
+│  Vector versions: knowhere C++ → [Min, Current, Max]  │
+│  Scalar versions: Go constants → [Min, Current, Max]  │
+│                                                       │
+│  Written to etcd Session                              │
+└────────────────────────┬──────────────────────────────┘
+                         │
+                         ▼
+┌──────────────────────────────────────────────────────┐
+│          DataCoord IndexEngineVersionManager          │
+│                                                       │
+│  Vector aggregation:                                  │
+│    GetCurrentIndexEngineVersion()  = MIN(all QN.Cur)  │
+│    GetMinimalIndexEngineVersion()  = MAX(all QN.Min)  │
+│    GetMaximumIndexEngineVersion()  = MIN(all QN.Max)  │
+│                                                       │
+│  Scalar aggregation:                                  │
+│    GetCurrentScalarIndexEngineVersion()  = MIN(...)   │
+│    GetMinimalScalarIndexEngineVersion()  = MAX(...)   │
+│    GetMaximumScalarIndexEngineVersion()  = MIN(...)   │
+│                                                       │
+│  Resolve methods (target override + max clamp):       │
+│    ResolveVecIndexVersion()                           │
+│    ResolveScalarIndexVersion()                        │
+└────────────┬─────────────────────┬────────────────────┘
+             │                     │
+             ▼                     ▼
+┌────────────────────────┐  ┌───────────────────────────┐
+│  Build New Index        │  │  Compaction Trigger        │
+│  (prepareJobRequest)    │  │  (ShouldRebuildSegIndex)   │
+│                         │  │                            │
+│  Vec: ResolveVec()      │  │  Path 1: autoUpgrade       │
+│  Scalar: ResolveScalar()│  │    vec+scalar, version <   │
+│                         │  │                            │
+│  Stats task:            │  │  Path 2: forceRebuild vec  │
+│    ResolveScalar()      │  │    resolved target !=      │
+│                         │  │                            │
+│  Mix compaction:        │  │  Path 3: forceRebuild      │
+│    ResolveScalar()      │  │    scalar (NEW)            │
+└────────────────────────┘  │    resolved target !=       │
+                            └───────────────────────────┘
+```
+
+### Maximum Version Aggregation
+
+Takes MIN across all QNs — Maximum represents "highest version a node can handle", so the cluster-safe upper bound is the lowest across all nodes.
+
+```go
+func (m *versionManagerImpl) GetMaximumIndexEngineVersion() int32 {
+    maximum := int32(math.MaxInt32)
+    for _, version := range m.versions {
+        if version.MaximumIndexVersion == 0 {
+            continue // skip old QNs that don't report Maximum
+        }
+        if version.MaximumIndexVersion < maximum {
+            maximum = version.MaximumIndexVersion
+        }
+    }
+    if maximum == math.MaxInt32 {
+        return math.MaxInt32 // all QNs are old, no upper bound check
+    }
+    return maximum
+}
+```
+
+### Resolve Methods
+
+Centralize the version resolution logic (target override + MaxVersion clamp) to avoid duplication across task_index, task_stats, and compaction_task_mix:
+
+```go
+func (m *versionManagerImpl) ResolveScalarIndexVersion() int32 {
+    version := m.GetCurrentScalarIndexEngineVersion()
+    if Params.DataCoordCfg.TargetScalarIndexVersion.GetAsInt64() != -1 {
+        if Params.DataCoordCfg.ForceRebuildScalarSegmentIndex.GetAsBool() {
+            version = Params.DataCoordCfg.TargetScalarIndexVersion.GetAsInt32()
+        } else {
+            version = max(version, Params.DataCoordCfg.TargetScalarIndexVersion.GetAsInt32())
+        }
+    }
+    if maxVersion := m.GetMaximumScalarIndexEngineVersion(); version > maxVersion {
+        version = maxVersion
+    }
+    return version
+}
+```
+
+### Compaction Trigger: Force Rebuild
+
+The compaction trigger's force-rebuild paths also use the resolved (clamped) target to compare against, ensuring that when target > cluster maximum, the trigger converges after a single rebuild (since the built index version matches the resolved target).
+
+### DataNode Scalar Version Clamp
+
+The DataNode's `getCurrentScalarIndexVersion` is fixed to clamp against `MaximumScalarIndexEngineVersion` instead of `CurrentScalarIndexEngineVersion`, aligning with how the vector side clamps against `C.GetMaximumIndexVersion()`.
+
+DataNode text index paths (stats task and sort compaction) are fixed to use the request-carried scalar version (from DataCoord's Resolve method) instead of hardcoding `common.CurrentScalarIndexEngineVersion`.
+
+## Compatibility, Deprecation, and Migration Plan
+
+### Rolling Upgrade Compatibility
+
+The only cross-node persistent data format change is the **etcd Session JSON structure** — adding `MaximumIndexVersion` field.
+
+| Scenario | Behavior |
+|----------|----------|
+| **Old QN + New DataCoord** | Old QN's Session JSON lacks `MaximumIndexVersion`, Go deserializes it as zero. `GetMaximumIndexEngineVersion()` skips zero-valued entries. If all QNs are old, returns `math.MaxInt32` (no upper bound check). |
+| **New QN + Old DataCoord** | New QN's `MaximumIndexVersion` field is silently ignored by old DataCoord (Go `encoding/json` ignores unknown fields). No impact. |
+| **New QN + New DataCoord** | All QNs report `MaximumIndexVersion`, aggregation works normally, upper bound check fully effective. |
+
+New config parameters (`TargetScalarIndexVersion` default -1, `ForceRebuildScalarSegmentIndex` default false) are only read by DataCoord. Default behavior is identical to the old version.
+
+## Test Plan
+
+- Unit tests for `GetMaximumIndexEngineVersion()` and `GetMaximumScalarIndexEngineVersion()` aggregation, including mixed old/new QN scenarios
+- Unit tests for `ResolveVecIndexVersion()` and `ResolveScalarIndexVersion()` with various target/force/max combinations
+- Unit tests for scalar force-rebuild compaction trigger path
+- Integration test: set `targetScalarIndexVersion` and verify scalar indexes are built with correct version
+- Integration test: set `forceRebuildScalarSegmentIndex=true` and verify all scalar indexes are rebuilt
+- Verify rolling upgrade compatibility: old QN (no MaximumIndexVersion) + new DataCoord works correctly
+
@@ -0,0 +1,250 @@
+# MEP: Delegator-Side Segment Pruning via PK Predicate Hints
+
+- **Created:** 2026-03-24
+- **Author(s):** @xiaofan-luan
+- **Status:** Implemented
+- **Component:** Proxy, QueryNode (Delegator)
+- **Related Issues:** #47804
+- **Released:** TBD
+
+## Summary
+
+When a search or query request contains a primary-key (PK) predicate, the shard delegator prunes its sealed-segment dispatch list before fanning out sub-tasks to worker nodes. The delegator evaluates a lightweight PK predicate IR against each segment's `pkoracle.Candidate` — which already holds a bloom filter and min/max PK statistics in memory — and removes segments that are provably non-matching. A proxy-side hint field avoids plan deserialization on the delegator for the common case where no PK predicate exists.
+
+## Motivation
+
+The shard delegator fans out every search/query to all worker nodes that own sealed segments within the requested time range. For collections with many sealed segments this is wasteful when the query carries a PK predicate such as `id == 5`, `id IN [1, 2, 3]`, or `1 < id < 100`: segments whose PK range does not overlap the predicate will return empty results after paying the full cost of a C++ vector-search kernel invocation.
+
+The delegator already maintains a `pkoracle.Candidate` (a `BloomFilterSet`) for every sealed segment it manages. Each candidate holds:
+
+- A **bloom filter** over all PKs in the segment, supporting `BatchPkExist` batch queries.
+- **Min/max PK statistics** via `GetMinPk()` / `GetMaxPk()`, computed as the global min/max across all stat blobs (current + historical).
+
+These are sufficient to definitively exclude a segment without touching C++:
+
+- **Point / IN queries** (`pk = X` or `pk IN [list]`): bloom filter plus min/max range can prove a segment cannot contain the target PKs.
+- **Range queries** (`pk > X`, `pk < Y`): min/max statistics can prove no overlap with the predicate.
+
+Filtering at the delegator — in Go, before any worker RPC is issued — means that excluded segments never reach C++. The entire cost of the RPC, segment pinning, and the vector-search kernel is avoided for those segments. Workers that end up with an empty segment list are skipped entirely by `organizeSubTask`.
+
+## Non-Goals
+
+- Pruning on non-PK predicates.
+- Filtering growing/streaming segments (their statistics are mutable under concurrent inserts).
+- Changes to the C++ expression evaluation path.
+
+## Public Interfaces
+
+### Proto Changes
+
+New field `pk_filter int32` on `SearchRequest` and `RetrieveRequest` in `internal.proto`:
+
+```protobuf
+message SearchRequest {
+    // existing fields ...
+    int32 pk_filter = N;
+}
+message RetrieveRequest {
+    // existing fields ...
+    int32 pk_filter = N;
+}
+```
+
+**Constants** (`pkg/common/common.go`):
+
+```go
+const (
+    PkFilterNotChecked  = int32(0) // proto wire default; set by old proxies (backward compat)
+    PkFilterHasPkFilter = int32(1) // proxy confirmed optimisable PK predicate exists
+    PkFilterNoPkFilter  = int32(2) // proxy confirmed no optimisable PK predicate
+)
+```
+
+### New Configuration Parameter
+
+| Key | Default | Hot-reload | Description |
+|-----|---------|------------|-------------|
+| `queryNode.enableSegmentFilter` | `true` | yes | Enable delegator-side PK predicate pruning |
+
+### New Metrics
+
+All three are `HistogramVec` with labels `{nodeID, collectionID, queryType}`:
+
+| Metric name | Description |
+|-------------|-------------|
+| `milvus_querynode_segment_filter_total_segment_num` | Sealed segments considered before pruning |
+| `milvus_querynode_segment_filter_skipped_segment_num` | Segments pruned (not dispatched) |
+| `milvus_querynode_segment_filter_hit_segment_num` | Segments that passed the filter |
+
+## Design Details
+
+### Architecture
+
+```
+Proxy
+  │  set PkFilter hint on SearchRequest / RetrieveRequest
+  │  (PkFilterHasPkFilter=1 or PkFilterNoPkFilter=2)
+  ▼
+Shard Delegator
+  │  1. PinReadableSegments → sealed []SnapshotItem
+  │     each SnapshotItem.Segments[i].Candidate = pkoracle.Candidate (BF + min/max)
+  │
+  │  2. PruneSealedSegmentsByPKFilter(sealed, pkFilter, plan)
+  │     ├─ skip if pkFilter == NoPkFilter
+  │     ├─ unmarshal plan → BuildPKFilterExpr → PKFilterExpr IR
+  │     └─ for each SnapshotItem:
+  │         collect entry.Candidate as []PKFilterTarget
+  │         CheckPKFilter(IR, targets) → matchedIDs
+  │         remove non-matching SegmentEntry in-place
+  │
+  │  3. organizeSubTask — skips workers with empty segment list
+  │
+  └─► Worker RPCs with pruned segment lists
+```
+
+### 1. Proxy: Setting the PK Filter Hint
+
+`checkSegmentFilter(plan)` in `internal/proxy/segment_filter_helper.go` calls `HasOptimizablePkPredicate` to analyse the expression tree and returns either `PkFilterHasPkFilter` or `PkFilterNoPkFilter`. It is called in both `SearchTask.Execute` and `QueryTask.Execute` and the result is written to `SearchRequest.PkFilter` / `RetrieveRequest.PkFilter` before the request is dispatched.
+
+`HasOptimizablePkPredicate` in `internal/util/exprutil/expr_checker.go` traverses the expression tree:
+
+| Expression | Optimisable? |
+|------------|--------------|
+| `TermExpr` on PK field | Yes |
+| `UnaryRangeExpr` on PK field (=, >, >=, <, <=) | Yes |
+| `BinaryRangeExpr` on PK field | Yes |
+| `AND(left, right)` | Yes if either side is optimisable |
+| `OR(left, right)` | Yes only if **both** sides are optimisable |
+| `NOT(inner)` | No — negation cannot narrow the segment set |
+
+**Backward compatibility:** `pk_filter` defaults to `0` (`PkFilterNotChecked`) on the wire. Old proxies that do not set the field cause the delegator to attempt IR compilation anyway — no correctness regression.
+
+### 2. `PKFilterTarget` Interface
+
+Defined in `internal/querynodev2/delegator/pk_filter.go`:
+
+```go
+type PKFilterTarget interface {
+    ID() int64
+    PkCandidateExist() bool
+    BatchPkExist(*storage.BatchLocationsCache) []bool
+    GetMinPk() *storage.PrimaryKey
+    GetMaxPk() *storage.PrimaryKey
+}
+```
+
+`pkoracle.Candidate` (implemented by `BloomFilterSet`) satisfies this interface. `GetMinPk` / `GetMaxPk` scan `currentStat` plus all `historyStats` to return the global min/max PK; they return `nil` when no statistics are available (segment treated as a match — conservative).
+
+`BloomFilterSet.BatchPkExist` ORs results across `currentStat` and all `historyStats` bloom filters, returning a `[]bool` hit array for the batch.
+
+Each `SegmentEntry` in the distribution carries a `Candidate pkoracle.Candidate` field; this is the data source for `PKFilterTarget`.
+
+### 3. PK Filter IR (`pk_filter.go`)
+
+`BuildPKFilterExpr(plan, pkFilter)` compiles the PK predicate from a `planpb.PlanNode` into a typed IR. It returns `nil` immediately when `pkFilter == PkFilterNoPkFilter`.
+
+```
+PKFilterExpr
+  ├── pkFilterTermExpr          pk IN [v1, v2, ...]  (or pk = x as single-element list)
+  ├── pkFilterUnaryRangeExpr    pk op value          (>, >=, <, <=)
+  ├── pkFilterBinaryRangeExpr   lo (</<=) pk (</<=) hi
+  ├── pkFilterLogicalExpr       AND / OR
+  └── pkFilterUnsupportedExpr   fallback — keeps all segments
+```
+
+Non-PK expressions and unsupported node types produce `pkFilterUnsupportedExpr`. `UnaryRangeExpr` with `Equal` is normalised to a single-element `pkFilterTermExpr` so bloom-filter checking applies.
+
+### 4. Evaluating the IR: `CheckPKFilter`
+
+`CheckPKFilter(expr, targets)` returns `(matchedIDs Set[int64], all bool)`:
+- `all == true` — keep all targets (returned for `pkFilterUnsupportedExpr` or empty target list).
+- `all == false` — `matchedIDs` is the explicit set of segment IDs that passed.
+
+Per-node evaluation:
+
+| Node | Logic |
+|------|-------|
+| `pkFilterTermExpr` | Build a `BatchLocationsCache` from the value list. For each segment: if `PkCandidateExist`, call `BatchPkExist`; check whether any value falls within `[minPk, maxPk]` and passes the bloom filter. Keep segment if any value matches. |
+| `pkFilterUnaryRangeExpr` | Boundary arithmetic on segment `[minPk, maxPk]`. E.g. `pk > X` → keep segment iff `maxPk > X`. |
+| `pkFilterBinaryRangeExpr` | Both lower and upper bound arithmetic on `[minPk, maxPk]`. |
+| `AND` | Evaluate left; short-circuit if already empty. Intersect with right. |
+| `OR` | Evaluate left; short-circuit if already `all`. Union with right. |
+| `Unsupported` | Return `{all: true}` — segment kept. |
+
+### 5. Delegator: `PruneSealedSegmentsByPKFilter`
+
+Located in `internal/querynodev2/delegator/segment_pruner.go`. Called in `delegator.Search`, `delegator.Query`, and `delegator.QueryStream` after `PinReadableSegments` returns the sealed snapshot but before `organizeSubTask` routes sub-requests:
+
+```go
+if paramtable.Get().QueryNodeCfg.EnableSegmentFilter.GetAsBool() {
+    PruneSealedSegmentsByPKFilter(ctx,
+        req.GetSerializedExprPlan(),
+        req.GetPkFilter(),
+        sealed,          // []SnapshotItem, pruned in-place
+        collectionID,
+        queryType,       // "search" or "query"
+    )
+}
+```
+
+Steps:
+
+1. If `pkFilter == PkFilterNoPkFilter`, return immediately.
+2. Unmarshal `serializedExprPlan` into `planpb.PlanNode`; log and return on error.
+3. `BuildPKFilterExpr(plan, pkFilter)` → IR; return if `nil`.
+4. For each `SnapshotItem`: collect `entry.Candidate` (non-nil only) as `[]PKFilterTarget`, call `CheckPKFilter`, remove non-matching entries in-place.
+5. Record the three Prometheus histograms (total / skipped / hit).
+
+`organizeSubTask` is unchanged; it already skips workers whose `Segments` slice is empty.
+
+### 6. Hint Forwarding to Workers
+
+`shallowCopySearchRequest` and `shallowCopyRetrieveRequest` in `delegator.go` copy `PkFilter` into the per-worker sub-request. Workers therefore also receive the hint, which can be used for future worker-side optimisations without any proxy or proto changes.
+
+## Correctness Guarantees
+
+- **No false negatives.** The bloom filter produces false positives only (a segment may be kept when unneeded), never false negatives. The min/max range check is exact. A segment is excluded only when it is proven not to contain a matching PK.
+- **Conservative on missing statistics.** If `GetMinPk()` or `GetMaxPk()` returns `nil`, the segment is kept.
+- **Growing segments untouched.** `PruneSealedSegmentsByPKFilter` only operates on the sealed `[]SnapshotItem`. Growing segments are always queried.
+- **Unsupported expressions kept.** Any expression that cannot be compiled into the IR becomes `pkFilterUnsupportedExpr` → `{all: true}` → all segments kept.
+- **Empty TermExpr.** `pk IN []` produces an empty match set, correctly pruning all sealed segments (the query is known to return no results).
+
+## Compatibility and Migration
+
+- **Rolling upgrade safe.** `PkFilter` defaults to `0` (`PkFilterNotChecked`) on the wire. Old proxies cause the delegator to attempt IR compilation on every request — semantically equivalent to the pre-feature behaviour, just with a small extra cost.
+- **Feature flag.** `queryNode.enableSegmentFilter` is `true` by default and can be toggled at runtime without restart.
+- **No API surface changes.** This is a pure internal optimisation; client-visible search/query APIs are unchanged.
+
+## Test Plan
+
+- **IR unit tests** (`pk_filter_test.go`): all node types (Term, UnaryRange, BinaryRange, AND, OR); edge cases (empty Term, nil plan, unsupported expr); AND/OR short-circuit paths.
+- **Pruner tests** (`distribution_test.go`): `PruneSealedSegmentsByPKFilter` removes non-matching entries in-place; no-ops when `pkFilter == NoPkFilter` or `EnableSegmentFilter == false`.
+- **pkoracle tests**: `BloomFilterSet.GetMinPk` / `GetMaxPk` returns correct global min/max across multiple stat blobs.
+- **Proxy tests** (`task_search_pk_hint_test.go`): `HasOptimizablePkPredicate` for all plan types and logical operators.
+- **E2E**: existing search/query integration tests confirm result correctness is preserved.
+
+## Rejected Alternatives
+
+### Worker-Side (Segment-Layer) Filtering
+
+An earlier design applied bloom-filter and min/max checks at the worker, just before the C++ call inside `validateOnHistorical`. This was rejected because:
+
+- The delegator already holds `pkoracle.Candidate` for every sealed segment. Filtering at the delegator eliminates entire worker RPCs rather than just the per-segment C++ cost.
+- Duplicating the filter IR and evaluation logic at the segments layer adds coupling and maintenance burden without additional benefit for the common case.
+- The delegator dispatch paths (`Search`, `Query`, `QueryStream`) are a single, well-defined choke point; a single call to `PruneSealedSegmentsByPKFilter` covers all three.
+
+### Passing Extracted PK Values via Proto
+
+An earlier design had the proxy extract PK values and send them alongside the serialised plan. Rejected because:
+
+- It duplicates data already present in the serialised plan, increasing message size proportionally to the IN-list length.
+- The hint field (`PkFilter`) achieves the avoidance of plan deserialization on the delegator at negligible cost.
+
+### Pruning Growing Segments
+
+Growing segments have bloom filters and statistics, but they are mutable under concurrent inserts. Pruning them would require locking or risk race conditions. Given that growing segments are typically small, the benefit does not justify the complexity.
+
+## References
+
+- PR: https://github.com/milvus-io/milvus/pull/47805
+- Issue: https://github.com/milvus-io/milvus/issues/47804
@@ -0,0 +1,115 @@
+# Arabic and Thai Analyzer Support
+
+## Summary
+
+Add built-in Arabic and Thai text analyzers to Milvus's tantivy-binding layer, enabling native full-text search for Arabic and Thai languages. This includes two new tokenizers/analyzers, two new token filters, and language-specific stop word lists.
+
+## Motivation
+
+Milvus currently supports full-text search for English, Chinese (Jieba), and a set of European languages via the standard/ICU tokenizer pipeline. Arabic and Thai have unique linguistic characteristics that require dedicated processing:
+
+- **Arabic**: Right-to-left script with diacritical marks (harakat), letter-form variations (hamza variants, teh marbuta, alef maksura), decorative stretching (tatweel/kashida), and its own digit system (Arabic-Indic numerals ٠-٩).
+- **Thai**: No whitespace between words — word boundaries must be determined by a segmentation model (LSTM-based ICU4X WordSegmenter).
+
+Without dedicated support, Arabic text search produces poor recall (diacritics and letter variants cause mismatches), and Thai text cannot be tokenized at all by whitespace-based tokenizers.
+
+## Design
+
+### Architecture Overview
+
+Both analyzers follow the existing pattern in tantivy-binding: a **tokenizer** splits text into tokens, then a chain of **filters** normalizes them.
+
+```
+Arabic:  StandardTokenizer → LowerCaser → DecimalDigitFilter → ArabicNormalizationFilter → Stemmer(Arabic) → StopWordFilter
+Thai:    ThaiTokenizer      → LowerCaser → DecimalDigitFilter → StopWordFilter
+```
+
+### New Components
+
+#### 1. ThaiTokenizer (`thai_tokenizer.rs`)
+
+- Uses **ICU4X `WordSegmenter::try_new_lstm()`** for LSTM-based Thai word segmentation.
+- Filters out non-word segments (whitespace, punctuation) — only tokens where `is_alphanumeric()` is true are emitted.
+- **Position scheme**: Character-based (Unicode scalar value count from input start), including skipped segments. This is consistent with `IcuTokenizer` and `JiebaTokenizer`.
+  - `position`: cumulative character offset from input start (counts characters in skipped segments too).
+  - `position_length`: character count of the current token segment.
+  - `offset_from` / `offset_to`: byte offsets into the original text.
+- Available as both a standalone tokenizer (`"tokenizer": "thai"`) and a built-in analyzer (`"type": "thai"`).
+
+#### 2. ArabicNormalizationFilter (`arabic_normalization_filter.rs`)
+
+Implements Lucene-compatible Arabic normalization:
+
+| Transformation | From | To |
+|---|---|---|
+| Hamza + Alef variants | آ أ إ (U+0622, U+0623, U+0625) | ا (U+0627, bare Alef) |
+| Teh Marbuta | ة (U+0629) | ه (U+0647, Heh) |
+| Alef Maksura | ى (U+0649) | ي (U+064A, Yeh) |
+| Harakat (diacritics) | U+064B..U+065F | removed |
+| Tatweel (kashida) | ـ (U+0640) | removed |
+
+Only runs the normalization pass when at least one normalizable character is detected (fast-path check).
+
+Available as a standalone filter: `"filter": ["arabic_normalization"]`.
+
+#### 3. DecimalDigitFilter (`decimal_digit_filter.rs`)
+
+Converts non-ASCII Unicode decimal digits (General Category Nd) to ASCII 0-9. Covers 34 digit systems including Arabic-Indic (٠-٩), Thai (๐-๙), Devanagari, Bengali, Fullwidth, etc.
+
+Uses a lookup table of known "zero" code points — since Unicode guarantees digits 0-9 are contiguous within each block, `ascii_value = '0' + (codepoint - block_zero)`.
+
+Available as a standalone filter: `"filter": ["decimaldigit"]`.
+
+#### 4. Stop Word Lists
+
+- **Arabic** (`arabic.txt`): 119 stop words sourced from Apache Lucene (BSD license, Jacques Savoy).
+- **Thai** (`thai.txt`): 115 stop words sourced from Apache Lucene.
+
+Both are registered in the stop word system and accessible via `"_arabic_"` / `"_thai_"` language identifiers.
+
+### Usage
+
+**Built-in analyzer** (recommended):
+
+```json
+{"type": "arabic"}
+{"type": "arabic", "stop_words": ["custom1", "custom2"]}
+
+{"type": "thai"}
+{"type": "thai", "stop_words": ["custom1", "custom2"]}
+```
+
+**Custom pipeline**:
+
+```json
+{
+  "tokenizer": "standard",
+  "filter": ["lowercase", "arabic_normalization", "decimaldigit"]
+}
+
+{
+  "tokenizer": "thai",
+  "filter": ["lowercase", "decimaldigit"]
+}
+```
+
+### Dependencies
+
+- **icu_segmenter** (ICU4X): Already used by the existing `IcuTokenizer`. The `ThaiTokenizer` uses the same crate with `try_new_lstm()` (LSTM model) instead of `try_new_auto()` (dictionary model), keeping it focused on Thai without pulling in CJK dictionary data.
+
+### Position Semantics (ThaiTokenizer)
+
+The position field uses **character-based absolute positioning** — each token's position equals the cumulative Unicode scalar count from the start of the input, counting characters in all segments (including skipped whitespace/punctuation).
+
+Example: `"สวัสดี ครับ"` (6 Thai chars + 1 space + 3 Thai chars)
+- Token "สวัสดี": position=0, position_length=6
+- Token "ครับ": position=7, position_length=3
+
+This matches the behavior of `IcuTokenizer` and `JiebaTokenizer`, ensuring consistent phrase query and proximity query semantics across all non-Latin tokenizers.
+
+## Test Plan
+
+- Unit tests for `ThaiTokenizer`: basic Thai segmentation, mixed Thai/English/CJK input, punctuation filtering, character-based position verification.
+- Unit tests for `ArabicNormalizationFilter`: hamza normalization, teh marbuta → heh, harakat removal, tatweel removal.
+- Unit tests for `DecimalDigitFilter`: Arabic-Indic and Thai digit conversion, ASCII passthrough.
+- Integration tests for built-in `arabic` and `thai` analyzers: end-to-end tokenization with stop words, custom stop words, digit conversion.
@@ -0,0 +1,500 @@
+# MEP: TEXT Storage with LOB and Growing-Segment Flush
+
+- **Created:** 2026-04-07
+- **Author(s):** @zhagnlu
+- **Status:** Draft
+- **Component:** Storage, QueryNode, DataCoord, DataNode
+- **Related Issues:** milvus-io/milvus#48783
+- **Related PR:** milvus-io/milvus#47567
+- **Released:** 3.0.0
+
+## Summary
+
+Introduce a new end-to-end storage path for the `TEXT` scalar type in Milvus
+3.0. The design has four pillars:
+
+1. **Unified V3 Manifest model.** TEXT data and metadata flow through the
+   Storage V3 manifest / column-group machinery already in `milvus-storage`.
+2. **C++-native read and write.** All TEXT IO is done inside the C++
+   `milvus-storage` library; the Go layer is a thin coordination wrapper. No
+   per-row data crosses the CGO boundary.
+3. **Growing-segment direct flush.** TEXT collections do *not* use the
+   StreamingNode write-buffer flush path. The QueryNode's Growing Segment
+   persists its own data, avoiding storing TEXT in two places.
+4. **Vortex on-disk format for LOB payloads.** Large TEXT payloads are stored
+   in Vortex files, replacing the Parquet path used by the previous LOB
+   experiment. Vortex gives better compression for long strings and native,
+   index-based random access.
+
+The on-disk layout is **hybrid**: a per-segment *reference binlog* lives
+inside the segment column-group manifest, while the actual LOB Vortex files
+live at the **partition** level so that multiple segments can share them and
+compaction can reuse files without copying bytes.
+
+## Motivation
+
+Milvus 3.0 promotes `TEXT` to a first-class scalar type with full-text
+search. Storing TEXT inline in the existing columnar files conflates two very
+different access patterns (long tail of small strings + a few very large
+documents) and causes:
+
+- **Memory pressure** in the growing segment from megabyte-class payloads.
+- **Compaction write amplification** — the entire TEXT column is rewritten
+  whenever any row in a segment changes.
+- **Double storage** of TEXT payloads if the StreamingNode write buffer is
+  used: the QueryNode growing segment already holds the rows for query,
+  and the StreamingNode would hold them again purely for flush. For
+  document-sized payloads this doubles RAM and adds WAL bandwidth.
+- **Go/CGO overhead** in the previous LOB prototype, which kept the LOB
+  manager and writer in Go and called into a Parquet writer through CGO
+  per row. The boundary cost dominated for large payloads.
+- **Parquet's weaker random-access story** for opaque large strings, where
+  Vortex's split index gives O(log N) row lookups natively.
+
+The design below addresses all four problems together. They are tightly
+coupled: moving TEXT IO into C++ is what makes growing-segment direct flush
+practical, and the partition-level Vortex layout is what makes compaction
+reuse possible without rewriting payloads.
+
+## Public Interfaces
+
+No SDK / proto / query-syntax change is exposed to end users. The change
+surface is:
+
+### New configuration
+
+| Key | Default | Refresh | Description |
+|-----|---------|---------|-------------|
+| `dataNode.text.inlineThreshold` | `65536` B | dynamic | TEXT values strictly smaller than this are encoded inline in the reference binlog; larger values go to a LOB Vortex file. |
+| `dataNode.text.maxLobFileBytes` | `67108864` B (64 MiB) | dynamic | Maximum size of a single LOB Vortex file. |
+| `dataNode.text.flushThresholdBytes` | `16777216` B (16 MiB) | dynamic | Spillover flush trigger for the growing segment LOB writer. |
+| `dataNode.compaction.lobHoleRatioThreshold` | `0.3` | dynamic | If overall hole ratio across source segments < threshold, compaction runs in REUSE_ALL mode; otherwise REWRITE_ALL. |
+| `dataCoord.gc.lob.enabled` | `true` | static | Enable the dedicated LOB GC. |
+| `dataCoord.gc.lob.safetyWindow` | `3600` s | dynamic | Minimum file age before LOB GC may delete an orphan. |
+| `dataCoord.gc.lob.checkInterval` | `1800` s | dynamic | LOB GC scan interval. |
+
+### New milvus-storage FFI surface
+
+Two FFI surfaces are used. For **compaction and import**, the Go side
+configures TEXT columns via `TextColumnConfig { FieldID, LobBasePath,
+InlineThreshold, MaxLobFileBytes, FlushThresholdBytes, RewriteMode }` and
+passes them to the `loon_segment_writer_*` FFI (`loon_segment_writer_new`,
+`loon_segment_writer_write`, `loon_segment_writer_close`). For **growing-
+segment flush**, the Go side calls `FlushGrowingSegmentData` in segcore,
+which extracts unflushed rows and writes them through the milvus-storage
+writer entirely in C++ — no per-row data crosses the CGO boundary. Inside
+the C++ writer, each row is classified by `InlineThreshold`, LOB-sized
+values are appended to Vortex files via `VortexFileWriter`, and references
+are encoded into the reference binlog. The C struct exposed to segcore is
+`CFlushConfig`; the C struct exposed to the segment-writer FFI is
+`LoonLobColumnConfig`.
+
+### Reused interfaces (no change)
+
+- The V3 column-group manifest / `Transaction<ColumnGroups>` API in
+  `milvus-storage`.
+- `flushcommon` `SyncPolicy`, `SyncManager`, `MetaWriter`,
+  `ChannelCheckpointUpdater`.
+- `DataCoord.SaveBinlogPaths`.
+
+## Design Details
+
+### Architecture overview
+
+```
+ ┌──────────────────────────────────────────────────────────────────────┐
+ │ Control plane                                                        │
+ │  ┌─────────────────────────┐    ┌────────────────────────────────┐   │
+ │  │ DataCoord               │    │ DataCoord LOB GC               │   │
+ │  │  - SegmentInfo          │◀──▶│  - scans reference binlogs     │   │
+ │  │  - SaveBinlogPaths      │    │  - safety window               │   │
+ │  └─────────────────────────┘    └────────────────────────────────┘   │
+ └──────────────────────────────────────────────────────────────────────┘
+ ┌──────────────────────────────────────────────────────────────────────┐
+ │ Data plane                                                           │
+ │                                                                      │
+ │  ┌──────────────────────────────────────────────────────────────┐    │
+ │  │ QueryNode                                                    │    │
+ │  │  pipeline → delegator → growing segment (segcore, C++)       │    │
+ │  │                                                              │    │
+ │  │  GrowingFlushManager (Go, new)                               │    │
+ │  │   reuses flushcommon SyncPolicy / SyncManager / MetaWriter   │    │
+ │  │   uses CheckpointTracker (Go, new)                           │    │
+ │  │   calls segcore FlushGrowingSegmentData (C FFI)              │    │
+ │  └──────────────────────────────────────────────────────────────┘    │
+ │                                                                      │
+ │  ┌──────────────────────────┐  ┌────────────────────────────────┐    │
+ │  │ DataNode compactor       │  │ DataNode importer              │    │
+ │  │  text-aware:             │  │  csv / json / parquet          │    │
+ │  │   REUSE_ALL  (file copy) │  │  routes large TEXT through     │    │
+ │  │   REWRITE_ALL            │  │  loon_segment_writer_* FFI     │    │
+ │  │   SKIP (L0 deletes)      │  │                                │    │
+ │  └──────────────────────────┘  └────────────────────────────────┘    │
+ │            │                                  │                      │
+ │            ▼                                  ▼                      │
+ │  ┌──────────────────────────────────────────────────────────────┐    │
+ │  │ C++ milvus-storage                                           │    │
+ │  │  Transaction<ColumnGroups>     (segment-level reference)     │    │
+ │  │  PackedWriter / Reader         (Parquet, reference binlog)   │    │
+ │  │  VortexFileWriter              (LOB payloads)                │    │
+ │  │  UUID file naming              (partition-level LOB files)    │    │
+ │  └──────────────────────────────────────────────────────────────┘    │
+ └──────────────────────────────────────────────────────────────────────┘
+```
+
+### On-disk model
+
+The layout is **hybrid**: reference data is per-segment, LOB payload data is
+per-partition.
+
+```
+{root}/insert_log/{collection_id}/{partition_id}/
+│
+├── {segment_id}/                       # SEGMENT level
+│   ├── _metadata/
+│   │   └── manifest-{version}.avro     # column-group manifest (existing V3)
+│   └── _data/
+│       ├── cg0_xxx.parquet             # normal columns: pk, vector, scalar
+│       └── cg1_yyy.parquet             # TEXT *reference binlog* (parquet)
+│                                       #   each row = inline bytes OR
+│                                       #              16-byte LOBReference
+│
+└── lobs/                               # PARTITION level (shared!)
+    └── {field_id}/
+        └── _data/
+            ├── {file_id_1}.vx          # Vortex file, append-only
+            ├── {file_id_2}.vx
+            └── ...
+```
+
+Key consequences of this layout:
+
+- **LOB files are NOT under the segment basePath.** Multiple segments can
+  reference the same `{file_id}.vx`. Compaction reuse is therefore a pure
+  metadata operation — the LOB bytes never move.
+- **LOB files do not have their own manifest.** A `file_id` *is* the path:
+  `{partition}/lobs/{field}/_data/{file_id}.vx`. Each writer generates a
+  UUID for the `file_id`, so multiple writers can share the same partition
+  without coordination through Go or etcd.
+- **Reference binlog reuses the existing column-group manifest.** From the
+  manifest's point of view, the TEXT field is just another column group of
+  Parquet files; the only special thing is the row encoding inside.
+- **Cross-segment LOB references are allowed**, but **cross-partition** is
+  not — every reference is local to a partition, so segment / partition
+  isolation is preserved.
+
+### Data model
+
+#### Reference encoding (inside the parquet reference binlog)
+
+Every row of a TEXT column carries a 1-byte flag prefix:
+
+```
+0x00  inline:    [0x00] [text bytes ...]                     (variable length)
+0x01  LOB ref:   [0x01] [pad x3] [file_id : uuid] [row_offset : int32]
+                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+                 24 bytes total, 4-byte aligned
+
+Byte layout of a LOB reference:
+  Offset  Size  Field
+  0       1     flag         = 0x01
+  1-3     3     padding      (reserved, 0x00)
+  4-19    16    file_id      binary UUID of the LOB Vortex file
+  20-23   4     row_offset   int32, row index in the Vortex file
+```
+
+The `file_id` is a 16-byte binary UUID; the standard string form
+(`xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx`) is used when building the
+object-store path `{partition}/lobs/{field}/_data/{file_id}.vx`.
+
+The `row_offset` is a **row index** inside the Vortex file, not a byte
+offset. Vortex's split index resolves it in O(log N) where N is the number
+of splits. Using row indices keeps the reference valid after Vortex
+re-encodes / recompresses splits.
+
+#### Invariants
+
+1. A LOB Vortex file is **immutable** once closed.
+2. A LOB file is reachable iff at least one live segment's reference binlog
+   contains a LOB reference naming its `file_id`.
+3. References never cross partition boundaries.
+
+### Components
+
+| Component | Layer | Responsibility |
+|-----------|-------|----------------|
+| `VortexFileWriter` | C++ milvus-storage | Native Vortex IO for LOB payloads. |
+| `loon_segment_writer_*` | FFI (C → Go) | Segment-writer FFI surface used by compaction and import. Accepts `LoonLobColumnConfig` for TEXT columns. |
+| `FlushGrowingSegmentData` | C FFI (segcore) | Extracts unflushed rows from a growing segment and writes them through the milvus-storage writer entirely in C++. Accepts `CFlushConfig`. |
+| `GrowingFlushManager` | Go (`querynodev2/segments`) | Per-channel scheduler that drives growing-segment flush for TEXT collections. |
+| `GrowingSegmentBuffer` | Go (`querynodev2/segments`) | Adapter that exposes a growing segment to the existing `SyncPolicy` interface (`IsFull`, `MinTimestamp`, `MemorySize`). |
+| `CheckpointTracker` | Go (`querynodev2/segments`) | Maps `(segmentID, row offset)` → `MsgPosition`, derived from the `StartPosition` already carried by `delegator.InsertData`. Drives checkpoint reporting. |
+
+### Why TEXT collections flush from the growing segment
+
+A normal collection writes through StreamingNode: WAL → write buffer →
+`SyncPolicy` → `SyncManager` → `MetaWriter`. The QueryNode then loads sealed
+segments from object storage.
+
+For TEXT collections that path would force the data to live in **two**
+nodes simultaneously: the QueryNode growing segment must hold it for queries,
+and the StreamingNode write buffer must also hold it for flush. For
+document-sized payloads this doubles RAM and adds WAL bandwidth that the
+data plane cannot afford.
+
+The growing segment already has the data in memory in segcore. By giving
+the QueryNode a flush manager that reuses the existing flushcommon
+infrastructure, we keep the same `SyncPolicy` semantics, the same
+`SaveBinlogPaths` contract, and the same checkpoint propagation, but
+without ever sending the TEXT bytes through the WAL or storing them in
+StreamingNode.
+
+This is the key architectural choice; everything in the write-path
+description below follows from it.
+
+### Write path
+
+1. **Pipeline → Delegator.** WAL consumer feeds `delegator.InsertData`
+   (which already carries `StartPosition`) to the growing segment.
+2. **Insert + record batch.** `ProcessInsert` writes rows into segcore as
+   today, and additionally calls
+   `CheckpointTracker.RecordBatch(segID, endOffset, position)` to remember
+   which `MsgPosition` covers which row range.
+3. **Background sync loop.** `GrowingFlushManager` ticks periodically. It
+   wraps each growing segment as a `GrowingSegmentBuffer` and runs the
+   existing flushcommon `SyncPolicy`s (`FullBuffer`, `StaleBuffer`,
+   `Sealed`, `FlushTs`). Selected segments are dispatched to a sync
+   worker.
+4. **C++ flush (single FFI call).** Go calls
+   `segment.FlushData(ctx, flushedOffset, currentOffset, flushConfig)`,
+   which invokes `FlushGrowingSegmentData` in segcore. The entire
+   extract → classify → write → commit sequence runs in C++:
+   - Segcore extracts unflushed rows from the growing segment's
+     `ConcurrentVector` for `[flushedOffset, currentOffset)`.
+   - Each TEXT row is classified by `InlineThreshold`.
+   - LOB-sized values are appended to the current Vortex file via
+     `VortexFileWriter`. The first time a LOB write happens in the writer
+     instance, a new UUID is generated as the `file_id` for the Vortex
+     file.
+   - The reference binlog is written via the existing `PackedWriter`,
+     with each row encoded as either `[0x00 | bytes]` or
+     `[0x01 | pad | file_id | row_offset]`.
+   - On completion, Vortex files are closed, the reference parquet is
+     closed, the column-group `Transaction` is committed, and the
+     manifest path + LOB `file_id`s are returned to Go.
+   No per-row data crosses the CGO boundary.
+5. **Metadata update.** Go updates the segment manifest via the existing
+   `MetaWriter` (`SaveBinlogPaths` with `Flushed=false`,
+   `WithFullBinlogs=false`). Checkpoint propagation reuses
+   `ChannelCheckpointUpdater`. After `SaveBinlogPaths` succeeds, the
+   `CheckpointTracker` updates the flushed offset and acknowledged
+   manifest version.
+
+Failure semantics: a crash before step 4 completes leaves an orphan
+Vortex file under `lobs/{field}/_data/`, never referenced from any
+`SegmentInfo`. The LOB GC sweeps it after the safety window.
+
+### Read path
+
+`GetText(rowID)` reads one row of the reference binlog and dispatches on
+the flag byte:
+
+- `0x00` → return the inline bytes after the flag.
+- `0x01` → decode `(file_id, row_offset)`, fetch through
+  `TextColumnReader::Take`, which:
+  1. Groups requested references by `file_id`.
+  2. For each file, opens (or reuses, via the per-segment LOB reader
+     cache) `{partition}/lobs/{field}/_data/{file_id}.vx`.
+  3. Calls `VortexFileReader::take(row_indices)` for batched, vectorized
+     random access.
+  4. Reorders results back to the caller's original index.
+
+The query and expression layers above `GetText` are unchanged.
+
+### Compaction
+
+Compaction decides upfront by scanning the reference binlogs of the source
+segments to collect per-`file_id` reference counts:
+
+```
+total_refs = Σ ref_count over all (segment, file)
+total_rows = Σ row_count over all distinct file_ids   (from Vortex file metadata)
+hole_ratio = 1 - total_refs / total_rows
+```
+
+- **REUSE_ALL** (`hole_ratio < threshold`, default 0.3): the TEXT column
+  in the output is built by **byte-copying** the encoded reference rows
+  from each source binlog. `file_id` and `row_offset` are unchanged. LOB
+  Vortex files are not touched. This is the common
+  case and is essentially free.
+- **REWRITE_ALL** (`hole_ratio ≥ threshold`): for each live row, the
+  compactor reads the original text (inline or via LOB) and feeds it
+  through `loon_segment_writer_*` again. The output gets brand-new LOB
+  files with dense `row_offset`s starting from 0. Old LOB files that
+  lose all references are reclaimed by GC.
+
+Some compaction kinds bypass the hole-ratio decision and force a fixed
+strategy because their data movement is predetermined:
+
+- **Clustering compaction** (including `ClusteringPartitionKeySortCompaction`):
+  always REWRITE_ALL — data is repartitioned across the output segments,
+  so LOB references cannot be reused as-is.
+- **Mix compaction with split (1 → N)**: always REWRITE_ALL — rows are
+  redistributed across multiple output segments.
+- **Sort compaction** (including `PartitionKeySortCompaction`): always
+  REUSE_ALL — only row order changes, the underlying byte content of
+  each row is unchanged, so reference rows can be byte-copied into the
+  output.
+- **L0 delete compaction**: always SKIP — only applies delete logs to
+  segments; LOB references in the source segments are unchanged.
+- **Non-split mix compaction**: uses the hole-ratio decision above.
+
+### Garbage collection
+
+A new `garbage_collector_lob.go` runs alongside the existing segment GC:
+
+- **Reachable set.** Scan the reference binlogs of all live (non-dropped)
+  segments, union all `file_id`s found in LOB references.
+- **Scan.** For each `lobs/{field_id}/_data/`, list files. Any file
+  whose `file_id` is not in the reachable set **and** whose mtime is
+  older than `dataCoord.gc.lob.safetyWindow` is deleted.
+- **Safety window.** Protects the gap between "Vortex file closed" and
+  "`lob_file_refs` committed in etcd".
+- The standard segment GC is updated to skip the partition `lobs/`
+  prefix so the two GCs do not race.
+
+### Configuration parameters
+
+See *Public Interfaces*. All parameters are exported in
+`configs/milvus.yaml` and registered in `paramtable`.
+
+## Compatibility, Deprecation, and Migration Plan
+
+**Impact on existing users.** None at the API level. SDK clients, the
+proto query language, and the rest of the schema system see no change.
+Behaviorally, large TEXT inserts that previously stressed memory or
+StreamingNode bandwidth now succeed cheaply.
+
+**V2 segments.** Untouched. The new path is V3-only.
+
+**Existing V3 segments without TEXT data.** Untouched. The TEXT
+column-group only appears in collections that declare a TEXT field.
+
+**Pre-LOB experimental TEXT path.** The previous Go-side LOB manager +
+Parquet writer prototype is removed in the same change. Any segment that
+had been written by the prototype is migrated by reading through the old
+reader and re-flushing through `TextColumnWriter` during the next
+compaction; no separate offline migration is required.
+
+**StreamingNode behavior change for TEXT collections.** For collections
+that contain a TEXT field, StreamingNode no longer flushes those
+segments — Growing Segment does. This is internal and gated by the
+collection schema. For non-TEXT collections, StreamingNode behavior is
+unchanged.
+
+**Phasing out the older behavior.** The legacy "TEXT inline in the
+columnar file" behavior is not removed; it is the natural fallback when
+every value is below `inlineThreshold`. Operators can effectively
+disable LOB by setting `inlineThreshold` to a very large value.
+
+**Migration tools.** None required.
+
+**Removal timeline.** No legacy path is deprecated by this MEP.
+
+**Rolling-upgrade ordering.** All Milvus components that touch TEXT
+segments must include the LOB read path before any component is upgraded
+to the LOB write path. Within this single feature branch all components
+are upgraded together, so the cluster-internal contract is consistent at
+the version boundary.
+
+## Test Plan
+
+System / integration:
+
+- End-to-end insert → growing-segment flush → query for mixed inline +
+  LOB workloads on V3, with value sizes that straddle `inlineThreshold`.
+- Compaction REUSE_ALL: synthetic deletes that keep `hole_ratio < τ`;
+  verify LOB Vortex files are not rewritten and the output segment's
+  `lob_file_refs` correctly merges source `ref_count`s.
+- Compaction REWRITE_ALL: synthetic deletes that push `hole_ratio ≥ τ`;
+  verify new LOB files are produced and old ones are reclaimed by GC
+  after the safety window.
+- Cross-segment LOB sharing: two segments referencing the same
+  `file_id`; verify both can be queried independently and that GC does
+  not delete the file while either is alive.
+- Orphan LOB file: kill QueryNode between Vortex close and metadata
+  commit; verify GC reclaims the orphan after the safety window and
+  not before.
+- Failure recovery: kill QueryNode mid-ingest; restart, replay WAL from
+  the last reported `MsgPosition`, and verify the segment converges
+  without data loss or duplication.
+- Import (CSV / JSON / Parquet) of large TEXT values; verify produced
+  segments have the same on-disk shape as inserted ones.
+- Python E2E suite: `tests/python_client/testcases/test_text_lob.py`.
+
+Lower-level coverage:
+
+- C++ unit: `VortexFileWriter`, `FlushGrowingSegmentData`,
+  LOB reference encoding / decoding.
+- Go unit: `growing_flush_manager`, `growing_segment_buffer`,
+  `checkpoint_tracker`, `garbage_collector_lob`, compaction REUSE/REWRITE
+  decision matrix.
+
+The implementation is considered correct when:
+
+1. The integration suite passes on V3 with TEXT enabled.
+2. A long-running soak test with continuous insert + delete + compaction
+   shows bounded LOB-file count, bounded object-store space, and
+   correct query answers throughout.
+
+## Rejected Alternatives
+
+- **Store TEXT inline in the columnar file, no LOB path.** Rejected:
+  reproduces the memory and write-amplification problems the feature is
+  meant to solve.
+
+- **Keep LOB files inside the segment basePath.** Rejected: makes
+  CopySegment / restore / GC simpler but **kills compaction reuse** —
+  the same payload would have to be copied into every output segment's
+  basePath. For a column with low churn this is exactly the cost we
+  want to avoid.
+
+- **Per-value object on the chunk manager (one S3 key per LOB).**
+  Rejected: real object-store per-object overhead destroys throughput
+  for millions of medium-sized values.
+
+- **Use Parquet for LOB files (the previous prototype).** Rejected:
+  weaker compression on long opaque strings, and Parquet's random-row
+  access is not as cheap as Vortex's split index. The previous
+  prototype also kept the writer in Go, so per-row CGO calls dominated.
+
+- **Manage `file_id` allocation in Go via etcd.** Rejected: forces
+  every C++ writer to round-trip through Go for each new file and
+  prevents segcore-side autonomy. UUID generation keeps allocation
+  entirely in C++ with no coordination overhead.
+
+- **Use a separate manifest file for LOB files.** Rejected:
+  `file_id → path` is computable; an extra manifest adds another
+  consistency surface and another concurrent-write hazard with no
+  benefit.
+
+- **Flush TEXT collections through StreamingNode like everything
+  else.** Rejected: would require holding TEXT payloads in two
+  nodes simultaneously and would push document-sized payloads through
+  the WAL. Growing-segment direct flush is the whole point of the
+  feature.
+
+- **Drop the inline path entirely; everything is a LOB.** Rejected:
+  short strings would pay an extra Vortex round-trip per row on read,
+  and the per-`file_id` open cost would amortize poorly for tag-like
+  columns.
+
+## References
+
+- Source commit: `e482d0259e1f4f895ddddff963e42e00a31910a7` —
+  `feat: support text lob`.
+- Internal design doc (Feishu): TEXT 类型的全新存储设计.
+- Related Milvus issue: milvus-io/milvus#48783.
+- Related Milvus PR: milvus-io/milvus#47567.
+- V3 column-group manifest format MEP: `20260226-manifest-format.md`.
+- Related JSON storage MEP: `20250308-json_storage.md`.
@@ -0,0 +1,546 @@
+# Regex Filter Expression Support
+
+## Background
+
+Milvus currently supports the `LIKE` operator for pattern matching on VARCHAR fields, which covers simple wildcard patterns (`%`, `_`). However, users frequently need more expressive pattern matching capabilities — for example, matching email addresses (`[a-zA-Z]+@gmail\.com`), extracting log entries with structured patterns (`ERROR.*timeout`), or filtering URLs by path segments (`/api/v[0-9]+/users`).
+
+While `LIKE` can handle some of these cases, it requires awkward workarounds or is simply incapable of expressing complex patterns. Regular expressions are the industry standard for this class of problems, supported natively by PostgreSQL (`~`), MySQL (`REGEXP`), ClickHouse (`match()`), and Elasticsearch (`regexp` query).
+
+---
+
+## Design Principles
+
+- Provide a first-class regex operator in the Milvus expression language, with syntax and semantics familiar to users of SQL databases.
+- Use **substring matching** semantics (match if any substring matches), which is the natural expectation for most users and consistent with `grep`, PostgreSQL `~`, and RE2 `PartialMatch`.
+- Leverage existing infrastructure (RE2, ngram index, expression execution framework) to minimize implementation surface area.
+- Support the same field types as `LIKE`: VARCHAR, JSON string values, and Array\<VARCHAR\>.
+
+---
+
+## Regex Flavor and Feature Scope
+
+The regex engine is [RE2](https://github.com/google/re2) (already a Milvus dependency), which provides a safe, linear-time regex implementation. RE2 is the same engine used by Google BigQuery, ClickHouse, and CockroachDB.
+
+### Supported Syntax
+
+| Category | Examples | Supported |
+|----------|---------|-----------|
+| Literals | `abc`, `hello world` | Yes |
+| Character classes | `[a-z]`, `[^0-9]`, `\d`, `\w`, `\s` | Yes |
+| Quantifiers | `*`, `+`, `?`, `{n}`, `{n,m}` | Yes |
+| Alternation | `cat\|dog` | Yes |
+| Grouping | `(abc)`, `(?:abc)` | Yes |
+| Anchors | `^` (start), `$` (end) | Yes |
+| Dot (any character) | `.` (matches any char including `\n`) | Yes |
+| Escape sequences | `\.`, `\*`, `\\` | Yes |
+| Unicode | `\p{Han}`, `\p{Latin}` | Yes |
+| Named groups | `(?P<name>...)` | Yes |
+| Flags | `(?i)` case-insensitive, `(?s)` dot-all, `(?-s)` disable dot-all | Yes |
+
+### NOT Supported (RE2 limitations)
+
+| Feature | Notes |
+|---------|-------|
+| Backreferences | `\1`, `\2` — not supported by RE2 (requires backtracking) |
+| Lookahead/Lookbehind | `(?=...)`, `(?<=...)` — not supported by RE2 |
+| Possessive quantifiers | `*+`, `++` — not supported by RE2 |
+| Atomic groups | `(?>...)` — not supported by RE2 |
+
+These features require backtracking and could lead to catastrophic exponential runtime. RE2's linear-time guarantee makes it safe for use in a database context where untrusted patterns may be submitted.
+
+### Matching Semantics
+
+**Substring matching**: The expression `field =~ "pattern"` returns `true` if any substring of the field value matches the given regex pattern. This is equivalent to `RE2::PartialMatch` and to `grep` behavior.
+
+Examples:
+- `field =~ "error"` matches `"connection error occurred"` (substring "error" matches)
+- `field =~ "^hello"` matches `"hello world"` but not `"say hello"` (anchored to start)
+- `field =~ "world$"` matches `"hello world"` but not `"world cup"` (anchored to end)
+- `field =~ "^hello world$"` matches only the exact string `"hello world"` (full-string match via anchors)
+- `field =~ "[0-9]{3}-[0-9]{4}"` matches any string containing a phone-number-like pattern
+
+Users can achieve full-string matching by using `^` and `$` anchors explicitly: `field =~ "^pattern$"`.
+
+---
+
+## Public Interfaces
+
+### Expression Syntax
+
+```
+field =~ "regex_pattern"       -- regex match (substring semantics)
+field !~ "regex_pattern"       -- regex not match (negation)
+```
+
+- `=~` is a new binary operator for regex matching. `!~` is its negation.
+- `!~` is syntactic sugar: `field !~ "pattern"` is equivalent to `NOT (field =~ "pattern")`.
+- The right operand must be a string literal containing a valid RE2 regex pattern.
+- The left operand must be a VARCHAR field, a JSON field path that resolves to a string, or an Array\<VARCHAR\> field (element-level matching).
+
+### SDK Examples
+
+```python
+# Python SDK
+collection.query(
+    expr='email =~ "[a-zA-Z0-9.]+@gmail\\.com"',
+    output_fields=["email"]
+)
+
+# Negation: exclude entries matching a pattern
+collection.query(
+    expr='message !~ "^DEBUG"',
+    output_fields=["message"]
+)
+
+# Filter log messages containing error codes
+collection.query(
+    expr='message =~ "E[0-9]{4}:"',
+    output_fields=["message"]
+)
+
+# JSON field regex — must use path specifier, not root node
+collection.query(
+    expr='metadata["tag"] =~ "v[0-9]+\\.[0-9]+"',
+    output_fields=["metadata"]
+)
+
+# Array<VARCHAR> field — must specify element index
+collection.query(
+    expr='tags[0] =~ "release-v[0-9]+"',
+    output_fields=["tags"]
+)
+
+# Combined with vector search
+collection.search(
+    data=[query_vector],
+    anns_field="embedding",
+    param={"metric_type": "L2", "params": {"nprobe": 10}},
+    filter='url =~ "/api/v[0-9]+/users"',
+    output_fields=["url"]
+)
+```
+
+### NULL Handling
+
+Following SQL standard three-valued logic:
+- `NULL =~ "pattern"` evaluates to `NULL` (unknown), not `false`.
+- `NULL !~ "pattern"` evaluates to `NULL`.
+- In filtering context, `NULL` results are treated as non-matching (rows are excluded).
+
+### Error Handling
+
+- Invalid regex patterns are rejected at parse time with a descriptive error message:
+  ```
+  Error: invalid regex pattern in =~ operator: missing closing ): `(unclosed`
+  ```
+- Applying `=~` to non-string fields is rejected at parse time:
+  ```
+  Error: regex match on non-string field is unsupported
+  ```
+
+---
+
+## Design Details
+
+### 1. Expression Parsing
+
+#### Grammar Changes
+
+The `=~` and `!~` operators are added to the ANTLR grammar (`Plan.g4`) as new tokens and rules:
+
+```
+REGEXMATCH: '=~';
+REGEXNOTMATCH: '!~';
+
+expr: ...
+    | expr REGEXMATCH StringLiteral       # RegexMatch
+    | expr REGEXNOTMATCH StringLiteral    # RegexNotMatch
+    ...
+```
+
+The `EscapeSequence` fragment is extended with a catch-all rule `'\\' ~[\r\n]` to accept regex escape sequences (like `\d`, `\w`, `\.`, `\p`) in string literals. Without this, ANTLR would reject patterns containing non-standard escapes at the lexer level.
+
+#### Proto Extension
+
+```protobuf
+// plan.proto
+enum OpType {
+  ...
+  InnerMatch = 15;
+  RegexMatch = 16;  // new
+}
+```
+
+#### String Extraction
+
+A dedicated `extractRegexPattern()` function handles the regex string literal, bypassing Go's `strconv.Unquote` which does not understand regex escapes like `\d`. The function strips surrounding quotes, converts escaped quotes to literal quotes, and passes all other backslash sequences through unchanged to RE2.
+
+#### Visitor Logic
+
+The Go visitor (`VisitRegexMatch`) validates the field type (VARCHAR / JSON / Array\<VARCHAR\>), validates the regex syntax using Go's `regexp.Compile`, and produces a `UnaryRangeExpr` with `OpType = RegexMatch`.
+
+`VisitRegexNotMatch` reuses `VisitRegexMatch` and wraps the result with a `NOT` expression. No separate `RegexNotMatch` OpType is needed.
+
+Before emitting `RegexMatch`, the parser attempts to optimize simple patterns to faster LIKE operations (see Section 5 below). Only patterns that cannot be optimized remain as `RegexMatch`.
+
+### 2. Execution Engine
+
+The C++ execution layer adds a `PartialRegexMatcher` class that wraps `RE2::PartialMatch`:
+
+```cpp
+struct PartialRegexMatcher {
+    explicit PartialRegexMatcher(const std::string& pattern);
+    bool operator()(const std::string& value) const;      // RE2::PartialMatch
+    bool operator()(const std::string_view& value) const;  // RE2::PartialMatch
+};
+```
+
+RE2 options: `dot_nl=true` (`.` matches `\n`), `log_errors=false`, `encoding=UTF8`.
+
+This contrasts with the existing `RegexMatcher` which uses `RE2::FullMatch` (used internally by `LIKE`'s `Match` op when the LIKE pattern has complex wildcards).
+
+#### Segment-Level Caching
+
+To avoid re-compiling RE2 and re-extracting literals on every batch, `PhyUnaryRangeFilterExpr` caches regex objects at the segment level via `EnsureRegexCache()`:
+
+```cpp
+// Constructed once per segment, reused across all batches:
+std::unique_ptr<PartialRegexMatcher> cached_regex_matcher_;
+std::string cached_volnitsky_literal_;     // longest extractable literal
+std::unique_ptr<VolnitskySearcher> cached_volnitsky_searcher_;
+```
+
+The `EnsureRegexCache()` method is called before the per-batch lambda and the cached pointers are captured by value, ensuring zero-cost reuse across batches.
+
+#### Volnitsky Pre-Filter
+
+For brute-force (no index) execution, a Volnitsky substring searcher pre-filters rows before invoking RE2. The algorithm:
+
+1. Extract literal substrings from the regex pattern (reusing `extract_literals_from_regex`).
+2. Select the longest literal as the Volnitsky needle.
+3. For each row: if `VolnitskySearcher::contains(row)` returns false, skip RE2 entirely.
+4. Only rows passing the Volnitsky pre-filter are verified with `RE2::PartialMatch`.
+
+The `VolnitskySearcher` implements the [Volnitsky algorithm](http://volnitsky.com/project/str_search/):
+- 64KB bigram hash table (fits L2 cache) with open-addressing linear probing.
+- O(n/m) average-case for needles ≥ 4 bytes (bigram skip distance).
+- SSE2 `memchr`-based fallback for short needles (< 4 bytes).
+- Word-aligned fast comparison before full `memcmp`.
+- Tail scanning after main loop to avoid missing matches near the end.
+
+For patterns with a long extractable literal and low match rate, the Volnitsky pre-filter avoids the vast majority of RE2 invocations (e.g., 5-11x CPU speedup observed on benchmarks).
+
+#### Execution Path Integration
+
+The `RegexMatch` op is integrated into the `UnaryExpr` execution pipeline:
+
+- **Raw data path**: For each row, optionally apply Volnitsky pre-filter, then `PartialRegexMatcher`. Both objects are constructed once per segment and reused across all batches. For JSON shared-data paths, a pre-built `PartialRegexMatcher` is stored in `UnaryCompareContext` to avoid per-row RE2 compilation.
+- **Scalar index path**: Indexes that support `PatternMatch()` (Sort, Marisa, Bitmap, Inverted) iterate unique values and apply matching, which is O(unique\_values) instead of O(total\_rows).
+- **Execution ordering**: Both `=~` and `!~` are classified as "heavy" operations (same as `LIKE`), so they are reordered after cheaper indexed expressions in conjunctive filters. For `!~`, `IsLikeExpr` recursively checks through the NOT wrapper.
+
+### 3. Index Optimization: Ngram Index (Primary Path)
+
+The ngram index is the primary optimization path for regex queries. It uses a two-phase approach:
+
+**Phase 1 — Coarse Filter (index-only)**:
+1. Extract fixed literal substrings from the regex pattern. For example:
+   - `"error.*timeout"` → `["error", "timeout"]`
+   - `"user_[0-9]+@gmail\.com"` → `["user_", "@gmail.com"]`
+   - `"[a-z]+"` → `[]` (no literals extractable)
+2. Break each literal into ngrams and query the ngram index.
+3. Intersect posting lists to produce a candidate bitmap.
+4. If no literals can be extracted, skip ngram optimization entirely (fall through to brute force).
+
+**Phase 2 — Fine Filter (raw data verification)**:
+1. For each candidate from Phase 1, load the raw field value.
+2. Apply `RE2::PartialMatch` to verify the match.
+3. Eliminate false positives from the coarse filter.
+
+This is the same two-phase architecture already used by `LIKE` on ngram indexes.
+
+**Literal extraction strategy** (`extract_literals_from_regex`): A conservative parser walks the regex pattern character-by-character, collecting runs of non-metacharacter bytes. Key behaviors:
+
+| Input | Handling |
+|-------|----------|
+| Escaped metacharacters (`\.`, `\*`) | Treated as literal (whitelist: only escaped `.[]+*?^${}()\|\\` are literal) |
+| Shorthand classes (`\d`, `\w`, `\s`, `\b`) | Split point (not literal) |
+| `\p{Han}`, `\P{Script}` | Consumes the `{...}` block, treated as split point |
+| `(?i)` flag | Detected by scanning `[imsU-]` flag chars; causes bail-out (ngram skipped) |
+| Named groups `(?P<name>...)` | Not confused with `(?i)` — only `[imsU-]` are flag chars |
+| Alternation (`\|`) at any level | Bail-out (entire extraction returns empty) |
+| `+` quantifier | Flush current literal and restart (char before `+` may repeat) |
+| `?` quantifier | Remove last char from current literal before flush (char is optional) |
+| `*` quantifier | Remove last char from current literal before flush |
+| `{n}` exact quantifier | Expand: repeat last char n times |
+| `{n,m}` variable quantifier | Expand first n copies, then flush and restart |
+| Non-optional groups `(...)` | Penetrated: literals inside are collected |
+| Optional groups `(...)?`, `(...)*`, `(...){0,...}` | Skipped entirely (content is optional) |
+
+Example extractions:
+
+| Regex Pattern | Extracted Literals |
+|--------------|-------------------|
+| `"hello"` | `["hello"]` |
+| `"abc.*def"` | `["abc", "def"]` |
+| `"user_[0-9]+@gmail\.com"` | `["user_", "@gmail.com"]` |
+| `"abc(de)fg"` | `["abcdefg"]` |
+| `"colou?r"` | `["colo", "r"]` |
+| `"a{3}"` | `["aaa"]` |
+| `"(?i)error"` | `[]` (ngram skipped — case-insensitive) |
+| `"[a-z]+"` | `[]` (no optimization, brute force) |
+| `"foo\|bar"` | `[]` (alternation bail-out) |
+| `"\d{3}-\d{4}"` | `["-"]` |
+
+### 4. Case-Insensitive Matching and Ngram
+
+RE2 supports inline flags such as `(?i)` for case-insensitive matching. The expression `field =~ "(?i)error"` matches `"Error"`, `"ERROR"`, `"error"`, etc.
+
+However, ngram indexes store tokens from the original (case-preserved) text. When a case-insensitive pattern is used, extracted literals like `"error"` may not match ngram terms like `"Err"` or `"ERR"` in the index. This causes the ngram coarse filter to produce false negatives — missing rows that should match.
+
+**Current behavior**: When the regex pattern contains a case-insensitive flag (`(?i)`), the literal extractor returns empty, causing ngram index optimization to be skipped entirely. The query falls back to brute-force `RE2::PartialMatch` on raw data with Volnitsky pre-filter. This is functionally correct but slower.
+
+**Future optimization**: Introduce a case-folding option for ngram indexes. When enabled, both the indexed text and query literals are lowercased before ngram tokenization, enabling ngram acceleration for case-insensitive patterns.
+
+### 5. Regex to LIKE Optimization
+
+Many real-world regex patterns are simple enough to be expressed as equivalent `LIKE` operations, which have dedicated fast paths (e.g., `memcmp` for prefix matching, `string::find` for substring matching) that are significantly faster than invoking the RE2 engine.
+
+The parser detects such patterns via `tryOptimizeRegexToLike()` and transparently downgrades them to the corresponding `LIKE` OpType:
+
+| Regex Pattern | Equivalent | OpType |
+|--------------|------------|--------|
+| `"abc"` (pure literal, no metacharacters) | `LIKE "%abc%"` | `InnerMatch("abc")` |
+| `"^abc"` (anchored start + literal) | `LIKE "abc%"` | `PrefixMatch("abc")` |
+| `"abc$"` (literal + anchored end) | `LIKE "%abc"` | `PostfixMatch("abc")` |
+| `"^abc$"` (both anchors + literal) | `== "abc"` | `Equal("abc")` |
+| `"^$"` (both anchors, empty literal) | `== ""` | `Equal("")` |
+
+The conversion is conservative: only patterns composed entirely of literal characters and `^`/`$` anchors are converted. Escaped metacharacters (e.g., `\.`, `\*`) are treated as literal. Any regex metacharacter (`.`, `*`, `+`, `?`, `[`, `(`, `{`, `\d`, etc.) causes the pattern to remain as `RegexMatch`.
+
+This optimization is applied in the Go parser layer (`VisitRegexMatch`), making it transparent to the execution engine and index layer.
+
+### 6. Index Support
+
+Regex filtering works with all index types that support LIKE, using the same or equivalent mechanisms. Wherever LIKE works, `=~` also works.
+
+| Index Type | Regex (`=~`) Strategy |
+|------------|----------------------|
+| **No index (brute force)** | Volnitsky pre-filter (if extractable literal exists) + RE2 PartialMatch on raw data |
+| **Ngram index** | Two-phase: ngram filter (literal extraction) + RE2 PartialMatch verify |
+| **Inverted index (tantivy)** | Convert pattern to tantivy-compatible syntax (see below), call `regex_query` on term dictionary |
+| **Sort index (StringIndexSort)** | Iterate unique values with RE2 PartialMatch, union posting lists. O(unique\_values). Both Memory and Mmap impls. |
+| **Marisa index (StringIndexMarisa)** | Iterate unique trie keys with RE2 PartialMatch, union row offsets. O(unique\_values). |
+| **Bitmap index** | Iterate unique keys with RE2 PartialMatch, union bitmaps. O(unique\_values). All three build modes (mmap/ROARING/BITSET). |
+
+All index types that support `PatternMatch()` iterate unique values rather than per-row `Reverse_Lookup`, making regex on indexed fields O(unique\_values) instead of O(total\_rows).
+
+For non-ngram indexes, the regex-to-LIKE optimization (Section 5) is important: simple patterns like `"^prefix"` are converted to `PrefixMatch` which can leverage sort index range queries and inverted index prefix queries natively.
+
+#### Inverted Index (tantivy) Pattern Conversion
+
+Since RE2 uses `dot_nl=true` (`.` matches `\n`) but tantivy's regex engine defaults to `.` not matching `\n`, the pattern must be converted via `regex_to_tantivy_pattern()`:
+
+1. **Dot replacement**: Unescaped `.` outside character classes is replaced with `[\s\S]` (matches any character including `\n`). Escaped dots (`\.`) and dots inside `[...]` are unchanged.
+2. **`(?s)`/`(?-s)` flag tracking**: The function maintains a `dot_all` state (initially `true`, matching RE2's `dot_nl=true`). Inline `(?s)` enables dot-all; `(?-s)` disables it. When dot-all is disabled, `.` is left as-is (tantivy's default behavior matches the intended semantics). Scoped flag groups `(?s:...)` and `(?-s:...)` are tracked with a stack to properly restore state on group close.
+3. **Substring wrapping**: The pattern is wrapped with `[\s\S]*(?:...)[\s\S]*` for substring matching semantics.
+
+### 7. Supported Field Types
+
+Regex supports the same field types and access patterns as LIKE:
+
+| Field Type | Syntax | Notes |
+|------------|--------|-------|
+| VARCHAR | `field =~ "pattern"` | Primary use case |
+| JSON (string path) | `metadata["key"] =~ "pattern"` | Requires JSON path specifier. `JSONField =~ "..."` on the root node is accepted (consistent with LIKE) but only matches when the JSON value itself is a bare string — object/array roots return false. |
+| Array\<VARCHAR\> (indexed) | `arr[0] =~ "pattern"` | Matches the element at the specified index, same as LIKE. Does NOT match "any element" — an explicit index is required. |
+| JSON (array of strings) | N/A | Not supported, same as LIKE |
+| INT/FLOAT/BOOL/other | N/A | Rejected at parse time |
+
+**Note on Array\<VARCHAR\>**: The `=~` operator on arrays follows the same semantics as LIKE — the user must specify an element index (e.g., `arr[0]`). "Match if any element matches" is not supported in V1. This is consistent with the existing LIKE behavior.
+
+---
+
+## Test Plan
+
+### Unit Tests (Go)
+- Parser correctly produces `UnaryRangeExpr` with `OpType_RegexMatch` for `=~` expressions.
+- Parser correctly produces `NOT(UnaryRangeExpr{RegexMatch})` for `!~` expressions.
+- Parser rejects invalid regex syntax with descriptive error.
+- Parser rejects `=~` on non-string fields.
+- JSON and Array\<VARCHAR\> fields are handled correctly.
+- Regex-to-LIKE optimization: `"abc"` → InnerMatch, `"^abc"` → PrefixMatch, `"abc$"` → PostfixMatch, `"^abc$"` → Equal.
+- Patterns with metacharacters remain as RegexMatch (not converted to LIKE).
+
+### Unit Tests (C++)
+- `PartialRegexMatcher` correctly implements substring matching: anchors, case-insensitive `(?i)`, dot\_nl, quantifiers, Unicode, named groups, control characters.
+- `UnaryExpr` with `RegexMatch` produces correct bitmaps on raw data (growing segment, 100K rows).
+- `extract_literals_from_regex`: 40+ direct extractor tests covering alternation bail-out, `(?i)` detection, `\p{...}` consumption, quantifier expansion, group penetration, whitelist escapes. Each test also verifies no false negatives (extracted literal must appear in any matching string).
+- `regex_to_tantivy_pattern`: `(?s)`/`(?-s)` flag awareness, scoped groups, escaped dots, character classes, non-capturing groups, combined flags.
+- ClickHouse edge case alignment: empty pattern, `.*`, dot\_nl, `(?-s)`, alternation, Unicode codepoints, emoji, word boundaries, backreference rejection.
+- Ngram two-phase execution (Phase1 + Phase2) produces correct results for `RegexMatch`.
+- NULL field values produce NULL results (excluded from filter).
+
+### Integration Tests
+- End-to-end: create collection with VARCHAR field, insert data, query with `=~` and `!~`, verify results.
+- With ngram index: create ngram index on field, verify regex queries use ngram acceleration.
+- All index types: inverted, sort, marisa, bitmap, ngram — verified across 25 test patterns.
+- Case-insensitive: verify `(?i)` patterns return correct results with and without ngram index.
+- JSON field: `metadata["key"] =~ "pattern"`.
+- Combined with vector search: hybrid search with regex filter.
+
+---
+
+## Relationship with LIKE
+
+`LIKE` and `=~` are independent operators that coexist. `LIKE` is not deprecated.
+
+| Aspect | `LIKE` | `=~` |
+|--------|--------|------|
+| Pattern language | SQL wildcards (`%`, `_`) | RE2 regex |
+| Matching semantics | Full-string | Substring |
+| Pattern decomposition | Yes (PrefixMatch, InnerMatch, etc.) | No (except regex-to-LIKE optimization) |
+| Execution engine | Specialized matchers (memcmp, string::find) | RE2 (with Volnitsky pre-filter) |
+| Index optimization | Ngram, inverted (PatternQuery), sort (prefix) | Ngram (primary), all index types (unique-value iteration) |
+
+For simple patterns, `LIKE` is faster because it avoids the regex engine entirely. Users should prefer `LIKE` when the pattern can be expressed with `%` and `_` wildcards. `=~` is for patterns that require regex expressiveness.
+
+---
+
+## Rejected Alternatives
+
+### 1. Full-string matching semantics (`^pattern$`)
+
+Full-string matching would allow tantivy's FST+Automaton to be used for index acceleration. However, most real-world regex use cases are substring matching (finding patterns within larger strings). Requiring users to write `.*pattern.*` for every substring match would be unintuitive and error-prone. PostgreSQL, MySQL, and grep all default to substring matching.
+
+### 2. Function syntax `REGEXP(field, pattern)`
+
+A function-based syntax like `REGEXP(field, "pattern")` was considered. While it's more extensible (easy to add flags parameter), it doesn't integrate as naturally into the expression language. The `=~` operator syntax is more concise and consistent with PostgreSQL and Ruby conventions. The operator approach also fits better into the existing `UnaryRangeExpr` infrastructure used by `LIKE`.
+
+### 3. Using tantivy's built-in regex for index acceleration
+
+Tantivy's `RegexQuery` uses FST+Automaton intersection for regex matching on the term dictionary. This is efficient for full-string matching with selective patterns. However, for substring matching (our chosen semantics), every pattern becomes `.*pattern.*`, which means the automaton must traverse nearly all FST branches — no pruning benefit. The ngram index approach is more effective for substring patterns because it directly identifies documents containing the literal substrings present in the regex.
+
+---
+
+## TODO — Remaining Work for Feature Release
+
+### P0 — Must Have
+
+- [x] **Brute-force regex (no index)**: Raw data scan with Volnitsky pre-filter + RE2 PartialMatch, cached at segment level.
+- [x] **All index types support regex**: Inverted (tantivy), Sort, Marisa, Bitmap, Ngram — all support `=~` via unique-value iteration or two-phase execution.
+- [x] **Tantivy `(?s)`/`(?-s)` alignment**: `regex_to_tantivy_pattern()` tracks inline dot-all flags, ensuring consistent semantics across all execution paths.
+- [ ] **E2E integration tests**: Python pymilvus end-to-end tests:
+  - Create collection → insert → query with `=~` and `!~` → verify results
+  - Test with no index, inverted index, ngram index configurations
+  - Hybrid search: vector search combined with regex filter
+  - JSON field: `metadata["key"] =~ "pattern"`
+  - Array\<VARCHAR\> field: element-level regex matching
+
+### P1 — Important
+
+- [ ] **Template expression support**: Verify `field =~ {pattern}` works with parameterized queries
+- [ ] **User documentation**: Expression syntax reference — document `=~` and `!~` operators, supported regex syntax (RE2), matching semantics (substring), and known limitations
+
+### P2 — Future Optimizations
+
+- [ ] **Alternation OR splitting for ngram**: Currently `abc|xyz` bails out of ngram entirely. Optimization: split on top-level `|`, extract literals from each branch independently, query ngrams per branch, OR the candidate bitmaps. Requires changing `ExecutePhase1` from a single `vector<string>` (AND) to `vector<vector<string>>` (OR of ANDs).
+- [ ] **Hyperscan multi-pattern optimization**: When multiple regex filters target the same field (e.g., `field =~ "pattern_a" || field =~ "pattern_b"`), compile all patterns into a single Hyperscan DFA and scan each row once. O(n) regardless of pattern count, vs O(n×k) for k independent RE2 evaluations.
+
+---
+
+## Comparison with Other Systems
+
+| Feature | **Milvus** | **PostgreSQL** | **ClickHouse** | **Elasticsearch** |
+|---------|-----------|----------------|----------------|-------------------|
+| **Regex Engine** | RE2 | Spencer/Tcl ARE | RE2 (+Hyperscan) | Lucene automaton |
+| **Matching Semantics** | Substring | Substring | Substring | Full-string (anchored) |
+| **Operators** | `=~`, `!~` | `~`, `~*`, `!~`, `!~*` | `match()`, `REGEXP` | `regexp` JSON query |
+| **Negation** | `!~` | `!~`, `!~*` | `NOT match()` | `bool.must_not` |
+| **Case-Insensitive** | `(?i)` flag | `~*` operator or `(?i)` | `(?i)` flag | `case_insensitive` param |
+| **Backreferences** | No (RE2) | Yes | No (RE2) | No |
+| **Lookahead/Lookbehind** | No (RE2) | Yes | No (RE2) | No |
+| **Named Groups** | Yes | Yes | Yes | No |
+| **Unicode `\p{}`** | Yes | Yes | Yes | No |
+| **`\d` `\w` `\s`** | Yes | Yes | Yes | No |
+| **Index Acceleration** | Ngram (two-phase) | pg_trgm GIN/GiST | ngrambf bloom skip index | Automaton × term dictionary |
+| **Pre-filter** | Volnitsky (bigram hash, SSE2) | None | Volnitsky (SSE2 StringSearcher) | None |
+| **Performance Guardrail** | RE2 linear time | None (use `statement_timeout`) | RE2 linear time | `max_determinized_states` |
+
+**Design rationale:** Milvus's regex support is most similar to ClickHouse — both use RE2 with substring matching semantics. RE2 is the preferred choice for database systems because it guarantees linear-time execution (no catastrophic backtracking / ReDoS), at the cost of not supporting backreferences and lookahead/lookbehind. PostgreSQL's Spencer engine supports these features but has no built-in protection against pathological patterns. Elasticsearch's Lucene regex is the most limited (no `\d`/`\w`/`\s`, no named groups) and uses full-string matching semantics.
+
+The ngram index acceleration approach is analogous to PostgreSQL's `pg_trgm` extension: extract literal substrings from the regex, look them up in the ngram/trigram index to narrow candidates, then verify with the full regex engine.
+
+### Detailed Alignment with ClickHouse
+
+Milvus's regex implementation is designed to align with ClickHouse's `match()` function semantics, since both use RE2 as the underlying engine.
+
+#### RE2 Options
+
+| Option | ClickHouse | Milvus | Aligned? |
+|--------|-----------|--------|----------|
+| `dot_nl` | ON by default (`.` matches `\n`; use `(?-s)` to disable) | ON | Yes |
+| `log_errors` | OFF | OFF | Yes |
+| `encoding` | UTF-8, with Latin-1 fallback on invalid UTF-8 | UTF-8 only | Acceptable difference |
+| `case_sensitive` | ON by default; OFF via flag or `(?i)` | ON by default; OFF via `(?i)` | Yes |
+| `max_mem` | Not explicitly set (RE2 default) | Not explicitly set | Yes |
+
+#### Matching Semantics
+
+| Behavior | ClickHouse | Milvus | Aligned? |
+|----------|-----------|--------|----------|
+| Anchoring | UNANCHORED (`RE2::Match` with `UNANCHORED`) | `RE2::PartialMatch` | Yes (equivalent) |
+| `match('Hello', '')` | Returns 1 (empty pattern matches everything) | `PartialMatch` returns true | Yes |
+| `match('x', '.*')` / `match('x', '.*?')` | Fast path: returns 1 without running RE2 | Runs RE2 (correct but no fast path) | Functionally equivalent |
+| `(?-s)` flag | Disables dot_nl (`.` stops matching `\n`) | Disables dot_nl (in both RE2 and tantivy paths) | Yes |
+
+#### Regex-to-Literal Optimization
+
+Both ClickHouse and Milvus optimize simple regex patterns to avoid the RE2 engine:
+
+| Aspect | ClickHouse (`OptimizedRegularExpression::analyze`) | Milvus |
+|--------|---------------------------------------------------|--------|
+| **Trivial literal detection** | If pattern has no metacharacters → `is_trivial=true`, uses `strstr()` | `tryOptimizeRegexToLike`: pure literal → `InnerMatch` (uses `string::find`) |
+| **Prefix detection** | Detects if required substring is at the start of the regex | `^literal` → `PrefixMatch` |
+| **Alternation handling** | Extracts list of alternative substrings for multi-pattern matching | Returns empty (bails out on `\|`) — more conservative |
+| **Required substring extraction** | Extracts longest required literal for Volnitsky pre-filter | Extracts all required literals for ngram AND-filtering + longest for Volnitsky pre-filter |
+| **Pre-filter algorithm** | Volnitsky on contiguous ColumnString buffer (single scan across all rows) | Volnitsky per-row on individual string\_view |
+
+#### Differences and Rationale
+
+| Difference | ClickHouse | Milvus | Rationale |
+|-----------|-----------|--------|-----------|
+| **UTF-8 fallback** | Falls back to Latin-1 if UTF-8 parsing fails | UTF-8 only, fails on invalid UTF-8 | Milvus VARCHAR fields are expected to be valid UTF-8. Binary data is not a target use case. |
+| **Fast path for `.*`** | Special-cased to return all-true without running RE2 | Runs RE2 (returns true for all inputs) | Correctness equivalent. Could be added as a performance optimization later. |
+| **Alternation optimization** | Extracts alternative substrings, tests each with Volnitsky | Bails out, falls back to brute-force RE2 | Conservative approach avoids false negatives. Could be optimized in P2. |
+| **Volnitsky scan scope** | Single scan over contiguous ColumnString buffer (cross-row bigram jumps) | Per-row scan on individual `string_view` | Milvus sealed segments have contiguous `StringChunk` layout — future optimization could do cross-row scanning similar to ClickHouse. |
+| **Capturing groups** | Supports `extract()`, `extractAll()`, `extractGroups()` | Not supported (filter-only, no extraction) | Milvus is a vector database; regex is used for filtering, not data extraction. |
+
+#### Verified Edge Case Alignment
+
+The following edge cases were tested against ClickHouse's documented behavior and confirmed to produce identical results in Milvus:
+
+| Pattern | Input | ClickHouse `match()` | Milvus `=~` | Aligned? |
+|---------|-------|---------------------|-------------|----------|
+| `""` (empty) | `"Hello"` | 1 | true | Yes |
+| `".*"` | `""` | 1 | true | Yes |
+| `"c.d"` | `"abc\ndef"` | 1 (dot_nl=true) | true | Yes |
+| `"(?-s)c.d"` | `"abc\ndef"` | 0 (dot_nl disabled) | false | Yes |
+| `"abc\|xyz"` | `"only_abc"` | 1 | true | Yes |
+| `"abc\|xyz"` | `"only_xyz"` | 1 | true | Yes |
+| `"abc(de)?fg"` | `"abcfg"` | 1 | true | Yes |
+| `"abc(de)?fg"` | `"abcdefg"` | 1 | true | Yes |
+| `"(?i)hello"` | `"HELLO"` | 1 | true | Yes |
+| `"\\bhello\\b"` | `"say hello world"` | 1 | true | Yes |
+| `"\\p{Han}+"` | `"中文"` | 1 | true | Yes |
+| `"\\d{3}-\\d{4}"` | `"555-1234"` | 1 | true | Yes |
+| `"(a)\\1"` | any | Error (RE2 rejects) | Error (RE2 rejects) | Yes |
+
+---
+
+## References
+
+- [RE2 Syntax Reference](https://github.com/google/re2/wiki/Syntax)
+- [RE2 Safety Guarantees](https://swtch.com/~rsc/regexp/regexp1.html) — linear-time matching, no catastrophic backtracking
+- [Google Code Search](https://swtch.com/~rsc/regexp/regexp4.html) — trigram index for regex acceleration (same approach as our ngram optimization)
+- [PostgreSQL Pattern Matching](https://www.postgresql.org/docs/current/functions-matching.html) — `~` operator with substring semantics
+- [Volnitsky Algorithm](http://volnitsky.com/project/str_search/) — O(n/m) substring search with bigram hash table
@@ -0,0 +1,359 @@
+# MEP: Support Sort/Bitmap/Hybrid Index Types for JSON Path Index
+
+- **Created:** 2026-04-10
+- **Author(s):** @zhengbuqian
+- **Status:** Draft
+- **Component:** Index | QueryNode | Proxy | DataCoord
+- **Related Issues:** TBD
+
+## Summary
+
+Extend JSON Path Index to support Sort, Bitmap, and Hybrid index types. AUTOINDEX for JSON Path Index routes to HYBRID, which dynamically selects the optimal index type based on data cardinality — matching the behavior of regular scalar columns.
+
+## Motivation
+
+JSON Path Index currently only supports the Inverted (Tantivy) index type. Different query patterns benefit from different index structures:
+
+- **Sort Index**: Efficient for range queries (`>`, `<`, `>=`, `<=`, `BETWEEN`) on numeric JSON keys. Binary search with O(log n) lookup.
+- **Bitmap Index**: Efficient for equality and `IN` queries on low-cardinality JSON keys (e.g., status codes, enum-like string values). Roaring bitmaps provide compact storage and fast set operations.
+- **Hybrid Index**: Automatically selects between Bitmap and Sort based on data cardinality at build time — the best default when the user doesn't know the data distribution.
+
+Regular scalar columns already support AUTOINDEX → HYBRID. JSON Path Index should have the same capability.
+
+## Public Interfaces
+
+### CreateIndex API
+
+No proto changes required. The existing `index_params` map already supports `index_type`, `json_path`, and `json_cast_type`. Users specify a different `index_type`:
+
+```python
+# Sort index on a numeric JSON key
+index_params = {
+    "index_type": "STL_SORT",
+    "json_path": "metadata['price']",
+    "json_cast_type": "DOUBLE",
+}
+
+# Bitmap index on a low-cardinality string JSON key
+index_params = {
+    "index_type": "BITMAP",
+    "json_path": "metadata['status']",
+    "json_cast_type": "VARCHAR",
+}
+```
+
+### Compatibility Matrix: json_cast_type x index_type
+
+| json_cast_type | INVERTED | STL_SORT | BITMAP | HYBRID |
+|---|---|---|---|---|
+| BOOL | Y | N | Y | Y |
+| DOUBLE | Y | Y | N | Y |
+| VARCHAR | Y | Y | Y | Y |
+| ARRAY_BOOL | Y | N | N | N |
+| ARRAY_DOUBLE | Y | N | N | N |
+| ARRAY_VARCHAR | Y | N | N | N |
+
+Rationale for exclusions:
+- **Sort + BOOL**: Boolean has only 2 values, sort is meaningless.
+- **Bitmap + DOUBLE**: Floating point has high cardinality, bitmap degrades.
+
+#### Why ARRAY_* cast types are excluded from Sort/Bitmap/Hybrid
+
+Regular Array columns currently do not support Sort or Bitmap indexes either (`STLSORTChecker` and `BITMAPChecker` reject top-level Array types). JSON Path ARRAY_* cast types follow the same constraint — no reason to support more index types for JSON arrays than for native arrays. HYBRID is also excluded to keep the upgrade path clean: when Array support is added for Sort/Bitmap later, HYBRID behavior goes from "not supported" → "Bitmap/Sort selection" without a breaking change.
+
+### AUTOINDEX Routing
+
+Update `scalarAutoIndex.params.build` default config to route JSON AUTOINDEX to **HYBRID** (was `INVERTED`):
+
+```json
+{"json": "HYBRID", ...}
+```
+
+### Single Index Per Path
+
+Milvus does not support multiple indexes on the same column, and for JSON fields does not support multiple indexes on the same path:
+
+- One index type per `(field_id, json_path)` pair.
+- Creating a second index on the same path with a different cast type or index type is rejected with an error ("at most one distinct index is allowed per field").
+- No query-time index selection logic needed.
+
+## Design Details
+
+### 1. Build-Time: JSON Data Extraction via Pre-processing
+
+The core approach is **pre-processing JSON field data into typed FieldData**, then feeding it to existing unmodified `ScalarIndexSort<T>::BuildWithFieldData()`, `BitmapIndex<T>::BuildWithFieldData()`, or `HybridScalarIndex<T>::BuildWithFieldData()`.
+
+`ProcessJsonFieldData<T>()` (`internal/core/src/index/JsonIndexBuilder.cpp`) already implements the complete JSON path extraction logic (path lookup, type casting, null/non-exist handling). A new function `ConvertJsonToTypedFieldData<T>()` reuses it with callbacks that populate a typed `FieldData<T>`:
+
+```cpp
+template <typename T>
+std::vector<FieldDataPtr> ConvertJsonToTypedFieldData(
+    const std::vector<FieldDataPtr>& json_field_datas,
+    const proto::schema::FieldSchema& schema,
+    const std::string& nested_path,
+    const JsonCastType& cast_type,
+    JsonCastFunction cast_function);
+```
+
+`ConvertJsonToTypedFieldData` returns a `JsonToTypedResult` containing:
+- **`field_datas`**: Typed FieldData with nullable semantics. Rows are marked invalid (`valid = false`) when the path is missing, null, OR the value fails to cast.
+- **`non_exist_offsets`**: Offsets of rows where the JSON path truly does not exist (row null, path missing, or path value null). This is a **subset** of invalid rows — rows where the path exists but the cast fails are NOT included.
+
+This distinction is critical for EXISTS semantics:
+- `valid_bitset_[i] = false` means row `i` cannot be indexed (path missing OR cast fail) → controls Range/In/NotIn filtering
+- `non_exist_offsets_` contains only rows where the path is absent → controls EXISTS queries
+
+### 2. EXISTS Query Support via non_exist_offsets
+
+`valid_bitset_` alone is insufficient for EXISTS. A row like `{'a': 'hello'}` with a DOUBLE index has `valid = false` (cast fails), but `EXISTS json_field['a']` must return true because the path exists.
+
+Each JSON wrapper (Sort, Bitmap, Hybrid) maintains its own `non_exist_offsets_` and overrides `Exists()` to compute the bitmap from it — the same approach used by `JsonInvertedIndex`. The offsets are serialized via `WriteEntries` and restored via `LoadEntries` to survive the Build → Upload → Load lifecycle across IndexNode and QueryNode.
+
+`IndexBase` provides a virtual `Exists()` method. `ExistsExpr` calls it uniformly:
+
+```cpp
+cached_index_chunk_res_ = std::make_shared<TargetBitmap>(index->Exists());
+```
+
+No cast_type switch or dynamic_cast to concrete index types needed.
+
+### 3. IndexFactory Routing (C++)
+
+`IndexFactory::CreateJsonIndex()` currently only accepts INVERTED and NGRAM. Extend to accept STL_SORT, BITMAP, and HYBRID.
+
+#### 3.1 Schema context and dual file-manager pattern
+
+Base indexes like `BitmapIndex` and `HybridScalarIndex` derive their internal type dispatching from the `FileManagerContext`'s `field_schema.data_type()`. If this is `JSON`, their switch statements hit the `default:` error branch. Additionally, `Build(const Config&)` calls `CacheRawDataAndFillMissing()` to read field data from storage — if the schema is modified to the cast type, this would produce typed FieldData instead of JSON FieldData, which would break `ConvertJsonToTypedFieldData`.
+
+Solution: each JSON wrapper maintains **two file managers**:
+- **Original-schema file manager** (`json_file_manager_`): for reading raw JSON binlog data
+- **Cast-schema file manager** (inherited from base via `MakeJsonCastContext`): for schema-dependent dispatching in the base index
+
+`MakeJsonCastContext()` creates a copy of the `FileManagerContext` with `field_schema.data_type()` set to the cast type's Milvus type (BOOL/DOUBLE/VARCHAR).
+
+#### 3.2 Wrapper for Sort/Bitmap
+
+`JsonScalarIndexWrapper<T, BaseIndex>` overrides `Build(const Config&)` — not `BuildWithFieldData` — to read raw JSON data via `json_file_manager_`, convert to typed FieldData, then call `BaseIndex::BuildWithFieldData` with the typed data:
+
+```cpp
+template <typename T, typename BaseIndex>
+class JsonScalarIndexWrapper : public BaseIndex {
+public:
+    // Base constructed with MakeJsonCastContext (cast-type schema)
+    // json_file_manager_ constructed with original JSON schema
+
+    void Build(const Config& config) override {
+        auto json_field_datas =
+            storage::CacheRawDataAndFillMissing(json_file_manager_, config);
+        auto typed_datas = ConvertJsonToTypedFieldData<T>(
+            json_field_datas, json_schema_, nested_path_, cast_type_, cast_function_);
+        BaseIndex::BuildWithFieldData(typed_datas);
+    }
+};
+```
+
+#### 3.3 Hybrid Index for JSON Path
+
+`JsonHybridScalarIndex<T>` extends `HybridScalarIndex<T>` and overrides `Build(const Config&)`. It cannot delegate to the base `Build` because `HybridScalarIndex::Build` reads data, does cardinality selection, and builds internally — all in one method. Instead, it reimplements the build flow using protected methods `SelectIndexBuildType` and `BuildInternal`:
+
+```cpp
+template <typename T>
+class JsonHybridScalarIndex : public HybridScalarIndex<T> {
+public:
+    void Build(const Config& config) override {
+        // Read config (cardinality limit, index type preferences)
+        // Read raw JSON data via json_file_manager_
+        // Convert to typed data
+        // SelectIndexBuildType(typed_datas)  -- cardinality selection
+        // BuildInternal(typed_datas)         -- delegates to internal Bitmap or Sort
+    }
+};
+```
+
+`SelectIndexBuildType` and `BuildInternal` are promoted from private to protected in `HybridScalarIndex` to enable this. The internal indexes created by `GetInternalIndex()` (e.g., `BitmapIndex`) use the cast-type `file_manager_context_` inherited from the base, so they see the correct data type in their schema.
+
+### 4. Segment Loading (C++)
+
+`ChunkedSegmentSealedImpl::LoadIndex()` stores JSON indexes into `json_indices`. The `JsonIndex` struct already has the necessary fields (`nested_path`, `field_id`, `cast_type`, `index`). Sort, Bitmap, and Hybrid JSON path indexes are stored in the same `json_indices` vector — no structural changes needed.
+
+### 5. PinJsonIndex Query Matching (C++)
+
+`PinJsonIndex()` matches query paths to indexed paths via exact path match + `IsDataTypeSupported` check. Since there is only one index per path and the wrapper exposes `GetCastType()`, the existing matching logic works unchanged.
+
+### 6. Query Execution Path (C++)
+
+For range/comparison expressions, the current dispatch dynamic_casts to `ScalarIndex<T>*` if the index is not a `JsonFlatIndex`. This already works for Sort, Bitmap, and Hybrid because they are all `ScalarIndex<T>` subclasses with `Range()`, `In()`, `NotIn()` implementations. **No changes needed.**
+
+### 7. Go-Side Parameter Validation
+
+#### STL_SORT Checker (`stl_sort_checker.go`)
+
+- `CheckValidDataType`: Add `JSON` to valid data types.
+- `CheckTrain`: When `dataType == JSON`, validate `json_cast_type` in `{"DOUBLE", "VARCHAR"}` and `json_path` present.
+
+#### Bitmap Checker (`bitmap_index_checker.go`)
+
+- `CheckValidDataType`: Add `JSON` to valid data types.
+- `CheckTrain`: When `dataType == JSON`, validate `json_cast_type` in `{"BOOL", "VARCHAR"}` and `json_path` present.
+
+#### HYBRID Checker (`hybrid_index_checker.go`)
+
+- `CheckValidDataType`: Add `JSON` to valid data types.
+- `CheckTrain`: When `dataType == JSON`, validate `json_cast_type` in `{"BOOL", "DOUBLE", "VARCHAR"}` and `json_path` present.
+
+HYBRID remains blocked for direct user specification (`task_index.go:289` rejects `IsHYBRIDChecker`). It is only reachable via AUTOINDEX routing, same as regular scalar columns.
+
+#### AUTOINDEX Config (`autoindex_param.go`)
+
+```go
+// Change default:
+// "json": "INVERTED" → "json": "HYBRID"
+DefaultValue: `{"int": "HYBRID","varchar": "HYBRID","bool": "BITMAP", "float": "HYBRID", "json": "HYBRID", "geometry": "RTREE", "timestamptz": "STL_SORT"}`
+```
+
+### 8. Rolling Upgrade Compatibility
+
+In a mixed-version cluster, some QueryNodes may not support JSON path Sort/Bitmap/Hybrid indexes.
+
+Each QN registers `ScalarIndexEngineVersion` in its session. DataCoord's `ResolveScalarIndexVersion()` returns MIN(all QNs' current version) — the highest version ALL QNs support.
+
+**Version gate at CreateIndex (collection level)**:
+
+Bump `CurrentScalarIndexEngineVersion` to 3 for QNs that support JSON path Sort/Bitmap/Hybrid. In `index_service.go`'s `CreateIndex`, when the request is for a JSON field with `index_type` in {`STL_SORT`, `BITMAP`, `HYBRID`}:
+
+```go
+if isJsonField && isNewJsonIndexType(indexType) {
+    clusterVersion := s.indexEngineVersionManager.ResolveScalarIndexVersion()
+    if clusterVersion < 3 {
+        return merr.Status(merr.WrapErrParameterInvalidMsg(
+            "index type %s on JSON path requires all QueryNodes to support scalar index version >= 3, "+
+            "but cluster resolved version is %d; please complete the rolling upgrade first",
+            indexType, clusterVersion)), nil
+    }
+}
+```
+
+This is a hard reject at index metadata creation time. During rolling upgrade, AUTOINDEX → HYBRID → rejected. Users needing a JSON Path Index during upgrade can explicitly specify `INVERTED`.
+
+## Architecture Summary
+
+```
+        CreateIndex(index_type=AUTOINDEX, json_path="/price", json_cast_type=DOUBLE)
+                                                |
+                                                v
+        ┌─────────────── Go Proxy ──────────────────────────────────────────┐
+        │  1. AUTOINDEX → HYBRID (via scalarAutoIndex config)              │
+        │  2. HYBRIDChecker.CheckTrain(): validate cast_type + path        │
+        └──────────────────────────────────────────────────────────────────┘
+                                                |
+                                                v
+        ┌─────────────── Go DataCoord ──────────────────────────────────────┐
+        │  1. Version check: ResolveScalarIndexVersion() >= 3              │
+        │  2. ParseAndVerifyNestedPath: json['price'] → /price             │
+        └──────────────────────────────────────────────────────────────────┘
+                                                |
+                                                v
+        ┌─────────────── C++ IndexFactory::CreateJsonIndex() ──────────────┐
+        │                                                                   │
+        │  INVERTED  → JsonInvertedIndex<T>                    (existing)  │
+        │  NGRAM     → NgramInvertedIndex                      (existing)  │
+        │  STL_SORT  → JsonScalarIndexWrapper<T, ScalarIndexSort<T>>       │
+        │  BITMAP    → JsonScalarIndexWrapper<T, BitmapIndex<T>>           │
+        │  HYBRID    → JsonHybridScalarIndex<T>                            │
+        └──────────────────────────────────────────────────────────────────┘
+                                                |
+                                                v
+        ┌─────────────── Build Phase ──────────────────────────────────────┐
+        │                                                                   │
+        │  1. ConvertJsonToTypedFieldData<T>()                             │
+        │     - path exists + cast OK → typed value, valid=true            │
+        │     - path missing / null / cast fail → valid=false              │
+        │  2. Delegate to base index's BuildWithFieldData()                │
+        │     - Sort: sorted array + binary search                         │
+        │     - Bitmap: per-value roaring bitmaps                          │
+        │     - Hybrid: count distinct → pick Bitmap or STL_SORT → build   │
+        │     - valid_bitset_ naturally tracks which rows have values       │
+        └──────────────────────────────────────────────────────────────────┘
+                                                |
+                                                v
+        ┌─────────────── Query Phase (mostly unchanged) ───────────────────┐
+        │                                                                   │
+        │  PinJsonIndex → exact path match → returns ScalarIndex<T>*       │
+        │  Range/In/NotIn → ScalarIndex<T> interface   (unchanged)         │
+        │  EXISTS → IndexBase::Exists() → non_exist_offsets_               │
+        └──────────────────────────────────────────────────────────────────┘
+```
+
+## File Change Summary
+
+| File | Change |
+|---|---|
+| `internal/core/src/index/IndexFactory.cpp` | `CreateJsonIndex()`: accept STL_SORT/BITMAP/HYBRID |
+| `internal/core/src/index/JsonScalarIndexWrapper.h` | **New**: template wrapper for Sort/Bitmap |
+| `internal/core/src/index/JsonHybridScalarIndex.h` | **New**: hybrid wrapper with cardinality-based selection |
+| `internal/core/src/index/JsonIndexBuilder.h/.cpp` | Add `ConvertJsonToTypedFieldData<T>()` |
+| `internal/core/src/index/Index.h` | Add virtual `Exists()` method returning `const TargetBitmap&` |
+| `internal/core/src/exec/expression/ExistsExpr.cpp` | Simplify to `index->Exists()` |
+| `internal/util/indexparamcheck/stl_sort_checker.go` | Accept JSON, validate json_cast_type/json_path |
+| `internal/util/indexparamcheck/bitmap_index_checker.go` | Accept JSON, validate json_cast_type/json_path |
+| `internal/util/indexparamcheck/hybrid_index_checker.go` | Accept JSON, validate json_cast_type/json_path |
+| `pkg/util/paramtable/autoindex_param.go` | Default `json` from `INVERTED` to `HYBRID` |
+| `pkg/common/common.go` | Bump `Current/MaximumScalarIndexEngineVersion` to 3 |
+| `internal/datacoord/index_service.go` | Version check in `CreateIndex` for JSON path Sort/Bitmap/Hybrid |
+
+## Test Plan
+
+### Unit Tests (C++)
+
+1. **ConvertJsonToTypedFieldData**: Normal values, missing paths, null values, cast failures, mixed rows.
+2. **Sort on JSON**: Build from JSON data, verify `Range()`, `In()`, `NotIn()`, `Exists()`.
+3. **Bitmap on JSON**: Build from JSON data, verify `In()`, `Range()` on strings, `Exists()`.
+4. **Hybrid on JSON**: Low cardinality → Bitmap selected; high cardinality → STL_SORT selected; query results correct in both cases.
+5. **Serialize/Deserialize roundtrip**: Build → Serialize → Load → query, verify results match.
+
+### Integration Tests (Go)
+
+1. Create collection with JSON field, insert mixed data (some rows have target key, some don't).
+2. Create Sort/Bitmap index on JSON paths, verify filter queries: range, equality, IN, EXISTS.
+3. Verify AUTOINDEX on JSON path creates HYBRID and queries work.
+4. Verify creating a second index on the same path with a different cast type is rejected.
+
+### Rolling Upgrade Tests
+
+1. Mixed cluster (some QN v2, some v3): CreateIndex with JSON path STL_SORT/BITMAP/HYBRID → expect clear error.
+2. Mixed cluster: CreateIndex with JSON path INVERTED → expect success.
+3. Full upgrade to v3: CreateIndex with JSON path HYBRID → success.
+4. Full upgrade: AUTOINDEX on JSON path → HYBRID → success.
+
+### Edge Cases
+
+- All rows missing the target path → index is all-null, EXISTS returns all-false.
+- Empty collection → index builds with 0 rows.
+- `json_cast_function` (STRING_TO_DOUBLE) with Sort/Bitmap/Hybrid indexes.
+
+## Rejected Alternatives
+
+### A. Subclassing Sort/Bitmap for JSON
+
+Creating dedicated `JsonSortIndex<T>` / `JsonBitmapIndex<T>` subclasses. Rejected: duplicates JSON extraction logic across classes and requires parallel class hierarchies. The wrapper approach achieves the same result with less code.
+
+### B. Using valid_bitset_ alone for EXISTS
+
+Using `valid_bitset_` from the base index (Sort/Bitmap) for EXISTS queries without maintaining separate `non_exist_offsets_`. Rejected: `valid_bitset_[i] = false` conflates two distinct cases — "path doesn't exist" and "path exists but cast fails". EXISTS must return true for the latter case (e.g., `{'a': 'hello'}` with a DOUBLE index). Separate `non_exist_offsets_` is required, matching the approach used by `JsonInvertedIndex`.
+
+### C. Silently degrade Hybrid + ARRAY_* to INVERTED
+
+When HYBRID encounters an ARRAY_* cast type, silently fall back to INVERTED. Rejected: when Bitmap/Sort ARRAY_* support is added later, HYBRID behavior would change from "INVERTED" to "Bitmap/Sort" — a breaking change. Explicit rejection keeps the upgrade path clean.
+
+## References
+
+- JSON Path Index: `internal/core/src/index/JsonInvertedIndex.h/.cpp`
+- JSON data extraction: `internal/core/src/index/JsonIndexBuilder.h/.cpp`
+- Sort index: `internal/core/src/index/ScalarIndexSort.h/.cpp`
+- Bitmap index: `internal/core/src/index/BitmapIndex.h/.cpp`
+- Hybrid index: `internal/core/src/index/HybridScalarIndex.h/.cpp`
+- Index factory: `internal/core/src/index/IndexFactory.cpp`
+- Go-side validation: `internal/util/indexparamcheck/`
+- AUTOINDEX config: `pkg/util/paramtable/autoindex_param.go`
+- Proxy AUTOINDEX routing: `internal/proxy/task_index.go:301-337`
+- Scalar index version: `pkg/common/common.go`, `internal/datacoord/index_engine_version_manager.go`
@@ -0,0 +1,765 @@
+# Design Document: Drop Collection Field / Function
+
+**Branch**: `feat/drop-collection-field`
+**Author**: sijie-ni-0214
+**Date**: April 2026
+**Scope**: 30 files, +2,182/-600 lines
+
+---
+
+## 1. Overview
+
+### 1.1 Motivation
+
+Schema evolution is essential for production vector databases. While Milvus already supports adding function fields via `AlterCollectionSchema` (PR #48810), there is no mechanism to remove obsolete fields or functions from a collection schema. Users who added experimental BM25 fields, deprecated scalar columns, or misconfigured function fields must currently recreate the entire collection and re-ingest all data.
+
+This feature enables users to dynamically drop fields and functions from existing collection schemas without data re-ingestion, completing the schema evolution lifecycle alongside the existing add-field capability.
+
+### 1.2 Key Requirements
+
+1. **Non-disruptive schema evolution**: Drop fields/functions without collection recreation
+2. **Backward compatibility**: Existing segments with dropped-field data must remain loadable and queryable
+3. **Cascade cleanup**: Indexes on dropped fields must be automatically removed
+4. **Field ID safety**: Dropped field IDs must never be reused to prevent data corruption
+5. **Concurrency safety**: Reuse the existing `AlterCollectionSchema` mutual exclusion and schema version consistency gateway
+6. **Unified API**: Integrate into the existing `AlterCollectionSchema` RPC rather than introducing a separate RPC
+
+### 1.3 Design Principles
+
+- **Schema-driven filtering**: All components use the latest schema to determine field accessibility; dropped fields are invisible without deleting underlying data
+- **Lazy cleanup**: Binlogs of dropped fields are not immediately deleted; they are simply skipped during segment loading
+- **Inline cascade**: Index deletion executes within the broadcast ack callback to avoid deadlocks, following the same pattern as `DropCollection`
+- **ID monotonicity**: A persistent `max_field_id` property ensures field IDs only increase, even after drops
+
+---
+
+## 2. Architecture Overview
+
+### 2.1 High-Level Data Flow
+
+```
+------------------------------------------------------------------------------+
+|                      AlterCollectionSchema (DropRequest)                      |
+------------------------------------------------------------------------------+
+                                        |
+                                        v
+------------------------------------------------------------------------------+
+|                                   PROXY                                      |
+|  * Mutual exclusion: alterSchemaInFlight (one schema change per collection)  |
+|  * Schema version consistency check (all segments aligned)                   |
+|  * Validate drop constraints (not PK, not partition key, not last vector...) |
+|  * Forward to RootCoord via MixCoord                                         |
+------------------------------------------------------------------------------+
+                                        |
+                                        v
+------------------------------------------------------------------------------+
+|                                ROOTCOORD                                     |
+|  * Build new schema (remove field/function, increment version)               |
+|  * Persist max_field_id to collection properties                             |
+|  * Broadcast AlterCollectionMessage to WAL (control + virtual channels)      |
+|  * Ack callback: cascade drop indexes on dropped fields                      |
+------------------------------------------------------------------------------+
+                                        |
+                    +-------------------+-------------------+
+                    v                                       v
+----------------------------------+   +-----------------------------------+
+|       QUERYNODE (Go + C++)       |   |            DATACOORD              |
+| * Segment loader: skip binlogs   |   | * DropIndex: tolerate dropped     |
+|   and indexes for dropped fields |   |   fields (skip vector-index       |
+| * C++ segcore: has_field() check |   |   validation when field is nil)   |
+|   in ComputeDiff*, LoadFieldData |   |                                   |
+| * Parquet reader: skip columns   |   |                                   |
+|   not in field_metas             |   |                                   |
+----------------------------------+   +-----------------------------------+
+```
+
+### 2.2 Component Responsibilities
+
+| Component | Responsibility |
+|-----------|----------------|
+| **Proxy** | Request validation, concurrency control, schema version consistency gate |
+| **RootCoord** | Schema construction, field ID management, WAL broadcast, cascade index deletion |
+| **QueryNode (Go)** | Filter dropped-field binlogs and indexes during segment loading |
+| **Segcore (C++)** | Schema-driven field filtering in `ComputeDiff*`, `LoadFieldData`, column group loading |
+| **DataCoord** | Tolerate dropped fields in `DropIndex` flow; skip vector-index validation for nil fields |
+
+### 2.3 What This Feature Does NOT Change
+
+The drop field feature reuses the existing `AlterCollectionSchema` infrastructure without modifying:
+
+- **Streaming/WAL layer**: No new message types; uses existing `AlterCollectionMessage`
+- **DataNode**: No backfill; dropped-field binlogs are left in place
+- **Schema version consistency**: Reuses the existing gateway from add-field (DataCoord reports consistent/total segment counts)
+
+---
+
+## 3. API Design
+
+### 3.1 RPC Endpoint
+
+Drop field/function uses the existing `AlterCollectionSchema` RPC with a new `DropRequest` action:
+
+```protobuf
+rpc AlterCollectionSchema(AlterCollectionSchemaRequest) returns (AlterCollectionSchemaResponse) {}
+```
+
+### 3.2 Proto Message Changes
+
+#### DropRequest Definition
+
+```protobuf
+message AlterCollectionSchemaRequest {
+    // ... existing fields (db_name, collection_name, etc.)
+
+    message DropRequest {
+        oneof identifier {
+            string field_name = 1;     // Drop field by name
+            int64 field_id = 2;        // Drop field by ID
+            string function_name = 3;  // Drop function (cascades to output fields)
+        }
+    }
+
+    message Action {
+        oneof op {
+            AddRequest add_request = 1;    // Existing: add field/function
+            DropRequest drop_request = 2;  // New: drop field/function
+        }
+    }
+
+    Action action = 5;
+}
+```
+
+#### AlterCollectionMessage Extension
+
+```protobuf
+// messages.proto - AlterCollectionMessageBody.Updates
+message Updates {
+    // ... existing fields (schema, properties, etc.)
+    repeated int64 dropped_field_ids = 9;  // Field IDs removed from schema,
+                                            // used to cascade-delete indexes
+                                            // in ack callback
+}
+```
+
+#### UpdateMask Difference: Drop vs Add
+
+| Operation | UpdateMask Fields |
+|-----------|-------------------|
+| **Add** (field/function) | `schema` only |
+| **Drop** (field/function) | `schema` + `properties` |
+
+Drop requires updating `properties` because it must persist `max_field_id` to prevent field ID reuse.
+
+### 3.3 Client SDK (pymilvus)
+
+```python
+def alter_collection_schema(
+    self,
+    collection_name: str,
+    *,
+    field_name: str = "",       # Drop by field name
+    field_id: int = 0,          # Drop by field ID
+    function_name: str = "",    # Drop by function name (cascade)
+    db_name: str = "",
+    timeout: Optional[float] = None,
+) -> None:
+    """Drop a field or function from the collection schema.
+
+    Exactly one of field_name, field_id, or function_name must be specified.
+    """
+```
+
+---
+
+## 4. Component Design Details
+
+### 4.1 Proxy Layer
+
+**Files**: `internal/proxy/impl.go`, `internal/proxy/task.go`
+
+#### Concurrency Control (Inherited from Add-Field)
+
+Drop operations share the same concurrency safeguards as add operations:
+
+1. **`alterSchemaInFlight`** (`sync.Map`): Ensures only one `AlterCollectionSchema` request per collection at a time. Key is `dbName/collectionName`.
+
+2. **Schema Version Consistency Gate**: Before any schema change, the proxy queries DataCoord for segment schema version statistics:
+   ```
+   consistentSegments == totalSegments  -->  proceed
+   consistentSegments != totalSegments  -->  reject (previous schema change still propagating)
+   ```
+
+3. **DDL Queue**: Tasks execute serially within the DDL queue, preventing race conditions.
+
+#### PreExecute: Drop Validation
+
+```go
+func (t *alterCollectionSchemaTask) preExecuteDrop(ctx context.Context) error {
+    dropReq := t.req.GetAction().GetOp().(*milvuspb.AlterSchemaAction_DropRequest).DropRequest
+
+    switch id := dropReq.GetIdentifier().(type) {
+    case *milvuspb.AlterCollectionSchemaRequest_DropRequest_FunctionName:
+        return validateDropFunction(t.oldSchema, id.FunctionName)
+    case *milvuspb.AlterCollectionSchemaRequest_DropRequest_FieldId:
+        // Resolve field ID to field name, then validate
+        fieldName := resolveFieldName(t.oldSchema, id.FieldId)
+        return validateDropField(t.oldSchema, fieldName)
+    case *milvuspb.AlterCollectionSchemaRequest_DropRequest_FieldName:
+        return validateDropField(t.oldSchema, id.FieldName)
+    }
+}
+```
+
+#### validateDropField Constraints
+
+| Constraint | Error Message |
+|-----------|---------------|
+| Field name is empty | `field name cannot be empty` |
+| Field not found in schema | `field not found: {name}` |
+| System field (`$rowid`, `$timestamp`, `$meta`, `$namespace`) | `cannot drop system field: {name}` |
+| Primary key field | `cannot drop primary key field: {name}` |
+| Partition key field | `cannot drop partition key field: {name}` |
+| Clustering key field | `cannot drop clustering key field: {name}` |
+| Dynamic field (`IsDynamic=true`) | `cannot drop dynamic field directly, use AlterCollection to disable` |
+| Last vector field in schema | `cannot drop the last vector field: {name}` |
+| Field is function input | `field is referenced by function {fn} as input` |
+| Field is function output | `field is referenced by function {fn} as output, drop the function instead` |
+| Target is a sub-field of a struct array field | `cannot drop sub-field of struct array field: {struct}.{sub}` |
+| Dropping whole struct array field would leave no vector | `cannot drop struct array field {name}: it would leave no vector field in the collection` |
+
+**Struct array field scope**:
+- Dropping a **whole struct array field by name or ID** is supported; it is equivalent to batch-dropping the struct entry plus all its sub-fields (`droppedFieldIds` covers the struct ID and every sub-field ID). The existing index cascade and segcore `has_field()` filtering cover the removal without any C++ changes.
+- Dropping an **individual sub-field** is not supported in this change. Milvus has no runtime API for adding a struct array field or an individual sub-field to an existing collection — struct array fields and their sub-fields are declared once at collection creation (neither `add_collection_field` nor `AlterCollectionSchema.AddRequest` accepts a struct-shaped payload). Dropping a single sub-field would therefore be asymmetric with no restore path, and is explicitly rejected.
+
+#### validateDropFunction Constraints
+
+| Constraint | Error Message |
+|-----------|---------------|
+| Function name is empty | `function name is empty` |
+| Function not found | `function not found: {name}` |
+| Cascade removal of output fields would leave no vector in schema | `cannot drop function {name}: it would leave no vector field in the collection` |
+
+
+### 4.2 RootCoord Layer
+
+**File**: `internal/rootcoord/ddl_callbacks_alter_collection_schema.go`
+
+#### broadcastAlterCollectionSchemaDrop
+
+```
+1. Resolve identifier to target (field or function)
+      |
+      v
+2. Build new schema:
+   * buildSchemaForDropField(coll, fieldName, fieldID)
+     OR buildSchemaForDropFunction(coll, functionName)
+      |
+      v
+3. Returns: (newSchema, newProperties, droppedFieldIds)
+      |
+      v
+4. Broadcast AlterCollectionMessage:
+   * UpdateMask: [schema, properties]
+   * Body: { schema, properties, droppedFieldIds }
+   * Channels: control channel + all virtual channels
+      |
+      v
+5. Ack Callback:
+   a. meta.AlterCollection() -- persist metadata
+   b. cascadeDropFieldIndexesInline() -- delete indexes
+   c. BroadcastAlteredCollection() -- notify proxies
+   d. ExpireCaches() -- invalidate metadata caches
+```
+
+#### buildSchemaForDropField
+
+```go
+func buildSchemaForDropField(coll *model.Collection, fieldName string, fieldID int64) (
+    *schemapb.CollectionSchema, []*commonpb.KeyValuePair, []int64, error,
+) {
+    // 1. Locate target field by name or ID
+    var targetField *model.Field
+    for _, field := range coll.Fields {
+        if (fieldName != "" && field.Name == fieldName) ||
+           (fieldID > 0 && field.FieldID == fieldID) {
+            targetField = field
+            break
+        }
+    }
+
+    // 2. Persist max_field_id to prevent ID reuse
+    maxFieldID := nextFieldID(coll) - 1
+    properties := updateMaxFieldIDProperty(coll.Properties, maxFieldID)
+
+    // 3. Build new schema without the target field
+    newFields := []*schemapb.FieldSchema{}
+    for _, field := range coll.Fields {
+        if field.FieldID != targetField.FieldID {
+            newFields = append(newFields, marshal(field))
+        }
+    }
+
+    schema := &schemapb.CollectionSchema{
+        Fields:    newFields,
+        Functions: marshalFunctions(coll.Functions),
+        Version:   coll.SchemaVersion + 1,
+    }
+
+    return schema, properties, []int64{targetField.FieldID}, nil
+}
+```
+
+#### buildSchemaForDropFunction
+
+Drops a function and **all its output fields** (cascade):
+
+```go
+func buildSchemaForDropFunction(coll *model.Collection, functionName string) (
+    *schemapb.CollectionSchema, []*commonpb.KeyValuePair, []int64, error,
+) {
+    // 1. Find target function
+    var targetFunc *model.Function
+    for _, fn := range coll.Functions {
+        if fn.Name == functionName { targetFunc = fn; break }
+    }
+
+    // 2. Collect output field IDs as droppedFieldIds
+    droppedFieldIds := targetFunc.OutputFieldIDs
+
+    // 3. Remove output fields from schema
+    outputFieldIDSet := make(map[int64]bool)
+    for _, id := range droppedFieldIds { outputFieldIDSet[id] = true }
+
+    newFields := filterFields(coll.Fields, outputFieldIDSet)
+
+    // 4. Remove function from schema
+    newFunctions := filterFunctions(coll.Functions, targetFunc.Name)
+
+    // 5. Persist max_field_id, increment schema version
+    // ...
+
+    return schema, properties, droppedFieldIds, nil
+}
+```
+
+### 4.3 Field ID Reuse Prevention
+
+**Problem**: If field IDs are assigned based on `max(current_fields)`, dropping the field with the highest ID would cause the next added field to reuse that ID. Since binlogs of the dropped field still exist on storage with the old field ID, this creates data corruption.
+
+**Solution**: Persist `max_field_id` in collection properties.
+
+```go
+// nextFieldID reads max field ID from three sources and returns max + 1:
+func nextFieldID(coll *model.Collection) int64 {
+    maxFieldID := int64(common.StartOfUserFieldID)  // 100
+
+    // Source 1: Current fields
+    for _, field := range coll.Fields {
+        if field.FieldID > maxFieldID { maxFieldID = field.FieldID }
+    }
+
+    // Source 2: Struct array sub-fields
+    for _, sf := range coll.StructArraySubFields { /* ... */ }
+
+    // Source 3: Persisted max_field_id (survives field deletion)
+    for _, kv := range coll.Properties {
+        if kv.Key == "max_field_id" {
+            if v, err := strconv.ParseInt(kv.Value, 10, 64); err == nil {
+                if v > maxFieldID { maxFieldID = v }
+            }
+        }
+    }
+
+    return maxFieldID + 1
+}
+```
+
+**Key**: `updateMaxFieldIDProperty` is called during every drop operation, ensuring the high-water mark is persisted even if the field with the highest ID is removed.
+
+### 4.4 Cascade Index Deletion
+
+**File**: `internal/rootcoord/ddl_callbacks_alter_collection_properties.go`
+
+#### Why Inline in Ack Callback?
+
+`broadcastAlterCollectionSchema()` holds a broadcast lock on the collection. Calling `DropIndex` as a separate RPC would attempt to acquire the same lock, causing a deadlock. Instead, index deletion is executed inline within the ack callback (the same pattern used by `DropCollection`).
+
+#### cascadeDropFieldIndexesInline
+
+```go
+func cascadeDropFieldIndexesInline(ctx context.Context, c *Core, msg *msgstream.AlterCollectionMsg) error {
+    droppedFieldIDs := msg.Body.Updates.DroppedFieldIds
+    if len(droppedFieldIDs) == 0 {
+        return nil  // No fields dropped, nothing to cascade
+    }
+
+    // 1. Query all indexes on the collection
+    resp, err := c.mixCoord.DescribeIndex(ctx, &indexpb.DescribeIndexRequest{
+        CollectionID: msg.CollectionID,
+    })
+
+    // 2. Find indexes on dropped fields
+    droppedSet := make(map[int64]bool)
+    for _, id := range droppedFieldIDs { droppedSet[id] = true }
+
+    for _, index := range resp.IndexInfos {
+        if droppedSet[index.FieldID] {
+            // 3. Broadcast DropIndex message via control channel
+            registry.CallMessageAckCallback(ctx, DropIndexMessage{
+                IndexID: index.IndexID,
+                // ...
+            })
+        }
+    }
+    return nil
+}
+```
+
+#### DataCoord Tolerance for Dropped Fields
+
+When `cascadeDropFieldIndexesInline` triggers `DropIndex`, the field no longer exists in the schema. The DataCoord `DropIndex` handler must tolerate this:
+
+```go
+// internal/datacoord/index_service.go
+field := typeutil.GetField(schema, index.FieldID)
+if field == nil {
+    // Field already dropped from schema -- skip vector-index validation,
+    // proceed with index cleanup
+    log.Info("field already dropped, proceeding with index drop")
+    continue
+}
+```
+
+Without this tolerance, the cascade would fail because `DropIndex` normally validates that vector indexes cannot be dropped on loaded collections.
+
+### 4.5 Disable Dynamic Field
+
+Disabling the dynamic schema (`AlterCollection` with `EnableDynamicField=false`) reuses the drop-field infrastructure:
+
+```go
+func broadcastDisableDynamicField(ctx context.Context, coll *model.Collection) error {
+    // 1. Find the $meta (dynamic) field
+    var dynamicFieldID int64
+    for _, field := range coll.Fields {
+        if field.IsDynamic { dynamicFieldID = field.FieldID; break }
+    }
+
+    // 2. Build schema without $meta, set EnableDynamicField=false
+    // 3. Broadcast with DroppedFieldIds = [dynamicFieldID]
+    // 4. Ack callback: cascadeDropFieldIndexesInline removes $meta indexes
+}
+```
+
+**Proxy-side gate coverage**: Enabling/disabling the dynamic field mutates the schema and bumps `SchemaVersion + 1`, so it must respect the same concurrency invariants as `AlterCollectionSchema` (see §6.1). The `AlterCollection` handler inspects `request.Properties` and, when `dynamicfield.enabled` is present, routes the request through the same `alterSchemaInFlight` + `checkSchemaVersionConsistency` gates as `AlterCollectionSchema`, sharing the same `sync.Map` so the two handlers are mutually exclusive per collection. Requests that only alter unrelated properties (e.g. `collection.ttl.seconds`, `mmap.enabled`) bypass the gate and keep their existing lightweight path.
+
+### 4.6 Segcore (C++) Layer
+
+**Files**: `internal/core/src/segcore/SegmentLoadInfo.cpp`, `ChunkedSegmentSealedImpl.cpp`, `SegmentGrowingImpl.cpp`
+
+#### Schema-Driven Filtering Strategy
+
+All C++ components use the **latest schema** (not the segment's original schema) to determine field visibility. The core check is:
+
+```cpp
+// Schema.h
+bool has_field(FieldId field_id) const {
+    return fields_.count(field_id) > 0;
+}
+```
+
+This single method gates all data access:
+
+| Function | File | Behavior |
+|----------|------|----------|
+| `ComputeDiffBinlogs` | SegmentLoadInfo.cpp | Skip binlog entries for dropped fields |
+| `ComputeDiffIndexes` | SegmentLoadInfo.cpp | Skip index entries for dropped fields |
+| `ComputeDiffColumnGroups` | SegmentLoadInfo.cpp | Skip column groups for dropped fields |
+| `ComputeDiffReloadFields` | SegmentLoadInfo.cpp | Exclude dropped fields from reload set |
+| `LoadFieldData` | ChunkedSegmentSealedImpl.cpp | Defense-in-depth: skip if field not in schema |
+| `load_field_data_internal` | SegmentGrowingImpl.cpp | Skip growing segment data for dropped fields |
+| `load_column_group_data_internal` | SegmentGrowingImpl.cpp | Skip column groups for dropped fields |
+| `TranslateGroupChunk` | GroupChunkTranslator.cpp | Skip parquet columns not in field_metas |
+
+**Key Design Decision**: Use `new_info.schema_` (the latest schema from the load request) rather than `this->schema_` (the segment's cached schema) in all `ComputeDiff*` functions. This ensures that even if the segment was loaded before the field was dropped, a reload/delta-load will correctly filter out the dropped field.
+
+### 4.7 QueryNode Segment Loader (Go)
+
+**File**: `internal/querynodev2/segments/segment_loader.go`
+
+The Go segment loader filters dropped fields at two points:
+
+```go
+// Index loading
+fieldSchema, err := schemaHelper.GetFieldFromID(fieldID)
+if err != nil {
+    log.Info("skip index for dropped field", zap.Int64("fieldID", fieldID))
+    continue  // Field dropped, skip its index
+}
+
+// Binlog/column group loading
+fieldSchema, err := schemaHelper.GetFieldFromID(fieldID)
+if err != nil {
+    log.Info("skip binlog for dropped field", zap.Int64("fieldID", fieldID))
+    continue  // Field dropped, skip it
+}
+```
+
+**Important**: The loader uses `continue` (not `return err`) to ensure that other fields in the same column group are still loaded correctly.
+
+---
+
+## 5. Drop Field vs Drop Function: Semantic Differences
+
+| Aspect | Drop Field | Drop Function |
+|--------|-----------|---------------|
+| **Target** | A single field | A function + all its output fields |
+| **DroppedFieldIds** | `[fieldID]` | `[outputFieldID1, outputFieldID2, ...]` |
+| **Input fields** | N/A | Preserved (not removed) |
+| **Index cascade** | Drops indexes on the field | Drops indexes on all output fields |
+| **Validation** | Cannot drop if referenced by a function | No restriction (output fields cascade) |
+
+### Difference from DropCollectionFunction (Existing API)
+
+| Aspect | DropCollectionFunction (existing) | AlterCollectionSchema Drop Function (this feature) |
+|--------|----------------------------------|-----------------------------------------------------|
+| Output fields | Preserved, marked `IsFunctionOutput=false` | **Cascade deleted** from schema |
+| Indexes | Not touched | Cascade deleted |
+| Field ID protection | No `max_field_id` update | `max_field_id` updated |
+
+---
+
+## 6. Concurrency and Consistency
+
+### 6.1 Three Layers of Protection
+
+Drop operations inherit the same concurrency model as add operations through the unified `AlterCollectionSchema` entry point:
+
+1. **`alterSchemaInFlight` Mutex** (`sync.Map`): Only one schema modification per collection at a time. A second request immediately receives an error response.
+
+2. **DDL Queue Serial Execution**: All DDL tasks (including schema changes) execute serially within the proxy's DDL queue.
+
+3. **Schema Version Consistency Gate**: The proxy checks DataCoord's segment statistics before allowing any schema change. If a previous change (e.g., add-field backfill) hasn't propagated to all segments, the new request is rejected.
+
+**Coverage across RPC entry points**: Because the proxy rootcoord lock serializes concurrent *writes* but not *writer-vs-in-flight-backfill*, layers (1) and (3) must guard every RPC entry that mutates the schema. In this PR, both `AlterCollectionSchema` and the dynamic-field branch of `AlterCollection` go through the same two gates and share the same `alterSchemaInFlight` `sync.Map`, so any two schema mutations targeting the same collection — regardless of which RPC they arrived on — are mutually exclusive, and neither can begin until the previous mutation's backfill has reached 100% of segments.
+
+### 6.2 Ack Callback Idempotency
+
+The ack callback (`cascadeDropFieldIndexesInline` + `meta.AlterCollection`) may be retried with exponential backoff if it fails. Both operations are idempotent:
+- `meta.AlterCollection`: Overwrites metadata; repeated calls produce the same result
+- `MarkIndexAsDeleted`: Skips already-deleted indexes
+
+### 6.3 Cascade Intermediate Window
+
+A drop proceeds in two phases within the ack-callback pipeline:
+
+1. **Metadata phase**: `meta.AlterCollection` removes the field from `coll.Fields`. After this step `DescribeCollection` no longer reports the field.
+2. **Index cascade phase**: `cascadeDropFieldIndexesInline` broadcasts `DropIndex` for each surviving index on the dropped field ID; once those acks complete, `indexMeta` entries are cleared.
+
+**Observable window**: Between phase 1 and phase 2 — typically milliseconds, but bounded only by the `DropIndex` broadcast round-trip — an external observer can see:
+
+- `DescribeCollection` → field is gone
+- `ListIndexes` → an index on the dropped field name is still reported
+
+This state is **transient**, not an orphaned-index bug. It is equivalent to the window that `DropCollection` exposes between deleting the collection meta and the cascade cleanup of its indexes.
+
+**Retry convergence**: `cascadeDropFieldIndexesInline` is driven by the standard broadcast ack-callback framework — if the `DropIndex` broadcast fails (network, rootcoord restart, etc.), the ack infrastructure retries until success, and rootcoord re-drives outstanding callbacks from the broadcast log after restart. Final consistency is guaranteed; the window does not accumulate.
+
+**On-call diagnosis**: to distinguish a transient drop-field cascade window from a true orphaned-index bug:
+
+- RootCoord logs `cascade dropping index on dropped field` with `fieldID`, `indexName`, `indexID` for every cascade target; the presence of a recent entry for the observed index confirms the current state is transient.
+- Re-query after ~30s — if `ListIndexes` still returns the dropped field, the cascade has genuinely failed to converge and should be treated as an orphaned-index bug through the existing runbook.
+
+### 6.4 In-flight Request Semantics
+
+Segcore's schema-driven filtering (§4.6) gates **loading**, not **query plan compilation or execution**. This section specifies how in-flight search / query / groupby requests behave relative to a concurrent drop.
+
+**A request "observes" a dropped field only when it actually references the field**, through any of:
+- `anns_field`
+- `filter` / expression
+- `output_fields` (explicit or via `"*"` expansion)
+- `group_by_field`
+- partition-key expression
+
+Requests that do not reference the dropped field pass through unaffected — plan compilation never looks up the removed field, and the executor never issues a column access for it.
+
+**When a request does reference the dropped field**, one of three outcomes applies, determined by the race between the proxy's `globalMetaCache` invalidation and the querynode's segment reload:
+
+| Case | Proxy cache state | QueryNode segment state | Outcome |
+|------|-------------------|-------------------------|---------|
+| 1. Post-invalidation (common) | new schema (N+1) | any | proxy's `CreateSearchPlanArgs` schema-helper cannot resolve the field; `task_search.go` wraps the error as `ErrParameterInvalid: "failed to create query plan: field not found: <field>"` — returned to the client |
+| 2. Cross-version (narrow) | stale (N) | new schema (N+1), segment reloaded without the field | plan compiles against schema N, reaches querynode; segcore's `chunk_data_impl`/`chunk_array_view_impl` check `field_data_ready_bitset_`, the bit is 0 for the dropped field, `AssertInfo` raises `SegcoreError` — surfaced to the client as an error status |
+| 3. No race | stale or new | still has the field | executes normally |
+
+Cases 1 and 2 both return a clear error to the client; **neither aborts or corrupts state**. Case 2 is bounded in time by the proxy cache invalidation latency (rootcoord broadcast → per-proxy cache invalidate, milliseconds to seconds in practice), and further bounded by the schema-version-consistency gate in §6.1: a second schema mutation cannot begin until the previous one has propagated, so the cross-version window is always tied to a single in-flight mutation and never accumulates.
+
+SDKs cache collection schema in a process-wide `GlobalCache`. `add_collection_field` and `add_collection_function` previously did not invalidate this cache, relying on `@retry_on_schema_mismatch` on `insert_rows` / `upsert_rows` to recover from staleness on the write path.
+
+Introducing drop-field / drop-function surfaces a case this passive model cannot recover from: for bytes-input search, the SDK consults the cache for the `anns_field` vector type to encode the placeholder. After `drop_collection_field` followed by `add_collection_field` re-creating the field with a different type, the stale cache returns the old type, the server rejects the request, and because `search` has no schema-mismatch retry (unlike `insert_rows`), the error persists on every subsequent call.
+
+To close this, all four schema-mutating public methods — `drop_collection_field`, `drop_collection_function`, `add_collection_field` and `add_collection_function` (sync + async) — now invalidate the cache on success, so the next request re-fetches fresh schema from the server.
+
+---
+
+## 7. Sequence Diagram
+
+### 7.1 Drop Field Flow
+
+```
+Client              Proxy           RootCoord        WAL/Streaming      QueryNode (C++)
+  |                   |                 |                |                    |
+  | AlterCollectionSchema               |                |                    |
+  | (DropRequest: field_name="vec2")    |                |                    |
+  |------------------>|                 |                |                    |
+  |                   |                 |                |                    |
+  |                   | alterSchemaInFlight.LoadOrStore  |                    |
+  |                   | (check no concurrent schema op) |                    |
+  |                   |                 |                |                    |
+  |                   | checkSchemaVersionConsistency   |                    |
+  |                   | (all segments aligned?)         |                    |
+  |                   |                 |                |                    |
+  |                   | validateDropField               |                    |
+  |                   | (not PK, not last vector, etc.) |                    |
+  |                   |                 |                |                    |
+  |                   | AlterCollectionSchema           |                    |
+  |                   |---------------->|                |                    |
+  |                   |                 |                |                    |
+  |                   |                 | buildSchemaForDropField             |
+  |                   |                 | (remove field, version+1,          |
+  |                   |                 |  persist max_field_id)             |
+  |                   |                 |                |                    |
+  |                   |                 | Broadcast AlterCollectionMessage   |
+  |                   |                 |--------------->|                    |
+  |                   |                 |                |                    |
+  |                   |                 |         (WAL append + flush)       |
+  |                   |                 |                |                    |
+  |                   |                 | Ack Callback:  |                    |
+  |                   |                 | 1. AlterCollection (persist)       |
+  |                   |                 | 2. cascadeDropFieldIndexesInline   |
+  |                   |                 |    -> DropIndex for vec2 indexes   |
+  |                   |                 | 3. BroadcastAlteredCollection      |
+  |                   |                 | 4. ExpireCaches                    |
+  |                   |                 |                |                    |
+  |                   |<----------------|                |                    |
+  |<------------------|                 |                |                    |
+  |                   |                 |                |                    |
+  |                   |                 |                | (on next load/reload)
+  |                   |                 |                |                    |
+  |                   |                 |                | ComputeDiffBinlogs |
+  |                   |                 |                | has_field("vec2")  |
+  |                   |                 |                | -> false, skip     |
+  |                   |                 |                |                    |
+```
+
+---
+
+## 8. Key Design Decisions
+
+### 8.1 Lazy Data Cleanup (No Immediate Binlog Deletion)
+
+**Decision**: Dropped-field binlogs remain on object storage; they are simply skipped during segment loading.
+
+**Rationale**:
+- Immediate deletion would require scanning all segments for the field's binlogs, which is expensive
+- Binlogs are naturally cleaned up during compaction (compacted segments only contain current-schema fields)
+- Avoids complex rollback logic if deletion fails partway
+- Storage cost is bounded and decreasing (compaction gradually removes stale data)
+
+### 8.2 Inline Cascade via Ack Callback
+
+**Decision**: Execute index deletion inline within the broadcast ack callback, not as a separate RPC call.
+
+**Rationale**:
+- `broadcastAlterCollectionSchema()` holds a broadcast lock on the collection
+- Calling `DropIndex` as a separate RPC would deadlock (attempts to acquire the same lock)
+- The ack callback pattern is proven: `DropCollection` uses the same approach for cascade cleanup
+- Ack callbacks support exponential-backoff retry with idempotent operations
+
+### 8.3 Persistent max_field_id
+
+**Decision**: Store the historical maximum field ID in collection properties (`max_field_id` key).
+
+**Rationale**:
+- Without this, dropping the highest-ID field would cause `nextFieldID()` to return a previously-used ID
+- Reused field IDs would cause QueryNode to incorrectly load stale binlogs for the new field
+- The property persists through metadata updates and is read by `nextFieldID()` alongside current field IDs
+- Minimal storage overhead (one key-value pair per collection)
+
+### 8.4 Unified Entry Point (No Separate RPC)
+
+**Decision**: Use `AlterCollectionSchema` with `DropRequest` instead of a dedicated `DropCollectionField` RPC.
+
+**Rationale**:
+- Inherits all existing concurrency controls (mutual exclusion, version consistency gate)
+- Single API surface for all schema evolution operations
+- Consistent with the Add/Drop symmetry pattern
+- Simplifies client SDK and documentation
+
+### 8.5 Schema-Driven Filtering in C++
+
+**Decision**: Use the new/latest schema (`new_info.schema_`) for all `ComputeDiff*` filtering, not the segment's cached schema.
+
+**Rationale**:
+- The segment's cached schema may predate the drop operation
+- Using the latest schema ensures correct filtering on reload/delta-load
+- Single source of truth: the schema from the load request always reflects current state
+- Defense-in-depth: multiple layers check `has_field()` to catch edge cases
+
+---
+
+## 9. Testing Strategy
+
+### 9.1 Unit Tests
+
+| Component | Test Scope | Test Count |
+|-----------|-----------|------------|
+| **Proxy: validateDropField** | All constraint violations (PK, partition key, clustering key, dynamic, last vector, function reference, system field, not found) | 12 |
+| **Proxy: preExecuteDrop** | DropRequest by field_name / field_id / function_name, nil action, unknown action | 11 |
+| **Proxy: validateDropFunction** | Function found / not found | 4 |
+| **RootCoord: broadcastAlterCollectionSchemaDrop** | Full broadcast flow with ack callback | 1 |
+| **RootCoord: buildSchemaForDropFunction** | Function removal, output field cascade, droppedFieldIds | 3 |
+| **RootCoord: nextFieldID** | Properties-based max_field_id, historical collection compat | 4 |
+| **RootCoord: updateMaxFieldIDProperty** | Property creation and update | 4 |
+| **C++: SegmentLoadInfo** | ComputeDiffBinlogs/Indexes/ColumnGroups with dropped fields | 8 |
+
+### 9.2 E2E Tests
+
+| # | Scenario | Verification |
+|---|----------|-------------|
+| 1 | Drop scalar field (empty collection) | Schema updated, field absent |
+| 2 | Drop scalar field (with data) | Query does not return dropped field |
+| 3 | Drop indexed field | Index cascade deleted |
+| 4 | Drop one of multiple vector fields | Remaining vector field searchable |
+| 5 | Insert after drop | New data lacks dropped field, search works |
+| 6 | Drop + add same-name field | New field gets different field ID, no crash |
+| 7 | Constraint rejection | PK / partition key / last vector correctly refused |
+| 8 | Dynamic field disable/enable | Idempotent toggle, index cascade |
+| 9 | Loaded collection drop + reload | Search and query work after reload |
+| 10 | Drop BM25 function | Function + output fields + indexes all removed, input preserved |
+| 11 | Drop function input field | Rejected (must drop function first) |
+| 12 | Field ID reuse prevention | Drop + Add same-name different-type, search old data no crash |
+| 13 | Add + Drop serial interaction | Schema version consistency gate works |
+
+---
+
+## 10. Future Enhancements
+
+### 10.1 Physical Binlog Cleanup
+
+Add a background GC task that scans object storage and removes binlogs for fields no longer present in any segment's schema.
+
+### 10.2 Batch Drop
+
+Support dropping multiple fields/functions in a single `AlterCollectionSchema` request.
+
+### 10.3 Drop Field with Data Migration
+
+Support dropping a field while migrating its data to another field (e.g., renaming).
+
+---
+
+## 11. References
+
+- Add Function Field Design: [20260129-add-function-field-design.md](./20260129-add-function-field-design.md)
+- AlterCollectionSchema RPC: PR [#48810](https://github.com/milvus-io/milvus/pull/48810)
+- Milvus Architecture: [docs/architecture.md](../architecture.md)
@@ -0,0 +1,518 @@
+# MEP: Search Embedded Aggregation
+
+- **Created:** 2026-04-13
+- **Author(s):** @MrPresent-Han
+- **Status:** Draft (revised 2026-04-22)
+- **Component:** Proxy, QueryNode, Segcore, SDK
+- **Related Issues:** TBD
+- **Released:** TBD
+
+## Summary
+
+Extend Milvus vector search to support **per-group aggregation metrics**, **multi-field grouping**, and **hierarchical (nested) grouping** in a single request, so that one search call can return "top-K groups → per-group metrics (avg / sum / count / max / min) → per-group top-K hits (optionally sorted) → sub-groups". This is the combined capability that Elasticsearch exposes through nested `terms` + `top_hits` + metric sub-aggregations.
+
+The segcore foundation for multi-field composite-key grouping is already landed (milvus-io/milvus#48970, `feat: support multi-field composite group_by for vector search`). This MEP defines the full feature surface on top of that foundation: user-facing nested API, leaf-level metric aggregations, within-bucket document sort, bucket-level ordering, and a proxy-side reconstruction of arbitrarily deep grouping trees.
+
+## Motivation
+
+### Gap to Elasticsearch
+
+Milvus today supports single-field `group_by_field` + `group_size` for grouping vector search results, but lacks several capabilities that modern vector+analytics workloads expect from Elasticsearch:
+
+| # | Capability | ES | Milvus master | milvus-io/milvus#48970 | Gap |
+|---|------------|----|--------------|----------|-----|
+| R1 | Single-field grouping | yes | yes | — | covered |
+| R2 | Per-group top-K hits | yes (`top_hits.size`) | yes (`group_size`) | — | covered |
+| R3 | Multi-field flat (`multi_terms`) | yes | no | segcore only | Go / Proxy / SDK |
+| R4 | **Per-group metrics** (avg / sum / count / max / min + stats / cardinality / percentiles) | yes (20+) | **no** | no | **core gap** |
+| R5 | Non-vector sort **inside** a group (`top_hits.sort`) | yes | no | no | uncovered |
+| R6 | Hierarchical (nested) grouping | yes (nested `terms`) | no | no | uncovered |
+| R7 | **Bucket-level ordering** (`terms.order` / `multi_terms.order`) | yes | no | no | uncovered |
+
+R4 is the dominant gap. The most common vector-search analytics requests — "for each document, return the top-2 most similar chunks **and** the average similarity + chunk count of that document" — are impossible to express today without issuing two separate requests and stitching results on the client.
+
+**R5 vs R7 — two independent sort dimensions, often conflated.** ES exposes them through two different parameters, and we follow the same split:
+
+- **R5 (document sort, inside a bucket)** — `top_hits.sort` in ES. Controls the order of the `top_hits.size` hits returned **within** one bucket. Display-only.
+- **R7 (bucket sort, across buckets)** — `terms.order` / `multi_terms.order` in ES. Controls the order of the top-K buckets returned in the response, and — crucially — selects **which** buckets land in the top-K. Can sort by `_count`, `_key`, or any sub-metric alias.
+
+### Concrete Use Cases
+
+1. **E-commerce faceted search.** For each brand, return the 3 most relevant products and the average / minimum price of all matched products in that brand.
+2. **Document retrieval with signals.** For each document, return the top-2 most similar chunks and the average similarity, max similarity, and total chunk count of all chunks that fell in the retrieval pool.
+3. **Category → brand drill-down.** Group top-5 categories by total revenue; inside each category, group top-3 brands by average rating; inside each brand, return the 3 cheapest items.
+
+### Design Goals
+
+1. **ES semantics alignment.** Each level follows ES `multi_terms` (flat tuple keys per level). Hierarchy is expressed via a recursive `sub_group` just like ES nested `terms`.
+2. **Zero segcore rewrite for hierarchy.** Segcore keeps a single, flat composite-key contract; nested execution is reconstructed at the proxy. This is the same trade-off ES itself makes when splitting aggregation responsibilities between shards and the coordinator.
+3. **Strict layer isolation.** Segcore and the QN/Delegator know nothing about metrics, ordering, or levels. The proxy owns the entire aggregation model.
+4. **Reuse the query aggregation framework.** Metric accumulators (`AggregateBase`, type checking) already exist in `internal/agg/`. This MEP extends their use to the search path without reinventing them.
+5. **Honest approximation.** Where exact semantics are incompatible with ANN early-stop, document the approximation and expose a tunable knob rather than pretend to be exact. Elasticsearch itself already operates this way at the shard level.
+
+## Public Interfaces
+
+The user-facing API uses a **single recursive class** `GroupBy`, plus a small `TopHits` helper. Hierarchy is expressed by chaining `sub_group`. `metrics` is a `Dict[alias, spec]`; `order` and `top_hits.sort` are explicit lists for deterministic priority.
+
+### New classes (PyMilvus)
+
+```python
+class GroupBy:
+    fields:    List[str]           # required — composite key for this level
+    size:      int                 # required — max buckets returned at this level
+    metrics:   Dict[str, dict]     # optional — alias → {"op": "<op>", "field": "<name>|_score|*"}
+    order:     List[dict]          # optional — [{"<key>": "desc"}, ...] (priority-ordered)
+    top_hits:  TopHits             # optional — if absent, this level returns buckets only
+    sub_group: GroupBy             # optional — recursive child level
+
+class TopHits:
+    size: int                      # required — how many hits per bucket
+    sort: List[dict]               # optional — intra-bucket document sort criteria
+```
+
+`metrics` key is the **alias**: it is also what `order` references, and what appears in the response. Supported ops in Phase 1: `avg`, `sum`, `count`, `max`, `min`. The special field `_score` refers to the vector similarity distance. `count` follows SQL/ES semantics: `count("*")` counts **all rows** in the bucket; `count("field")` counts **only rows where `field` is non-null** — giving distinct, useful meaning on nullable fields. For non-nullable fields the two coincide. `avg` / `sum` / `max` / `min` skip nulls.
+
+### New `search()` parameter
+
+```python
+client.search(
+    ...,
+    group_by: GroupBy = None,       # replaces / supersedes group_by_field + group_size
+    ...
+)
+```
+
+When `group_by` is set, `limit` must **not** also be set — the root `GroupBy.size` controls the top-level bucket count and each level has its own `size`, so a user-supplied `limit` has no meaningful role. Sending both returns `merr.WrapErrParameterInvalid` with a message pointing users to `GroupBy.size`.
+
+### Examples
+
+**Flat single-level with metrics and bucket ordering:**
+
+```python
+results = client.search(
+    data=[query_vector], anns_field="embedding",
+    output_fields=["name", "price"],
+    group_by=GroupBy(
+        fields=["brand", "color"], size=10,
+        metrics={
+            "avg_price": {"avg":   "price"},
+            "doc_count": {"count": "*"},
+        },
+        order=[{"avg_price": "desc"}, {"_count": "desc"}],
+        top_hits=TopHits(size=3, sort=[{"field": "rating", "order": "desc"}]),
+    ),
+)
+```
+
+**Two-level nested:**
+
+```python
+group_by=GroupBy(
+    fields=["category"], size=5,
+    metrics={"total_revenue": {"sum": "price"}},
+    order=[{"total_revenue": "desc"}],
+    top_hits=TopHits(size=2),                        # level-1 also returns docs
+    sub_group=GroupBy(
+        fields=["brand"], size=3,
+        metrics={"avg_rating": {"avg": "rating"}},
+        order=[{"avg_rating": "desc"}],
+        top_hits=TopHits(size=3, sort=[{"field": "price", "order": "asc"}]),
+    ),
+)
+```
+
+**Three-level nested with pure-aggregation intermediate levels:** omit `top_hits` on a level to skip document return at that level (bucket key + metrics + sub-group only). See `pymilvus-search-agg-api-design-chn.md` §5.5 for the full example.
+
+### Response shape
+
+```json
+{
+  "groups": [
+    {
+      "key":     {"category": "electronics"},
+      "metrics": {"total_revenue": 48200.0, "avg_score": 0.87},
+      "hits":    [{"score": 0.95, "fields": {...}}, ...],
+      "sub_groups": [
+        {"key": {"brand": "Samsung"}, "metrics": {...}, "hits": [...], "sub_groups": []}
+      ]
+    }
+  ]
+}
+```
+
+`hits` is absent when `top_hits` is not configured at that level. `sub_groups` is absent at leaf levels.
+
+### Compatibility aliases
+
+- `group_by_field` + `group_size` (legacy) continue to work unchanged — they are **translated to** `GroupBy(fields=[field], size=topk, top_hits=TopHits(size=group_size))` at the proxy boundary.
+- `group_by_field_ids` (proto field 17, `repeated int64`, added by milvus-io/milvus#48970) is used only by the legacy SearchGroupBy reduce path (see §5.1). The new SearchAggregation path does **not** use field 17; it carries composite key values inside `fields_data`.
+- **Mutual exclusion**: setting both `group_by_field` and `group_by` in the same request is a `ParamError`.
+
+## Approximation Contract
+
+Approximation is accepted **by design**. ANN early-stop means exact semantics are not a goal at this layer — same trade-off as Elasticsearch distributed `terms` aggregation at shard level: partial collection visibility traded for latency and cost. Users who need exact results on the full post-filter collection should issue an independent scalar `query` aggregation request (see §Future Extensions).
+
+### Exactness by output type
+
+| Output | Exact? | Condition |
+|--------|--------|-----------|
+| Composite `_key` value of a returned row | ✓ | Always correct for rows that reach the proxy |
+| Bucket existence | ✗ | Keys whose rows never entered the ANN retrieval pool are invisible |
+| Bucket `_count` | ✗ | Counts rows in the retrieval pool, not the full collection |
+| Leaf-level metrics | ✗ | Computed over rows returned to the proxy |
+| Parent-level metrics (nested) | ✗✗ | Aggregate over child-bucket rows already subject to early-stop loss; bias compounds per level |
+| Top-K bucket selection by `order` | ✗ | Orders over observed (already-approximate) values |
+| Top-K hits within a bucket | Best-effort | Bounded by ANN pool × early-stop |
+
+### Bias amplification across nesting levels
+
+Approximation is not a single error source — it **cascades**. A parent-level metric aggregates over child-bucket rows that are themselves subject to early-stop loss. Using such a metric as an `order` key turns "the value is biased" into "the winner selection is biased." This is **accepted behavior, not a bug**. Users observing unacceptable bias can:
+
+1. Raise `group_count_safe_factor` to widen the retrieval pool.
+2. Fall back to scalar `query` aggregation for exact results.
+3. Restructure the query so only leaf-level metrics drive `order`.
+
+### Mitigation knobs
+
+| Knob | Default | Error source | Effect |
+|------|---------|--------------|--------|
+| `group_count_safe_factor` | 4 | Tuple coverage — small buckets crowded out by dominant ones | Segcore over-fetches `Πsize × factor` tuples; proxy applies `order` / `size` truncation. ES `shard_size` analog. |
+| `metric_safe_factor` | 1 (Phase 2) | Per-bucket sample size — metric accumulator sees only rows that cleared early-stop | Widens segcore per-bucket window. Reserved in proto; segcore implementation deferred. |
+
+Both knobs are orthogonal. Neither produces exact results alone; together they reduce both the likelihood and severity of approximation errors at the cost of retrieval work.
+
+## Design Details
+
+### 5.1 Layer isolation — the foundational principle
+
+| Layer | Responsibility |
+|-------|----------------|
+| **PyMilvus / Go SDK** | Serialize user's `GroupBy` tree into `SearchRequest.group_by` (proto) |
+| **Proxy** | Parse `GroupBy` → build `SearchAggregationContext`; build `SearchAggregationInfo` (flat field-id list) to send down; receive raw shard results; reconstruct hierarchy + metrics + ordering locally |
+| **QN / Delegator** | Merge growing + sealed segment results for each shard. **Zero awareness** of metrics, levels, or ordering — only does "multi-column group reduce" |
+| **Segcore** | Per-segment composite-key group-by (already done in milvus-io/milvus#48970) |
+
+**Two separate paths never mix at the wire:**
+
+| Path | Trigger | Wire carrier | Reduce logic |
+|------|---------|-------------|--------------|
+| SearchGroupBy (legacy) | `group_by_field_id > 0` | `SearchResultData.group_by_field_value` (field 8) | Single-field dedup |
+| **SearchAggregation (new)** | `agg_info != nil` | `SearchResultData.fields_data` (group-by + metric source fields + output fields) | Multi-field composite-key dedup |
+
+No proto translation between the two. Segcore decides behavior from `QueryInfo.group_by_field_id` vs `QueryInfo.agg_info`.
+
+### 5.2 Segcore — already landed in milvus-io/milvus#48970
+
+Key pieces shipped:
+
+- `SearchInfo.group_by_field_ids_` — `vector<FieldId>` replaces `optional<FieldId>`.
+- `CompositeGroupKey` (a small inline-storage vector of `GroupByValueType`) + `CompositeGroupKeyHash` using `folly::bits::hashMix`.
+- `MultiFieldDataGetter` — type-erases per-field value getters into `std::function<GroupByValueType(int64_t)>`.
+- `CompositeGroupByMap` — enforces `group_capacity` (topk) and per-group `group_size` with a `try_emplace + rollback` idiom.
+- `AssembleCompositeGroupByValues` — serializes each group-by field as a separate `FieldData` under `SearchResultData.group_by_field_values`.
+
+For the SearchAggregation path, segcore will additionally:
+
+- Accept `QueryInfo.agg_info` (new `SearchAggregationInfo` message).
+- When `agg_info` is set, emit group-by values + metric-source fields into `fields_data` instead of `group_by_field_values`. No new C++ types; the existing composite-key accumulator is reused verbatim.
+
+No changes to the composite-key accumulator are needed for Phase 1 — the per-bucket metric window equals `group_size` implicitly, same as the `top_hits` window. Higher-precision windows are tracked under §5.8 as a Phase-2 knob.
+
+### 5.3 QN / Delegator — unified `SearchGroupByReduce`
+
+`internal/querynodev2/segments/search_reduce.go::SearchGroupByReduce` is **upgraded in place**, not forked. A single struct handles both the single-column (field 8) and multi-column (`fields_data`) modes, unified via a `buildKeyExtractors` helper:
+
+```go
+func InitSearchReducer(info *reduce.ResultInfo) SearchReduce {
+    if info.GetGroupByFieldId() > 0 || info.GetAggInfo() != nil {
+        return &SearchGroupByReduce{}          // same struct, unified implementation
+    }
+    return &SearchCommonReduce{}
+}
+
+func buildKeyExtractors(srds []*schemapb.SearchResultData, info *reduce.ResultInfo) []func(int) string {
+    if info.GetAggInfo() != nil {
+        // multi-column: read composite from fields_data by group_by_field_ids
+        return buildMultiFieldExtractors(srds, info.GetAggInfo().GetGroupByFieldIds())
+    }
+    return buildSingleFieldExtractors(srds)    // single-column: read field 8
+}
+```
+
+Output is symmetric:
+
+| Mode | Output |
+|------|--------|
+| Single-column | `GroupByFieldValue` (field 8) preserved |
+| Multi-column | `fields_data` preserved; accepted rows only, rejected rows dropped |
+
+The reduce contract at this layer is the same for both modes: dedup PK, enforce `group_size` per composite key, cap at `topK` distinct keys. Nothing about metrics, nested levels, or ordering is visible here. See `qn-reduce-implementation-plan.md` for implementation detail.
+
+### 5.4 Proxy — `SearchAggregationContext` and `SearchAggregationComputer`
+
+A new Go package `internal/proxy/search_agg/` owns the aggregation model.
+
+#### 5.4.1 Context building (`context_builder.go`)
+
+```go
+func BuildSearchAggregationContext(
+    groupBy *commonpb.GroupBySpec,
+    schema  *schemapb.CollectionSchema,
+    nq      int64,
+) (*SearchAggregationContext, error)
+```
+
+Walks the `GroupBy` tree depth-first and produces:
+
+- `Levels []LevelContext` — one per level, root → leaf, with **`OwnFieldIDs` in user-specified order** (required for `_key` lexicographic comparison).
+- `GroupByFieldSlotIdx` — fieldID → slot index in each `SearchResultData`'s composite-key column set.
+- `FieldsDataSlotIdx` — fieldID → slot index in `fields_data` for metric-source reads and hit materialization.
+- `UserOutputFieldIDs` vs `InternalFieldIDs` — lets the proxy send metric-source / sort-key fields to QN without leaking them into the user-visible response.
+
+Validation at build time: metric source fields exist in schema; `order` keys reference a declared metric alias or the reserved `_count` / `_key`; no duplicate field ID across levels; per-level `size > 0`.
+
+#### 5.4.2 The computer (`computer.go`)
+
+```go
+type SearchAggregationComputer struct {
+    ctx     *SearchAggregationContext
+    results []*internalpb.SearchResults  // raw shard results, read-only
+}
+
+// One call returns one result tree per query vector. No internal state.
+func (c *SearchAggregationComputer) Compute() ([][]*AggBucketResult, error)
+```
+
+**Zero-copy row references.** No row is ever copied; the computer uses `RowRef{ResultIdx, RowIdx}` index pairs that point back into the original `SearchResults`. Per-field reads go through fieldID → slot lookups in the context.
+
+**One-pass top-down recursive descent.** For each level, a single sweep over the rows simultaneously accumulates:
+
+1. group key canonicalization → bucket map insert
+2. `Count` increment (used by `order: _count`)
+3. Per-metric `AggregateBase.Update` for the alias set at that level
+4. Top-K heap push if `TopHits` is configured
+5. `[]RowRef` collection (only if the level has a `sub_group`)
+
+After the sweep, finalize each bucket's metrics, apply `order`, truncate to `Size`, and recurse into `sub_group` with the per-bucket `[]RowRef`. The recursion is bounded by the user's `GroupBy` tree depth.
+
+#### 5.4.3 Pipeline integration
+
+`internal/proxy/search_pipeline.go` gets a new variant `searchWithAggPipe` consisting of a single `aggregateOp` that takes raw `[]*internalpb.SearchResults` and produces `*milvuspb.SearchResults` carrying new `SearchResultData.agg_buckets` / `agg_topks` proto fields. No intermediate `searchReduceOp` and no `endOp` / `highlightNode` post-processing — the computer assembles the final response directly. Routing picks this pipeline when `searchTask.aggCtx != nil`.
+
+### 5.5 Per-group metrics (R4)
+
+Metrics are accumulated at the proxy over the rows that reached the proxy from QN. This matches ES's scoping: "a metric sub-aggregation sees the documents that matched the query and fell into this bucket." Approximation characteristics are captured in the Approximation Contract section above.
+
+#### 5.5.1 Reuse vs new code
+
+**Reused from `internal/agg/`:**
+- `AggregateBase` interface, `SumAggregate`, `CountAggregate`, `MinAggregate`, `MaxAggregate`.
+- Type check rules from `internal/agg/type_check.go`.
+- `FieldValue` / `FieldAccessor` for physical field reads.
+
+**New in `search_agg/`:**
+- `bucketState` — pairs per-bucket `AggregateBase`s with the top-K heap and recursion rows.
+- `_score` synthetic source — reads `SearchResults.Scores[rowIdx]` instead of a segcore field, so metrics like `avg(_score)` / `max(_score)` work end-to-end. Restricted to `avg / max / min / sum` in Phase 1.
+- Metric proto (see §6) and response serialization.
+
+#### 5.5.2 Phased priority
+
+| Phase | Metric | Partial state | Notes |
+|-------|--------|---------------|-------|
+| **1** | `count` / `sum` / `avg` / `max` / `min` | `int64` / `double` / `(sum, count)` / value / value | MVP — exactly the ops with associative merge already in `internal/agg/` |
+| 2 | `cardinality` | HyperLogLog sketch | HLL merge |
+| 2 | `stats` | `(count, sum, min, max)` | elementwise |
+| 3 | `percentiles` | t-digest | digest merge |
+
+#### 5.5.3 Parent-level metrics and approximation
+
+Nested `sub_group` parents carrying metrics are supported. Parent metrics aggregate over all child-level retrieval pools — they are subject to bias amplification (see Approximation Contract). Users requiring exact values on the full post-filter collection should fall back to scalar `query` aggregation.
+
+### 5.6 Bucket ordering (R7) and intra-bucket sort (R5)
+
+#### 5.6.1 R7 — `order` selects the top-K buckets
+
+`order` is an ordered list (priority descending). Keys may be:
+
+- `_count` — bucket size (the computer's `Count` field)
+- `_key` — composite grouping value, compared lexicographically over `OwnFieldIDs` in user-specified order
+- any metric alias declared in the same level's `metrics`
+
+Multiple criteria give explicit tiebreakers; each has independent `asc`/`desc`.
+
+Default (when `order` is omitted): keep today's behavior for `group_by_field` — buckets ordered by best hit score. This is applied as an implicit `order=[{'_score': 'desc'}]` where `_score` reads the best (highest) hit score in the bucket. Milvus normalizes score direction at the kernel level so that higher = more similar regardless of metric type (L2 / IP / cosine), so a single `desc` default is uniform across metrics. This default is an **internal implicit rule** applied only when the user omits `order`; it is independent of the Q4 restriction on user-supplied explicit `_score` order keys (which still require `avg/max/min(_score)` wrapping).
+
+#### 5.6.2 R5 — `TopHits.sort` orders docs inside a bucket
+
+`TopHits.sort` applies only to the `TopHits.size` docs returned per bucket. It does **not** change which docs entered the metric accumulator or the bucket count — same display-only semantics as ES `top_hits.sort`. Phase 1 allowed sort keys: numeric + varchar + `_score`.
+
+#### 5.6.3 Early-stop impact
+
+For `order` keys other than `_score`, arrival-order early-stop is no longer safe (a small bucket could become the winning bucket later in the iterator). Mitigated via `group_count_safe_factor` — see Approximation Contract.
+
+### 5.7 Nested grouping (R6) — proxy-side reconstruction
+
+#### 5.7.1 Decision: no segcore nesting
+
+A tree-shaped partial-result schema, tree-shaped reduce, per-parent accumulators, and a new plan node type in segcore would be a large engine change, and the value-to-cost ratio is poor: vector-search analytics rarely exceed 2–3 grouping levels, and ES itself caps effective depth via `search.max_buckets` (default 65535 ≈ 4 levels at `size=10`).
+
+#### 5.7.2 Design: flatten at the proxy
+
+The proxy flattens all user levels into a single flat composite-key request for QN/segcore. The field id list sent down is the union of every level's `OwnFieldIDs`, deterministically ordered. Inflated capacity:
+
+```
+segcore_topk = Π(level.size for each level) * group_count_safe_factor
+               (hard cap: 65535, aligned with ES max_buckets)
+```
+
+Default `group_count_safe_factor = 4`. The proxy then reconstructs the tree per `computeLevel`, using the per-bucket `[]RowRef` pool for each parent's child recursion — no re-retrieval, no second query-agg trip.
+
+**Overflow rejection.** If `Π(level.size) × group_count_safe_factor > 65535`, the proxy rejects at parse time with `merr.WrapErrParameterInvalid`, naming the product, factor, and cap. No silent clamp. QN/Delegator additionally enforces its existing per-request memory budget as a defense-in-depth guard against pathological nested requests that somehow bypass proxy validation.
+
+**Max nesting depth.** Recursive `GroupBy.sub_group` is capped at `MAX_NEST_DEPTH = 10`. Requests exceeding this are rejected at parse time with `merr.WrapErrParameterInvalid` — guards against stack overflow and resource exhaustion in the proxy `computeLevel` recursion.
+
+### 5.8 Approximation knobs — proto placement
+
+Knob semantics and trade-offs are documented in the Approximation Contract above. Proto placement: `SearchAggregationInfo.group_count_safe_factor` (int32, default 4). `metric_safe_factor` is reserved in the proto but its segcore implementation is deferred to Phase 2; the Phase-1 release treats it as a no-op.
+
+## 6. Proto additions
+
+```protobuf
+// internal.proto — sent from Proxy to QN/Delegator
+message SearchAggregationInfo {
+    repeated int64 group_by_field_ids       = 1;  // union of all levels' OwnFieldIDs, flat
+    repeated int64 metric_field_ids         = 2;  // all metric-source fields, deduped
+    int32          group_count_safe_factor  = 3;  // default 4
+    int32          metric_safe_factor       = 4;  // default 1, Phase-2
+}
+
+// plan.proto — per-node plan carrier
+message QueryInfo {
+    ...
+    SearchAggregationInfo agg_info = <N>;           // non-nil → agg path
+}
+
+// milvus.proto — user-facing request
+message SearchRequest {
+    ...
+    common.GroupBySpec group_by = <N>;              // non-nil → agg path
+}
+
+// common.proto — recursive user spec
+message GroupBySpec {
+    repeated string            fields      = 1;
+    int64                      size        = 2;
+    map<string, MetricAggSpec> metrics     = 3;
+    repeated OrderSpec         order       = 4;
+    TopHitsSpec                top_hits    = 5;
+    GroupBySpec                sub_group   = 6;
+}
+message MetricAggSpec { string op = 1; string field = 2; }
+message OrderSpec     { string key = 1; string dir  = 2; }  // dir: "asc" | "desc"
+message TopHitsSpec   { int64  size = 1; repeated SortSpec sort = 2; }
+message SortSpec      { string field = 1; string order = 2; }
+
+// schema.proto — response carrier
+message SearchResultData {
+    ...
+    repeated AggBucket agg_buckets = 18;            // top-level buckets for all nq, flattened
+    repeated int64     agg_topks   = 19;            // number of top-level buckets per nq
+}
+message AggBucket {
+    repeated BucketKeyEntry key        = 1;
+    int64                   count      = 2;
+    map<string, double>     metrics    = 3;
+    repeated AggHit         hits       = 4;
+    repeated AggBucket      sub_groups = 5;
+}
+```
+
+All proto additions are optional / new-message fields; older binaries that don't understand them see unchanged behavior.
+
+## Compatibility, Deprecation, and Migration Plan
+
+- **No breaking changes.** `group_by_field` + `group_size` continue to work exactly as today. At the proxy boundary they are translated into `GroupBy(fields=[field], size=topk, top_hits=TopHits(size=group_size))`, but the translation reuses the **legacy SearchGroupBy path** (field 8) — no behavior drift.
+- **Routing invariant.** Path selection is keyed on the **original request source**, not the normalized form: if the incoming `SearchRequest` carried `group_by_field`, the request routes through the legacy SearchGroupBy path even though the proxy has materialized an internal `GroupBy` struct; only requests that originally carried `group_by` take the new SearchAggregation path. This prevents any chance of a legacy request being misrouted into the new agg pipeline after normalization.
+- **Proto additions are additive.** All new fields / messages are optional; mixed-version clusters are safe as long as all segcore / QN / proxy binaries reach the version that understands `QueryInfo.agg_info` before any client starts sending `SearchRequest.group_by`.
+- **Mutual exclusion** at the proxy: setting both `group_by_field_id` and `group_by` on the same request returns `merr.WrapErrParameterInvalid`. Prevents silent confusion about which path executed.
+- **Feature flag** `proxy.search.embeddedAggregation.enabled` (paramtable), default `off` for the first release. When off, `SearchRequest.group_by` is rejected with a clear error; `group_by_field` + `group_size` still work.
+- **Disallowed combinations in Phase 1.** The proxy hard-rejects any request that combines `group_by` with an incompatible feature, returning `merr.WrapErrParameterInvalid` rather than silently ignoring the conflicting parameter. Phase 1 disallows: hybrid search (multi-vector rerank), `highlight`, more than one JSON field in `GroupBy.fields` across all levels, and `limit` set together with `group_by` (use `GroupBy.size` instead). Revisited as these features are extended in later phases.
+
+## Test Plan
+
+1. **Segcore unit tests** (`test_search_group_by.cpp`, `test_group_by_json.cpp`).
+   - Multi-field composite key — sealed / growing / cross-segment reduce / single-field via composite path (covered by milvus-io/milvus#48970).
+   - `agg_info` present → group-by values emitted into `fields_data`; field 8 / 17 untouched.
+   - Nullable field in a composite key — `nullopt == nullopt` semantics.
+   - Hash collision on distinct keys → `operator==` separates them.
+2. **QN/Delegator unit tests** (`internal/querynodev2/segments/search_reduce_test.go`).
+   - Single-column backward compat (field 8 in → field 8 out).
+   - Multi-column single field (1 field in `fields_data`).
+   - Multi-column multi-field (2–3 fields, composite dedup).
+   - `group_size` / `topK` enforcement for both modes.
+   - Cross-segment merge — same composite key from different segments merged.
+   - `InitSearchReducer` routing on `GroupByFieldId > 0` vs `AggInfo != nil`.
+3. **Proxy unit tests** (`internal/proxy/search_agg/*_test.go`).
+   - `BuildSearchAggregationContext` field-id resolution (including JSON paths).
+   - `SearchAggregationComputer.Compute()` — flat / 2-level / 3-level fixtures with known ground truth.
+   - `order` by `_count`, `_key`, metric alias; multi-criterion tiebreaker priority.
+   - `TopHits.sort` independence from `order`.
+   - `normalizeGroupBy` translating legacy `group_by_field` to `GroupBySpec`.
+   - Mutual-exclusion validation.
+4. **Integration tests** (`tests/integration/`).
+   - Leaf-only metric happy path, cross-node.
+   - `group_count_safe_factor` over-fetch recovers small level-1 buckets under skewed distribution.
+   - `order` by `_count`, `_key`, metric alias — verify top-K buckets match expected ranking.
+   - Feature flag off → new params rejected.
+5. **E2E (pytest)** (`tests/python_client/milvus_client_v2/test_milvus_client_search_group_by.py`).
+   - End-to-end round-trip with 1-, 2-, 3-level `GroupBy`.
+   - Disallowed-combination rejection: hybrid search + `GroupBy`, `highlight` + `GroupBy`, multi-JSON `GroupBy.fields` — each must return `merr.WrapErrParameterInvalid`.
+
+## Rejected Alternatives
+
+### A1 — Run ANN to exhaustion, no early-stop
+
+Dropping early-stop gives ES-exact semantics but collapses search latency by orders of magnitude. Not recoverable by tuning.
+
+### A2 — Two-phase execution (search for hits + query-agg for metrics)
+
+Coordinator fires one search + one query-agg in parallel. Metric precision is perfect, but request latency roughly doubles and the two semantics don't compose naturally — the metric is computed over the entire post-filter collection, not the ANN retrieval pool, which is usually a different number than what users asking for "avg similarity of the chunks I actually retrieved" expect. Rejected for Phase 1; can be offered later as an explicit `metric_source="query"` option.
+
+### A3 — Native segcore nested execution
+
+Implement ES `terms -> terms` natively in segcore: tree-shaped partials, per-parent accumulators, tree-shaped reduce. Most ES-faithful, but a large engine change (new plan node type, new reduce code, new proto schema), and the user value over proxy-side reconstruction is low. Revisit if workloads demand level-1 metrics over deeply nested hierarchies.
+
+### A4 — Flat API only, no nesting at all
+
+Expose only `group_by_fields=[...]` (multi-field flat), let users cope with missing small buckets. Rejected: distribution-skew failure mode (small level-1 buckets silently vanishing) is a correctness surprise users cannot debug, and the hierarchy use case is one of the top-3 user requests.
+
+### A5 — Single `precision` knob instead of two `safe_factor`s
+
+One knob is easier to document but cannot independently control the two orthogonal error sources (tuple coverage vs per-bucket sample size). Rejected.
+
+## Future Extensions
+
+**Scalar query aggregation.** The operators (`count` / `sum` / `avg` / `min` / `max`) and the `GroupBy` class shape are not intrinsically tied to vector search — they reuse the same `internal/agg/` accumulators that the scalar query path already has. A natural follow-up is to adopt the same `GroupBy` API for scalar `query` calls, giving Milvus a unified GROUP BY surface across both retrieval paths. `_score` and `TopHits` would not apply in scalar mode; results would be exact (no ANN early-stop, no `safe_factor`). Out of scope for this MEP: migration from the existing `QueryPlanNode.Aggregate` surface needs its own proposal.
+
+## Open Questions
+
+1. **Per-bucket metric window (`metric_safe_factor`) — Phase 2.** Whether to widen segcore's per-bucket `group_size` window for metric accumulation beyond what `top_hits` needs. Landing this means touching the segcore operator; tentatively deferred until Phase 1 ships and we have data on metric precision complaints.
+2. **Metric on JSON dynamic types.** `sum(json_path)` requires a type cast. Follow `internal/agg/type_check.go` rules, or relax for search? **Tentative:** same rules.
+3. **`order` referencing an undeclared alias.** Hard error at parse time, or fall through to `_count`? **Tentative:** hard error.
+4. **`_score` as an `order` key directly.** ES forbids this and requires `avg/max/min(_score)` wrapping. Follow ES? **Tentative:** follow ES.
+5. **`group_count_safe_factor` / `metric_safe_factor` placement in the Python API.** Inside `GroupBy` (design plan's current choice), or on `search()` kwargs? **Tentative:** inside root `GroupBy`.
+6. **Cardinality backend.** Vendor HyperLogLog or reuse an internal library? Pending audit of existing HLL code in the codebase.
+7. **Hybrid search + `group_by`.** **Resolved — disallowed in Phase 1** (see Compatibility). Hybrid search has its own reducer and a clean composition with the agg pipeline needs a dedicated investigation; deferred to a later phase.
+
+## References
+
+- milvus-io/milvus#48970 — `feat: support multi-field composite group_by for vector search` (segcore composite-key foundation).
+- ES `multi_terms`: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-multi-terms-aggregation.html
+- ES `terms` approximation semantics: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html (see `shard_size`, `show_term_doc_count_error`)
+- Internal design docs (authoritative implementation plans):
+  - `~/hc-claude-projects/milvus/search-aggregation/pymilvus-search-agg-api-design-chn.md` — Python SDK API
+  - `~/hc-claude-projects/milvus/search-aggregation/segcore-implementation-plan.md` — C++ segcore
+  - `~/hc-claude-projects/milvus/search-aggregation/qn-reduce-implementation-plan.md` — QN / Delegator reduce
+  - `~/hc-claude-projects/milvus/search-aggregation/proxy-implementation-plan.md` — Go proxy: `SearchAggregationContext`, `SearchAggregationComputer`, pipeline integration
+- Milvus query aggregation framework (reused by proxy metrics): `internal/agg/` and `internal/core/src/exec/operator/query-agg/`
+- Milvus grouping search user doc: https://milvus.io/docs/grouping-search.md
@@ -0,0 +1,111 @@
+# GetReplicateConfiguration API Design
+
+**Date:** 2026-01-28
+
+## Overview
+
+Add a new public API `GetReplicateConfiguration` that allows cluster administrators to view the current cross-cluster replication topology. The API returns configured replication relationships without exposing sensitive connection parameters like tokens.
+
+## Motivation
+
+Operators need visibility into the current replication setup to:
+- Verify replication topology is configured correctly
+- Troubleshoot replication issues
+- Audit cluster configuration
+
+Currently, `UpdateReplicateConfiguration` exists but there's no corresponding read API to inspect the current state.
+
+## API Definition
+
+Add to `milvus.proto` on the `MilvusService`:
+
+```protobuf
+rpc GetReplicateConfiguration(GetReplicateConfigurationRequest)
+    returns (GetReplicateConfigurationResponse) {}
+
+message GetReplicateConfigurationRequest {
+  option (common.privilege_ext_obj) = {
+    object_type: Global
+    object_privilege: PrivilegeGetReplicateConfiguration
+    object_name_index: -1
+  };
+}
+
+message GetReplicateConfigurationResponse {
+  common.Status status = 1;
+  common.ReplicateConfiguration configuration = 2;
+}
+```
+
+### Response Behavior
+
+- Returns the existing `common.ReplicateConfiguration` structure
+- `ConnectionParam.token` fields are cleared/redacted before returning
+- If no replication is configured, returns empty configuration with success status
+
+## Security
+
+### Authorization
+
+- Requires **ClusterAdmin** privilege
+- Unauthenticated or unauthorized requests return permission denied error
+
+### Data Sanitization
+
+The implementation must sanitize sensitive fields before returning:
+- `MilvusCluster.connection_param.token` → empty string
+
+## Implementation Flow
+
+```
+Client SDK
+    │
+    ▼
+Proxy (MilvusService)
+    │  - Check ClusterAdmin permission
+    │  - Forward to StreamingCoord
+    ▼
+StreamingCoord
+    │  - Retrieve current ReplicateConfiguration
+    │  - Sanitize: clear token fields
+    │  - Return response
+    ▼
+Proxy
+    │  - Return to client
+    ▼
+Client SDK
+```
+
+## Files to Modify
+
+### Proto Changes (milvus-proto repo)
+- `proto/milvus.proto` - Add RPC definition and request/response messages
+
+### Proxy Layer
+- `internal/proxy/impl.go` - Add `GetReplicateConfiguration` method
+- `internal/proxy/proxy.go` - Wire up the new API
+
+### StreamingCoord Layer
+- `internal/streamingcoord/server/service/` - Add handler to fetch and sanitize config
+
+### Permission
+- Register the API with ClusterAdmin privilege check
+
+### Tests
+- Unit tests for sanitization logic
+- Integration test for the full API flow
+
+## Alternatives Considered
+
+### 1. Expose via StreamingCoordStateService only (internal)
+Rejected: Requires direct access to internal services; not accessible via standard SDKs.
+
+### 2. Return full configuration including tokens
+Rejected: Security risk; tokens should not be readable via API.
+
+### 3. Include health/status information
+Deferred: Keeping initial implementation simple to match the Update API symmetry. Health info can be added later if needed.
+
+## Open Questions
+
+None.
@@ -0,0 +1,238 @@
+# Force Promote for Primary-Secondary Failover
+
+**Date:** 2026-02-02
+
+## Overview
+
+Add a `force_promote` flag to the `UpdateReplicateConfiguration` API that allows a secondary cluster to immediately become a standalone primary when the original primary is unavailable. This enables active-passive failover for Milvus cross-cluster replication.
+
+## Motivation
+
+In Milvus cross-cluster replication, a secondary cluster applies configuration changes by waiting for the primary cluster to broadcast the `AlterReplicateConfigMessage` via CDC. If the primary becomes unreachable, the secondary blocks indefinitely because it can never receive the replicated message.
+
+Operators need a mechanism to:
+- Promote a secondary cluster to primary during disaster recovery
+- Resume service without waiting for the unreachable primary
+- Handle incomplete transactions and broadcasts left in an inconsistent state after failover
+
+## Current Replication Flow
+
+**Primary Cluster:**
+1. Receives `UpdateReplicateConfiguration` request
+2. Broadcasts `AlterReplicateConfigMessage` to all pchannels
+3. Returns after broadcast completes
+4. Configuration persisted via broadcast callback
+
+**Secondary Cluster:**
+1. Receives `UpdateReplicateConfiguration` request
+2. Attempts broadcast but fails with `ErrNotPrimary`
+3. Waits for CDC to replicate the `AlterReplicateConfigMessage` from primary
+4. Returns only when configuration matches
+
+**Problem:** Secondary clusters block indefinitely if the primary is unreachable.
+
+## Design
+
+### API Change
+
+Add `force_promote` field to the existing `UpdateReplicateConfigurationRequest`:
+
+```protobuf
+// In milvus.proto
+message UpdateReplicateConfigurationRequest {
+  common.ReplicateConfiguration replicate_configuration = 1;
+  bool force_promote = 2;  // Immediately promote secondary to standalone primary
+}
+```
+
+Add `force_promote` and `ignore` fields to the internal message header:
+
+```protobuf
+// In messages.proto
+message AlterReplicateConfigMessageHeader {
+  common.ReplicateConfiguration replicate_configuration = 1;
+  bool force_promote = 2;
+  bool ignore = 3;  // Skip processing of this message (used for incomplete broadcasts)
+}
+```
+
+Add metadata field to track force-promoted configurations:
+
+```protobuf
+// In streaming.proto
+message ReplicateConfigurationMeta {
+  common.ReplicateConfiguration replicate_configuration = 1;
+  bool force_promoted = 2;
+}
+```
+
+### Force Promote Constraints
+
+For safety, force promote requires an empty configuration and auto-constructs the standalone primary config from existing meta:
+
+| Constraint | Validation | Rationale |
+|---|---|---|
+| Secondary cluster only | Use `WithSecondaryClusterResourceKey()` API; returns error if primary | Only secondary clusters need emergency promotion |
+| Empty clusters field | `len(config.Clusters) == 0` | Config is auto-constructed from existing meta |
+| Empty topology field | `len(config.CrossClusterTopology) == 0` | Config is auto-constructed from existing meta |
+
+The auto-constructed configuration:
+- Contains a single cluster entry for the current cluster
+- Uses existing pchannels from the cluster's meta
+- Has no cross-cluster topology (standalone primary)
+
+### Force Promote Flow
+
+```
+Client SDK
+    │  UpdateReplicateConfiguration(config={}, force_promote=true)
+    ▼
+Proxy
+    │  Forward to StreamingCoord
+    ▼
+StreamingCoord (Assignment Service)
+    │  1. Validate empty cluster/topology fields
+    │  2. Use WithSecondaryClusterResourceKey() to acquire lock and verify secondary
+    │  3. Auto-construct standalone primary config from existing meta
+    │  4. Build message with AckSyncUp=true (disable fast DDL ack)
+    │  5. Broadcast AlterReplicateConfigMessage with ForcePromote=true
+    ▼
+StreamingNode (TxnBuffer)
+    │  6. Detect ForcePromote && !Ignore in message
+    │  7. Roll back all uncommitted transactions via RollbackAllUncommittedTxn()
+    ▼
+StreamingNode (Replicate Interceptor)
+    │  8. Detect ForcePromote && !Ignore in message header
+    │  9. Switch replication mode to primary
+    ▼
+StreamingCoord (Broadcast Callback)
+    │  10. Skip if Ignore=true (incomplete old message)
+    │  11. Fix incomplete broadcasts: mark with Ignore=true, supplement to remaining vchannels
+    │  12. Persist config with ForcePromoted=true flag
+    ▼
+Done — cluster is now standalone primary
+```
+
+### Handling Incomplete Messages
+
+When force promote executes, incomplete messages from the old topology must be handled:
+
+#### Transaction Rollback (via TxnBuffer)
+
+When TxnBuffer processes the forced `AlterReplicateConfigMessage`:
+
+1. TxnBuffer detects `ForcePromote == true && Ignore == false` in the message header
+2. Calls `RollbackAllUncommittedTxn()` to clean up all pending transactions
+3. All buffered transaction messages are discarded
+4. Rollback happens before the message is passed to downstream consumers
+
+```go
+// TxnBuffer method
+func (b *TxnBuffer) RollbackAllUncommittedTxn() {
+    for txnID := range b.builders {
+        b.rollbackTxn(txnID)
+    }
+    b.logger.Info("Rolled back all uncommitted transactions in TxnBuffer due to force promote")
+}
+```
+
+No remote detection or coordinator intervention is needed — each vchannel's TxnBuffer handles its own transactions.
+
+#### Incomplete Broadcast Fixing (In Callback on StreamingCoord)
+
+During force promote, incomplete broadcasts from previous operations (e.g., failed switchover) must be handled to prevent their callbacks from overwriting the force promote configuration.
+
+In the `alterReplicateConfiguration()` callback:
+
+1. Skip processing if `Ignore == true` (this is an old incomplete message)
+2. For force promote messages, call `FixIncompleteBroadcastsForForcePromote()`
+3. Mark incomplete `AlterReplicateConfigMessage` broadcasts with `Ignore=true`
+4. Supplement marked messages to their remaining vchannels
+5. This ensures old callbacks don't overwrite the new force promote config
+
+```go
+// Broadcaster internal method
+func (bm *broadcastTaskManager) FixIncompleteBroadcastsForForcePromote(ctx context.Context) error {
+    // 1. Find incomplete AlterReplicateConfig broadcasts
+    // 2. Update task messages with Ignore=true
+    // 3. Persist updated tasks to catalog
+    // 4. Supplement to remaining vchannels
+}
+```
+
+#### The `ignore` Field
+
+The `ignore` field in `AlterReplicateConfigMessageHeader` prevents processing of messages that were broadcast before force promote but completed after:
+
+| Location | Behavior when `Ignore=true` |
+|----------|----------------------------|
+| TxnBuffer | Skip transaction rollback |
+| Replicate Interceptor | Skip replication mode switch |
+| DDL ACK Callback | Skip config update and DDL fixing |
+| CDC Channel Replicator | Skip replication removal check |
+| CDC Stream Client | Skip message handling |
+| Replicate Service | Skip message overwrite |
+| Recovery Storage | Skip checkpoint and config update |
+
+## Files Modified
+
+### Proto Changes
+- `pkg/proto/messages.proto` — Add `force_promote` and `ignore` fields to `AlterReplicateConfigMessageHeader`
+- `pkg/proto/streaming.proto` — Add `force_promoted` to `ReplicateConfigurationMeta`
+
+### Core Implementation
+- `internal/streamingcoord/server/service/assignment.go` — Add `handleForcePromote()`, ignore field checks in ACK callback
+- `internal/streamingcoord/server/balancer/channel/manager.go` — Persist force promote flag in configuration meta
+- `internal/streamingcoord/server/broadcaster/broadcast_manager.go` — Add `WithSecondaryClusterResourceKey()`, `FixIncompleteBroadcastsForForcePromote()`
+- `internal/streamingcoord/server/broadcaster/broadcaster.go` — Add methods to `Broadcaster` interface
+- `internal/streamingnode/server/wal/utility/txn_buffer.go` — Add `RollbackAllUncommittedTxn()`, force promote detection in `HandleImmutableMessages()`
+- `internal/streamingnode/server/wal/interceptors/replicate/replicate_interceptor.go` — Add ignore field check
+- `internal/streamingnode/server/wal/recovery/recovery_storage_impl.go` — Add ignore field check
+
+### CDC Integration
+- `internal/cdc/replication/replicatemanager/channel_replicator.go` — Add ignore field check
+- `internal/cdc/replication/replicatestream/replicate_stream_client_impl.go` — Add ignore field check
+- `internal/cdc/util/util.go` — Add ignore field check in `IsReplicationRemovedByAlterReplicateConfigMessage()`
+
+### Client & Proxy
+- `internal/proxy/impl.go` — Pass through `force_promote` flag
+- `client/milvusclient/replicate_builder.go` — Add `WithForcePromote()` builder method
+- `internal/distributed/streaming/replicate_service.go` — Accept request object, add ignore field check
+- `internal/distributed/streaming/streaming.go` — Update `ReplicateService` interface
+
+### Tests
+- `internal/streamingcoord/server/service/assignment_test.go` — Force promote validation, ignore field, and DDL fixing tests
+- `internal/streamingnode/server/wal/utility/txn_buffer_test.go` — TxnBuffer rollback and force promote tests
+- `tests/integration/replication/force_promote_test.go` — Integration tests
+
+## Edge Cases
+
+1. **Primary cluster rejection** — Force promote rejected via `WithSecondaryClusterResourceKey()` returning `ErrNotSecondary`
+2. **Non-empty config rejection** — Force promote requires empty clusters/topology fields; non-empty configs are rejected
+3. **Concurrent force promotes** — `WithSecondaryClusterResourceKey()` acquires exclusive cluster-level lock
+4. **Idempotency** — `proto.Equal()` check skips duplicate updates
+5. **Incomplete switchover messages** — Marked with `ignore=true` before supplementing, preventing config overwrite
+6. **Empty pending broadcasts** — DDL fixing is a no-op when no incomplete broadcasts exist
+7. **Ignored messages** — All 7 locations check `ignore` field and skip processing
+
+## Alternatives Considered
+
+### 1. Append RollbackTxn messages to WAL for each transaction
+Rejected: Requires enumerating all in-flight transactions at the coordinator level and appending individual rollback messages. The `TxnBuffer.RollbackAllUncommittedTxn()` approach is simpler and handles transactions locally in each vchannel's buffer.
+
+### 2. Handle transaction rollback during WAL recovery
+Rejected: Force promote is not a WAL recovery event. The `AlterReplicateConfigMessage` propagates naturally through the WAL, and TxnBuffer's message handling is the correct place to trigger rollback.
+
+### 3. Separate API endpoint for force promote
+Rejected: Force promote is a specialized mode of `UpdateReplicateConfiguration`. Adding a separate endpoint would duplicate validation logic and complicate the client SDK.
+
+### 4. User-specified config for force promote
+Rejected: Allowing user-specified clusters/topology creates opportunities for configuration mismatches. Auto-constructing the config from existing meta ensures consistency and simplifies the API.
+
+### 5. Timestamp-based detection of incomplete messages
+Rejected: Using a `force_promote_timestamp` field to detect stale messages is fragile and requires clock synchronization. The `ignore` field approach is explicit and doesn't depend on timing.
+
+## Related Issues
+
+- https://github.com/milvus-io/milvus/issues/47351
+- https://github.com/milvus-io/milvus/pull/47352
@@ -0,0 +1,280 @@
+# Data Salvage for Force Failover
+
+- **Created:** 2026-02-05
+- **Author(s):** @bigsheeper
+- **Status:** Draft
+- **Component:** Proxy | StreamingCoord | StreamingNode
+- **Related Issues:** #47598
+- **Related Design:** [Force Promote for Primary-Secondary Failover](./20260202-force_promote_failover.md)
+
+## Summary
+
+This document describes the data salvage mechanism for recovering unreplicated data after a force failover. It introduces a salvage checkpoint that captures the last synced position from the old primary during force promote, and a `DumpMessages` API that allows external tools to retrieve WAL messages for data recovery.
+
+## Motivation
+
+When a force promote occurs, the secondary cluster immediately becomes a standalone primary. However, there may be data on the old primary cluster that was never replicated to the secondary:
+
+```
+Timeline:
+  Primary:     [msg1] [msg2] [msg3] [msg4] [msg5] [msg6] ---> (becomes unreachable)
+                                     ^
+                                     └── Last synced to secondary
+
+  Secondary:   [msg1] [msg2] [msg3] [msg4] ---> (force promotes to primary)
+                                     ^
+                                     └── Salvage checkpoint captures this position
+
+  Lost data:   [msg5] [msg6] (need recovery mechanism)
+```
+
+After the old primary becomes available again, operators need a way to:
+
+1. **Identify the data gap** — Know exactly which messages were not replicated
+2. **Retrieve lost messages** — Dump WAL messages from the old primary starting from the salvage checkpoint
+3. **Replay data** — Re-insert the recovered data into the new primary
+
+## Public Interfaces
+
+### New API: GetReplicateInfo
+
+Extends the existing `GetReplicateInfo` API response with salvage checkpoint:
+
+```protobuf
+message GetReplicateInfoResponse {
+  common.Status status = 1;
+  common.ReplicateCheckpoint checkpoint = 2;
+  common.ReplicateCheckpoint salvage_checkpoint = 3;  // NEW: Last synced position before force promote
+}
+```
+
+### New API: DumpMessages
+
+A streaming RPC that dumps WAL messages from a specified starting position:
+
+```protobuf
+// Dumps messages from a WAL channel for data recovery
+rpc DumpMessages(DumpMessagesRequest) returns (stream DumpMessagesResponse) {}
+
+message DumpMessagesRequest {
+  string pchannel = 1;                    // Physical channel to dump from
+  common.MessageID start_message_id = 2;  // Starting position (from salvage checkpoint)
+  uint64 start_timetick = 3;              // Optional: filter messages after this timetick
+  uint64 end_timetick = 4;                // Optional: stop dumping after this timetick (0 = no limit)
+  bool start_position_exclusive = 5;      // If true, skip the message at start_message_id
+}
+
+message DumpMessagesResponse {
+  oneof response {
+    common.Status status = 1;             // Error status (only on failure)
+    common.ImmutableMessage message = 2;  // Dumped message (success path)
+  }
+}
+```
+
+### Message Types Dumped
+
+Only replicable data messages are dumped. System messages are filtered out:
+
+| Dumped (Replicable) | Filtered (System) |
+|---------------------|-------------------|
+| Insert | TimeTick |
+| Delete | CreateSegment |
+| CreateCollection | Flush (system flush) |
+| DropCollection | RollbackTxn |
+| CreatePartition | |
+| DropPartition | |
+| BeginTxn | |
+| CommitTxn | |
+| Txn | |
+| Import | |
+| ManualFlush | |
+
+## Design Details
+
+### Salvage Checkpoint Capture
+
+During force promote, the salvage checkpoint is captured from the current replicate checkpoint:
+
+```
+Force Promote Flow (with salvage checkpoint):
+
+1. UpdateReplicateConfiguration(config, force_promote=true)
+       │
+       ▼
+2. StreamingCoord validates force promote constraints
+       │
+       ▼
+3. For each pchannel:
+   ┌─────────────────────────────────────────────────────────┐
+   │  salvageCheckpoints[pchannel] = currentReplicateCheckpoint │
+   └─────────────────────────────────────────────────────────┘
+       │
+       ▼
+4. Broadcast AlterReplicateConfigMessage with force_promote=true
+       │
+       ▼
+5. Persist ReplicateConfigurationMeta with:
+   - force_promoted = true
+   - force_promote_timestamp = current timestamp
+   - salvage_checkpoints = map[pchannel]checkpoint
+```
+
+### Salvage Checkpoint Storage
+
+Salvage checkpoints are stored in the replicate configuration metadata:
+
+```protobuf
+// In streaming.proto
+message ReplicateConfigurationMeta {
+  common.ReplicateConfiguration replicate_configuration = 1;
+  bool force_promoted = 2;
+  uint64 force_promote_timestamp = 3;
+  map<string, common.ReplicateCheckpoint> salvage_checkpoints = 4;  // NEW
+}
+```
+
+### DumpMessages Implementation
+
+The `DumpMessages` API uses the WAL scanner to stream messages:
+
+```go
+func (node *Proxy) DumpMessages(req *DumpMessagesRequest, stream DumpMessagesServer) error {
+    // 1. Validate request
+    if req.Pchannel == "" || req.StartMessageId == nil {
+        return status.Error(codes.InvalidArgument, "pchannel and start_message_id required")
+    }
+
+    // 2. Create WAL scanner from start position
+    scanner := streaming.WAL().Read(ctx, ReadOption{
+        VChannel:     req.Pchannel,  // Uses pchannel as vchannel for raw access
+        DeliverPolicy: StartFrom(req.StartMessageId, req.StartPositionExclusive),
+    })
+    defer scanner.Close()
+
+    // 3. Stream messages until end condition
+    for {
+        msg, err := scanner.Next()
+        if err == io.EOF {
+            return nil
+        }
+
+        // Filter system messages
+        if !shouldDumpMessage(msg.MessageType()) {
+            continue
+        }
+
+        // Check end timetick
+        if req.EndTimetick > 0 && msg.TimeTick() > req.EndTimetick {
+            return nil
+        }
+
+        // Filter by start timetick
+        if req.StartTimetick > 0 && msg.TimeTick() < req.StartTimetick {
+            continue
+        }
+
+        // Send message
+        stream.Send(&DumpMessagesResponse{
+            Response: &DumpMessagesResponse_Message{
+                Message: convertToImmutableMessage(msg),
+            },
+        })
+    }
+}
+```
+
+### Data Recovery Workflow
+
+After force failover, operators can recover data using these steps:
+
+```
+Data Recovery Workflow:
+
+1. Get salvage checkpoint from new primary (force-promoted cluster):
+
+   resp = new_primary.GetReplicateInfo(pchannel)
+   salvage_checkpoint = resp.salvage_checkpoint
+
+2. Get current checkpoint from old primary (when it becomes available):
+
+   resp = old_primary.GetReplicateInfo(pchannel)
+   current_checkpoint = resp.checkpoint
+
+3. Dump messages from old primary starting at salvage checkpoint:
+
+   stream = old_primary.DumpMessages(
+       pchannel = pchannel,
+       start_message_id = salvage_checkpoint.message_id,
+       end_timetick = current_checkpoint.timetick,  // Optional bound
+   )
+
+   for msg in stream:
+       # Process message (Insert, Delete, DDL, etc.)
+       replay_message_to_new_primary(msg)
+
+4. Verify data consistency between clusters
+```
+
+### Security Considerations
+
+The `DumpMessages` API exposes raw WAL data, which may contain sensitive information:
+
+1. **Authentication** — Standard Milvus authentication applies
+2. **Authorization** — Requires admin/root privileges (to be enforced)
+3. **Audit logging** — All DumpMessages calls should be logged for compliance
+4. **Network isolation** — Recommend using internal network for data recovery operations
+
+## Files Modified
+
+### Proto Changes
+- `proto/milvus.proto` — Add `DumpMessages` RPC, `DumpMessagesRequest`, `DumpMessagesResponse`
+- `proto/common.proto` — Add `salvage_checkpoint` to `GetReplicateInfoResponse` (if not already added)
+
+### Core Implementation
+- `internal/streamingcoord/server/service/assignment.go` — Capture salvage checkpoints during force promote
+- `internal/streamingcoord/server/balancer/channel/manager.go` — Store/retrieve salvage checkpoints
+- `internal/proxy/impl.go` — Implement `DumpMessages` handler
+- `internal/distributed/proxy/service.go` — Add gRPC service method
+- `internal/distributed/streaming/replicate_service.go` — Add `GetSalvageCheckpoint` method
+
+### Tests
+- `internal/proxy/dump_messages_test.go` — Unit tests for message filtering
+- `tests/integration/replication/data_salvage_test.go` — Integration tests for DumpMessages and salvage checkpoint
+
+## Test Plan
+
+1. **Unit Tests**
+   - Test `shouldDumpMessage` correctly filters system messages
+   - Test salvage checkpoint capture during force promote
+
+2. **Integration Tests**
+   - Test `GetReplicateInfo` returns salvage checkpoint after force promote
+   - Test `DumpMessages` streams messages from specified position
+   - Test `DumpMessages` respects timetick filters
+   - Test `DumpMessages` handles missing pchannel/message_id
+
+3. **E2E Recovery Test**
+   - Set up primary-secondary replication
+   - Insert data on primary
+   - Simulate primary failure and force promote secondary
+   - Recover old primary
+   - Dump messages from old primary using salvage checkpoint
+   - Verify data can be replayed to new primary
+
+## Rejected Alternatives
+
+### 1. Automatic data sync after old primary recovery
+Rejected: Automatic sync could cause conflicts if the new primary has diverged. Manual recovery gives operators control over the reconciliation process.
+
+### 2. Store salvage data in object storage during force promote
+Rejected: Would require significant additional infrastructure and increase force promote latency. The WAL already contains all needed data.
+
+### 3. Extend CDC to handle salvage recovery
+Rejected: CDC is designed for continuous replication, not point-in-time recovery. A dedicated API provides clearer semantics and better debugging.
+
+## References
+
+- [Force Promote Design Document](./20260202-force_promote_failover.md)
+- [Milvus Issue #47598](https://github.com/milvus-io/milvus/issues/47598)
+- [Implementation PR #47599](https://github.com/milvus-io/milvus/pull/47599)
@@ -0,0 +1,380 @@
+# Per-Cluster mTLS for CDC Cross-Cluster Replication
+
+## Overview
+
+Starting from v2.6.x, Milvus CDC supports **per-cluster mTLS configuration** for cross-cluster replication. Each target cluster can have its own CA certificate, client certificate, and client key, enabling secure replication across clusters that use independent certificate authorities.
+
+Key capabilities:
+
+- **Independent CA per cluster**: Each cluster can have its own Certificate Authority, preventing cross-cluster credential misuse
+- **Per-cluster client identity**: CDC uses a distinct client certificate for each target cluster, enabling fine-grained access control and audit
+
+## Quick Start
+
+A minimal 2-cluster example: cluster A (by-dev1) replicates to cluster B (by-dev2), each with its own CA.
+
+**1. Generate per-cluster certs**
+
+```bash
+for i in 1 2; do
+    openssl genrsa -out ca-dev${i}.key 4096 2>/dev/null
+    openssl req -new -x509 -key ca-dev${i}.key -out ca-dev${i}.pem -days 3650 -subj "/CN=CA-dev${i}"
+
+    openssl genrsa -out server-dev${i}.key 2048 2>/dev/null
+    openssl req -new -key server-dev${i}.key -out server-dev${i}.csr -subj "/CN=server-dev${i}"
+    openssl x509 -req -in server-dev${i}.csr -CA ca-dev${i}.pem -CAkey ca-dev${i}.key \
+        -CAcreateserial -out server-dev${i}.pem -days 3650 \
+        -extfile <(printf "subjectAltName=DNS:localhost,IP:127.0.0.1")
+
+    openssl genrsa -out client-dev${i}.key 2048 2>/dev/null
+    openssl req -new -key client-dev${i}.key -out client-dev${i}.csr -subj "/CN=cdc-to-dev${i}"
+    openssl x509 -req -in client-dev${i}.csr -CA ca-dev${i}.pem -CAkey ca-dev${i}.key \
+        -CAcreateserial -out client-dev${i}.pem -days 3650
+done
+rm -f *.csr *.srl
+```
+
+**2. Add per-cluster TLS to each cluster's `milvus.yaml`**
+
+Each cluster uses its own server cert and CA. The `tls.clusters` section (for CDC outbound) is the same on all clusters.
+
+Cluster A (`milvus.yaml`):
+
+```yaml
+tls:
+  serverPemPath: /certs/server-dev1.pem
+  serverKeyPath: /certs/server-dev1.key
+  caPemPath: /certs/ca-dev1.pem
+  clusters:
+    by-dev1:
+      caPemPath: /certs/ca-dev1.pem
+      clientPemPath: /certs/client-dev1.pem
+      clientKeyPath: /certs/client-dev1.key
+    by-dev2:
+      caPemPath: /certs/ca-dev2.pem
+      clientPemPath: /certs/client-dev2.pem
+      clientKeyPath: /certs/client-dev2.key
+```
+
+Cluster B (`milvus.yaml`):
+
+```yaml
+tls:
+  serverPemPath: /certs/server-dev2.pem
+  serverKeyPath: /certs/server-dev2.key
+  caPemPath: /certs/ca-dev2.pem
+  clusters:
+    by-dev1:
+      caPemPath: /certs/ca-dev1.pem
+      clientPemPath: /certs/client-dev1.pem
+      clientKeyPath: /certs/client-dev1.key
+    by-dev2:
+      caPemPath: /certs/ca-dev2.pem
+      clientPemPath: /certs/client-dev2.pem
+      clientKeyPath: /certs/client-dev2.key
+```
+
+**3. Start clusters with mTLS**
+
+```bash
+# Each cluster uses its own server cert
+export COMMON_SECURITY_TLSMODE=2
+
+# Cluster A
+TLS_CAPEMPATH=/certs/ca-dev1.pem TLS_SERVERPEMPATH=/certs/server-dev1.pem \
+    TLS_SERVERKEYPATH=/certs/server-dev1.key ./milvus run standalone
+
+# Cluster B (separate machine or different ports)
+TLS_CAPEMPATH=/certs/ca-dev2.pem TLS_SERVERPEMPATH=/certs/server-dev2.pem \
+    TLS_SERVERKEYPATH=/certs/server-dev2.key ./milvus run standalone
+```
+
+**4. Configure replication (A -> B)**
+
+```python
+from pymilvus import MilvusClient
+
+client = MilvusClient(
+    uri="https://localhost:19530", token="root:Milvus",
+    ca_pem_path="/certs/ca-dev1.pem",
+    client_pem_path="/certs/client-dev1.pem",
+    client_key_path="/certs/client-dev1.key",
+)
+
+client.update_replicate_configuration(
+    clusters=[
+        {"cluster_id": "by-dev1",
+         "connection_param": {"uri": "https://cluster-a:19530", "token": "root:Milvus"},
+         "pchannels": [f"by-dev1-rootcoord-dml_{i}" for i in range(16)]},
+        {"cluster_id": "by-dev2",
+         "connection_param": {"uri": "https://cluster-b:19531", "token": "root:Milvus"},
+         "pchannels": [f"by-dev2-rootcoord-dml_{i}" for i in range(16)]},
+    ],
+    cross_cluster_topology=[
+        {"source_cluster_id": "by-dev1", "target_cluster_id": "by-dev2"},
+    ],
+)
+```
+
+CDC will now use `ca-dev2.pem` + `client-dev2.pem` when connecting to cluster B, and log `"CDC outbound TLS enabled" [targetCluster=by-dev2]`.
+
+## Background
+
+Milvus CDC replicates data between clusters by consuming change streams from a source cluster and writing to one or more target clusters. When clusters enforce mTLS (`tlsMode=2`), CDC must present a valid client certificate trusted by each target cluster's CA.
+
+Previous versions required a **shared CA** across all clusters, meaning a single `caPemPath` was used for all outbound connections. This has two drawbacks:
+
+1. **No isolation**: If CDC accidentally uses the wrong client cert for a target, the shared CA still trusts it, masking configuration errors
+2. **Operational risk**: Compromising the shared CA affects all clusters simultaneously
+
+Per-cluster mTLS solves both problems by allowing each cluster to operate under its own CA.
+
+## Configuration
+
+### Parameters
+
+Per-cluster TLS is configured under the `tls.clusters` section in `milvus.yaml`. Each entry is keyed by the cluster ID (the value used in `UpdateReplicateConfiguration`).
+
+| Parameter | Type | Description |
+|---|---|---|
+| `tls.clusters.<clusterID>.caPemPath` | string | Path to the CA certificate used to verify the target cluster's server certificate |
+| `tls.clusters.<clusterID>.clientPemPath` | string | Path to the client certificate presented to the target cluster |
+| `tls.clusters.<clusterID>.clientKeyPath` | string | Path to the client private key |
+
+**Activation rule**: TLS is enabled for a target cluster only when **both** `clientPemPath` and `clientKeyPath` are set. If only `caPemPath` is set, TLS is not activated. TLS 1.3 is enforced as the minimum version.
+
+### Server-Side TLS Parameters
+
+Each cluster's Milvus server also needs its own TLS configuration:
+
+| Parameter | Type | Description |
+|---|---|---|
+| `common.security.tlsMode` | int | `0` = disabled, `1` = one-way TLS, `2` = mTLS |
+| `tls.caPemPath` | string | CA certificate for verifying client certs (server-side) |
+| `tls.serverPemPath` | string | Server certificate (must have SAN matching the server's hostname/IP) |
+| `tls.serverKeyPath` | string | Server private key |
+
+### Configuration Example
+
+A 3-cluster deployment where each cluster has its own CA:
+
+```yaml
+# milvus.yaml — CDC outbound TLS configuration
+tls:
+  serverPemPath: /certs/server-dev1.pem
+  serverKeyPath: /certs/server-dev1.key
+  caPemPath: /certs/ca-dev1.pem
+  clusters:
+    by-dev1:
+      caPemPath: /certs/ca-dev1.pem
+      clientPemPath: /certs/client-dev1.pem
+      clientKeyPath: /certs/client-dev1.key
+    by-dev2:
+      caPemPath: /certs/ca-dev2.pem
+      clientPemPath: /certs/client-dev2.pem
+      clientKeyPath: /certs/client-dev2.key
+    by-dev3:
+      caPemPath: /certs/ca-dev3.pem
+      clientPemPath: /certs/client-dev3.pem
+      clientKeyPath: /certs/client-dev3.key
+```
+
+> **Note**: The top-level `tls.serverPemPath`, `tls.serverKeyPath`, and `tls.caPemPath` configure the cluster's own server-side TLS. The `tls.clusters.*` section configures CDC's **outbound** client credentials per target cluster.
+
+## Certificate Generation
+
+### Per-Cluster CA and Certificates
+
+Generate an independent CA and certificates for each cluster. Below is an example for 3 clusters:
+
+```bash
+#!/bin/bash
+CERT_DIR="./certs"
+DAYS=3650
+mkdir -p "${CERT_DIR}"
+
+for i in 1 2 3; do
+    # CA (self-signed)
+    openssl genrsa -out "${CERT_DIR}/ca-dev${i}.key" 4096
+    openssl req -new -x509 -key "${CERT_DIR}/ca-dev${i}.key" \
+        -out "${CERT_DIR}/ca-dev${i}.pem" -days ${DAYS} \
+        -subj "/CN=MilvusCA-dev${i}"
+
+    # Server cert (SAN must match how clients connect)
+    openssl genrsa -out "${CERT_DIR}/server-dev${i}.key" 2048
+    openssl req -new -key "${CERT_DIR}/server-dev${i}.key" \
+        -out "${CERT_DIR}/server-dev${i}.csr" -subj "/CN=milvus-server-dev${i}"
+    openssl x509 -req -in "${CERT_DIR}/server-dev${i}.csr" \
+        -CA "${CERT_DIR}/ca-dev${i}.pem" -CAkey "${CERT_DIR}/ca-dev${i}.key" \
+        -CAcreateserial -out "${CERT_DIR}/server-dev${i}.pem" -days ${DAYS} \
+        -extfile <(printf "subjectAltName=DNS:localhost,IP:127.0.0.1")
+
+    # CDC client cert (for CDC connecting TO this cluster)
+    openssl genrsa -out "${CERT_DIR}/client-dev${i}.key" 2048
+    openssl req -new -key "${CERT_DIR}/client-dev${i}.key" \
+        -out "${CERT_DIR}/client-dev${i}.csr" -subj "/CN=cdc-to-dev${i}"
+    openssl x509 -req -in "${CERT_DIR}/client-dev${i}.csr" \
+        -CA "${CERT_DIR}/ca-dev${i}.pem" -CAkey "${CERT_DIR}/ca-dev${i}.key" \
+        -CAcreateserial -out "${CERT_DIR}/client-dev${i}.pem" -days ${DAYS}
+done
+
+rm -f "${CERT_DIR}"/*.csr "${CERT_DIR}"/*.srl
+```
+
+### What Gets Generated
+
+For each cluster `dev{i}`:
+
+| File | Signed By | Purpose |
+|---|---|---|
+| `ca-dev{i}.pem` / `ca-dev{i}.key` | Self-signed | Certificate Authority for cluster by-dev{i} |
+| `server-dev{i}.pem` / `server-dev{i}.key` | `ca-dev{i}` | Server certificate (used by Milvus proxy/nodes) |
+| `client-dev{i}.pem` / `client-dev{i}.key` | `ca-dev{i}` | CDC client certificate for connecting TO by-dev{i} |
+
+### Why Separate CAs Matter
+
+With a shared CA, using the wrong `caPemPath` for a target cluster silently succeeds — the shared CA trusts all certificates. With per-cluster CAs:
+
+- `ca-dev1.pem` only trusts `server-dev1.pem` and `client-dev1.pem`
+- Connecting to cluster B (`server-dev2.pem`) with cluster A's CA (`ca-dev1.pem`) fails with `CERTIFICATE_VERIFY_FAILED`
+- This ensures CDC configuration errors are caught immediately rather than silently allowing misrouted connections
+
+## Setting Up Replication
+
+### Step 1: Configure mTLS on Each Cluster
+
+Each cluster's server-side TLS is configured via environment variables or `milvus.yaml`:
+
+```bash
+# Cluster A (by-dev1)
+export COMMON_SECURITY_TLSMODE=2
+export TLS_CAPEMPATH=/certs/ca-dev1.pem
+export TLS_SERVERPEMPATH=/certs/server-dev1.pem
+export TLS_SERVERKEYPATH=/certs/server-dev1.key
+```
+
+### Step 2: Configure Per-Cluster CDC Outbound TLS
+
+Add the `tls.clusters` section to `milvus.yaml` as shown in [Configuration Example](#configuration-example). Since cluster IDs contain dashes (e.g., `by-dev1`), these **must** be configured in `milvus.yaml` — environment variables cannot represent dashes in nested key names.
+
+Point `MILVUSCONF` to the directory containing the modified `milvus.yaml`:
+
+```bash
+export MILVUSCONF=/path/to/config-dir
+```
+
+### Step 3: Configure Replication Topology
+
+Use the `UpdateReplicateConfiguration` API via PyMilvus. The API only carries `uri` and `token` — no TLS fields:
+
+```python
+from pymilvus import MilvusClient
+
+# Connect to each cluster with its own CA and client cert
+client_a = MilvusClient(
+    uri="https://cluster-a:19530", token="root:Milvus",
+    ca_pem_path="/certs/ca-dev1.pem",
+    client_pem_path="/certs/pymilvus-dev1.pem",
+    client_key_path="/certs/pymilvus-dev1.key",
+)
+
+# Build replication config: A -> B, A -> C
+config = {
+    "clusters": [
+        {
+            "cluster_id": "by-dev1",
+            "connection_param": {"uri": "https://cluster-a:19530", "token": "root:Milvus"},
+            "pchannels": [f"by-dev1-rootcoord-dml_{i}" for i in range(16)],
+        },
+        {
+            "cluster_id": "by-dev2",
+            "connection_param": {"uri": "https://cluster-b:19531", "token": "root:Milvus"},
+            "pchannels": [f"by-dev2-rootcoord-dml_{i}" for i in range(16)],
+        },
+        {
+            "cluster_id": "by-dev3",
+            "connection_param": {"uri": "https://cluster-c:19532", "token": "root:Milvus"},
+            "pchannels": [f"by-dev3-rootcoord-dml_{i}" for i in range(16)],
+        },
+    ],
+    "cross_cluster_topology": [
+        {"source_cluster_id": "by-dev1", "target_cluster_id": "by-dev2"},
+        {"source_cluster_id": "by-dev1", "target_cluster_id": "by-dev3"},
+    ],
+}
+
+# Apply to ALL clusters (the current primary processes the config change)
+client_a.update_replicate_configuration(**config)
+```
+
+> **Important**: Send `UpdateReplicateConfiguration` to **all clusters** in parallel. Only the current primary processes it, but if you don't know which cluster is primary (e.g., after a switchover), sending to all ensures the config is applied.
+
+## Troubleshooting
+
+### Verify Per-Cluster Cert Selection in CDC Logs
+
+When CDC establishes outbound connections, it logs the cert paths for each target:
+
+```
+[INFO] [cluster/milvus_client.go:78] ["CDC outbound TLS enabled"]
+    [targetCluster=by-dev2]
+    [caPemPath=/certs/ca-dev2.pem]
+    [clientPemPath=/certs/client-dev2.pem]
+    [clientKeyPath=/certs/client-dev2.key]
+```
+
+Verify that:
+- Each target cluster has a **different** `caPemPath` (e.g., `ca-dev2.pem` for by-dev2, not `ca.pem`)
+- Each target cluster has the **matching** `clientPemPath` (e.g., `client-dev2.pem` for by-dev2)
+
+### Common Errors
+
+| Symptom | Cause | Fix |
+|---|---|---|
+| `CERTIFICATE_VERIFY_FAILED` in CDC logs | Wrong `caPemPath` for target cluster | Ensure `tls.clusters.<id>.caPemPath` matches the CA that signed the target's server cert |
+| `TLSV1_ALERT_UNKNOWN_CA` in CDC logs | Target cluster doesn't trust CDC's client cert | Ensure the client cert is signed by the CA configured as the target's server-side `tls.caPemPath` |
+| `TLSV1_ALERT_CERTIFICATE_REQUIRED` | CDC connecting without client cert | Ensure both `clientPemPath` and `clientKeyPath` are set in `tls.clusters.<id>` |
+| CDC log shows no "CDC outbound TLS enabled" | Missing or incomplete config | Verify `tls.clusters` section exists in the `milvus.yaml` at `MILVUSCONF` path, not the source checkout |
+| `UpdateReplicateConfiguration` hangs | Sent to a secondary cluster only | Send to all clusters in parallel so the actual primary processes it |
+
+### Verify Cross-CA Isolation
+
+To confirm that per-cluster CAs are actually enforced, connect to one cluster using another cluster's CA — it should fail:
+
+```bash
+# This should FAIL — ca-dev1 does not trust server-dev2
+curl --cacert /certs/ca-dev1.pem \
+     --cert /certs/client-dev1.pem \
+     --key /certs/client-dev1.key \
+     https://cluster-b:19531/healthz
+# Expected: SSL certificate problem: certificate verify failed
+```
+
+## Switchover
+
+When switching the primary (e.g., from A to B), simply call `UpdateReplicateConfiguration` with the new topology on **all clusters**:
+
+```python
+# Switchover: B becomes primary, replicates to A and C
+config["cross_cluster_topology"] = [
+    {"source_cluster_id": "by-dev2", "target_cluster_id": "by-dev1"},
+    {"source_cluster_id": "by-dev2", "target_cluster_id": "by-dev3"},
+]
+
+# Send to all clusters in parallel
+from concurrent.futures import ThreadPoolExecutor
+with ThreadPoolExecutor(max_workers=3) as executor:
+    for client in [client_a, client_b, client_c]:
+        executor.submit(client.update_replicate_configuration, **config)
+```
+
+No certificate changes are needed — each cluster's CDC already has the per-cluster TLS config for all possible targets.
+
+## Version Compatibility
+
+| Feature | Minimum Version |
+|---|---|
+| CDC cross-cluster replication | v2.6.x |
+| Per-cluster mTLS (`tls.clusters.*`) | v2.6.x |
+| `UpdateReplicateConfiguration` API | v2.6.x |
+| PyMilvus `update_replicate_configuration` | v2.6.x |
@@ -0,0 +1,85 @@
+# CDC Replication Overview
+
+Milvus CDC (Change Data Capture) replicates data changes from one Milvus cluster to another. Starting from Milvus v2.6, CDC can be used to build a primary-standby disaster recovery topology.
+
+In a primary-standby topology, one cluster acts as the primary and accepts writes. One or more standby clusters continuously receive changes from the primary and can serve read traffic. When the primary cluster is unavailable or needs maintenance, you can switch service traffic to a standby cluster.
+
+## Architecture
+
+A typical topology contains:
+
+- **Primary cluster**: The source cluster for replication. It accepts reads and writes.
+- **Standby cluster**: A target cluster for replication. It receives changes from the primary and is read-only while it remains a standby.
+- **CDC node**: A Milvus component that forwards WAL changes from the current primary to standby clusters. Deploy CDC on each cluster that may become primary after switchover or failover.
+- **Replication topology**: The configured source-to-target relationship, such as `cluster-a -> cluster-b`.
+
+The most common setup is one primary and one standby:
+
+![Milvus CDC replication architecture](./assets/cdc-replication-architecture.svg)
+
+```text
+Application writes
+      |
+      v
+Primary cluster A  -- CDC replication -->  Standby cluster B
+```
+
+Milvus also supports a single-primary, multi-standby topology:
+
+```text
+Primary cluster A  -- CDC replication -->  Standby cluster B
+                  \-- CDC replication -->  Standby cluster C
+```
+
+## Primary and Standby Behavior
+
+| Role | Reads | Writes | Replication behavior |
+|---|---:|---:|---|
+| Primary | Yes | Yes | Sends changes to standby clusters |
+| Standby | Yes | No | Receives replicated changes from the primary |
+
+A standby cluster rejects direct write requests. This prevents split brain and keeps the topology consistent.
+
+## Failover Options
+
+Milvus provides two ways to move service traffic from the primary to a standby cluster.
+
+| Operation | Use when | Data loss | Expected recovery behavior |
+|---|---|---|---|
+| **[Planned switchover](./03-cdc-planned-switchover.md)** | The primary is still reachable, or you are doing maintenance | RPO = 0 | Waits for remaining replicated data before roles change |
+| **[Force failover](./04-cdc-force-failover.md)** | The primary is completely unavailable and cannot be recovered quickly | Possible | Promotes the standby immediately so writes can resume |
+
+Use planned switchover whenever the primary can still respond. Use force failover only when restoring availability is more important than waiting for the original primary.
+
+## CDC Lag
+
+CDC lag is the amount of data that has been written to the primary cluster but has not yet been applied to a standby cluster.
+
+CDC lag affects failover behavior:
+
+- During planned switchover, lower CDC lag usually means the switchover completes faster.
+- During force failover, CDC lag represents the data window that may be lost if the original primary is unavailable.
+
+Monitor CDC lag continuously and keep it as low as possible. [CDC Replication Quick Start](./02-cdc-replication-quick-start.md) includes a PromQL example for estimating CDC lag.
+
+## FAQ
+
+### Can a standby cluster serve queries?
+
+Yes. A standby cluster can serve read traffic. It cannot accept writes until it becomes primary.
+
+### Does CDC support active-active writes?
+
+No. CDC replication is designed for a single-primary topology. Writing to multiple clusters at the same time can cause split brain and data divergence.
+
+### Does planned switchover lose data?
+
+No. Planned switchover waits for the remaining data to be replicated before the standby becomes primary.
+
+### Does force failover lose data?
+
+It can. Any data written to the old primary but not yet replicated to the standby may be lost.
+
+### How much data can be lost during force failover?
+
+The potential data loss is bounded by CDC lag at the time the primary becomes unavailable.
@@ -0,0 +1,332 @@
+# CDC Replication Quick Start
+
+This guide shows how to deploy two standalone Milvus clusters with Milvus Operator and configure CDC replication from a source cluster to a target cluster.
+
+The examples use:
+
+- `source-cluster` as the primary cluster.
+- `target-cluster` as the standby cluster.
+- `milvus` as the namespace for Milvus clusters.
+- `milvus-operator` as the namespace for Milvus Operator.
+
+Before you begin, read [CDC Replication Overview](./01-cdc-replication-overview.md) to understand the primary-standby model and failover options.
+
+## Prerequisites
+
+- Milvus v2.6.16 or later.
+- Milvus Operator v1.3.4 or later.
+- A Kubernetes cluster is available.
+- The source and target clusters can connect to each other over the network.
+- You have admin credentials for both Milvus clusters.
+- You know the physical channel count for each cluster.
+
+## Step 1: Upgrade Milvus Operator
+
+Add the Milvus Operator Helm repository:
+
+```bash
+helm repo add zilliztech-milvus-operator https://zilliztech.github.io/milvus-operator/
+```
+
+Update the repository:
+
+```bash
+helm repo update zilliztech-milvus-operator
+```
+
+Install or upgrade Milvus Operator:
+
+```bash
+helm -n milvus-operator upgrade --install milvus-operator \
+  zilliztech-milvus-operator/milvus-operator \
+  --create-namespace
+```
+
+Check that the operator pod is running:
+
+```bash
+kubectl get pods -n milvus-operator
+```
+
+Example output:
+
+```text
+NAME                               READY   STATUS    RESTARTS   AGE
+milvus-operator-6f7d8c9c7d-xm4tj   1/1     Running   0          54s
+```
+
+## Step 2: Deploy the Source Cluster
+
+Create a file named `milvus_source_cluster.yaml`:
+
+```yaml
+apiVersion: milvus.io/v1beta1
+kind: Milvus
+metadata:
+  name: source-cluster
+  namespace: milvus
+  labels:
+    app: milvus
+spec:
+  mode: standalone
+  components:
+    image: milvusdb/milvus:v2.6.16
+    cdc:
+      replicas: 1
+  dependencies:
+    msgStreamType: woodpecker
+```
+
+Apply the configuration:
+
+```bash
+kubectl create namespace milvus
+kubectl apply -f milvus_source_cluster.yaml
+```
+
+Check that the source cluster pods are running:
+
+```bash
+kubectl get pods -n milvus
+```
+
+Example output:
+
+```text
+NAME                                                   READY   STATUS    RESTARTS   AGE
+source-cluster-etcd-0                                  1/1     Running   0          3m
+source-cluster-minio-6d8f7d9b9f-9t7j2                  1/1     Running   0          3m
+source-cluster-milvus-standalone-7f8d9c8f6d-r2m5x      1/1     Running   0          2m
+source-cluster-milvus-cdc-66d64747bd-sckxj             1/1     Running   0          2m
+```
+
+Make sure the CDC pod, such as `source-cluster-milvus-cdc-...`, is in the `Running` state.
+
+## Step 3: Deploy the Target Cluster
+
+Create a file named `milvus_target_cluster.yaml`:
+
+```yaml
+apiVersion: milvus.io/v1beta1
+kind: Milvus
+metadata:
+  name: target-cluster
+  namespace: milvus
+  labels:
+    app: milvus
+spec:
+  mode: standalone
+  components:
+    image: milvusdb/milvus:v2.6.16
+    cdc:
+      replicas: 1
+  dependencies:
+    msgStreamType: woodpecker
+```
+
+The CDC component is enabled on the target cluster as well. It is idle while the target is a standby, but it is needed if the target later becomes the primary after planned switchover.
+
+Apply the configuration:
+
+```bash
+kubectl apply -f milvus_target_cluster.yaml
+```
+
+Check that the target cluster pods are running:
+
+```bash
+kubectl get pods -n milvus | grep -E 'NAME|target-cluster'
+```
+
+Example output:
+
+```text
+NAME                                                   READY   STATUS    RESTARTS   AGE
+target-cluster-etcd-0                                  1/1     Running   0          3m
+target-cluster-minio-5f7c8d9b6f-k8s2q                  1/1     Running   0          3m
+target-cluster-milvus-standalone-66dc8d9f7f-5n6bp      1/1     Running   0          2m
+target-cluster-milvus-cdc-7f8c9d6b8c-q4t9m             1/1     Running   0          2m
+```
+
+## Step 4: Prepare Cluster Information
+
+Get the Milvus service addresses for both clusters:
+
+```bash
+kubectl get svc -n milvus | grep -E 'NAME|source-cluster|target-cluster'
+```
+
+Example output:
+
+```text
+NAME                                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)              AGE
+source-cluster-milvus                 ClusterIP   10.98.124.90     <none>        19530/TCP,9091/TCP   8m
+target-cluster-milvus                 ClusterIP   10.109.234.172   <none>        19530/TCP,9091/TCP   3m
+```
+
+Prepare two types of addresses:
+
+- Cluster addresses are written to the replication configuration and used by CDC components. These addresses must be reachable from the CDC pods.
+- Client addresses are used only by your Python client when calling Milvus APIs. If you run the Python client outside the Kubernetes cluster, expose the Milvus services through your normal access method, such as a load balancer, ingress, or port-forward.
+
+Prepare the connection information and pchannel lists for both clusters:
+
+```python
+source_cluster_addr = "http://source-cluster-milvus.milvus.svc.cluster.local:19530"
+target_cluster_addr = "http://target-cluster-milvus.milvus.svc.cluster.local:19530"
+
+source_client_addr = source_cluster_addr
+target_client_addr = target_cluster_addr
+
+# If your Python client runs outside the Kubernetes cluster, replace only
+# source_client_addr and target_client_addr with externally reachable addresses.
+# Keep source_cluster_addr and target_cluster_addr reachable from CDC pods.
+# For example:
+# source_client_addr = "http://127.0.0.1:19530"
+# target_client_addr = "http://127.0.0.1:19531"
+
+source_cluster_token = "root:Milvus"
+target_cluster_token = "root:Milvus"
+
+source_cluster_id = "source-cluster"
+target_cluster_id = "target-cluster"
+
+pchannel_num = 16
+source_cluster_pchannels = [
+    f"{source_cluster_id}-rootcoord-dml_{i}"
+    for i in range(pchannel_num)
+]
+target_cluster_pchannels = [
+    f"{target_cluster_id}-rootcoord-dml_{i}"
+    for i in range(pchannel_num)
+]
+```
+
+Replace the addresses with the actual Milvus service addresses in your environment. Do not set `source_cluster_addr` or `target_cluster_addr` to a local port-forward address unless the CDC pods can also reach that address. The pchannel list must match your Milvus deployment. Do not copy the example values without checking your cluster configuration.
+
+## Step 5: Create the Replication Configuration
+
+Create a replication configuration from `source-cluster` to `target-cluster`:
+
+```python
+replicate_config = {
+    "clusters": [
+        {
+            "cluster_id": source_cluster_id,
+            "connection_param": {
+                "uri": source_cluster_addr,
+                "token": source_cluster_token,
+            },
+            "pchannels": source_cluster_pchannels,
+        },
+        {
+            "cluster_id": target_cluster_id,
+            "connection_param": {
+                "uri": target_cluster_addr,
+                "token": target_cluster_token,
+            },
+            "pchannels": target_cluster_pchannels,
+        },
+    ],
+    "cross_cluster_topology": [
+        {
+            "source_cluster_id": source_cluster_id,
+            "target_cluster_id": target_cluster_id,
+        }
+    ],
+}
+```
+
+## Step 6: Apply the Replication Configuration
+
+Apply the same configuration to both clusters:
+
+```python
+from pymilvus import MilvusClient
+
+source_client = MilvusClient(
+    uri=source_client_addr,
+    token=source_cluster_token,
+)
+target_client = MilvusClient(
+    uri=target_client_addr,
+    token=target_cluster_token,
+)
+
+try:
+    source_client.update_replicate_configuration(**replicate_config)
+    target_client.update_replicate_configuration(**replicate_config)
+finally:
+    source_client.close()
+    target_client.close()
+```
+
+For production automation, use separate short-lived clients for this control-plane operation. This avoids sharing the same gRPC channel with application DML traffic while the cluster role is changing.
+
+After the configuration is applied, changes written to `source-cluster` are replicated to `target-cluster`.
+
+## Step 7: Verify Data Replication
+
+To verify that replication works:
+
+1. Connect to `source-cluster`.
+2. Create a collection.
+3. Insert data into the collection.
+4. Load the collection and run a query or search on `source-cluster`.
+5. Connect to `target-cluster`.
+6. Run the same query or search on `target-cluster` without manually loading the collection on the standby cluster.
+7. Confirm that the expected data is visible on both clusters.
+
+The target cluster is a standby cluster in this topology. Do not run manual DDL or DCL operations, such as `load_collection`, on the standby cluster. Those operations should be performed on the source cluster and replicated to the target cluster.
+
+The exact verification code depends on your collection schema. For a basic Milvus collection workflow, see the Milvus quick start documentation.
+
+## CDC Lag
+
+CDC lag is the data window between the primary and standby clusters. You should monitor it continuously after replication is configured.
+
+CDC lag can increase when:
+
+- The primary write rate is high.
+- Network latency or packet loss increases between clusters.
+- The standby cluster is overloaded.
+- CDC nodes are under-provisioned.
+- Large DDL or import operations are running.
+
+Use CDC lag to guide operational decisions:
+
+- If lag is low, planned switchover should complete faster.
+- If lag is high, force failover may lose more data.
+
+You can estimate CDC lag with the following PromQL query:
+
+```promql
+clamp_min(
+  max by (channel_name) (
+    milvus_wal_last_confirmed_time_tick
+  )
+  -
+  min by (channel_name) (
+    milvus_cdc_last_replicated_time_tick
+  ),
+  0
+)
+```
+
+The result is in seconds. For each source channel, the query compares the latest confirmed WAL timetick with the last timetick replicated by CDC. If a primary replicates to multiple standby clusters, the `min by (channel_name)` expression reports the slowest replication progress for that channel.
+
+If Prometheus scrapes multiple Milvus clusters, add label filters that match your deployment, such as `namespace` or `app_kubernetes_io_instance`, to avoid mixing metrics from different clusters.
+
+## FAQ
+
+### Do I need to call `update_replicate_configuration` on both clusters?
+
+Yes. Apply the same topology to all participating clusters. If one cluster is not primary at the time of the call, it waits until the topology is applied through CDC.
+
+### How should I choose `cluster_id`?
+
+Use a stable, unique ID for each cluster. The ID is also used in pchannel names and replication topology references.
+
+### Can I change pchannels after replication is configured?
+
+You can update the topology, but the pchannel list must match the cluster layout. Treat pchannel changes as an advanced operation and verify replication carefully afterward.
@@ -0,0 +1,156 @@
+# Planned Switchover
+
+Planned switchover changes the primary-standby direction without data loss. Use it when the current primary cluster is still reachable, or when you need to move traffic for maintenance.
+
+This guide assumes the current topology is:
+
+```text
+cluster-a (primary)  ->  cluster-b (standby)
+```
+
+After switchover, the topology becomes:
+
+```text
+cluster-b (primary)  ->  cluster-a (standby)
+```
+
+## When to Use Planned Switchover
+
+Use planned switchover when:
+
+- You are doing maintenance on the current primary.
+- The primary is partially degraded but can still respond to requests.
+- You need RPO = 0 and cannot accept data loss.
+
+Do not use planned switchover if the primary is completely unavailable. In that case, use [Force Failover](./04-cdc-force-failover.md).
+
+## Before You Begin
+
+Check the following before starting:
+
+- Both clusters are reachable.
+- CDC replication is healthy.
+- CDC lag is low enough for your recovery time target.
+- Application writes can be paused or retried during the role change.
+- You have prepared the new topology configuration.
+
+Planned switchover guarantees no data loss, but the operation time depends on how much data remains to be replicated.
+
+## Build the New Topology
+
+Create a full replacement configuration where `cluster-b` becomes the source and `cluster-a` becomes the target.
+
+```python
+# If you followed the Quick Start, cluster A is the original source cluster,
+# and cluster B is the original target cluster.
+cluster_a_id = source_cluster_id
+cluster_a_addr = source_cluster_addr
+cluster_a_client_addr = source_client_addr
+cluster_a_token = source_cluster_token
+cluster_a_pchannels = source_cluster_pchannels
+
+cluster_b_id = target_cluster_id
+cluster_b_addr = target_cluster_addr
+cluster_b_client_addr = target_client_addr
+cluster_b_token = target_cluster_token
+cluster_b_pchannels = target_cluster_pchannels
+
+switchover_config = {
+    "clusters": [
+        {
+            "cluster_id": cluster_a_id,
+            "connection_param": {
+                "uri": cluster_a_addr,
+                "token": cluster_a_token,
+            },
+            "pchannels": cluster_a_pchannels,
+        },
+        {
+            "cluster_id": cluster_b_id,
+            "connection_param": {
+                "uri": cluster_b_addr,
+                "token": cluster_b_token,
+            },
+            "pchannels": cluster_b_pchannels,
+        },
+    ],
+    "cross_cluster_topology": [
+        {
+            "source_cluster_id": cluster_b_id,
+            "target_cluster_id": cluster_a_id,
+        }
+    ],
+}
+```
+
+## Apply the New Topology
+
+Apply the same configuration to both clusters. Send the request to the current primary first, and then send it to the standby. If you later switch back, reverse the order because `cluster-b` is the current primary.
+
+```python
+from pymilvus import MilvusClient
+
+client_a = MilvusClient(uri=cluster_a_client_addr, token=cluster_a_token)
+client_b = MilvusClient(uri=cluster_b_client_addr, token=cluster_b_token)
+
+try:
+    client_a.update_replicate_configuration(**switchover_config)
+    client_b.update_replicate_configuration(**switchover_config)
+finally:
+    client_a.close()
+    client_b.close()
+```
+
+The old primary demotes to standby and rejects new writes. The old standby waits for remaining replicated data, promotes itself to primary, and then accepts writes.
+
+If the request fails because of a transient network or service error, retry with the same configuration.
+
+## Redirect Application Traffic
+
+After `cluster-b` becomes primary:
+
+1. Point write traffic to `cluster-b`.
+2. Confirm reads and writes succeed on `cluster-b`.
+3. Confirm `cluster-a` is no longer receiving application writes.
+4. Keep monitoring replication from `cluster-b` back to `cluster-a`.
+
+## Verify the Result
+
+Verify that `cluster-b` is serving as the new primary and that data remains consistent. Common checks include:
+
+- Compare row counts for important collections.
+- Query known primary keys from both clusters.
+- Run a representative search on the new primary and old primary.
+- Run a small write on `cluster-b` and confirm it is replicated to `cluster-a`.
+
+## Switch Back
+
+To switch back later, apply the original topology again:
+
+```text
+cluster-a -> cluster-b
+```
+
+Use the same planned switchover flow. Make sure the current primary is reachable and replication is healthy before switching back.
+
+## FAQ
+
+### Does planned switchover lose data?
+
+No. Planned switchover waits for remaining data to be replicated before the standby becomes primary.
+
+### Do I need to stop application writes?
+
+You should pause writes or make writes retryable during the role change. Writes sent to the old primary after it demotes are rejected.
+
+### Why does switchover take longer than expected?
+
+The most common reason is CDC lag. The new primary must receive remaining data before it can safely take over with RPO = 0.
+
+### Can I retry a failed switchover request?
+
+Yes. Retry with the same target topology.
+
+### What happens to the old primary?
+
+The old primary becomes a standby. It should no longer receive application writes.
@@ -0,0 +1,168 @@
+# Force Failover
+
+Force failover promotes a standby cluster to a standalone primary when the original primary is completely unavailable. It is an availability-first operation and may lose data that was not replicated before the failure.
+
+This guide assumes the original topology is:
+
+```text
+cluster-a (primary)  ->  cluster-b (standby)
+```
+
+After force failover, `cluster-b` becomes a standalone primary:
+
+```text
+cluster-b (primary)
+```
+
+## When to Use Force Failover
+
+Use force failover only when:
+
+- The original primary cannot respond to requests.
+- The primary cannot be recovered within an acceptable time.
+- Restoring write availability is more important than waiting for the old primary.
+
+If the primary is still reachable, use [Planned Switchover](./03-cdc-planned-switchover.md) instead. Planned switchover avoids data loss.
+
+## Data Loss Risk
+
+Force failover does not wait for the original primary. Any data written to the old primary but not yet replicated to the standby may be lost.
+
+The possible data loss is determined by CDC lag at the time the primary became unavailable.
+
+Before running force failover, understand the tradeoff:
+
+| Goal | Planned switchover | Force failover |
+|---|---|---|
+| Restore writes while primary is unreachable | No | Yes |
+| Avoid data loss | Yes | Not guaranteed |
+| Requires old primary to respond | Yes | No |
+
+## Before You Begin
+
+Confirm the following:
+
+- The original primary is unavailable.
+- You have decided not to wait for primary recovery.
+- Application traffic can be redirected to the standby.
+- Traffic automation will not send writes back to the old primary if it recovers.
+- You have the standby cluster ID, address, token, and pchannels.
+
+The most important safety requirement is to prevent split brain. After force failover, only the promoted standby should accept application writes.
+
+## Build the Force Failover Configuration
+
+Build a configuration that contains only the standby cluster and no replication topology. Set `force_promote=True`.
+
+```python
+# If you followed the Quick Start, cluster B is the original target cluster.
+cluster_b_id = target_cluster_id
+cluster_b_addr = target_cluster_addr
+cluster_b_client_addr = target_client_addr
+cluster_b_token = target_cluster_token
+cluster_b_pchannels = target_cluster_pchannels
+
+force_failover_config = {
+    "clusters": [
+        {
+            "cluster_id": cluster_b_id,
+            "connection_param": {
+                "uri": cluster_b_addr,
+                "token": cluster_b_token,
+            },
+            "pchannels": cluster_b_pchannels,
+        }
+    ],
+    "cross_cluster_topology": [],
+    "force_promote": True,
+}
+```
+
+## Promote the Standby
+
+Send the request to the standby cluster.
+
+```python
+from pymilvus import MilvusClient
+
+client_b = MilvusClient(uri=cluster_b_client_addr, token=cluster_b_token)
+
+try:
+    client_b.update_replicate_configuration(**force_failover_config)
+finally:
+    client_b.close()
+```
+
+If the request succeeds, `cluster-b` becomes a standalone primary and can accept writes.
+
+## Redirect Application Traffic
+
+After promotion:
+
+1. Redirect write traffic to `cluster-b`.
+2. Remove `cluster-a` from write endpoints, load balancers, DNS records, and automation.
+3. Verify that `cluster-b` accepts writes.
+4. Keep `cluster-a` isolated until it is decommissioned.
+
+Example write verification:
+
+```python
+client_b = MilvusClient(uri=cluster_b_client_addr, token=cluster_b_token)
+
+try:
+    client_b.insert(
+        collection_name="test_collection",
+        data=[{"id": 1, "vector": [0.1] * 128}],
+    )
+finally:
+    client_b.close()
+```
+
+Adjust the collection name and schema fields to match your deployment.
+
+## Verify the Result
+
+Verify the promoted cluster directly:
+
+- Writes succeed on `cluster-b`.
+- Reads return expected data.
+- No application component writes to `cluster-a`.
+
+## Handling the Old Primary
+
+After force failover, decommission `cluster-a`. Do not send application writes to it if it becomes reachable again. It may contain data that was never replicated to `cluster-b`, and `cluster-b` may already contain new writes after failover.
+
+Do not reconnect `cluster-a` to the old topology.
+
+## Minimizing Data Loss
+
+You cannot remove all data-loss risk from force failover, but you can reduce it:
+
+- Monitor CDC lag continuously.
+- Keep standby clusters provisioned to handle the primary write rate.
+- Keep cross-cluster network latency and packet loss low.
+- Make application writes idempotent.
+- Retry writes whose success is uncertain after failover.
+- Prefer planned switchover whenever the primary can still respond.
+
+## FAQ
+
+### Does force failover always lose data?
+
+No, but it can. If all writes were already replicated before the primary failed, no data is lost. If CDC lag existed, the lagging data may be lost.
+
+### How long does force failover take?
+
+It typically completes within seconds, depending on cluster state and control-plane availability on the standby.
+
+### Can I run force failover on the primary?
+
+No. Force failover is intended for a standby cluster. If the current primary is available, use planned switchover.
+
+### Can the old primary rejoin automatically?
+
+No. After force failover, the old primary must be treated as stale and decommissioned.
+
+### How do I avoid split brain?
+
+Ensure that only the promoted cluster receives writes. Remove the old primary from all write paths before it can recover and accept traffic.
@@ -0,0 +1,70 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="1500" height="640" viewBox="0 0 1500 640" role="img" aria-labelledby="title desc">
+  <title id="title">Milvus CDC replication architecture</title>
+  <desc id="desc">CDC nodes in a source cluster consume messages from source streaming nodes and replicate them to target proxies. Target proxies write DDL, DCL, and DML messages to target streaming nodes, which append them to WAL.</desc>
+  <defs>
+    <marker id="arrow-green" markerWidth="10" markerHeight="10" refX="8" refY="5" orient="auto" markerUnits="strokeWidth">
+      <path d="M 0 0 L 10 5 L 0 10 z" fill="#4d9b63"/>
+    </marker>
+    <marker id="arrow-purple" markerWidth="10" markerHeight="10" refX="8" refY="5" orient="auto" markerUnits="strokeWidth">
+      <path d="M 0 0 L 10 5 L 0 10 z" fill="#8162cc"/>
+    </marker>
+    <marker id="arrow-red" markerWidth="10" markerHeight="10" refX="8" refY="5" orient="auto" markerUnits="strokeWidth">
+      <path d="M 0 0 L 10 5 L 0 10 z" fill="#d35d59"/>
+    </marker>
+  </defs>
+
+  <rect width="1500" height="640" fill="#ffffff"/>
+
+  <rect x="30" y="35" width="690" height="570" rx="6" fill="none" stroke="#7f8b94" stroke-width="1.8" stroke-dasharray="6 6"/>
+  <text x="40" y="75" font-family="Arial, Helvetica, sans-serif" font-size="32" font-weight="700" fill="#20262d">Source Cluster</text>
+
+  <rect x="85" y="130" width="585" height="180" rx="8" fill="#ffd0b2"/>
+  <text x="95" y="163" font-family="Arial, Helvetica, sans-serif" font-size="24" font-weight="700" fill="#20262d">Streaming Nodes</text>
+  <rect x="116" y="185" width="155" height="72" rx="7" fill="#f7f7f7"/>
+  <text x="129" y="224" font-family="Arial, Helvetica, sans-serif" font-size="19" fill="#2b3036">Streaming Node</text>
+  <rect x="300" y="185" width="155" height="72" rx="7" fill="#f7f7f7"/>
+  <text x="313" y="224" font-family="Arial, Helvetica, sans-serif" font-size="19" fill="#2b3036">Streaming Node</text>
+  <rect x="485" y="185" width="155" height="72" rx="7" fill="#f7f7f7"/>
+  <text x="498" y="224" font-family="Arial, Helvetica, sans-serif" font-size="19" fill="#2b3036">Streaming Node</text>
+
+  <rect x="85" y="415" width="585" height="180" rx="8" fill="#9fd5b2"/>
+  <text x="95" y="448" font-family="Arial, Helvetica, sans-serif" font-size="24" font-weight="700" fill="#20262d">CDC Nodes</text>
+  <rect x="116" y="470" width="155" height="72" rx="7" fill="#f7f7f7"/>
+  <text x="153" y="512" font-family="Arial, Helvetica, sans-serif" font-size="19" fill="#2b3036">CDC Node</text>
+  <rect x="300" y="470" width="155" height="72" rx="7" fill="#f7f7f7"/>
+  <text x="337" y="512" font-family="Arial, Helvetica, sans-serif" font-size="19" fill="#2b3036">CDC Node</text>
+  <rect x="485" y="470" width="155" height="72" rx="7" fill="#f7f7f7"/>
+  <text x="522" y="512" font-family="Arial, Helvetica, sans-serif" font-size="19" fill="#2b3036">CDC Node</text>
+
+  <path d="M 377 412 L 377 325" fill="none" stroke="#4d9b63" stroke-width="3" marker-end="url(#arrow-green)"/>
+  <text x="338" y="363" font-family="Arial, Helvetica, sans-serif" font-size="18" fill="#2b3036">Consume</text>
+
+  <rect x="880" y="35" width="580" height="570" rx="6" fill="none" stroke="#7f8b94" stroke-width="1.8" stroke-dasharray="6 6"/>
+  <text x="890" y="75" font-family="Arial, Helvetica, sans-serif" font-size="32" font-weight="700" fill="#20262d">Target Cluster</text>
+
+  <rect x="935" y="130" width="142" height="465" rx="7" fill="#d2daf9"/>
+  <text x="945" y="163" font-family="Arial, Helvetica, sans-serif" font-size="24" font-weight="700" fill="#20262d">Proxy</text>
+  <rect x="956" y="185" width="100" height="46" rx="7" fill="#f7f7f7"/>
+  <text x="983" y="216" font-family="Arial, Helvetica, sans-serif" font-size="19" fill="#2b3036">Proxy</text>
+  <rect x="956" y="286" width="100" height="46" rx="7" fill="#f7f7f7"/>
+  <text x="983" y="317" font-family="Arial, Helvetica, sans-serif" font-size="19" fill="#2b3036">Proxy</text>
+  <rect x="956" y="392" width="100" height="46" rx="7" fill="#f7f7f7"/>
+  <text x="983" y="423" font-family="Arial, Helvetica, sans-serif" font-size="19" fill="#2b3036">Proxy</text>
+  <rect x="956" y="500" width="100" height="46" rx="7" fill="#f7f7f7"/>
+  <text x="983" y="531" font-family="Arial, Helvetica, sans-serif" font-size="19" fill="#2b3036">Proxy</text>
+
+  <rect x="1200" y="130" width="215" height="155" rx="7" fill="#ffd0b2"/>
+  <text x="1215" y="217" font-family="Arial, Helvetica, sans-serif" font-size="25" font-weight="700" fill="#20262d">Streaming Nodes</text>
+
+  <rect x="1200" y="445" width="215" height="150" rx="7" fill="#8ebcf0"/>
+  <text x="1285" y="525" font-family="Arial, Helvetica, sans-serif" font-size="25" font-weight="700" fill="#20262d">WAL</text>
+
+  <path d="M 672 505 H 795 Q 797 505 797 503 V 212 Q 797 210 799 210 H 930" fill="none" stroke="#4d9b63" stroke-width="3" marker-end="url(#arrow-green)"/>
+  <text x="760" y="306" font-family="Arial, Helvetica, sans-serif" font-size="19" fill="#2b3036">Replicate</text>
+
+  <path d="M 1080 365 H 1125 Q 1140 365 1140 350 V 217 Q 1140 208 1149 208 H 1195" fill="none" stroke="#8162cc" stroke-width="3" marker-end="url(#arrow-purple)"/>
+  <text x="1082" y="305" font-family="Arial, Helvetica, sans-serif" font-size="19" fill="#2b3036">DDL/DCL/DML</text>
+
+  <path d="M 1308 290 V 430" fill="none" stroke="#d35d59" stroke-width="3" marker-end="url(#arrow-red)"/>
+  <text x="1275" y="376" font-family="Arial, Helvetica, sans-serif" font-size="19" fill="#2b3036">Append</text>
+</svg>
@@ -0,0 +1,411 @@
+# LoadDiff-Based Segment Self-Managed Loading Design Document
+
+## Overview
+
+This document describes the LoadDiff-based segment self-managed loading architecture in Milvus. The design moves segment loading orchestration from the Go layer to the C++ segcore layer, enabling segments to autonomously manage their loading process and support incremental updates (reopen) without full segment reloads.
+
+## Related Issues and PRs
+
+### Issues
+- **[#45060](https://github.com/milvus-io/milvus/issues/45060)**: Make segment itself manage loading
+- **[#46358](https://github.com/milvus-io/milvus/issues/46358)**: Support reopen segment on query nodes
+
+### Key PRs (Merged)
+| PR | Description | Status |
+|----|-------------|--------|
+| [#45061](https://github.com/milvus-io/milvus/pull/45061) | Add `NewSegmentWithLoadInfo` API to support segment self-managed loading | Merged |
+| [#45488](https://github.com/milvus-io/milvus/pull/45488) | Move segment loading logic from Go layer to segcore | Merged |
+| [#46359](https://github.com/milvus-io/milvus/pull/46359) | Support reopen segment for data/schema changes | Merged |
+| [#46394](https://github.com/milvus-io/milvus/pull/46394) | QueryCoord support segment reopen when manifest path changes | Merged |
+| [#46536](https://github.com/milvus-io/milvus/pull/46536) | Unify segment Load and Reopen through diff-based loading | Merged |
+| [#46598](https://github.com/milvus-io/milvus/pull/46598) | Handle legacy binlog format (v1) in segment load diff computation | Merged |
+| [#47061](https://github.com/milvus-io/milvus/pull/47061) | Support lazy load for index-has-raw-data fields in Storage V3 LoadDiff | Merged |
+| [#47412](https://github.com/milvus-io/milvus/pull/47412) | Integrate default value filling into LoadDiff computation | Merged |
+
+## Architecture
+
+### Previous Architecture
+
+In the previous design, the Go layer (`segment_loader.go`) was responsible for:
+- Resource estimation and requests
+- Concurrency control
+- Loading orchestration (Load → LoadSegment → loadSealedSegment/LoadMultiFieldData)
+- Waiting and synchronization
+
+The C++ layer (segcore) was passive:
+- Passively receiving data (LoadFieldData, LoadIndex, LoadDeletedRecord)
+- No autonomous scheduling capability
+
+### New Architecture
+
+```
+┌─────────────────────────────────────────────────────────────────────────────┐
+│                              QueryCoord                                      │
+│  ┌─────────────────────────────────────────────────────────────────────┐    │
+│  │                     SegmentChecker                                   │    │
+│  │  - Detects segment state changes (lacks, redundancies, updates)     │    │
+│  │  - Creates Load/Reopen/Reduce tasks based on diff with target       │    │
+│  │  - getSealedSegmentDiff() checks ManifestPath changes               │    │
+│  └─────────────────────────────────────────────────────────────────────┘    │
+└─────────────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    │ SegmentLoadInfo
+                                    ▼
+┌─────────────────────────────────────────────────────────────────────────────┐
+│                              QueryNode (Go)                                  │
+│  ┌─────────────────────────────────────────────────────────────────────┐    │
+│  │                     segmentLoader                                    │    │
+│  │  - Resource estimation and protection                               │    │
+│  │  - Delegates actual loading to segcore via C API                    │    │
+│  │  - ReopenSegments() for incremental updates                         │    │
+│  └─────────────────────────────────────────────────────────────────────┘    │
+│                                    │                                         │
+│  ┌─────────────────────────────────────────────────────────────────────┐    │
+│  │                     LocalSegment                                     │    │
+│  │  - Load() / Reopen() pass SegmentLoadInfo to C++                    │    │
+│  └─────────────────────────────────────────────────────────────────────┘    │
+└─────────────────────────────────────────────────────────────────────────────┘
+                                    │
+                                    │ C API (SegmentLoad / ReopenSegment)
+                                    ▼
+┌─────────────────────────────────────────────────────────────────────────────┐
+│                              Segcore (C++)                                   │
+│  ┌─────────────────────────────────────────────────────────────────────┐    │
+│  │                     SegmentLoadInfo                                  │    │
+│  │  - Wraps protobuf SegmentLoadInfo with caching                      │    │
+│  │  - ComputeDiff(new_info) → LoadDiff                                 │    │
+│  │  - GetLoadDiff() → LoadDiff (for initial load from empty state)    │    │
+│  └─────────────────────────────────────────────────────────────────────┘    │
+│                                    │                                         │
+│  ┌─────────────────────────────────────────────────────────────────────┐    │
+│  │                     ChunkedSegmentSealedImpl                         │    │
+│  │  - SetLoadInfo(proto) stores segment load configuration             │    │
+│  │  - Load() calls GetLoadDiff() + ApplyLoadDiff()                     │    │
+│  │  - Reopen(new_info) calls ComputeDiff() + ApplyLoadDiff()           │    │
+│  │  - ApplyLoadDiff() executes incremental changes                     │    │
+│  └─────────────────────────────────────────────────────────────────────┘    │
+└─────────────────────────────────────────────────────────────────────────────┘
+```
+
+## Core Components
+
+### 1. SegmentLoadInfo Class
+
+Location: `internal/core/src/segcore/SegmentLoadInfo.h/cpp`
+
+The `SegmentLoadInfo` class wraps the protobuf `SegmentLoadInfo` message and provides:
+
+- **Accessor methods**: GetSegmentID(), GetNumOfRows(), GetIndexInfos(), GetBinlogPaths(), etc.
+- **Caching**: Builds internal caches for quick field lookups
+- **Index conversion**: Converts FieldIndexInfo to LoadIndexInfo with mmap settings
+- **Diff computation**: ComputeDiff() and GetLoadDiff() methods
+
+```cpp
+class SegmentLoadInfo {
+public:
+    explicit SegmentLoadInfo(const ProtoType& info, SchemaPtr schema);
+
+    // Compute diff between current and new load info (for Reopen)
+    LoadDiff ComputeDiff(SegmentLoadInfo& new_info);
+
+    // Compute diff from empty state (for initial Load)
+    LoadDiff GetLoadDiff();
+
+    // Column groups support (Storage V3)
+    std::shared_ptr<milvus_storage::api::ColumnGroups> GetColumnGroups();
+
+private:
+    void BuildCache();
+    void ComputeDiffIndexes(LoadDiff& diff, SegmentLoadInfo& new_info);
+    void ComputeDiffBinlogs(LoadDiff& diff, SegmentLoadInfo& new_info);
+    void ComputeDiffColumnGroups(LoadDiff& diff, SegmentLoadInfo& new_info);
+    void ComputeDiffReloadFields(LoadDiff& diff, SegmentLoadInfo& new_info);
+};
+```
+
+### 2. LoadDiff Structure
+
+Location: `internal/core/src/segcore/SegmentLoadInfo.h`
+
+The `LoadDiff` struct represents the difference between two segment load states:
+
+```cpp
+struct LoadDiff {
+    // Indexes to load (field_id -> list of LoadIndexInfo)
+    std::unordered_map<FieldId, std::vector<LoadIndexInfo>> indexes_to_load;
+
+    // Binlogs to load (Storage V1/V2)
+    std::vector<std::pair<std::vector<FieldId>, proto::segcore::FieldBinlog>> binlogs_to_load;
+
+    // Column groups to load (Storage V3 manifest mode)
+    std::vector<std::pair<int, std::vector<FieldId>>> column_groups_to_load;
+
+    // Column groups for lazy loading (index has raw data)
+    std::vector<std::pair<int, std::vector<FieldId>>> column_groups_to_lazyload;
+
+    // Fields to reload (when index raw data availability changes)
+    std::vector<FieldId> fields_to_reload;
+
+    // Fields to fill with default values (schema evolution)
+    std::vector<FieldId> fields_to_fill_default;
+
+    // Indexes to drop
+    std::set<FieldId> indexes_to_drop;
+
+    // Field data to drop
+    std::unordered_set<FieldId> field_data_to_drop;
+
+    // Manifest update flag
+    bool manifest_updated = false;
+    std::string new_manifest_path;
+
+    bool HasChanges() const;
+    std::string ToString() const;  // For debugging
+};
+```
+
+### 3. ApplyLoadDiff Method
+
+Location: `internal/core/src/segcore/ChunkedSegmentSealedImpl.cpp`
+
+The `ApplyLoadDiff` method executes incremental changes based on the computed diff:
+
+```cpp
+void ChunkedSegmentSealedImpl::ApplyLoadDiff(
+    SegmentLoadInfo& segment_load_info,
+    LoadDiff& diff,
+    milvus::OpContext* op_ctx) {
+
+    // 1. Load new indexes
+    if (!diff.indexes_to_load.empty()) {
+        LoadBatchIndexes(trace_ctx, diff.indexes_to_load, op_ctx);
+    }
+
+    // 2. Reload fields (MUST before drop index)
+    if (!diff.fields_to_reload.empty()) {
+        ReloadColumns(diff.fields_to_reload, op_ctx);
+    }
+
+    // 3. Drop indexes (MUST after reload to maintain data availability)
+    if (!diff.indexes_to_drop.empty()) {
+        for (auto field_id : diff.indexes_to_drop) {
+            DropIndex(field_id);
+        }
+    }
+
+    // 4. Load column groups (eager + lazy)
+    if (!diff.column_groups_to_load.empty()) {
+        LoadColumnGroups(..., diff.column_groups_to_load, true, op_ctx);
+    }
+    if (!diff.column_groups_to_lazyload.empty()) {
+        LoadColumnGroups(..., diff.column_groups_to_lazyload, false, op_ctx);
+    }
+
+    // 5. Load field binlogs (Storage V1/V2)
+    if (!diff.binlogs_to_load.empty()) {
+        LoadBatchFieldData(trace_ctx, diff.binlogs_to_load, op_ctx);
+    }
+
+    // 6. Fill default values for schema evolution
+    if (!diff.fields_to_fill_default.empty()) {
+        FillDefaultValueFields(diff.fields_to_fill_default);
+    }
+
+    // 7. Drop field data
+    if (!diff.field_data_to_drop.empty()) {
+        for (auto field_id : diff.field_data_to_drop) {
+            DropFieldData(field_id);
+        }
+    }
+}
+```
+
+### 4. Unified Load and Reopen
+
+Both Load and Reopen operations now use the same diff-based mechanism:
+
+**Initial Load:**
+```cpp
+void ChunkedSegmentSealedImpl::Load(
+    milvus::tracer::TraceContext& trace_ctx,
+    milvus::OpContext* op_ctx) {
+
+    auto diff = segment_load_info_.GetLoadDiff();
+    ApplyLoadDiff(segment_load_info_, diff, op_ctx);
+}
+```
+
+**Reopen (Incremental Update):**
+```cpp
+void ChunkedSegmentSealedImpl::Reopen(
+    const milvus::proto::segcore::SegmentLoadInfo& new_load_info) {
+
+    SegmentLoadInfo new_seg_load_info(new_load_info, schema_);
+
+    // Store old info and update to new
+    SegmentLoadInfo current(segment_load_info_);
+    segment_load_info_ = new_seg_load_info;
+
+    // Compute diff between old and new
+    auto diff = current.ComputeDiff(new_seg_load_info);
+    ApplyLoadDiff(new_seg_load_info, diff);
+}
+```
+
+## Storage Mode Support
+
+### Storage V1 (Legacy Binlog)
+
+- Each field has its own binlog files
+- `child_fields` is empty, field_id == group_id
+- Diff computed via `ComputeDiffBinlogs()`
+
+### Storage V2 (Column Groups)
+
+- Multiple fields can share column groups
+- `child_fields` contains field IDs in the group
+- Diff computed via `ComputeDiffBinlogs()`
+
+### Storage V3 (Manifest Mode)
+
+- Uses Loon manifest for column group metadata
+- Supports lazy loading for fields with index raw data
+- Diff computed via `ComputeDiffColumnGroups()`
+- `ManifestPath` changes trigger segment reopen
+
+## QueryCoord Integration
+
+### Segment Checker
+
+Location: `internal/querycoordv2/checkers/segment_checker.go`
+
+The segment checker detects segments that need reopen:
+
+```go
+func (c *SegmentChecker) getSealedSegmentDiff(...) (toLoad, loadPriorities, toRelease, toUpdate) {
+    isSegmentUpdate := func(segment *datapb.SegmentInfo) bool {
+        segInDist, existInDist := distMap[segment.ID]
+        return existInDist && segInDist.ManifestPath != segment.GetManifestPath()
+    }
+    // ... detect updated segments and create reopen tasks
+}
+
+func (c *SegmentChecker) createSegmentReopenTasks(...) []task.Task {
+    // Creates ActionTypeReopen tasks for updated segments
+}
+```
+
+### Action Types
+
+```go
+const (
+    ActionTypeGrow   ActionType = iota + 1
+    ActionTypeReduce
+    ActionTypeUpdate
+    ActionTypeReopen  // New action type for segment reopen
+)
+```
+
+## Current Capabilities
+
+| Feature | Status | Description |
+|---------|--------|-------------|
+| Initial Load | ✅ Implemented | Load segment from empty state via GetLoadDiff() |
+| Index Load | ✅ Implemented | Load new indexes (vector/scalar) |
+| Index Drop | ✅ Implemented | Drop indexes when removed from target |
+| Binlog Load | ✅ Implemented | Load field binlogs (Storage V1/V2) |
+| Column Group Load | ✅ Implemented | Load column groups (Storage V3) |
+| Lazy Loading | ✅ Implemented | Skip loading fields with index raw data |
+| Field Reload | ✅ Implemented | Reload when index raw data changes |
+| Schema Evolution | ✅ Implemented | Fill default values for new fields |
+| Manifest Update | ✅ Implemented | Reopen when manifest path changes |
+| Legacy Format | ✅ Implemented | Handle V1 binlog format |
+
+## Future Goals
+
+The ultimate goal is for **all load state changes** to use this LoadDiff-based mechanism:
+
+### Planned Extensions
+
+1. **Online Index Building**
+   - Detect when new index is built for a field
+   - Reopen segment to load the new index
+   - Drop field data if index has raw data
+
+2. **Online Schema Changes**
+   - Add new fields with default values
+   - Drop fields (remove field data)
+   - Modify field properties
+
+3. **Index Type Changes**
+   - Switch between index types
+   - Handle index parameter changes
+
+4. **Compaction Integration**
+   - Update segment after compaction
+   - Handle segment merging
+
+5. **Tiered Storage**
+   - Move data between tiers
+   - Update storage locations
+
+## Benefits
+
+1. **Clear Responsibilities**: Loading logic is entirely managed by the segment itself
+2. **Performance Optimization**: Reduces Go-C++ cross-language calls
+3. **Precise Resource Management**: C++ layer directly handles resource allocation
+4. **Caching Integration**: Segments can directly interact with the caching layer
+5. **Proactive Schema Evolution**: Segments are aware of all their own information
+6. **Incremental Updates**: Avoid full segment reloads for minor changes
+7. **Unified API**: Single ApplyLoadDiff() handles all loading scenarios
+
+## Proto Definition
+
+Location: `pkg/proto/segcore.proto`
+
+```protobuf
+message SegmentLoadInfo {
+  int64 segmentID = 1;
+  int64 partitionID = 2;
+  int64 collectionID = 3;
+  int64 dbID = 4;
+  int64 flush_time = 5;
+  repeated FieldBinlog binlog_paths = 6;
+  int64 num_of_rows = 7;
+  repeated FieldBinlog statslogs = 8;
+  repeated FieldBinlog deltalogs = 9;
+  repeated int64 compactionFrom = 10;
+  repeated FieldIndexInfo index_infos = 11;
+  string insert_channel = 13;
+  int64 readableVersion = 14;
+  int64 storageVersion = 15;
+  bool is_sorted = 16;
+  map<int64, TextIndexStats> textStatsLogs = 17;
+  repeated FieldBinlog bm25logs = 18;
+  map<int64, JsonKeyStats> jsonKeyStatsLogs = 19;
+  common.LoadPriority priority = 20;
+  string manifest_path = 21;  // Storage V3 manifest path
+}
+```
+
+## Execution Order
+
+The order of operations in `ApplyLoadDiff()` is critical:
+
+1. **Load new indexes** - Can run in parallel
+2. **Reload columns** - MUST happen before dropping indexes
+3. **Drop indexes** - MUST happen after reload to maintain data availability
+4. **Load column groups** - Eager and lazy loading
+5. **Load field binlogs** - For Storage V1/V2
+6. **Fill default values** - For schema evolution
+7. **Drop field data** - Clean up removed fields
+
+## Testing
+
+Unit tests are located in `internal/core/src/segcore/SegmentLoadInfoTest.cpp`:
+
+- Test diff computation for indexes
+- Test diff computation for binlogs (V1 and V4 formats)
+- Test diff computation for column groups
+- Test mixed format handling
+- Test schema evolution scenarios
@@ -0,0 +1,36 @@
+# Segcore Search Design
+
+init: 7.23.2021, by [FluorineDog](https://github.com/FluorineDog)
+
+update: 2.10.2022, by [zhuwenxing](https://github.com/zhuwenxing)
+
+## Search
+
+Search now supports two modes: json DSL mode and Boolean Expr mode. We will talk about the latter one in detail because the former has been deprecated and is only used in tests.
+
+The execution mode of Boolean Expr works as follows:
+
+1. client packs search expr, topk, and query vector into proto and sends to Proxy node.
+2. Proxy Node unmarshals the proto, parses it to logical plan, makes a static check, and generates protobuf IR.
+3. Query Node unmarshals the plan, generates an executable plan AST, and queries in the segcore.
+
+See details of expression usage at [Boolean Expression Rules](https://milvus.io/docs/v2.0.0/boolean.md)
+
+## Segcore Search Process
+
+After obtaining the AST, the execution engine uses the visitor mode to explain and executes the whole AST tree:
+
+1. Each node includes two steps, a mandatory vector search and an optional predicate.
+
+   1. If Predicate exists, execute predicate expression stage to generate bitset as the vector search bitmask.
+   2. If Predicate does not exist, the vector search bitmask will be empty.
+   3. Bitmask will be used to mark filtered out / deleted entities in the vector execution engine.
+
+2. Currently, Milvus supports the following node on the AST, visitor mode is used to interpret and execute from top to bottom and generate the final bitmask.
+
+   1. LogicalUnaryExpr: not expression
+   2. LogicalBinaryExpr: and or expression
+   3. TermExpr: in expression `A in [1, 2, 3]`
+   4. CompareExpr: compare expression `A > 1` `B <= 1`
+
+3. TermExpr and CompareExpr are leaf nodes of execution.
--- a/Show More
+++ b/Show More