issue: #51191
Fixes three robustness defects in the copy-segment restore pipeline
(`internal/datacoord/copy_segment_*.go`), each with a unit test covering
the failure mode.
### 1. `checkPendingJob` cannot resume a partially created job (high)
Task creation is a multi-step sequence (per-group `AllocID` + `AddTask`,
each persisted individually, then the `Pending → Executing` transition).
If any step fails mid-way (etcd hiccup, DataCoord restart), the job
stayed `Pending` with only a subset of tasks persisted, and the old
early-return `if len(tasks) > 0 { return }` fired on every subsequent
round — the restore hung until the job timeout with no way to progress.
Now the transition is resume-safe: it computes the source segments
already covered by persisted tasks, creates tasks only for the uncovered
mappings, and always (re-)applies the idempotent `Executing` transition
(so a round that persisted all tasks but failed the job update also
recovers).
### 2. Transient `QueryCopySegment` RPC error failed the whole restore
(medium)
A single network blip or DataNode restart during a status poll
permanently failed the task and its parent restore job. This was
inconsistent with the import pipeline, where a `QueryImport` RPC error
resets the task for retry instead of failing the job.
Now a query RPC error is logged and the task is kept `InProgress` for
re-query on the next tick. The job-level timeout still bounds a
permanently unreachable node; worker-reported task failures keep the
fail-fast behavior.
### 3. `copySegmentTask.Clone` mutated shared state before persisting
(medium)
`Clone` copied the `atomic.Pointer` value, not the protobuf, so the
"clone" used by `UpdateTask` shared the same `*datapb.CopySegmentTask`
as the cached entry. Update actions mutate the proto in place, so the
in-memory task was mutated *before* `SaveCopySegmentTask` — a failed
catalog save left the coordinator cache reflecting the new state while
etcd kept the old one. Now `Clone` deep-copies with `proto.Clone`,
matching `copySegmentJob.Clone`.
### Testing
- `TestCheckPendingJob_ResumesPartialTaskCreation`,
`TestCheckPendingJob_AllTasksExistTransitionsToExecuting`
- `TestQueryTaskOnWorker_RPCErrorKeepsTaskInProgress`
- `TestClone_DeepCopiesProto`,
`TestUpdateTask_SaveFailureLeavesCacheUnchanged`
- Full `./internal/datacoord/` package: PASS
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
issue: #49156
## Summary
- Record `cost_time` / `cost_cpu_num` on DataNode index build tasks and
surface them through `QueryTask` response `Properties`.
- Reserve the unified `cost_time` / `cost_cpu_num` keys under
`pkg/taskcommon` so future PRs can wire up the other
`CreateTask`-dispatched task types.
- Introduce small `internal/datanode/taskcost` helper (`NowMs`,
`ElapsedMs`, `EstimateIndexBuildCPUNum`) for index build task cost
accounting.
- Final state/failReason and execution-end cost are written in a single
`TaskManager` critical section, and `QueryTask` reads them from a single
snapshot, so state and cost in one response are always consistent.
## Notes
This PR is producer-side only. DataCoord / proxy / metrics exporter do
not consume these properties yet; downstream consumption should land in
a follow-up linked from #49156.
`cost_cpu_num` is a coarse, statistics-only estimate for observability
(vector index ≈ knowhere build pool size = NumCPU in cluster mode;
scalar index = 1). It is not used for coordinator-side scheduling, so
the standalone pool-ratio deviation (NumCPU × buildIndexThreadPoolRatio)
and pool sharing across concurrent builds are intentionally not modeled.
Supersedes #49157 (that PR's head branch was rebased onto latest master
and force-pushed, so GitHub no longer allows reopening it). Content is
identical, rebased to a single commit on top of current master.
## Test plan
- [x] `go test ./taskcommon/...` from `pkg/`
- [x] `go test -tags dynamic,test -gcflags="all=-N -l" -count=1
./internal/datanode/taskcost`
- [ ] `go test -tags dynamic,test -gcflags="all=-N -l" -count=1
./internal/datanode/index -run
'TestIndexTaskSchedulerRecordsIndexTaskCost|Test_statsTaskInfoSuite/Test_IndexTaskCostMethods'`
- [ ] Manual sanity: `QueryTask` returns `cost_time` / `cost_cpu_num`
for DataNode index build task success and failure paths.
---------
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
## What
The datacoord `globalScheduler.pickNode` used **first-fit**: it returned
the first node in map-iteration order whose available slots could
satisfy the task. Because Go map iteration order is non-deterministic
and the search stops at the first fit, scheduled tasks (import /
compaction / index / stats) were not spread evenly across DataNodes — a
cluster could land most tasks on one pod while others stayed idle.
This PR maintains a per-round **max-heap** of nodes keyed by available
slots and always assigns each task to the **most-available
(least-loaded)** node (water-filling on available slots). The heap is
built once per `schedule()` round and reused across all picks, so later
picks observe the decremented slots.
The previous fallback (drain the most-available node when no node can
fully satisfy `taskSlot`) and the empty-cluster / no-slot behaviors are
**preserved**.
## Why
Related discussion:
https://github.com/milvus-io/milvus/discussions/50872 — bulk import on a
multi-DataNode cluster spent ~90% of wall-clock on a single pod. One
root cause is on the **scheduling layer**: first-fit selection did not
balance tasks across nodes. This PR addresses that layer.
> Note: the import-specific regrouping in `RegroupImportFiles` (which
can merge multiple files into a single task) is a separate,
complementary issue and is **not** addressed here.
## Test
- Reworked `TestGlobalScheduler_pickNode` (tie / least-loaded / fallback
/ single-node decrement / all-empty / empty-cluster).
- Added `TestGlobalScheduler_pickNode_Balancing`: 3 nodes × 100 slots,
30 tasks × 10 slots → each node receives exactly 10 and is drained to 0.
- `go vet` clean; full `internal/datacoord/task` package passes.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
issue: #50914
---------
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Add the design document for online shard split of namespace
(multi-tenant) collections.
The design covers:
- Range routing over `big_endian(hash(namespace)) || namespace` with a
versioned routing table derived from the collection meta.
- Write switching via a `SplitShard` WAL fence message (`T_switch`) with
reject-and-refetch on the proxy; target vchannels are initialized with a
`BarrierTimeTick` so every message of the new WALs is strictly greater
than `T_switch`.
- In-place child delegators fronted by the source delegator during the
transition window (sealed segments stay single-loaded, deletes and
timeticks are forwarded back).
- Multi-round metadata-only relabel redistribution by DataCoord, with
the release-safety analysis (frozen targets, merged recovery view,
register-then-release).
- One-shot QueryCoord adoption with in-place delegator handoff,
consistency guarantees, engineering constraints, configuration, and
failure handling.
issue: #50463🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
issue: #50145
## Summary
Make `mixCompactionTask` build text indexes inline for `enable_match`
fields (mirroring `sortCompactionTask`) so that mix-compaction output
segments arrive at QueryNode already carrying `TextStatsLogs`. This
eliminates the CGO_LOAD `CreateTextIndex` fallback path in segcore —
observed in production to redundantly rebuild text indexes during
segment load for tens of seconds up to 13+ minutes per segment.
## Background
For a freshly mix-compacted segment with text-match fields, the load
pipeline currently races:
1. Vector index task completes within ~10s (event-triggered)
2. QueryCoord treats the segment as loadable once the vector index is
ready
3. The `TextIndexJob` (stats task) is event-less — only the 60s polling
ticker submits it
4. The job itself takes ~2 minutes for typical text-heavy segments
5. QueryNode receives the segment before its persistent text index is
written; segcore detects `text_stats_logs == nil` and calls
`CreateTextIndex` in-place at load time (`[CGO_LOAD]` scope)
`sortCompactionTask` already avoids this by building the text index
inline as part of compaction
(`internal/datanode/compactor/sort_compaction.go:469-503`).
`mixCompactionTask` did not — this PR closes the gap.
## What changes
- **`internal/datanode/compactor/mix_compactor.go`** — Add `cm
storage.ChunkManager` field; constructor takes one extra arg; add a thin
`createTextIndex` method wrapper that delegates to the existing
package-level `createTextIndex` helper (in `compactor_common.go`,
already used by sort compaction); after `applyLOBCompaction`, loop over
each non-empty **sorted** output segment, build text indexes, update the
V3 manifest, and assign `TextStatsLogs`. Emits a `create_text_index`
stage latency metric. (No new shared helper is introduced — this PR
reuses `compactor_common.go:createTextIndex` and `sort_compaction.go` is
untouched.)
- **`internal/datanode/compactor/namespace_compactor.go`** — Forward
`cm` to the inner `mixCompactionTask`.
- **`internal/datanode/services.go`** — Updated call sites (pass `cm`).
- **`internal/datacoord/compaction_task_mix.go`** — Extend the
`FileResources` plumbing condition to include `MixCompaction` so custom
analyzers (ref mode) reach the inline build. Previously only
`SortCompaction` got `FileResources`, which would have caused
mix-compacted text indexes to use default tokenization → silent search
regressions.
- **`internal/datacoord/compaction_task_clustering.go` /
`compaction_inspector.go`** — Namespace-enabled `ClusteringCompaction`
is routed on the DataNode to `NewNamespaceCompactor`, which delegates to
`mixCompactionTask.Compact()` and now builds the text index inline for
sorted-by-namespace outputs. Wire `IndexEngineVersionManager` into
`clusteringCompactionTask` and, in `BuildCompactionRequest()`, set
`CurrentScalarIndexVersion` (otherwise the inline text index metadata is
emitted with version 0) and fetch `FileResources` for namespace-enabled
clustering plans in ref mode (otherwise the custom analyzer resources
are not downloaded and the text index falls back to the default
analyzer). Addresses review feedback from @aoiasd.
- **`internal/datanode/compactor/mix_compactor_test.go` /
`namespace_compactor_test.go`** — Updated existing constructor call
sites to pass the new `cm` arg.
- **`internal/datanode/compactor/mix_compactor_text_index_test.go`
(new)** — Mockey-based unit tests covering: the wrapper's identifier
propagation to the package helper, error propagation, and real
`Compact()` inline-loop semantics (empty + unsorted segments skipped,
`TextStatsLogs` assigned for sorted, V3 manifest rewritten,
build/manifest errors abort).
- **`internal/datacoord/compaction_task_mix_test.go`** — Test that
`MixCompaction` plans receive `FileResources` in ref mode.
## Test plan
- [x] `go test -tags dynamic,test -gcflags="all=-N -l" -count=1 -run
"TestMixCompaction|TestNamespace|TestSort"
./internal/datanode/compactor/...` — PASS (post-rebase)
- [x] `go test -tags dynamic,test -gcflags="all=-N -l" -count=1 -run
"TestMixCompaction|TestCompactionTask" ./internal/datacoord/...` — PASS
- [x] New unit tests pass:
`TestMixCompaction_createTextIndex_Delegates`,
`TestMixCompaction_createTextIndex_PropagatesError`,
`TestMixCompaction_Compact_InlineTextIndex`,
`TestMixCompaction_Compact_InlineTextIndex_BuildError`,
`TestMixCompaction_Compact_InlineTextIndex_ManifestError`, and
(datacoord)
`TestMixCompactionTaskSuite/TestBuildCompactionRequest_MixFileResourcesInRefMode`,
`TestClusteringCompactionTaskSuite/TestBuildCompactionRequest_NamespaceFileResourcesInRefMode`
- [ ] CI builds against fresh C++ artifacts (local worktree build was
skipped because the C++ shared libs in this workspace are stale relative
to upstream's recent cgo changes — unrelated to this PR's diff)
- [ ] Manual / integration: verify a freshly mix-compacted segment with
text-match fields lands on QueryNode with non-empty `text_stats_logs`
and does NOT trigger the `[CGO_LOAD] CreateTextIndex` slow path in
QueryNode logs
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
issue: #50377
## Problem
Standalone crashes with SIGSEGV (exit 139) when an `ST_WITHIN` query
runs on a nullable GEOMETRY field of a **growing** segment concurrently
with schema-evolution (add field).
The reported stack:
```
GISFunctionFilterExpr.cpp -> segment_->bulk_subscript(...)
SegmentGrowingImpl.cpp -> insert_record_.get_valid_data(field_id)
/usr/include/c++/12/bits/unordered_map.h:880 (SIGSEGV)
```
## Root cause
`InsertRecordGrowing` keeps the per-field columns in two plain
`std::unordered_map`s (`data_` / `valid_data_`).
- The query path (`SegmentGrowingImpl::bulk_subscript` → `get_data_base`
/ `get_valid_data`, and every other growing-segment reader that funnels
through them) reads these maps **without any lock**.
- Schema evolution (`Reopen` → `fill_empty_field` → `append_field_meta`
→ `append_data` / `append_valid_data`) `emplace`s into the maps, which
can **rehash** the bucket array.
- A rehash concurrent with a lock-free `find()` / `at()` on a query
thread walks freed bucket memory → SIGSEGV. The GIS `ST_WITHIN` path
just happens to be a frequent reader (`bulk_subscript`).
Only `Insert` (shared) and `Reopen` (unique) took `sch_mutex_`; the
query path took nothing, so nothing serialized a reader against the
rehash.
## Fix
Add a dedicated `field_map_mutex_` to `InsertRecordGrowing`:
- structural writers (`append_*`, `drop_field_data`, `clear`) take it
**unique**;
- lookups (`get_data_base`, `get_valid_data`, `is_data_exist`,
`is_valid_data_exist`) take it **shared**.
The `append_data` paths resolve their reader-dependent values before
taking the unique lock to avoid recursive locking on the non-recursive
`shared_mutex`. The mutex is kept separate from the existing
`shared_mutex_` (which serializes pk inserts) so the hot read path does
not contend with inserts. Because all growing-segment readers funnel
through these accessors, this covers `bulk_subscript`, vector search and
expression chunk reads uniformly.
## Test
`internal/core/unittest/test_growing_concurrent_reopen.cpp` races
single-offset `bulk_subscript` readers against `Reopen`-driven field
additions. It reliably fails before the fix (failed map lookup under
debug/ASan; SIGSEGV in Release) and passes after.
---------
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Issue
issue: #50188
## Problem
For external collections, `query_iterator` returns hundreds of thousands
of rows and then fails with `MilvusException: (code=65535,
message=fieldID(1) not found)`. Rebuilding the iterator from the last
primary-key cursor reproduces the error immediately and
deterministically.
## Root Cause
The proxy unconditionally appends the reserved system Timestamp field
(`common.TimeStampField`, fieldID=1) to `OutputFieldsId` for the
iterator MVCC dedup (`internal/proxy/task_query.go:610`).
On the QueryNode side, `FillRetrieveResultIfEmpty` is invoked when a
query produces a **fully empty** result. It iterates `OutputFieldsId`
and resolves each field id through the collection `SchemaHelper` to
synthesize empty columns. System fields are never part of the
user/collection schema, so for external collections (whose schema
contains no system fields, and whose segcore synthesizes the timestamp
as a constant Int64 column via `SynthesizeExternalSystemFields` /
`init_timestamps_constant`) the lookup `GetFieldFromID(1)` fails with
`fieldID(1) not found`.
This surfaces when the iterator paginates past the last matching row so
the next batch matches zero rows, and reproduces deterministically when
the iterator is rebuilt from the last cursor (the query is again empty,
hence the ~20ms fail-fast).
Internal collections do not hit this because segcore returns
empty-but-present field columns for zero-match queries, bypassing the
empty-fill path; the underlying bug is that the Go empty-fill path does
not special-case system fields the way the C++ retrieve path does.
## Fix
Special-case reserved system fields (RowID / Timestamp) in
`FillRetrieveResultIfEmpty` and emit them as empty Int64 columns —
matching the non-empty retrieve path (segcore emits system fields as
Int64) — instead of resolving them through the schema. The proxy already
strips system fields via `filterSystemFields`, so the user-facing result
is unchanged.
## Test
- Added table-driven cases to `TestFillIfEmpty` covering an
external-collection schema (no system fields) with `OutputFieldsId`
carrying `TimeStampField`, and a RowID+Timestamp case. Both assert no
error and that empty Int64 columns are emitted.
- Changed code covered at 100%.
- `go test -tags dynamic,test -gcflags="all=-N -l"
./internal/util/typeutil/` passes.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
issue: #50098
## Summary
- Apply `common.arrow.ioThreadPoolCoefficient` /
`ioThreadPoolMaxCapacity` to DataNode at startup, mirroring the
QueryNode wiring from #49208/#49623. Without this, the paramtable keys
exist but have no effect on DataNode and the Arrow IO pool stays at
Arrow's hard-coded default of 8.
- Also call `initcore.InitArrowReaderConfig` so the parquet reader
range-coalescing limits (`hole`/`range` size) apply.
- Register paramtable watchers so capacity changes hot-reload without
restart, matching QueryNode behavior.
- Raise `dataCoord.import.fileNumPerSlot` default from 1 to 4 to reduce
per-import slot demand by 4x and leave more arrow IO pool headroom for
concurrent compactions on the same worker.
## Why
See #50098 for the full reproducer and analysis. Briefly: production
observed a 1.5M-row sort compaction taking >1 hour while the worker's
CPU sat at 12% and network at 14% of baseline — all 5 large
SortCompactions on different pods finished at nearly identical wall
times (variance < 1%), the classic shared-pool saturation signature.
Local bench at 50ms RTT with 100 concurrent readers:
| ARROW_IO_THREADS | conc=100 wall_ms | per-task p50_ms |
|---|---|---|
| 8 (default) | 36,917 | 35,908 |
| 32 | 18,174 | 16,074 |
| 64 | 17,320 | 15,702 |
## Test plan
- [x] `go test -count=1 -run TestComponentParam ./util/paramtable/` (pkg
module)
- [x] `go test -tags dynamic,test -gcflags="all=-N -l" -count=1 -run
TestRegisterArrowIOThreadPoolWatchers ./internal/datanode/index/`
- [x] `go test -tags dynamic,test -gcflags="all=-N -l" -count=1 -run
"TestImport" ./internal/datacoord/`
- [ ] CI
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Related to #49585
## What changed
This PR refactors DataCoord GC scheduling and moves dropped segment
index cleanup into the dropped segment recycle path.
- Split the previous serial `meta` GC loop into independent GC tasks so
dropped segment recycling no longer blocks index, segment-index,
analyze, JSON/text stats, snapshot, orphan, or LOB cleanup work.
- Broadcast GC pause commands to all pausable GC tasks instead of only
the old `meta` worker.
- Recycle dropped segments concurrently, while preserving the
per-segment cleanup order.
- When recycling a dropped segment, delete segment object files and
segment index files first, then remove segment-index meta, and finally
remove segment meta.
- Keep orphan index file/meta GC as a fallback for old data and partial
failure recovery.
- Add `GetAllSegmentIndexes` so dropped segment GC can clean
segment-index meta even when the field index has already been marked
deleted and would be filtered by `GetSegmentIndexes`.
## Failure and safety behavior
- If object file deletion fails, segment-index meta and segment meta are
kept for retry.
- If segment-index meta deletion fails, segment meta is kept for retry.
- Snapshot protection is checked for both segment IDs and index build
IDs before deleting files or metadata.
- V3 segment data still uses `RemoveWithPrefix` for the manifest base
path, and index files are removed from recorded segment-index file keys.
## Tests
Added coverage for:
- GC task registry and pause worker fan-out.
- Dropped segment concurrent worker signaling.
- Dropped segment cleanup of binlog plus segment-index files/meta.
- Deleted field-index filtering case, where `GetSegmentIndexes` returns
empty but dropped segment GC still finds segment-index meta.
- File deletion failure, segment-index meta deletion failure, and
segment meta deletion failure retry behavior.
- Snapshot-blocked index build IDs.
- V3 dropped segment file deletion success and failure paths.
- `GetAllSegmentIndexes` nil/clone behavior.
Local verification:
- `gofmt`
- `git diff --check`
I also attempted targeted `go test -tags dynamic,test
./internal/datacoord ... -coverprofile=...`, but this local workspace
has a stale C++ core output mismatch and fails at link time with
`undefined reference to loon_properties_inject_external_spec`. CI should
run against a matching core build.
---------
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
issue: #49136
## Summary
Add `collection_id` to the `Properties` map of every task that
datacoord's `Cluster.createTask` dispatches to datanode (Compaction,
PreImport, Import, Index, Stats, Analyze, ExternalCollection,
CopySegment). This is groundwork for worker-side attribution and
telemetry without decoding every task payload; this PR only propagates
the wire field and does not add the datanode-side consumer yet.
## What changed
- Add `CollectionIDKey`, `AppendCollectionID`, `GetCollectionID` helpers
in `pkg/taskcommon.Properties`.
- `GetCollectionID` returns `(int64, error)` so future consumers can
distinguish a missing / invalid property from an explicit `0` value.
- For requests whose proto already exposes `CollectionID` —
`PreImportRequest`, `ImportRequest`, `CreateJobRequest`,
`CreateStatsRequest`, `AnalyzeRequest`,
`UpdateExternalCollectionRequest` — read it directly from the request.
- `CompactionPlan` and `CopySegmentRequest` have no top-level
`CollectionID` field, so `CreateCompaction` / `CreateCopySegment` now
take an explicit `collectionID int64` argument (consistent with the
existing `taskSlot` parameter of `CreatePreImport` / `CreateImport`).
Callers pass `t.GetTaskProto().GetCollectionID()` /
`t.GetCollectionId()`, which are already tracked in task metadata.
- Update `MockCluster`, existing callers and existing tests accordingly.
- Add a dedicated unit test `TestCluster_CreateProperties_CollectionID`
asserting that `collection_id` is carried in the
`CreateTaskRequest.Properties` for all eight task types, plus round-trip
tests for the new properties helpers.
## Follow-up
- Add the datanode-side consumer for `collection_id` once the
worker-side attribution / telemetry path lands.
- Consider adding top-level `collection_id` fields to `CompactionPlan`
and `CopySegmentRequest` so all `Create*` paths can read collection IDs
directly from their request protos.
## Test plan
- [x] `go test ./taskcommon/...` from `pkg/` —
`TestProperties_CollectionID` and pre-existing tests pass.
- [x] `go test -tags=test ./internal/datacoord/session/...` — previously
passed for `TestCluster_CreateProperties`,
`TestCluster_CreateProperties_CollectionID`,
`TestCluster_{Compaction,Import,Index,Stats,Analyze,CopySegment}`.
- [x] `go test -tags=test -run
"TestL0CompactionTask|TestClusteringCompactionTask|TestMixCompactionTask|TestCopySegment"
./internal/datacoord/` — affected compaction / copy-segment suites pass.
- [x] `go build ./internal/datacoord/...` — clean build.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
issue: #49977
## Summary
AWS SDK C++ 1.11.692 (bumped from 1.11.352 in #47810) changed two
defaults that together break PutObject against non-AWS S3-compatible
storage:
1. `PutObjectRequest::GetChecksumAlgorithmName()` returns `"crc64nvme"`
by default
2. `ClientConfiguration::checksumConfig::requestChecksumCalculation`
defaults to `WHEN_SUPPORTED`
`ChecksumInterceptor` then attaches a request hash to every PutObject,
and the V4 signer switches the request to `Transfer-Encoding:
aws-chunked` + `x-amz-content-sha256:
STREAMING-UNSIGNED-PAYLOAD-TRAILER`. Aliyun OSS, Tencent COS, Huawei
OBS, and GCP all reject this streaming form.
This makes any `MinioChunkManager::PutObjectBuffer` call fail in an
infinite `JobStateRetry` loop — json_stats / bson_inverted
shared_key_index builds and any non-DiskANN-style index upload (IVF_*,
HNSW, FLAT) are affected.
## Fix
Restrict `requestChecksumCalculation` to `WHEN_REQUIRED` on non-AWS
providers so the SDK only adds checksums when the operation model
demands them (PutObject does not). Same approach as
milvus-io/milvus-storage#500.
Patched in both client construction paths:
- `MinioChunkManager::MinioChunkManager` (storage_type=minio,
address-based dispatch) — gated on `storageType != S3`
- `ChunkManager.cpp` subclasses (storage_type=remote,
cloud_provider-based dispatch) — applies in `GcpChunkManager`,
`AliyunChunkManager`, `TencentCloudChunkManager`,
`HuaweiCloudChunkManager`. `AwsChunkManager` keeps the default
`WHEN_SUPPORTED`.
CARDINAL_TIERED / DiskANN indexes already work because they upload via
Arrow's `S3FileSystem` provided by milvus-storage, which was fixed in
#500.
## Test plan
- [x] Added `TEST(MinioChecksumConfig, OverridesAreWhenRequired)` in
`test_storage.cpp` — asserts the override flips both directions from
`WHEN_SUPPORTED` to `WHEN_REQUIRED`
- [ ] Build / CI green
- [ ] Manual smoke test on Aliyun OSS endpoint — index build (IVF_SQ8)
and json_stats no longer fail with `x-oss-ec=0017-00000804`. CI cannot
regression-detect this since MinIO doesn't reject `aws-chunked`; a real
Aliyun OSS endpoint is required.
## Workaround (no code change)
Setting `AWS_REQUEST_CHECKSUM_CALCULATION=when_required` and
`AWS_RESPONSE_CHECKSUM_VALIDATION=when_required` on milvus pods also
fixes this — the AWS SDK reads them at `ClientConfiguration`
default-construct time, so it works across all bundled SDK copies
without rebuilding. Useful for hotfix on running clusters until this PR
ships.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
- Map `analyzing` state to `InProgress` in `FromCompactionState` so the
global scheduler keeps the task in `runningTasks` instead of dropping it
- Guard `QueryTaskOnWorker` to skip DataNode queries while the task is
in `analyzing` state, preventing premature state reset
## Root cause
When FloatVector is used as a clustering key, `CreateTaskOnWorker` calls
`doAnalyze()` which sets the task state to `analyzing`. However,
`FromCompactionState` did not map `analyzing` to any scheduler state,
returning `None`. This caused the global scheduler to drop the task from
both `pendingTasks` and `runningTasks`. After `processAnalyzing()`
transitioned the state back to `pipelining`, no one re-scheduled
`doCompact()`, leaving the task stuck forever.
## Test plan
- [x] `TestFromCompactionState` — verifies all compaction states map
correctly, including `analyzing → InProgress`
- [x] `TestQueryTaskOnWorkerSkipAnalyzing` — verifies
`QueryTaskOnWorker` returns immediately when task is in `analyzing`
state without querying DataNode
issue: #47540🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
## Summary
Fix nil pointer dereference in `MilvusRegistry.Gather()` that occurs
during QueryTask after etcd chaos recovery, caused by CGo memory
corruption overwriting the `GoRegistry` pointer to nil.
- Add nil guard for `GoRegistry` before calling `GoRegistry.Gather()` —
if nil, return `(nil, nil)` instead of panicking
- Add nil guard for `CRegistry` before calling `CRegistry.Gather()` — if
nil, return Go metrics only
- Change error return from uninitialized `res` variable to explicit
`nil` for clarity
issue: #48383
## Test plan
- [x] `TestMilvusRegistryGather_NilGoRegistry` — verifies no panic when
both `GoRegistry` and `CRegistry` are nil
- [x] `TestMilvusRegistryGather_NilCRegistry` — verifies Go metrics are
returned correctly when only `CRegistry` is nil
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Summary
- `shallowCopySearchRequest` omits `EntityTtlPhysicalTime` when
distributing search requests to worker nodes
- The fallback computes physical time from `MvccTimestamp`, which may
lag behind when `speedupGuranteeTS` reduces the guarantee timestamp
(e.g. no writes after flush with local WAL)
- This causes entity-level TTL filtering to use a stale timestamp,
allowing expired entities to appear in search results
- Add `EntityTtlPhysicalTime` to both `shallowCopySearchRequest` and the
advanced search request construction path
issue: #48260
## Test plan
- [x] Unit test added:
`TestShallowCopySearchRequest_EntityTtlPhysicalTime`
- [ ] Existing entity TTL e2e tests pass
(`test_search_without_filter_expired_entity_data`)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
## Summary
Fixes#47540
- **Bug 1 (MixCoord panic):** `compactionInspector.analyzeScheduler` was
never initialized in production code. When FloatVector is used as
ClusteringKey, the `doAnalyze()` path calls `analyzeScheduler.Enqueue()`
on a nil pointer → panic. Fixed by passing `analyzeScheduler` to
`newCompactionInspector`.
- **Bug 2 (DataNode round-robin panic):** `validateClusteringKey()` had
no field type whitelist, so JSON/Bool/Array passed schema validation.
During clustering compaction, `NewScalarFieldValue()` panics on
unsupported types. Fixed by adding `IsClusteringKeyType()` check to
reject unsupported types at collection creation time.
## Test plan
- [x] `TestIsClusteringKeyType` — verifies supported/unsupported type
classification
- [x] `TestClusteringKey` — new sub-tests for JSON, Bool, Array as
ClusteringKey (all rejected)
- [ ] Existing `TestClusteringKey` sub-tests (normal, multiple keys,
vector key) still pass
- [ ] `TestCompactionPlanHandler*` tests pass with updated
`newCompactionInspector` signature
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Summary
- **Bug**: `pickNode` always set `AvailableSlots=0` for the selected
node, limiting the scheduler to at most one task per node per scheduling
cycle regardless of actual capacity
- **Fix**: Deduct only the required slot count (`AvailableSlots -=
taskSlot`) when a node has sufficient capacity; zero-out only as a
fallback when no node has enough slots
- **Tests**: Updated `TestGlobalScheduler_pickNode` to verify proper
slot deduction across multiple consecutive picks and fallback behavior
issue: #48169
## Test plan
- [x] Updated `TestGlobalScheduler_pickNode` to cover:
- Normal pick: verifies `AvailableSlots` decremented by `taskSlot` (not
zeroed)
- Multiple consecutive picks from same node: verifies cumulative
deduction
- Fallback path: when `taskSlot > AvailableSlots`, picks max-available
node and zeros it
- No available slots: returns `NullNodeID`
- [ ] CI: ut-go passes
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
combineOrInWithNotEqual used `newBoolConstExpr(true) (a ValueExpr)` when
detecting tautologies like `(a IN S) OR (a != d) where d ∈ S`. This
ValueExpr was not recognized by foldBinary's `IsAlwaysTrueExpr` check,
causing it to pass through to the C++ execution engine. The C++ side
compiled it into `PhyValueExpr` which returns a ConstantVector<bool>,
but FilterBitsNode expects a ColumnVector, resulting in an assertion
error.
Fix by using `newAlwaysTrueExpr()` so the expression is properly
recognized and handled throughout the pipeline.
issue: #47997
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
When a collection has ttl_field configured, search() without a filter
expression still returns expired entities. This is because entity-level
TTL filtering is injected in CompileExpressions, which is only called
when a FilterBitsNode exists in the plan. Search without filter builds
the plan as MvccNode -> VectorSearchNode with no FilterBitsNode, so TTL
filtering is completely skipped.
Add an AlwaysTrueExpr FilterBitsNode when no user filter is present but
entity-level TTL is enabled, so CompileExpressions is invoked and merges
the TTL filter expression at execution time.
issue: #47977
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Problem
In strong consistency mode, when query_timestamp is significantly older
than current physical time (e.g., replaying old data with
guaranteeTimestamp from the past), entity-level TTL filtering
incorrectly used query_timestamp instead of current physical time. This
caused:
- Valid data to be incorrectly filtered as "expired"
- Zero search/query results even when data should be visible
## Root Cause
The TTL filter expression (`ttl_field > physical_us OR ttl_field IS
NULL`) was using physical time derived from query_timestamp, which could
be hours or days old in strong consistency scenarios. Entity TTL should
always use current physical time, not query timestamp.
## Solution
1. **C++ Core Changes**:
- Add `physical_time_us_` parameter to QueryContext (defaults to
query_timestamp)
- Pass physical_time_us through Search/Retrieve API chain:
* QueryContext constructor → Search/Retrieve C API → segment
implementation
- CreateTTLFieldFilterExpression now uses
QueryContext::get_physical_time_us()
- Add ExecuteQueryExpr overload with physical_time_us for testing
2. **Go Layer Changes**:
- Extract physical time from guaranteeTimestamp in query tasks:
* query_task.go: Search operations
* query_stream_task.go: Streaming search operations
- Use tsoutil.ParseHybridTs() to convert TSO timestamp to physical time
- Pass physical_time_us to C++ Search/Retrieve calls via
plan.go/segment.go
3. **Testing**:
- EntityTTLTest.cpp: Core TTL filtering with physical_time_us
- EntityTTLEdgeCaseTest.cpp: Edge cases (no TTL field, zero
physical_time_us)
- Tests verify QueryContext stores/returns physical_time_us correctly
- Tests validate TTL filtering uses physical_time_us, not
query_timestamp
## Behavior
- **Before**: TTL filter used query_timestamp (old in strong
consistency)
- **After**: TTL filter uses current physical time (from
guaranteeTimestamp)
- **Backward compatible**: When physical_time_us=0, falls back to
query_timestamp
## Technical Details
- TSO timestamp conversion: `tsoutil.ParseHybridTs(timestamp)` →
physical_ms
- Physical time in microseconds: `physical_us = physical_ms * 1000`
- TTL expression unchanged: `ttl_field > physical_us OR ttl_field IS
NULL`
Fixes#47413
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
issue: #47285
Fix critical DoS vulnerability where filter expressions containing
division or modulo by zero operations cause Milvus server to crash with
SIGFPE .
Any client with query access could crash the server remotely using
expressions like `"int_field / {d} == 1", params:{"d": 0}`.
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
issue： #46033
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Pull Request Summary: Entity-Level TTL Field Support
### Core Invariant and Design
This PR introduces **per-entity TTL (time-to-live) expiration** via a
dedicated TIMESTAMPTZ field as a fine-grained alternative to
collection-level TTL. The key invariant is **mutual exclusivity**:
collection-level TTL and entity-level TTL field cannot coexist on the
same collection. Validation is enforced at the proxy layer during
collection creation/alteration (`validateTTL()` prevents both being set
simultaneously).
### What Is Removed and Why
- **Global `EntityExpirationTTL` parameter** removed from config
(`configs/milvus.yaml`, `pkg/util/paramtable/component_param.go`). This
was the only mechanism for collection-level expiration. The removal is
safe because:
- The collection-level TTL path (`isEntityExpired(ts)` check) remains
intact in the codebase for backward compatibility
- TTL field check (`isEntityExpiredByTTLField()`) is a secondary path
invoked only when a TTL field is configured
- Existing deployments using collection TTL can continue without
modification
The global parameter was removed specifically because entity-level TTL
makes per-entity control redundant with a collection-wide setting, and
the PR chooses one mechanism per collection rather than layering both.
### No Data Loss or Behavior Regression
**TTL filtering logic is additive and safe:**
1. **Collection-level TTL unaffected**: The `isEntityExpired(ts)` check
still applies when no TTL field is configured; callers of
`EntityFilter.Filtered()` pass `-1` as the TTL expiration timestamp when
no field exists, causing `isEntityExpiredByTTLField()` to return false
immediately
2. **Null/invalid TTL values treated safely**: Rows with null TTL or TTL
≤ 0 are marked as "never expire" (using sentinel value `int64(^uint64(0)
>> 1)`) and are preserved across compactions; percentile calculations
only include positive TTL values
3. **Query-time filtering automatic**: TTL filtering is transparently
added to expression compilation via `AddTTLFieldFilterExpressions()`,
which appends `(ttl_field IS NULL OR ttl_field > current_time)` to the
filter pipeline. Entities with null TTL always pass the filter
4. **Compaction triggering granular**: Percentile-based expiration (20%,
40%, 60%, 80%, 100%) allows configurable compaction thresholds via
`SingleCompactionRatioThreshold`, preventing premature data deletion
### Capability Added: Per-Entity Expiration with Data Distribution
Awareness
Users can now specify a TIMESTAMPTZ collection property `ttl_field`
naming a schema field. During data writes, TTL values are collected per
segment and percentile quantiles (5-value array) are computed and stored
in segment metadata. At query time, the TTL field is automatically
filtered. At compaction time, segment-level percentiles drive
expiration-based compaction decisions, enabling intelligent compaction
of segments where a configurable fraction of data has expired (e.g.,
compact when 40% of rows are expired, controlled by threshold ratio).
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
issue: #46687
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
- Core invariant: raw-data cleanup must be scoped to (segment_id,
field_id) so deleting temporary raw files for one field never removes
raw files for other fields in the same segment (prevents cross-field
deletion during index builds).
- Root cause and fix (bug): VectorDiskIndex::Build() and
BuildWithDataset() called RemoveDir on the segment-level path; this
removed rawdata/{segment_id}/. The fix changes both calls to remove
storage::GenFieldRawDataPathPrefix(local_chunk_manager, segment_id,
field_id) instead, limiting cleanup to rawdata/{segment_id}_{field_id}/
(field-scoped).
- Logic removed/simplified: the old helper GetSegmentRawDataPathPrefix
was removed and callers were switched to GenFieldRawDataPathPrefix;
cleanup logic is simplified from segment-level to field-level path
generation and removal, eliminating redundant broad deletions.
- Why this does NOT cause data loss or regress behavior: the change
narrows RemoveDir() to the exact field path used when caching raw data
and offsets earlier in Build (offsets_path and CacheRawDataToDisk
produce field-scoped local paths). Build still writes/reads offsets and
raw data from GenFieldRawDataPathPrefix(...) and then removes that same
prefix after successful index.Build(); therefore only temporary files
for the built field are deleted and other fields’ raw files under the
same segment are preserved. This fixes issue #46687 by preventing
accidental deletion of other fields’ raw data.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
issue: #46576
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
- Core invariant: During meta load, only tasks that are truly
terminal-cleaned (states cleaned or unknown) should be dropped; all
other non-terminal tasks (including timeout and completed) must be
restored so the inspector can reattach them to executing/cleaning queues
and finish their cleanup lifecycle.
- Removed/simplified logic: loadMeta no longer uses the broad
isCompactionTaskFinished predicate (which treated timeout, completed,
cleaned, unknown as terminal). It now uses the new
isCompactionTaskCleaned predicate that only treats cleaned/unknown as
terminal. This removes the redundant exclusion of timeout/completed
tasks and simplifies the guard to drop only cleaned/unknown tasks.
- Bug fix (root cause & exact change): Fixes issue #46576 — the previous
isCompactionTaskFinished caused timeout/completed tasks to be skipped
during meta load and thus not passed into restoreTask(). The PR adds
isCompactionTaskCleaned and replaces the finished check so timeout and
completed tasks are included in restoreTask() and re-attached to the
inspector’s existing executing/cleaning queues.
- No data loss or regression: Tasks in cleaned/unknown remain dropped
(isCompactionTaskCleaned still returns true for cleaned/unknown).
Non-terminal timeout/completed tasks now follow the same restoreTask()
control path used previously for restored tasks — they are enqueued into
the inspector’s queue/executing/cleaning flows rather than being
discarded. No exported signatures changed and all restored tasks flow
into existing handlers, avoiding behavior regression or data loss.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
### **Description**
- Add `-buildvcs=false` flag to Go test commands in coverage script
- Increase default session TTL from 10s to 15s
- Update SessionTTL parameter default value from 30 to 15
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Co-authored-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: chyezh <chyezh@outlook.com>
Co-authored-by: czs007 <zhenshan.cao@zilliz.com>
issue: #46274
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Performance Improvements**
* Field-level text index creation and JSON-key statistics now run
concurrently, reducing overall indexing time and speeding task
completion.
* **Observability Enhancements**
* Per-task and per-field logging expanded with richer context and
per-phase elapsed-time reporting for improved monitoring and
diagnostics.
* **Refactor**
* Node slot handling simplified to compute slot counts on demand instead
of storing them.
<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
### **PR Type**
Enhancement
___
### **Description**
- Only flush and fence segments for schema-changing alter collection
messages
- Skip segment sealing for collection property-only alterations
- Add conditional check using messageutil.IsSchemaChange utility
function
___
### Diagram Walkthrough
```mermaid
flowchart LR
A["Alter Collection Message"] --> B{"Is Schema Change?"}
B -->|Yes| C["Flush and Fence Segments"]
B -->|No| D["Skip Segment Operations"]
C --> E["Set Flushed Segment IDs"]
D --> E
E --> F["Append Operation"]
```
<details><summary><h3>File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>shard_interceptor.go</strong><dd><code>Conditional
segment sealing based on schema changes</code>
</dd></summary>
<hr>
internal/streamingnode/server/wal/interceptors/shard/shard_interceptor.go
<ul><li>Added import for <code>messageutil</code> package to access
schema change detection <br>utility<br> <li> Modified
<code>handleAlterCollection</code> to conditionally flush and fence
<br>segments only for schema-changing messages<br> <li> Wrapped segment
flushing logic in <code>if
</code><br><code>messageutil.IsSchemaChange(header)</code> check<br>
<li> Skips unnecessary segment sealing when only collection properties
are <br>altered</ul>
</details>
</td>
<td><a
href="https://github.com/milvus-io/milvus/pull/46488/files#diff-c1acf785e5b530e59137b21584cf567ccd9aeeb613fb3684294b439289e80beb">+9/-3</a>
</td>
</tr>
</table></td></tr></tbody></table>
</details>
___
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Optimized collection schema alteration to conditionally perform
segment allocation operations only when schema changes are detected,
reducing unnecessary overhead in unmodified collection scenarios.
<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
issue: #45847
After a collection is successfully loaded, the shard-leader state on the
QC may still not be marked as serviceable. It becomes serviceable only
after the scheduled distribution update runs, which will also invalidate
the shard-leader cache on the proxy. Therefore, even if queries are
already executable, the shard-leader mapping on the proxy may still
change afterward.
Try to ensure—as much as possible—that the proxy’s shard-leader cache
remains stable before killing the mixcoord.
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
issue: #45314
This PR only ensures that no panic occurs. However, we still need to
provide protection for the delegator handling ongoing query tasks.
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
issue: #44961
This PR fixes 3 geometry related bugs:
1. Implement `ToString` interface for GisFunctionFilter.
2. Ignore GisFunctionFilter `MoveCursor` for growing segment.
3. Don't skip null geometry for building R-Tree index, should be record
in null_offsets.
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
issue: #44648
If the value is `null` during insertion, it will be omitted instead of
being filled with nil. Therefore, when performing checks, there’s no
need to retrieve data based on the valid offset.
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
issue: #44802
After a Geometry object is serialized into WKB, the resulting binary may
contain '\0' bytes.
When growing mmap is enabled, the append data logic uses strcpy, which
stops copying at the first '\0' bytes.
This causes only part of the WKB---typically the portion up to the
geometry type field to be copied, leading to corrupted data.
As a result, during parsing, all POINT geometries are incorrectly
interperted as POINT(0 0).
To fix this issue, memcpy will be used instead of strcpy.
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
issue: #43427
The GISFunction asserts that the segment_offsets cannot be nullptr. When
size is 0, the segment_offsets is nullptr, so the loop is skiped.
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
issue: #43427
This pr's main goal is merge #37417 to milvus 2.5 without conflicts.
# Main Goals
1. Create and describe collections with geospatial type
2. Insert geospatial data into the insert binlog
3. Load segments containing geospatial data into memory
4. Enable query and search can display geospatial data
5. Support using GIS funtions like ST_EQUALS in query
6. Support R-Tree index for geometry type
# Solution
1. **Add Type**: Modify the Milvus core by adding a Geospatial type in
both the C++ and Go code layers, defining the Geospatial data structure
and the corresponding interfaces.
2. **Dependency Libraries**: Introduce necessary geospatial data
processing libraries. In the C++ source code, use Conan package
management to include the GDAL library. In the Go source code, add the
go-geom library to the go.mod file.
3. **Protocol Interface**: Revise the Milvus protocol to provide
mechanisms for Geospatial message serialization and deserialization.
4. **Data Pipeline**: Facilitate interaction between the client and
proxy using the WKT format for geospatial data. The proxy will convert
all data into WKB format for downstream processing, providing column
data interfaces, segment encapsulation, segment loading, payload
writing, and cache block management.
5. **Query Operators**: Implement simple display and support for filter
queries. Initially, focus on filtering based on spatial relationships
for a single column of geospatial literal values, providing parsing and
execution for query expressions.Now only support brutal search
7. **Client Modification**: Enable the client to handle user input for
geospatial data and facilitate end-to-end testing.Check the modification
in pymilvus.
---------
Signed-off-by: Yinwei Li <yinwei.li@zilliz.com>
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Co-authored-by: ZhuXi <150327960+Yinwei-Yu@users.noreply.github.com>
