## What
Enforce a function–field binding principle for function DDL (add / drop
/ alter function on an existing collection), so a function and its
output field stay coupled and already-stored vectors can never be
silently invalidated by DDL.
BM25 and MinHash follow the strict binding: add/drop only via the
unified `add_function_field` / `drop_function_field`, output field
always coupled. **TextEmbedding is a temporary carve-out** — its legacy
`AddCollectionFunction` / `DropCollectionFunction` paths are retained,
so attach-over-existing-field and detach are still possible for it,
because `add_function_field` embedding backfill is not yet implemented.
It will be folded into the unified path in a follow-up. So the "always
coupled" invariant currently holds for BM25/MinHash, not TextEmbedding.
## Changes
- **AlterCollectionSchema invariant**: reject standalone add-function (a
function must be added together with its new output field), reject
detaching a function without dropping its output field, and reject
function cascade (a new function's input being another function's
output).
- **Legacy RPCs**: `AddCollectionFunction` / `DropCollectionFunction`
are rejected for BM25/MinHash (must use `add_function_field` /
`drop_function_field`) and retained only for TextEmbedding (see
carve-out). `AlterCollectionFunction` is kept and gains the whitelist
below. In the alter and legacy-add paths, function field IDs are always
re-derived from field names (never trusted from the request), so a
request cannot inject an unrelated field ID that a later
`drop_function_field` would delete.
- **alter_function whitelist**: only connection/runtime params may
change (TextEmbedding: `url`, `credential`, `timeout_ms`,
`max_client_batch_size`, `region`, `location`, `projectid`, `user`);
BM25/MinHash have no alterable params. Function identity (type, name,
input/output fields) and output-shaping params (`dim`, `model_name`,
`endpoint` — the TEI model identity, `normalize`, `truncate*`, prompts,
...) are immutable. Params are normalized (keys lowercased, duplicate
keys rejected) so a crafted duplicate key cannot bypass the diff.
- **drop_function_field**: allow any output-producing function (dropping
needs no backfill), removing the previous BM25/MinHash-only restriction.
## Not in scope / follow-up
- Extending `add_function_field` to TextEmbedding (post-creation add
would require backfilling every existing row through the external
embedding model) — deferred; folding the TextEmbedding legacy paths into
the unified binding follows this.
- MinHash input-field analyzer immutability lives on a different DDL
path (`AlterCollectionField`); tracked as a separate follow-up.
## Breaking changes
- `DropCollectionFunction` on a **MinHash** function (detach — remove
the function but keep its output field) is no longer supported; use
`drop_function_field` instead (drops the function together with its
output field). If a released version supported MinHash detach, migrate
any such call to `drop_function_field`.
issue: #51348🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
issue: https://github.com/milvus-io/milvus/issues/44358
design doc:
docs/design-docs/design_docs/20260609-external-snapshot-export-restore.md
Part 1/3 of the external snapshot cross-bucket restore stack.
This PR defines the API contract and entry surfaces for external
snapshot export and restore:
- Adds the consolidated design document for cross-bucket external
snapshot restore.
- Adds public gRPC, REST, and Go SDK API surfaces for
RestoreExternalSnapshot and ExportSnapshot.
- Adds internal DataCoord proto plumbing and generated code needed by
later implementation PRs.
- Wires Proxy RBAC grouping and database interceptor behavior for the
new APIs.
- Keeps the request contract on a single external_spec field and keeps
db_name for namespace routing rather than permission scoping.
Validation copied from the commit:
- GOTOOLCHAIN=go1.25.10 go test -c -tags dynamic,test -gcflags="all=-N
-l" -ldflags="-r ${RPATH}" -o /tmp/datacoord-commit1.test
github.com/milvus-io/milvus/internal/datacoord
- cd client && GOTOOLCHAIN=go1.25.10 go test -c -o
/tmp/client-milvusclient-commit1.test ./milvusclient
- internal/proxy package compile was blocked locally by missing C++
header internal/core/output/include/segcore/search_result_export_c.h
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
issue: #49967
## What was changed
- Add `X-Milvus-Trace-Id` to REST responses for `/v1/vector/...` and
`/v2/vectordb/...` routes.
- Create a REST entry trace span before access logging so response
headers and REST logs use the same request trace ID.
- Keep the v2 wrapper from overwriting the entry trace ID when one
already exists.
- Add timeout and early-return coverage for the trace ID header.
## Verification
- `gofumpt -l internal/distributed/proxy/httpserver/handler_v2.go
internal/distributed/proxy/httpserver/traceid_test.go
internal/distributed/proxy/httpserver/utils.go
internal/distributed/proxy/httpserver/constant.go
internal/distributed/proxy/httpserver/timeout_middleware.go
internal/distributed/proxy/service.go`
- `git diff --check`
- `zsh -c 'source scripts/setenv.sh && go test -v -count=1 -tags
dynamic,test -gcflags="all=-N -l" -ldflags="-r
${MILVUS_WORK_DIR}/cmake_build/lib -r
${MILVUS_WORK_DIR}/internal/core/output/lib"
./internal/distributed/proxy/httpserver -run
"TestHTTPReturnWritesTraceIDHeader|TestHTTPReturnSkipsTraceIDHeaderWhenMissing|TestTraceIDHandlerFuncCreatesRequestTraceID|TestTraceIDHandlerFuncCoversV1AndV2RestRoutes|TestTraceIDHandlerFuncCreatesTraceIDWhenUnsampled|TestTraceIDHandlerFuncReturnsUniqueTraceID|TestTraceIDHandlerFuncCoversEarlyAbort|TestWrapperPostBadJSONReturnsTraceIDHeader|TestWrapperPostKeepsRequestTraceID|TestTimeoutMiddlewareReturnsTraceIDHeader"
-timeout 300s'`
- Manual curl verification against local standalone: REST response
header and REST access log trace IDs matched; 10 repeated search
requests returned 10 valid and unique 32-hex trace IDs.
## Note
Full `go test ./internal/distributed/proxy/httpserver` currently hits an
existing unrelated `TestSearchV2/search#09` assertion mismatch: expected
`Mismatch type uint8`, actual contains `Mismatch type []uint8 ...`.
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
## Summary
Implements Two-Phase Commit (2PC) for Import in primary/secondary
replication clusters. Data stays invisible (`is_importing=true`) until
an explicit commit signal is delivered via WAL, ensuring primary and
secondary clusters reach the same visible state at the same logical
position.
- **New proto**: `CommitImport=44`, `RollbackImport=45` WAL message
types; `Uncommitted=8`, `Committing=9` `ImportJobState` values;
`CommitImport`, `AbortImport`, `HandleCommitVchannel` RPCs on DataCoord;
`committed_vchannels` + `auto_commit` fields on `ImportJob`
- **WAL broadcast**: DataCoord broadcasts
`CommitImportMessage`/`RollbackImportMessage` to all vchannels via DDL
broadcast; CDC replicates to secondary clusters verbatim
- **DDL ack callbacks**: `commitImportV2AckCallback` CAS
`Uncommitted→Committing`; `rollbackImportV2AckCallback` CAS `*→Failed` +
segment drop
- **WAL flusher**: `wal_flusher.dispatch()` intercepts
`CommitImportMessage` per-vchannel, calls `wbMgr.FlushChannel` +
`DataCoord.HandleCommitVchannel`; no-op handler for
`RollbackImportMessage`
- **ImportChecker**: new `Uncommitted` case (auto-commits when
`auto_commit=true`); new `Committing` case (transitions to `Completed`
when all vchannels confirmed)
- **auto_commit option**: default `true` (backward compatible); `false`
lets replication platform control commit timing
- **RESTful API**: `POST /v2/vectordb/jobs/import/commit` and `POST
/v2/vectordb/jobs/import/abort`; `GetImportProgress` surfaces
`Uncommitted` and `Committing` states
**Out of scope**: `commit_timestamp` propagation (handled in companion
PR)
## Test Plan
- [ ] Unit tests for `IsAutoCommit` helper
- [ ] Unit tests for `HandleCommitVchannel` idempotency
- [ ] Unit tests for `CommitImport`/`AbortImport` RPC handlers (state
validation, mutex, broadcast)
- [ ] Unit tests for DDL ack callbacks (CAS races for commit/abort)
- [ ] Unit tests for `ImportChecker` `Uncommitted`/`Committing` cases
- [ ] Unit tests for WAL flusher dispatch (CommitImport/RollbackImport
cases)
- [ ] Unit tests for RESTful commit/abort handlers (valid jobId, invalid
jobId error path)
- [ ] `GetImportProgress` surfaces `Uncommitted` and `Committing` states
- [ ] E2E: non-replication cluster with `auto_commit=true` (default)
behaves identically to pre-2PC
issue: #48525
design doc: https://github.com/milvus-io/milvus-design-docs/pull/29🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Yihao Dai <yihao.dai@zilliz.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Related to #49398
Bump Go module references from pkg/v2 and milvus-proto/go-api/v2 to
pkg/v3 and milvus-proto/go-api/v3 so the client tracks the Milvus 3.x
release line.
This prepares the repository for the upcoming 3.x.y release by aligning
imports, module dependencies, and proto API references with the new
major-version module paths.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #45881
design doc:
https://github.com/milvus-io/milvus-design-docs/blob/main/design_docs/20260105-external_table.md
## What
Support external collection operations in REST v2.
- Add schema-level external source/spec handling for REST v2 create and
describe collection.
- Add REST v2 external collection refresh, describe-job, and list-job
routes.
- Reject top-level external config and quick-create external collection
requests.
- Reject add-field requests that try to add external field mappings.
- Support listing all refresh jobs when collection name is empty.
## Tests
- `go build -tags dynamic,test github.com/milvus-io/milvus/internal/...
github.com/milvus-io/milvus/pkg/...`
- `go test -tags dynamic,test -gcflags="all=-N -l" -ldflags="-r
${RPATH}" github.com/milvus-io/milvus/internal/proxy -run
'TestProxy_ListRefreshExternalCollectionJobs_(ListAll|ByCollection)'
-count=1 -v`
Known local limitation:
- Full `internal/datacoord` package coverage run is blocked by an
existing compaction test failure:
`TestCompactionTriggerManagerSuite/TestGetExpectedSegmentSize/all_DISKANN`
expects `209715200` but gets `104857600`.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
## Summary
- Inject Milvus zap logger into hook plugin via `SetZapLogger` type
assertion after plugin load, enabling the plugin to use Milvus's
centralized logging
- Inject connection manager into hook plugin via `SetClientInfoProvider`
type assertion, allowing the plugin to look up SDK info (sdk_type,
sdk_version, host) for non-Connect requests by connection identifier
- Pass `*gin.Context` directly via `GinParamsKey` instead of
`ginCtx.Keys` map, enabling the hook plugin to access `ClientIP()`,
`FullPath()`, `GetHeader()` and other gin context methods for RESTful
requests
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
#44212
Also, record metrics only when storageUsageTracking is enabled.
Use MB for scanned_remote counter and scanned_total counter metrics to
avoid overflow.
---------
Signed-off-by: chasingegg <chao.gao@zilliz.com>
The restful API default timeout was hard-coded. This PR make this
timeout value configurable via paramtable.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
if nq>2, restful will flatten all the res. If one nq res has duplicate
pks, the length of this slice will be less then topk. This pr
will attach topks in the output.
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
Custom `jsonRender` that encodes JSON data directly into the response
stream, it uses less memory since it does not buffer the entire JSON
structure before sending it, unlike `c.JSON` in `HTTPReturn`, which
serializes the JSON fully in memory before writing it to the response.
Benchmark testing shows that, using the custom render incurs no
performance loss and reduces memory consumption by nearly 50%.
issue: https://github.com/milvus-io/milvus/issues/37671
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Support the new RESTful URL for retrieving/describing import progress:
`/v2/vectordb/jobs/import/describe`.
Deprecate the old URL: `/v2/vectordb/jobs/import/get_progress`.
issue: https://github.com/milvus-io/milvus/issues/36752
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
issue: #32225#31978#32360
1. v1 can only accept one vector, but v2 accept list of vectors #32225
2. cannot get dbName from AliasReq #31978
3. enhance: [restful v2]support config DB-Name in the http header #32360
---------
Signed-off-by: PowderLi <min.li@zilliz.com>
1. complete the output of describe collection #31180
2. add a new parameter `params` for create collection #31181#31149
3. create index according to indexConfig while create collection #31199
4. autoID: false while create a collection quickly, take upsert into
consideration
5. polish error information while search without annsField #31010
fix: [restful v1] insert / upsert with default partitionName JIRA-4952
---------
Signed-off-by: PowderLi <min.li@zilliz.com>
bug list
1. element data type is needed while create a collection with an array
field #30638
2. spelling mistake about metricsType #30643
3. new collection enable dynamic field / auto id as user defined #30665
4. convert rowCount from string to int #30661
5. try it's best to create a new collection #30652
6. int64 percious #20415
7. insert into collection which has multi vector fields #30674
8. cannot rename a collection to other database #30700
9. update the request body of "indexes/create" #30769
10. got [] while list indexes of a collection which has no index #30722
11. restful need encode password before call
CreateCredential/UpdateCredential #30730
12. some parameters missed the required label #30737
13.define the field to be or not to be a partition key while create a
collection #30797
enhance: support opentelemetry
enhance: support dataType: Float16Vector & BFloat16Vector #22837
enhance: describe collection will show the field is partition key or not
#30789
Signed-off-by: PowderLi <min.li@zilliz.com>
issue: #28521#29732
include
1. list collection's import jobs
2. create a new import job
3. get the progress of an import job
fix:
1. mix the order of dbName & collectionName #29728
2. trace log keep the same as v1
3. support traceID
4. azure precheck, blob name cannot end with / #29703
---------
Signed-off-by: PowderLi <min.li@zilliz.com>
