## What
Enforce a function–field binding principle for function DDL (add / drop
/ alter function on an existing collection), so a function and its
output field stay coupled and already-stored vectors can never be
silently invalidated by DDL.
BM25 and MinHash follow the strict binding: add/drop only via the
unified `add_function_field` / `drop_function_field`, output field
always coupled. **TextEmbedding is a temporary carve-out** — its legacy
`AddCollectionFunction` / `DropCollectionFunction` paths are retained,
so attach-over-existing-field and detach are still possible for it,
because `add_function_field` embedding backfill is not yet implemented.
It will be folded into the unified path in a follow-up. So the "always
coupled" invariant currently holds for BM25/MinHash, not TextEmbedding.
## Changes
- **AlterCollectionSchema invariant**: reject standalone add-function (a
function must be added together with its new output field), reject
detaching a function without dropping its output field, and reject
function cascade (a new function's input being another function's
output).
- **Legacy RPCs**: `AddCollectionFunction` / `DropCollectionFunction`
are rejected for BM25/MinHash (must use `add_function_field` /
`drop_function_field`) and retained only for TextEmbedding (see
carve-out). `AlterCollectionFunction` is kept and gains the whitelist
below. In the alter and legacy-add paths, function field IDs are always
re-derived from field names (never trusted from the request), so a
request cannot inject an unrelated field ID that a later
`drop_function_field` would delete.
- **alter_function whitelist**: only connection/runtime params may
change (TextEmbedding: `url`, `credential`, `timeout_ms`,
`max_client_batch_size`, `region`, `location`, `projectid`, `user`);
BM25/MinHash have no alterable params. Function identity (type, name,
input/output fields) and output-shaping params (`dim`, `model_name`,
`endpoint` — the TEI model identity, `normalize`, `truncate*`, prompts,
...) are immutable. Params are normalized (keys lowercased, duplicate
keys rejected) so a crafted duplicate key cannot bypass the diff.
- **drop_function_field**: allow any output-producing function (dropping
needs no backfill), removing the previous BM25/MinHash-only restriction.
## Not in scope / follow-up
- Extending `add_function_field` to TextEmbedding (post-creation add
would require backfilling every existing row through the external
embedding model) — deferred; folding the TextEmbedding legacy paths into
the unified binding follows this.
- MinHash input-field analyzer immutability lives on a different DDL
path (`AlterCollectionField`); tracked as a separate follow-up.
## Breaking changes
- `DropCollectionFunction` on a **MinHash** function (detach — remove
the function but keep its output field) is no longer supported; use
`drop_function_field` instead (drops the function together with its
output field). If a released version supported MinHash detach, migrate
any such call to `drop_function_field`.
issue: #51348🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
issue: #51170
Expose the existing server-side query ORDER BY capability (query param
key `order_by_fields`, already consumed by proxy `task_query.go`)
through the two remaining client surfaces:
- **Go SDK**: `queryOption.WithOrderByFields(fields ...string)` — each
spec is `"field"`, `"field:asc"` or `"field:desc"`, joined into the
`order_by_fields` query param.
- **RESTful v2**: `QueryReqV2` gains `orderByFields` (`[]string`, same
spec format), forwarded by the query handler as the `order_by_fields`
query param.
Validation (sortable type, explicit-limit requirement, iterator
exclusion) stays server-side in the proxy, consistent with how
limit/offset/group-by are handled at these layers. Interface shape
mirrors pymilvus (`order_by=["price:desc"]`).
Tests: Go SDK option unit test, RESTful v2 handler unit test (asserts
the KV pair is forwarded, and absent when not requested), and a
go_client e2e case (desc / asc-default / no-limit rejection).
Follow-up (out of scope here): search-side `order_by` first-class parity
for RESTful v2 / Go SDK (currently reachable via raw search params
only).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
## What
Validate REST v2 quick-create enum-like fields before creating the
collection.
This PR fixes `vectorFieldType` handling for quick collection creation:
- rejects invalid `vectorFieldType` values instead of silently using
`FloatVector`
- supports `FloatVector`, `BinaryVector`, `Float16Vector`,
`BFloat16Vector`, and `SparseFloatVector`
- defaults quick-create index metric by vector type: dense `COSINE`,
binary `HAMMING`, sparse `IP`
- validates explicit binary/sparse `metricType` before collection
creation
- rejects `SparseFloatVector` quick-create with `dimension`
It also fixes top-level quick-create `consistencyLevel` handling:
- parses and validates top-level `consistencyLevel`
- preserves existing `params.consistencyLevel` behavior
- rejects conflicting top-level and `params.consistencyLevel` values
Fixes#51085Fixes#51084
## Why
Previously, `/v2/vectordb/collections/create` ignored top-level
`vectorFieldType` and always generated a `FloatVector` schema in
quick-create mode. A typo such as `InvalidVectorType` could return
success and mask client-side bugs.
Binary and sparse quick-create requests without `metricType` could also
create the collection first and then fail index creation because the old
fallback metric was `COSINE`, which is incompatible with those vector
types.
For #51084, top-level `consistencyLevel` was not present in
`CollectionReq`, so JSON decoding silently dropped it. Invalid values
such as `"consistencyLevel": "Invalid"` were ignored and the collection
could be created with the default consistency level.
## Tests
- `./bin/gofumpt -w internal/distributed/proxy/httpserver/request_v2.go
internal/distributed/proxy/httpserver/handler_v2.go
internal/distributed/proxy/httpserver/handler_v2_test.go`
- `./bin/gci write internal/distributed/proxy/httpserver/request_v2.go
internal/distributed/proxy/httpserver/handler_v2.go
internal/distributed/proxy/httpserver/handler_v2_test.go
--skip-generated -s standard -s default -s
"prefix(github.com/milvus-io)" --custom-order`
- `git diff --check`
- `python3 -m py_compile
tests/restful_client_v2/testcases/test_collection_operations.py`
- `rg -n
"string\\(paramtable\\.(BinaryVectorDefaultMetricType|SparseFloatVectorDefaultMetricType)\\)|interface\\("
internal/distributed/proxy/httpserver/handler_v2.go
internal/distributed/proxy/httpserver/handler_v2_test.go`
Attempted locally but blocked by missing native dependencies:
```
go test ./internal/distributed/proxy/httpserver -run 'TestCreateCollection(QuickVectorFieldType|TopLevelConsistencyLevel)$' -count=1
# missing rocksdb.pc and milvus_core.pc
```
CI investigation before this push:
```
# build-ut-cov/code-check failed on c6 predecessor because of unconvert:
# internal/distributed/proxy/httpserver/handler_v2.go:2054:16 unnecessary conversion
# internal/distributed/proxy/httpserver/handler_v2.go:2056:16 unnecessary conversion
# internal/distributed/proxy/httpserver/handler_v2_test.go:2185:28 unnecessary conversion
# internal/distributed/proxy/httpserver/handler_v2_test.go:2217:28 unnecessary conversion
# Fixed by removing string(...) around paramtable default metric constants.
# e2e-default failed in job 40810 on:
# testcases/test_geometry_operations.py::TestGeometryCollection::test_spatial_query_and_search[False-True-sealed-ST_CROSSES]
# error: Connection refused to /v2/vectordb/collections/flush on port 19530
# the same job showed querynode/streamingnode restarts and proxy ContainerCreating during cleanup.
# REST collection tests in that job had already passed, so this looked unrelated to this PR.
# e2e-default failed again in dispatcher 13914 / job-1 32439 on:
# testcases/test_collection_operations.py::TestCreateCollectionNegative::test_create_collection_quick_setup_with_invalid_consistency_level
# root cause: the test used collection_create(), whose wrapper auto-injects params.consistencyLevel=Strong.
# That made the request hit the new conflict branch instead of the intended top-level invalid consistencyLevel branch.
# Fixed in 3e190eee56 by sending the request through raw client.post() to avoid wrapper mutation.
```
Signed-off-by: Yanliang Qiao <yanliang.qiao@zilliz.com>
issue: https://github.com/milvus-io/milvus/issues/44358
design doc:
docs/design-docs/design_docs/20260609-external-snapshot-export-restore.md
Part 1/3 of the external snapshot cross-bucket restore stack.
This PR defines the API contract and entry surfaces for external
snapshot export and restore:
- Adds the consolidated design document for cross-bucket external
snapshot restore.
- Adds public gRPC, REST, and Go SDK API surfaces for
RestoreExternalSnapshot and ExportSnapshot.
- Adds internal DataCoord proto plumbing and generated code needed by
later implementation PRs.
- Wires Proxy RBAC grouping and database interceptor behavior for the
new APIs.
- Keeps the request contract on a single external_spec field and keeps
db_name for namespace routing rather than permission scoping.
Validation copied from the commit:
- GOTOOLCHAIN=go1.25.10 go test -c -tags dynamic,test -gcflags="all=-N
-l" -ldflags="-r ${RPATH}" -o /tmp/datacoord-commit1.test
github.com/milvus-io/milvus/internal/datacoord
- cd client && GOTOOLCHAIN=go1.25.10 go test -c -o
/tmp/client-milvusclient-commit1.test ./milvusclient
- internal/proxy package compile was blocked locally by missing C++
header internal/core/output/include/segcore/search_result_export_c.h
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
issue: https://github.com/milvus-io/milvus/issues/50571
design doc: docs/design-docs/design_docs/20260624-function-chain-api.md
Add the FunctionChain proto-to-runtime path for ordinary Search L2
rerank. The change converts public FunctionChain protos into ChainRepr,
derives caller-independent read/write metadata, validates
Search-specific inputs in Proxy, and executes public chains through the
existing rerank pipeline.
Add request validation for duplicate stages, unsupported stages,
unsupported system inputs/outputs, unknown schema fields, unsupported
input field types, and hybrid-search usage. Extend REST v2 request
conversion to accept function_chains.
Replace score_combine with num_combine and add typed parameter readers,
repr-based FuncChain construction, chain optimization/pruning helpers,
and model-rerank parameter handling. Add coverage for chain repr
conversion, function/operator behavior, Proxy planning, search pipeline
integration, REST conversion, and REST API cases.
Add the Function Chain API design document and update milvus-proto to
the latest upstream pseudo-version.
Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
## Summary
Reject REST import create requests when `options.auto_commit` is present
but not `"true"` or `"false"`. The omitted option remains accepted and
keeps the default auto-commit behavior.
Fixes#50460
## Changes
- Add strict REST JSON validation for `options.auto_commit`.
- Add handler coverage for omitted, valid, null, empty, and invalid
values.
## Test Plan
- `GOTOOLCHAIN=auto make static-check`
- `GOTOOLCHAIN=auto go test ./util/merr` from `pkg/`
- `GOTOOLCHAIN=auto go test ./internal/json`
Note: local `static-check` used `milvus-cpp-share` to reuse the main
checkout C++ artifacts and a temporary ignored
`cmd/tools/migration/legacy/legacypb/legacy.pb.go` copied from the main
checkout, because this generated file is ignored but required by the
migration package during local lint loading.
---------
Signed-off-by: Yihao Dai <yihao.dai@zilliz.com>
- Persist role descriptions when creating roles, expose them in role
list/describe results, and support updating descriptions through the
AlterRole path without changing role names.
- Store role descriptions in the role value body while keeping role
names in keys, and tolerate legacy empty role values plus undecodable
stored values by returning an empty description for that role row.
- Reject built-in/default roles and over-limit descriptions before WAL
writes, and share the role-description length validator between proxy
and rootcoord.
related: #50183
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
- Wire credential descriptions through create/update/read paths,
including proxy, HTTP v2, Go SDK options, length validation, optional
update presence handling, and internal credential proto/model
persistence.
- Preserve existing encrypted passwords and descriptions during
RootCoord credential updates when either field is omitted, reject
all-empty or partial direct password updates, and keep sha256 passwords
cache-only.
- Skip auth-cache refresh for description-only credential updates so
existing authentication state is not blanked, while still refreshing
caches for password updates.
- Return persisted user descriptions from select_user and HTTP v2
describe_user, including include_role_info=false, and reuse
already-loaded credential rows in user listing and RBAC backup to avoid
duplicate etcd reads while ignoring malformed credential keys.
- Use the upstream milvus-proto go-api/v3 version that includes the
credential description proto change; the temporary fork replaces in
root, pkg, client, and tests/go_client modules have been removed.
design doc:
docs/design-docs/design_docs/20260601-rbac-user-description.md
related: #50179
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
## Summary
Add struct array (ArrayOfVector) support to the RESTful v2 API:
- Schema: declare struct fields with sub-fields in `createCollection`
- Data: ingest struct array rows in `insert` / `upsert`, auto-detecting
per-element-object vs. parallel-column row shapes
- Search: query a struct sub-vector via `annsField:
"<struct>.<subField>"`,
with dim/type validated against the schema
- Reject `nullable` / `defaultValue` on sub-fields (not supported by
engine)
## Test plan
- [x] Unit tests in `struct_array_v2_test.go` cover schema validation,
row parsing for both shapes, and search request building
- [ ] Manual e2e via curl against a local standalone
issue: #45496
---------
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
issue: #49241
This PR exposes partial update field operations through REST upsert so
ARRAY_APPEND and ARRAY_REMOVE requests can reach proxy validation.
Changes:
- Accept fieldOps in v1 and v2 REST upsert request payloads.
- Convert REST op strings into FieldPartialUpdateOp on UpsertRequest.
- Add request helper coverage for supported and unsupported ops.
Validation:
- make milvus
- Manual REST validation on standalone:
- v2 ARRAY_APPEND updated tags from [1,2] to [1,2,3,4]
- v2 ARRAY_REMOVE updated tags to [1,3]
- v1 ARRAY_APPEND updated tags to [1,3,8]
- unsupported op ARRAY_EXTEND returned parameter error
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Related to #49398
Bump Go module references from pkg/v2 and milvus-proto/go-api/v2 to
pkg/v3 and milvus-proto/go-api/v3 so the client tracks the Milvus 3.x
release line.
This prepares the repository for the upcoming 3.x.y release by aligning
imports, module dependencies, and proto API references with the new
major-version module paths.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #45881
design doc:
https://github.com/milvus-io/milvus-design-docs/blob/main/design_docs/20260105-external_table.md
## What
Support external collection operations in REST v2.
- Add schema-level external source/spec handling for REST v2 create and
describe collection.
- Add REST v2 external collection refresh, describe-job, and list-job
routes.
- Reject top-level external config and quick-create external collection
requests.
- Reject add-field requests that try to add external field mappings.
- Support listing all refresh jobs when collection name is empty.
## Tests
- `go build -tags dynamic,test github.com/milvus-io/milvus/internal/...
github.com/milvus-io/milvus/pkg/...`
- `go test -tags dynamic,test -gcflags="all=-N -l" -ldflags="-r
${RPATH}" github.com/milvus-io/milvus/internal/proxy -run
'TestProxy_ListRefreshExternalCollectionJobs_(ListAll|ByCollection)'
-count=1 -v`
Known local limitation:
- Full `internal/datacoord` package coverage run is blocked by an
existing compaction test failure:
`TestCompactionTriggerManagerSuite/TestGetExpectedSegmentSize/all_DISKANN`
expects `209715200` but gets `104857600`.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
## Summary
- Add `GetCollectionName()` method to 12 RESTful API request types that
were missing it
- Ensures `ProxyFunctionCall` metrics report the correct collection name
label for all RESTful endpoints
- Affected types: `RenameCollectionReq`, `QueryReqV2`,
`CollectionIDReq`, `CollectionFilterReq`, `CollectionDataReq`,
`SearchReqV2`, `HybridSearchReq`, `PartitionsReq`, `GrantV2Req`,
`IndexParamReq`, `CollectionReq`, `RunAnalyzerReq`
issue: #48461
## Test plan
- [ ] Verify existing unit tests pass
- [ ] Confirm RESTful API metrics now include collection name for
search/query/hybrid_search endpoints
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
issue: #29735
Implement partial field update functionality for upsert operations,
supporting scalar, vector, and dynamic JSON fields without requiring all
collection fields.
Changes:
- Add queryPreExecute to retrieve existing records before upsert
- Implement UpdateFieldData function for merging data
- Add IDsChecker utility for efficient primary key lookups
- Fix JSON data creation in tests using proper map marshaling
- Add test cases for partial updates of different field types
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Custom `jsonRender` that encodes JSON data directly into the response
stream, it uses less memory since it does not buffer the entire JSON
structure before sending it, unlike `c.JSON` in `HTTPReturn`, which
serializes the JSON fully in memory before writing it to the response.
Benchmark testing shows that, using the custom render incurs no
performance loss and reduces memory consumption by nearly 50%.
issue: https://github.com/milvus-io/milvus/issues/37671
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
1. support isClusteringKey in restful api;
2. throw err if passed invalid 'enableDynamicField' params
3. parameters in indexparams are not processed properly, related with
#36365
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
