- Rebuild TiKV write transactions on retriable begin, build-stage IO,
and classified-transient commit failures while aborting deterministic
build failures and unsafe commit outcomes.
- Preserve original errors for TiKV undetermined commit results and
caller cancellation before string-wrapping; keep private
`retry.Unrecoverable` markers inside the retry helper so they never leak
to callers.
- Stop `ReliableWriteMetaKv` from replaying undetermined TiKV write
results, correct the TiKV begin/requestTimeout comment, and add
regression tests for commit aborts, retry exhaustion, deadline stopping,
and outer reliable-write behavior.
related: #51425
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Related: #50236
- Reuse the initially loaded grantee keys/values when ListPolicy checks
legacy grantee ID ownership.
- Avoid reloading the full grantee-privileges tree once per legacy
grant.
- Add a regression test that verifies ListPolicy performs only one full
grantee-prefix load for legacy grants.
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
- Split the metastore catalog interfaces into coordinator-specific files
so follow-up coordinator refactors can move their catalog surfaces
independently.
- Preserve the existing interfaces, signatures, and behavior; the rework
only fixes gci import grouping in the split RootCoord and StreamingNode
catalog files.
related: #50917
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
related: #46442
This commit adds proper RBAC (Role-Based Access Control) support for the
/expr HTTP endpoint, replacing the previous root-only authentication.
Changes:
- Add new rbac.go with CheckPrivilege function for HTTP endpoints
- Support HTTP Basic Auth only (removed non-standard Bearer token
format)
- Integrate with existing Casbin RBAC framework
- Add PrivilegeExpr to GlobalLevelPrivileges and ClusterAdminPrivileges
- Register GetUserRoleFunc callback in meta_cache.go
- Update tests for new RBAC behavior
Security features:
- Authentication required when authorization is enabled
- Root user bypass when RootShouldBindRole is false
- Proper 401/403 status code differentiation
- Integration with privilege result cache
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
PR #50178 added an unconditional collection-description length check in
proxy `alterCollectionTask.PreExecute`. The check runs on every incoming
`collection.description` value, including values that are not actually
changing.
### Bug
A collection created before the limit existed (e.g. a 2 KB description)
gets rejected when a client resends that unchanged description alongside
a genuine change (e.g. a TTL update). Proxy fails it with
`ErrParameterInvalid`, even though the description is not being
modified. This breaks the PR's own invariant: existing collection
metadata is not re-validated.
### Change
Validate the description only when its value actually differs from the
stored one. A pre-existing collection can resend its (possibly
oversized) description unchanged while altering other properties; that
unchanged value is now skipped. A new or genuinely changed oversized
description still returns `ErrParameterInvalid`.
This keeps the check in `alterCollectionTask.PreExecute`, the same layer
as every other AlterCollection property validator (TTL, mmap,
consistency level), so the fix stays minimal and consistent with the
surrounding code.
## Tests
`internal/proxy/task_test.go` —
`TestAlterCollectionTaskValidateDescription` rewritten as table cases:
- resending an unchanged oversized description while altering TTL is
accepted (the bug);
- changing a valid description to an oversized one is rejected;
- changing an oversized description to a different oversized value is
rejected;
- shrinking an oversized description to a valid value is accepted;
- with duplicate description keys, a changed oversized value is still
rejected per-property.
related: #50173
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
- Skip malformed RBAC catalog entries whose parsed etcd key segments are
empty or whitespace-only when listing roles, user-role mappings, grants,
and policies, so describe_user no longer surfaces ghost empty role names
from historical trailing-slash keys.
- Keep the fix read-side only and inline with the existing invalid-key
handling; write, replay, restore, and cleanup paths remain unchanged.
- Add regression coverage for empty-segment RBAC keys across
describe-user role lookup, ListRole, ListUserRole, ListGrant, and
ListPolicy.
related: #49989
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
- Persist role descriptions when creating roles, expose them in role
list/describe results, and support updating descriptions through the
AlterRole path without changing role names.
- Store role descriptions in the role value body while keeping role
names in keys, and tolerate legacy empty role values plus undecodable
stored values by returning an empty description for that role row.
- Reject built-in/default roles and over-limit descriptions before WAL
writes, and share the role-description length validator between proxy
and rootcoord.
related: #50183
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
- Wire credential descriptions through create/update/read paths,
including proxy, HTTP v2, Go SDK options, length validation, optional
update presence handling, and internal credential proto/model
persistence.
- Preserve existing encrypted passwords and descriptions during
RootCoord credential updates when either field is omitted, reject
all-empty or partial direct password updates, and keep sha256 passwords
cache-only.
- Skip auth-cache refresh for description-only credential updates so
existing authentication state is not blanked, while still refreshing
caches for password updates.
- Return persisted user descriptions from select_user and HTTP v2
describe_user, including include_role_info=false, and reuse
already-loaded credential rows in user listing and RBAC backup to avoid
duplicate etcd reads while ignoring malformed credential keys.
- Use the upstream milvus-proto go-api/v3 version that includes the
credential description proto change; the temporary fork replaces in
root, pkg, client, and tests/go_client modules have been removed.
design doc:
docs/design-docs/design_docs/20260601-rbac-user-description.md
related: #50179
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
- Add a refreshable proxy.maxCollectionDescriptionLength setting with a
default 1024-byte limit and document it in milvus.yaml.
- Validate collection descriptions during CreateCollection and
AlterCollection PreExecute, including duplicate alter properties, while
leaving existing metadata loading unchanged.
- Document that restore or replication flows that recreate collections
through CreateCollection must satisfy the configured limit or raise it
first.
- Cover byte-length boundaries, CJK byte overflow, create rejection,
alter rejection, refreshability, duplicate alter-property rejection, and
generated YAML consistency in tests.
related: #50173
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
- Add a dedicated RBAC grantee ID helper that keeps the full 32-hex MD5
digest while preserving the legacy `MD5` helper for existing callers.
- Switch grantee-id write paths and collection-name grant migration to
full-length IDs, and use dual-format lookup so upgraded clusters can
still read unique legacy 16-hex metadata.
- Harden legacy grantee-id ownership checks so shared legacy IDs fail
closed on reads and lazy migration, unique legacy child keys are cleaned
up during migration, and drop/rename cleanup does not copy or remove
shared child subtrees while survivor parent keys still reference them.
- Cover the upgrade path with tests for legacy list/grant/revoke
behavior, shared-ID read/migration/drop/rename safety, computed-MD5
orphan exclusion, and full-length collision resistance.
related: #49857
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
…ate drift
Built-in RBAC privilege groups were shipping both in
pkg/util/constant.go Go
arrays and in configs/milvus.yaml. Whenever a new privilege was added to
the
Go arrays, the yaml was frequently forgotten; stale yaml then overrode
the fresh Go defaults at
runtime, returning PERMISSION_DENIED on built-in group roles.
This change flips Export=false on the 9 privilege group ParamItems so
generated milvus.yaml no longer ships them as defaults. Runtime falls
through
to DefaultValue (Go constants). Deployments that need to override still
work
unchanged — FileSource still wins over DefaultValue when the user sets
the
key in their own yaml.
Also removes the deprecated
common.security.rbac.overrideBuiltInPrivilegeGroups.enabled
flag (zero production references).
issue: #49167
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
related: #48587
Provide default userName/password (etcdadmin) for etcd authentication
and add fail-fast validation that panics with a clear message when
etcd.auth.enabled=true but credentials are empty, instead of failing
with an opaque etcd connection error.
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
- Catalog.RestoreRBAC routed IsAnyWord through the
IsPrivilegeNameDefined /else branches and ended up calling
PrivilegeGroupNameForMetastore("*"), writing
'grantee-id/<id>/PrivilegeGroup*'.
- MetaTable.CheckIfRBACRestorable rejected wildcard grants with
'privilege [*] does not exist' before broadcastRestoreRBACV2 ever
reached the catalog, so the catalog fix alone was unreachable on the
in-process restore path.
related: #48963
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
related: https://github.com/milvus-io/milvus/issues/48373
- Add missing `return` in `GetSegmentTaskNum` for early exit when no
filters
- Rename `SetAssignedScore` to `AddAssignedScore` to match `+=`
semantics
- Remove deprecated `rand.Seed` init (Go 1.20+ auto-seeds)
- Add `ScoreAwareAssignPolicy` interface to eliminate type assertions
- Extract `BalanceReplicaHelper` to deduplicate streaming service checks
across balancers
- Remove unused `meta` field from RoundRobin/RowCount/ScoreBased
balancer structs
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
…ent deadlock
related: #48308
When hundreds of channels need rebalancing (e.g. during upgrade),
channel tasks could fill the entire per-node executor pool, blocking
segment/leader tasks and causing a deadlock. This fix splits the single
pool into two independent pools:
- Channel task pool: controlled by channelTaskCapFraction (default 0.1)
- Non-channel task pool: remainder of total capacity
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
related: #48061
related: #48062
related: #48137
- Reconstruct logical etcd keys to avoid double rootPath prefix in
delete/migrate operations
- Use typeutil.After for privilege name extraction instead of broken
length-based substring
- Match wildcard dbName and use DefaultTenant consistently
- Move grant cleanup to DropCollection ack callback
- Enable resolveAliasForPrivilege by default and fix alias cache
- passed rbac alias e2e tests
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
Signed-off-by: Li Liu <li.liu@zilliz.com>
Co-authored-by: Li Liu <li.liu@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
related: https://github.com/milvus-io/milvus/issues/47850
1. Privilege interceptor: resolve collection alias to real collection
name
before RBAC permission check, so that operating via alias checks
permission against the real collection, not the alias name.
2. MetaCache alias cache: add aliasInfo cache with positive/negative
entries to avoid repeated DescribeAlias RPC calls. Cache is
invalidated on alias removal, collection removal, and database removal.
3. Catalog grant cleanup: add DeleteGrantByCollectionName and
MigrateGrantCollectionName to RootCoordCatalog interface and kv
implementation. On collection drop, delete all associated grants;
on collection rename, migrate grants to the new name.
4. Feature flag: add proxy.resolveAliasForPrivilege config to
enable/disable alias resolution in the privilege interceptor.
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
related: https://github.com/milvus-io/milvus/issues/46928
Add segment_storage_version label to track segment counts by storage
version (V1=0, V2=2). This enables monitoring segment distribution
across different storage formats.
Changes:
- Add segmentStorageVersionLabelName constant in metrics.go
- Update DataCoordNumSegments GaugeVec with new label
- Update segMetricMutation struct to track storage version dimension
- Update all WithLabelValues calls in meta.go
- Add unit tests for the new label
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
related: https://github.com/milvus-io/milvus/issues/46442
core changes:
- Add config (default: false) to disable /expr endpoint by default
- On Proxy nodes, require root user authentication via HTTP Basic Auth
when enabled
- On non-Proxy nodes, keep original auth parameter behavior for backward
compatibility
- Add HasRegistered() and AuthBypass to expr package for node type
detection
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
The Out of Memory (OOM) error occurs because a handler retains the
entire ImportRecordBatch in memory. Consequently, even when child arrays
within the batch are flushed, the memory for the complete batch is not
released. We temporarily fixed by deep copying record batch in #43724.
The proposed fix is to split the RecordBatch into smaller sub-batches by
column group. These sub-batches will be transferred via CGO, then
reassembled before being written to storage using the Storage V2 API.
Thus we can achieve zero-copy and only transferring references in CGO.
related: #43310
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
related: #39173
Core Features
* Parquet File Analysis: Analyze Milvus binlog Parquet files with
metadata extraction
* MinIO Integration: Direct connection to MinIO storage for remote file
analysis
* Vector Data Deserialization: Specialized handling of Milvus vector
data in binlog files
* Interactive CLI: Command-line interface with interactive exploration
Analysis Capabilities
* Metadata & Vector Analysis: Extract schema info, row counts, and
vector statistics
* Data Export: Export data to JSON format with configurable limits
* Query Functionality: Search for specific records by ID
* Batch Processing: Analyze multiple Parquet files simultaneously
User Experience
* Verbose Output: Detailed logging for debugging
* Error Handling: Robust error handling for file access and parsing
* Flexible Output: Support for single file and batch analysis formats
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
Co-authored-by: nico <109071306+NicoYuan1986@users.noreply.github.com>
The root cause of the issue lies in the fact that when a sealed segment
contains multiple row groups, the get_cells function may receive
unordered cids. This can result in row groups being written into
incorrect cells during data retrieval.
Previously, this issue was hard to reproduce because the old Storage V2
writer had a bug that caused it to write row groups larger than 1MB.
These large row groups could lead to uncontrolled memory usage and
eventually an OOM (Out of Memory) error. Additionally, compaction
typically produced a single large row group, which avoided the incorrect
cell-filling issue during query execution.
related: https://github.com/milvus-io/milvus/issues/43388,
https://github.com/milvus-io/milvus/issues/43372,
https://github.com/milvus-io/milvus/issues/43464, #43446, #43453
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
use meta to get chunked column memory size to avoid getting cells
actually from storage.
related: #39173
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
Fix issues in end-to-end tests:
1. **Split column groups based on schema**, rather than estimating by
average chunk row size. **Ensure column group consistency within a
segment**, to avoid errors caused by loading multiple column group
chunks simultaneously.
2. **Use sorted segmentId** when generating the stats binlog path, to
ensure consistent and correct file path resolution.
3. **Determine field IDs as follows**:
For multi-column column groups, retrieve the field ID list from
metadata.
For single-column column groups, use the column group ID directly as the
field ID.
related: #39173fix: #42862
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
Each cell of a field should be written to its own mmap file, rather than
writing all cells of the field into a single mmap file.
related: #39173
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
storage v2 chunked seal segment loading is based on caching layer. A
cell unit in storage v2 is a parquet row group in remote object storage,
containing all fields. Therefore, each field needs a proxy to do related
one field operations.
<img width="965" alt="Screenshot 2025-04-28 at 10 59 30"
src="https://github.com/user-attachments/assets/83e93a10-3b1d-4066-ac17-b996d5650416"
/>
related: #39173
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
support parallel loading sealed and growing segments with storage v2
format by async reading row groups.
related: #39173
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
The fields and partitions information are stored and fetched with
different prefixes in the metadata. In the CreateCollectionTask, the
RootCoord checks the existing collection information against the
metadata. This check fails if the order of the fields or partitions info
differs, leading to an error after restarting Milvus. To resolve this,
we should use a map in the check logic to ensure consistency.
related: https://github.com/milvus-io/milvus/issues/40955
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
- Feat: Support Mix compaction. Covering tests include compatibility and
rollback ability.
- Read v1 segments and compact with v2 format.
- Read both v1 and v2 segments and compact with v2 format.
- Read v2 segments and compact with v2 format.
- Compact with duplicate primary key test.
- Compact with bm25 segments.
- Compact with merge sort segments.
- Compact with no expiration segments.
- Compact with lack binlog segments.
- Compact with nullable field segments.
- Feat: Support Clustering compaction. Covering tests include
compatibility and rollback ability.
- Read v1 segments and compact with v2 format.
- Read both v1 and v2 segments and compact with v2 format.
- Read v2 segments and compact with v2 format.
- Compact bm25 segments with v2 format.
- Compact with memory limit.
- Enhance: Use serdeMap serialize in BuildRecord function to support all
Milvus data types.
related: #39173
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
* use the new packed reader and writer api to be compatible with current
etcd meta
* For the new packed writer API: column groups and paths are explicitly
defined by users and won't split column groups by memory in storage v2.
Packed writer follows the user-defined column groups to split arrow
record and write into the corresponding file path.
* For the new packed reader API: read paths are explicitly defined by
users.
related: #39173
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
related: #37031
* built-in privilege group privileges in listPrivilegeGroups() should be
the same as in milvus.yaml
* collections granted by collection level built-in privilege group
should be list in showCollections()
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
related issue: https://github.com/milvus-io/milvus/issues/37031
fixed issues:
#37974: better error messages for grant v2 interface
#37903: fix meta built-in privilege group object name
#37843: better error messages for custom privilege group interface
#38002: fix built-in privilege group meta to pass proxy interceptor
check
#38008: fix revoke v2 to support revoking v1 granted privileges
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
issue: #34357
Go Parquet uses dictionary encoding by default, and it will fall back to
plain encoding if the dictionary size exceeds the dictionary size page
limit. Users can specify custom fallback encoding by using
`parquet.WithEncoding(ENCODING_METHOD)` in writer properties. However,
Go Parquet [fallbacks to plain
encoding](https://github.com/apache/arrow/blob/e65c1e295d82c7076df484089a63fa3ba2bd55d1/go/parquet/file/column_writer_types.gen.go.tmpl#L238)
rather than custom encoding method users provide. Therefore, this patch
only turns off dictionary encoding for the primary key.
With a 5 million auto ID primary key benchmark, the parquet file size
improves from 13.93 MB to 8.36 MB when dictionary encoding is turned
off, reducing primary key storage space by 40%.
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
issue: #34123
Benchmark case: The benchmark run the go benchmark function
`BenchmarkDeltalogFormat` which is put in the Files changed. It tests
the performance of serializing and deserializing from two different data
formats under a 10 million delete log dataset.
Metrics: The benchmarks measure the average time taken per operation
(ns/op), memory allocated per operation (MB/op), and the number of
memory allocations per operation (allocs/op).
| Test Name | Avg Time (ns/op) | Time Comparison | Memory Allocation
(MB/op) | Memory Comparison | Allocation Count (allocs/op) | Allocation
Comparison |
|---------------------------------|------------------|-----------------|---------------------------|-------------------|------------------------------|------------------------|
| one_string_format_reader | 2,781,990,000 | Baseline | 2,422 | Baseline
| 20,336,539 | Baseline |
| pk_ts_separate_format_reader | 480,682,639 | -82.72% | 1,765 | -27.14%
| 20,396,958 | +0.30% |
| one_string_format_writer | 5,483,436,041 | Baseline | 13,900 |
Baseline | 70,057,473 | Baseline |
| pk_and_ts_separate_format_writer| 798,591,584 | -85.43% | 2,178 |
-84.34% | 30,270,488 | -56.78% |
Both read and write operations show significant improvements in both
speed and memory allocation.
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
