pr: https://github.com/milvus-io/milvus/pull/49922
## Summary
Route generic LIKE Match operations to raw-data scan when a varchar
inverted index is present, while keeping PrefixMatch on the
inverted-index path. Update planner policy and sealed-segment coverage
for a complex LIKE pattern.
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
relate: #50304
## Summary
Add the local format design doc, use the storage writer format constant,
and introduce local_format metadata propagation with column-group split
policy support.
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
issue: #50961
## Summary
Allow AlterCollectionField to set or delete analyzer configuration on
string fields before they are used by text match or BM25. Keep analyzer
params immutable once text match is enabled or a BM25 function depends
on the field.
When analyzer params change, RootCoord revalidates the new analyzer
config, reserves newly referenced file resources before broadcasting,
and lets MetaTable.AlterCollection reconcile file resource refCnt for
request, replay, replicated, and recovery paths.
## Verification
- `git diff --check`
- `go test -tags dynamic,test -gcflags="all=-N -l" -count=1
./util/typeutil -run TestIsBm25FunctionInputField` from `pkg/`
- `go test -tags dynamic,test -gcflags="all=-N -l" -count=1
./internal/rootcoord -run
'TestMetaTableAlterCollectionFileResourceRefCnt|TestDDLCallbacksAlterCollectionFieldAnalyzerValidation'`
blocked locally: missing `milvus_core.pc`
- `go test -tags dynamic,test -gcflags="all=-N -l" -count=1
./internal/proxy -run TestAlterCollectionField` blocked locally: missing
`milvus_core.pc` and `milvus-storage.pc`
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
issue: #50931
## Summary
Clean analyzer option update helpers on master to match the 2.6
follow-up.
Remove the exported UpdateParams wrapper and return C status handling
directly during option initialization.
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Summary
Move the RBAC check before the hook interceptor.
Write the resolved roles for the current request into the request
context.
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
issue: #50931
## Summary
Initialize analyzer runtime options with Lindera download URLs and the
default dictionary path. Register runtime config callbacks so analyzer
yaml updates are propagated to the Rust analyzer layer.
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
issue: #50865
## Summary
Preserve existing insert log paths when building segment insert files
for StorageV2 text index builds.
Fall back to constructing paths from log IDs when the binlog path is
absent.
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
issue: #46308
## Summary
Normalize analyzer names before running multi-analyzer highlight and
analyzer requests: omitted names use default, one name is broadcast, and
per-text names must match the text count. Also add default analyzer
names for lexical highlight query texts and request the analyzer-name
field for multi-analyzer query-only highlight.
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
relate: #49716
pr: https://github.com/milvus-io/milvus/pull/49717
## Summary
Split FunctionRunnerManager lifecycle references between WAL and
delegator keys while still sharing runners by function signature. Add
latest-version lookup support for schemaVersion 0 or nil schema in the
manager; master call sites still pass the current schema version
explicitly.
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
issue: #46308
## Summary
Keep lexical highlight corpus texts aligned with search result rows when
highlighted string fields are nullable or empty. Null and empty rows now
still produce highlight data with an empty fragments list.
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
issue: #50542
## Summary
Keep cloned empty BM25 stats writable so recovered growing segments
cannot register nil stats in the IDF oracle. Add a regression test for
updating a growing segment after empty BM25 stats registration.
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
relate: #49716
## Summary
Move BM25/MinHash function execution from DN/QN pipeline embedding nodes
to the write-before path before WAL append. Add collection-scoped
function runner caching and keep pipeline-side compatibility fill for
old insert messages without generated function output fields.
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
issue: #50206
## Summary
- Support dynamically updating access log configuration (formatters,
methods, writer settings) without restarting, by watching all
`proxy.accessLog` config keys instead of only the enable switch.
- Add RESTful API paths (`/search`, `/query`, `/v2/vectordb/entities/*`)
to the default search/query formatter methods, so RESTful requests also
get detailed access log formatting.
- Fix resource leak: close old `RotateWriter` before re-initializing
when config changes.
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
relate: #50113
## Summary
Make function runner text tokenization concurrency configurable through
function.analyzer.runner_concurrency, preserving the default value of 8.
BM25 and multi-analyzer BM25 now read the parameter for each
BatchRun/BatchAnalyze call so runtime config changes can take effect
without recreating runners. Also register a config event handler for
function.analyzer.concurrency_per_cpu_core so the shared analyzer pool
resizes dynamically when that setting changes.
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
issue: #50005
## Summary
Separate pattern match capability from planner policy so inverted
indexes keep regex filters on the index path while suffix/contains LIKE
can fall back to raw-data scan when appropriate.
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
relate: #49907
## Summary
Make TextMatch upload return relative file names consistently, including
the unified scalar index path. Restore full object keys in DataNode text
stats task results and compaction metadata so mixed-version rolling
upgrades remain compatible with older Coord/QueryNode consumers.
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
relate: #49684
## Summary
Strip a UTF-8 BOM from the first line when loading analyzer word files
so BOM-prefixed stop words are matched correctly. Add coverage for BOM +
CRLF stop word files.
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
relate: #49650
## Summary
Skip BM25 sparse vector fields in segcore ShouldLoadField decisions so
BM25 function outputs are not treated as loadable raw field data. Cover
both BM25 function output fields and vector fields whose metric type is
BM25 with unit tests.
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
relate: #49692
## Summary
Return a validation error instead of panicking when ST_DWITHIN point
validation receives valid non-POINT WKT. Add coverage for POLYGON WKT
passed to checkValidPoint.
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
relate: #48612
## Summary
Enable AckSyncUp for all DropCollection broadcasts so drop
acknowledgement waits for downstream DDL handling instead of using the
default FastAck path.
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
relate: #40168
## Summary
Add `user_dict` param to the lindera tokenizer, aligned with other
tokenizers' file resource config — supports both `type: local` (local
path) and `type: remote` (file resource from object storage). Dispatches
by file extension: `.csv` via `load_user_dictionary_from_csv` with the
configured `DictionaryKind`, `.bin` via `load_user_dictionary_from_bin`.
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
## Summary
- Inject Milvus zap logger into hook plugin via `SetZapLogger` type
assertion after plugin load, enabling the plugin to use Milvus's
centralized logging
- Inject connection manager into hook plugin via `SetClientInfoProvider`
type assertion, allowing the plugin to look up SDK info (sdk_type,
sdk_version, host) for non-Connect requests by connection identifier
- Pass `*gin.Context` directly via `GinParamsKey` instead of
`ginCtx.Keys` map, enabling the hook plugin to access `ClientIP()`,
`FullPath()`, `GetHeader()` and other gin context methods for RESTful
requests
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
relate: https://github.com/milvus-io/milvus/issues/48612
## Summary
- Increment `fileResourceRefCnt` during `validateSchema` instead of in
the async ack callback's `AddCollection`, closing the TOCTOU race where
`RemoveFileResource` could delete a resource between validation and
`AddCollection`
- On failure before `Broadcast`, refCnt is decremented immediately; on
restart, refCnt for pending broadcast tasks is recovered from etcd
before rootcoord becomes Healthy
- Remove refCnt++ from `addCollectionMeta` since it's now done at
validation time (reload path unchanged)
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Summary
Two improvements to BM25 stats memory accounting in idfOracle:
### 1. Per-entry estimate: 80 → 20 bytes/entry
The previous hardcoded `bm25StatsPerEntryBytes = 80` overestimates
`map[uint32]int32` memory by 4-7x. Empirical measurement on Go 1.24
(median of 5 runs, n ≥ 10K to avoid GC noise):
| n | heap delta | bytes/entry | overestimate vs 80 |
|---|---|---|---|
| 10K | 152 KB | 15.22 | 5.2× |
| 100K | 1.22 MB | 12.18 | 6.6× |
| 1M | 19.4 MB | 19.43 | 4.1× |
| 5M | 77.9 MB | 15.58 | 5.1× |
| 10M | 156 MB | 15.58 | 5.1× |
| 50M | 623 MB | 12.47 | 6.4× |
Steady-state cost oscillates between **12 and 19.5 bytes/entry**
depending on the swiss table fill ratio (peak ~19.5 right after a 2×
capacity grow, trough ~12 when near load factor 7/8).
Root cause: the original comment claimed "bucket overhead ~72B per
entry", but Go map buckets/groups hold 8 entries each, so amortized
overhead is ~10 bytes/entry, not 72.
**Fix**: lower the default to 20 (covers measured upper bound 19.5 with
small safety margin) and expose as
`queryNode.idfOracle.bm25StatsBytesPerEntry` (refreshable) for runtime
tuning.
### 2. Skip cgo Charge/Refund when tiered eviction is disabled
When `queryNode.segcore.tieredStorage.evictionEnabled = false` (the
default), the C++ caching layer's resource accounting is inert — no
eviction will be driven by it. The per-load `C.ChargeLoadedResource` /
`C.RefundLoadedResource` cgo calls are pure overhead in this case.
**Fix**: gate `syncResource` and `checkMemoryResource` on the eviction
flag. Skip entirely when eviction is off.
## Impact
- ~75% reduction in cache budget consumed by BM25 stats (when eviction
enabled)
- Eliminates per-segment cgo overhead in default deployments (eviction
off)
- Tunable per-entry estimate without redeploying
## Test plan
- [x] `TestIDFOracle` passes
- [x] `TestBM25Stats_MemSize` passes with the new configurable value
- [x] Verified memory measurement methodology with multiple sample sizes
issue: #46468🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
## Summary
- Wraps `cm.Reader` (S3/MinIO HTTP body) with `bufio.NewReader` in
`streamOneFile` before creating the TeeReader
- `DeserializeFromReader` calls `binary.Read` which reads 4/8 bytes per
call; without buffering, each small read hits the remote reader
directly, causing excessive IO operations during preload parsing
- With `bufio.Reader` (default 4KB buffer), small reads are served from
the buffer, reducing remote IO to ~1 per 4KB
## Test plan
- [x] Existing `TestIDFOracle` tests pass
- [x] Verified via profiling that IO syscall count drops significantly
during BM25 preload
issue: #46468🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
relate: #41424
## Summary
- Stream BM25 stats directly to local disk via TeeReader/io.Copy,
avoiding full in-memory download and re-serialization
- Add caching layer resource tracking (Charge/Refund) for BM25 memory
and disk usage
- Add `queryNode.idfOracle.preload` to defer stats parsing to first
SyncDistribution
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
When DataNode is configured with `sync` mode for file resource manager
but no storage address is set in yaml (i.e., remote storage type with
empty address), the initialization previously attempted to create a
ChunkManager and failed with an error, causing the node to fail to
start. This change detects the missing address upfront and gracefully
falls back to `close` mode instead.
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
relate: https://github.com/milvus-io/milvus/issues/41424
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Pull Request Summary
**Core Invariant**: File resources must physically exist in storage
before being registered in metadata; this PR enforces the invariant at
the entry point (AddFileResource) by validating file existence via a
storageClient abstraction before any metadata mutation.
**Enhancement: Pre-validation of file resource existence**
- Introduces a `storageClient` interface (`Exist(ctx, filePath) (bool,
error)`) that FileResourceManager uses to validate storage presence
before registration
- AddFileResource now checks file existence and returns an InputError if
the file is missing, preventing stale/phantom resource entries in
metadata
- Previously, resources could be registered without verifying the
underlying file existed, creating consistency issues when storage
operations failed or files were deleted externally
**Removes redundant synchronization boilerplate**
- Consolidates QC/DN synchronization logic into FileResourceManager;
AddFileResource and RemoveFileResource methods now internally call
Notify and syncQcFileResource(ctx) instead of repeating this pattern in
services.go
- services.go no longer needs explicit ListFileResource +
SyncQcFileResource calls after add/remove operations; it now simply
delegates to fileManager.AddFileResource/RemoveFileResource
- Proxy layer implementations (AddFileResource, RemoveFileResource,
ListFileResources) now forward to real mixCoord methods instead of "not
implemented" stubs, making the feature fully functional
**No data loss or behavior regression**
- Storage existence check is gated by qnMode == SyncMode (for QC sync)
and dnMode == SyncMode (for DN sync loop startup), preserving backward
compatibility for non-sync deployments
- Resources are only added to metadata after existence confirmation,
ensuring consistency; removal remains symmetric and does not require
existence checks
- The mode guards in Start() and Notify() ensure synchronization only
occurs when configured, preventing spurious sync attempts in disabled
modes
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
Add a thread pool to load BM25 stats and save them to local disk during
loading, reducing peak memory usage.
relate: https://github.com/milvus-io/milvus/issues/41424
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
- Core invariant: sealed-segment BM25 statistics are immutable and may
be persisted to disk independently from growing-segment stats; IDF state
is reconstructed by combining persisted sealed stats (preloaded from
disk) with in-memory growing stats at runtime (preloadSealed +
RegisterGrowing).
- Capability added: asynchronous BM25 loading via a dedicated
BM25LoadPool (config key common.threadCoreCoefficient.bm25Load and param
BM25LoadThreadCoreCoefficient) — delegator.loadBM25Stats is executed on
the pool to load sealed BM25 stats and call idfOracle.RegisterSealed,
reducing peak memory during segment load.
- Logic removed/simplified and why: the previous single Register(segID,
stats, state) + background local-writer/spool loop was split into
RegisterGrowing (in-memory) and RegisterSealed (sealed + on-disk
persistence) and the localloop removed; RegisterSealed writes sealed
stats directly (ToLocal) and uses singleflight to deduplicate,
eliminating redundant spooling and lifecycle complexity while clarifying
sealed vs growing flows.
- Why this does NOT introduce data loss or behavior regression: sealed
stats are still written and reloaded (RegisterSealed persists to
dirPath; preloadSealed merges disk and memory on first load), growing
segments continue to register in-memory via RegisterGrowing,
loadSegments now defers sealed BM25 handling to loadBM25Stats but still
registers sealed candidates after load, and tests were updated to
reflect RegisterSealed/RegisterGrowing usage—so
serialization/deserialization, preload semantics, and test coverage
preserve existing persisted state and runtime behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
relate: https://github.com/milvus-io/milvus/issues/46718
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Enhancement: Optimize Jieba and Lindera Analyzer Clone
**Core Invariant**: JiebaTokenizer and LinderaTokenizer must be
efficiently cloneable without lifetime constraints to support analyzer
composition in multi-language detection chains.
**What Logic Was Improved**:
- **JiebaTokenizer**: Replaced `Cow<'a, Jieba>` with
`Arc<jieba_rs::Jieba>` and removed the `<'a>` lifetime parameter. The
global JIEBA instance now wraps in Arc, enabling `#[derive(Clone)]` on
the struct. This eliminates lifetime management complexity while
maintaining zero-copy sharing via atomic reference counting.
- **LinderaTokenizer**: Introduced public `LinderaSegmenter` struct
encapsulating dictionary and mode state, and implemented explicit
`Clone` that properly duplicates the segmenter (cloning Arc-wrapped
dictionary), applies `box_clone()` to each boxed token filter, and
clones the token buffer. Previously, Clone was either unavailable or
incompletely handled trait objects.
**Why Previous Implementation Was Limiting**:
- The `Cow::Borrowed` pattern for JiebaTokenizer created explicit
lifetime dependencies that prevented straightforward `#[derive(Clone)]`.
Switching to Arc eliminates borrow checker constraints while providing
the same reference semantics for immutable shared state.
- LinderaTokenizer's token filters are boxed trait objects, which cannot
be auto-derived. Manual Clone implementation with `box_clone()` calls
correctly handles polymorphic filter duplication.
**No Data Loss or Behavior Regression**:
- Arc cloning is semantically equivalent to `Cow::Borrowed` for
read-only access; both efficiently share the underlying Jieba instance
and Dictionary without data duplication.
- The explicit Clone preserves all tokenizer state: segmenter (with
shared Arc dictionary), all token filters (via individual box_clone),
and the token buffer used during tokenization.
- Token stream behavior unchanged—segmentation and filter application
order remain identical.
- New benchmarks (`bench_jieba_tokenizer_clone`,
`bench_lindera_tokenizer_clone`) measure and validate clone performance
for both tokenizers.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
Add support for DataNode compaction using file resources in ref mode.
SortCompation and StatsJobs will build text indexes, which may use file
resources.
relate: https://github.com/milvus-io/milvus/issues/43687
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
- Core invariant: file resources (analyzer binaries/metadata) are only
fetched, downloaded and used when the node is configured in Ref mode
(fileresource.IsRefMode via CommonCfg.QNFileResourceMode /
DNFileResourceMode); Sync now carries a version and managers track
per-resource versions/resource IDs so newer resource sets win and older
entries are pruned (RefManager/SynchManager resource maps).
- Logic removed / simplified: component-specific FileResourceMode flags
and an indirection through a long-lived BinlogIO wrapper were
consolidated — file-resource mode moved to CommonCfg, Sync/Download APIs
became version- and context-aware, and compaction/index tasks accept a
ChunkManager directly (binlog IO wrapper creation inlined). This
eliminates duplicated config checks and wrapper indirection while
preserving the same chunk/IO semantics.
- Why no data loss or behavior regression: all file-resource code paths
are gated by the configured mode (default remains "sync"); when not in
ref-mode or when no resources exist, compaction and stats flows follow
existing code paths unchanged. Versioned Sync + resourceID maps ensure
newly synced sets replace older ones and RefManager prunes stale files;
GetFileResources returns an error if requested IDs are missing (prevents
silent use of wrong resources). Analyzer naming/parameter changes add
analyzer_extra_info but default-callers pass "" so existing analyzers
and index contents remain unchanged.
- New capability: DataNode compaction and StatsJobs can now build text
indexes using external file resources in Ref mode — DataCoord exposes
GetFileResources and populates CompactionPlan.file_resources;
SortCompaction/StatsTask download resources via fileresource.Manager,
produce an analyzer_extra_info JSON (storage + resource->id map) via
analyzer.BuildExtraResourceInfo, and propagate analyzer_extra_info into
BuildIndexInfo so the tantivy bindings can load custom analyzers during
text index creation.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
- Core invariant: config refresh events must reliably propagate updated
values and evict cached entries within a bounded time window; tests must
observe this deterministically without relying on fixed sleeps.
- Logic simplified: brittle fixed time.Sleep delays and separate error
assertions were replaced by assert.Eventually polling blocks that
combine value checks and cache-eviction verification, and consolidated
checks to reduce redundant assertions.
- Why no data loss / no behavior regression: only test synchronization
and assertions were changed—production config manager code paths (value
propagation, KV puts, cache eviction) are untouched; tests now wait for
the same outcomes more robustly, so no mutation of runtime behavior or
storage occurs.
- Enhancement scope: this is a test-stability improvement (no new
runtime capability); it fixes flaky unit tests (root cause: timing
assumptions) by replacing fixed waits with bounded polling and by using
t.Context for KV puts to align test context usage.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
- Core invariant: TestWait must deterministically verify that
FixedSizeAllocator.Wait() is notified when virtual resources are
released, so an allocation blocked due to exhausted virtual capacity
eventually succeeds after explicit deallocations.
- Removed/simplified logic: replaced the previous flaky pattern that
spawned 100 concurrent goroutines performing Reallocate with an explicit
channel-synchronized release goroutine that performs 100 sequential
negative Reallocate calls only after the test blocks on allocation. This
eliminates timing-dependent concurrency and the nondeterministic i-based
assertion.
- Why no data loss or behavior regression: only the test changed —
allocator implementation (Allocate/Reallocate/Release/Wait/notify) and
public APIs are unchanged. The test now exercises the same code paths
(Allocate fails, Wait blocks on cond, Reallocate/Release call notify),
but in a deterministic order, so the allocator semantics and resource
accounting (used, allocs map) remain unaffected.
- Change type and intent: Enhancement/refactor of unit test stability —
it tightens test synchronization to remove race-dependent assertions and
ensure the Wait/notify mechanism is reliably exercised without modifying
allocator logic.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
relate: https://github.com/milvus-io/milvus/issues/46498
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
- Core invariant: text fields configured with multi_analyzer_params must
include a "by_field" string that names another field containing per-row
analyzer choices; schemaInfo.GetMultiAnalyzerNameFieldID caches and
returns the dependent field ID (or 0 if none) and relies on that mapping
to make per-row analyzer names available to the highlighter.
- What changed / simplified: the highlighter is now schema-aware —
addTaskWithSearchText accepts *schemaInfo and uses
GetMultiAnalyzerNameFieldID to resolve the analyzer-name field;
resolution and caching moved into schemaInfo.multiAnalyzerFieldMap
(meta_cache.go), eliminating ad-hoc/typeutil-only lookups and duplicated
logic; GetMultiAnalyzerParams now gates on EnableAnalyzer(),
centralizing analyzer enablement checks.
- Why this fixes the bug (root cause): fixes#46498 — previously the
highlighter failed when the analyzer-by-field was not in output_fields.
The change (1) populates task.AnalyzerNames (defaulting missing names to
"default") when multi-analyzer is configured and (2) appends the
analyzer-name field ID to LexicalHighlighter.extraFields so FieldIDs
includes it; the operator then requests the analyzer-name column at
search time, ensuring per-row analyzer selection is available for
highlighting.
- No data-loss or regression: when no multi-analyzer is configured
GetMultiAnalyzerNameFieldID returns 0 and behavior is unchanged; the
patch only adds the analyzer-name field to requested output IDs (no
mutation of stored data). Error handling on malformed params is
preserved (errors are returned instead of silently changing data), and
single-analyzer behavior remains untouched.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
Support crate analyzer with file resource info, and return used file
resource ids when validate analyzer.
Save the related resource ids in collection schema.
relate: https://github.com/milvus-io/milvus/issues/43687
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
- Core invariant: analyzer file-resource resolution is deterministic and
traceable by threading a FileResourcePathHelper (collecting used
resource IDs in a HashSet) through all tokenizer/analyzer construction
and validation paths; validate_analyzer(params, extra_info) returns the
collected Vec<i64) which is propagated through C/Rust/Go layers to
callers (CValidateResult → RustResult::from_vec_i64 → Go []int64 →
querypb.ValidateAnalyzerResponse.ResourceIds →
CollectionSchema.FileResourceIds).
- Logic removed/simplified: ad‑hoc, scattered resource-path lookups and
per-filter file helpers (e.g., read_synonyms_file and other inline
file-reading logic) were consolidated into ResourceInfo +
FileResourcePathHelper and a centralized get_resource_path(helper, ...)
API; filter/tokenizer builder APIs now accept &mut
FileResourcePathHelper so all file path resolution and ID collection use
the same path and bookkeeping logic (redundant duplicated lookups
removed).
- Why no data loss or behavior regression: changes are additive and
default-preserving — existing call sites pass extra_info = "" so
analyzer creation/validation behavior and error paths remain unchanged;
new Collection.FileResourceIds is populated from resp.ResourceIds in
validateSchema and round‑tripped through marshal/unmarshal
(model.Collection ↔ schemapb.CollectionSchema) so schema persistence
uses the new list without overwriting other schema fields; proto change
adds a repeated field (resource_ids) which is wire‑compatible (older
clients ignore extra field). Concrete code paths: analyzer creation
still uses create_analyzer (now with extra_info ""), tokenizer
validation still returns errors as before but now also returns IDs via
CValidateResult/RustResult, and rootcoord.validateSchema assigns
resp.ResourceIds → schema.FileResourceIds.
- New capability added: end‑to‑end discovery, return, and persistence of
file resource IDs used by analyzers — validate flows now return resource
IDs and the system stores them in collection schema (affects tantivy
analyzer binding, canalyzer C bindings, internal/util analyzer APIs,
querynode ValidateAnalyzer response, and rootcoord/create_collection
flow).
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
relate: https://github.com/milvus-io/milvus/issues/46571
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
- Core invariant: the LexicalHighlighter API now expects the match
queries under the parameter name highlight_query (not queries); all call
sites must pass highlight_query to supply match data. This PR assumes
the underlying highlighter behavior and processing of those query values
are unchanged.
- Logic simplified/removed: removed the legacy keyword queries in tests
and updated calls to use highlight_query
(tests/python_client/milvus_client/test_milvus_client_highlighter.py).
This eliminates a redundant/incorrect keyword alias and aligns tests
with the consolidated LexicalHighlighter constructor parameter name.
- Why this does NOT introduce data loss or behavior regression: the
change is a parameter-name rename only — no parsing, matching, or
storage logic was modified. Tests now construct LexicalHighlighter with
pre_tags/post_tags/highlight_search_text/fragment_* and pass the query
list under highlight_query; the highlighter execution path
(client.search → highlighter processing → result['highlight']) is
untouched, so existing highlight outputs and stored data remain
unchanged.
- Other changes: bumped pymilvus test dependency to 2.7.0rc93 in
tests/python_client/requirements.txt to match the updated constructor
signature; scope of change is limited to tests and dependency pinning
(no production code changes).
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
relate: https://github.com/milvus-io/milvus/issues/43687
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* File resource operations (add, remove, list) are now unavailable and
return a not-implemented response.
* **Tests**
* Tests updated to expect error responses for those file resource
operations and removed some previous coordination-path assertions.
<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
relate: https://github.com/milvus-io/milvus/issues/42589
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## New Features
- Added `concurrency_per_cpu_core` configuration parameter for the
analyzer component, enabling customizable per-CPU concurrency tuning
(default: 8).
## Tests
- Added test coverage for batch analysis operations.
<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
relate:https://github.com/milvus-io/milvus/issues/43687
Support use file resource with sync mode.
Auto download or remove file resource to local when user add or remove
file resource.
Sync file resource to node when find new node session.
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
relate: https://github.com/milvus-io/milvus/issues/43687
We used to run the temporary analyzer and validate analyzer on the
proxy, but the proxy should not be a computation-heavy node. This PR
move all analyzer calculations to the streaming node.
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
relate: https://github.com/milvus-io/milvus/issues/44758
Wrong code which should be `(result.seg_offsets_[i] >= 0 &&
result.seg_offsets_[j] < 0)`, but was `(result.seg_offsets_[j] >= 0 &&
result.seg_offsets_[j] < 0) ` now.
But because all placeholder which was offset -1, will fill with worst
distance value.
For IP, L2 or COSIN, it will be +inf or -inf. So sort distance was
enough.
But when use BM25, it will be NAN. Will case sort out of ordered.
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
And support set function mode and boost mode when run search with boost.
RandomScore support get random function score between [0, weight).
FunctionMode decide how to calculate boost score for multiple boost
function scores.
BoostMode decide how to calculate final score for origin score and boost
score.
relate: https://github.com/milvus-io/milvus/issues/43867
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
