42f6bcb69a
fix: add PK stats to growing-source flush ( #51532 )
...
issue: #50425
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2026-07-18 22:26:40 +08:00
9ffdaee294
fix: gate growing-source flush on StorageV3 ( #51410 )
...
issue: #51250
Keep TEXT segments on StorageV3 while making growing-source flush an
explicit allowed mode instead of a mandatory TEXT path.
Propagate create-segment storage versions through WAL flusher and write
buffer, reject storage-version mismatches in DataCoord, and keep raw
segcore chunks sticky for segments that may be flushed from growing
source.
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2026-07-17 10:22:39 +08:00
5bce29fed1
fix: prepare release handoff before manual flush ( #51267 )
...
issue: #50425
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2026-07-14 19:58:36 +08:00
879312922b
fix: use binary arrays for nullable text lob refs ( #51124 )
...
issue: #50425
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2026-07-07 23:02:34 +08:00
84a0dcc2d4
fix: handle text lob refs and flush pressure accounting ( #51079 )
...
issue: #50877
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2026-07-07 10:22:43 +08:00
33f93c2aa1
fix: keep dual writes for growing-source flush ( #50999 )
...
issue: #50911
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2026-07-04 13:44:28 +08:00
05027eea17
fix: load vortex text lob refs as binary ( #50929 )
...
issue: #50906
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2026-07-01 14:12:28 +08:00
439f829377
fix: handle unaligned mmap-backed Arrow buffers ( #50893 )
...
issue: #50884
Align mmap allocations to max_align_t and copy fixed-width Arrow buffers
when source memory is not aligned for the element type.
Add coverage for mmap allocation alignment.
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2026-06-30 11:06:31 +08:00
f02ce55acb
fix: serialize cached text LOB reader access ( #50862 )
...
issue: #50828
---------
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2026-06-30 11:04:40 +08:00
70991957bb
fix: propagate json stats parquet close errors ( #50856 )
...
issue: #50821
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2026-06-29 11:30:27 +08:00
2310c93b15
fix: align growing source storage v3 flush ( #50738 )
...
issue: #50697
Summary:
1. Align StorageV3 column group layout for growing-source flush by
reusing the segment currentSplit/manifest layout, passing a schema-based
split pattern to C++, and using the schema_based writer policy instead
of always writing a single column group.
2. Project growing flush output to the target segment layout by passing
AllowedFieldIDs to C++, so ordinary fields, vector fields, text fields,
and BM25 output fields are all filtered to the old segment layout when
appending to an existing StorageV3 manifest.
3. Stop retrying non-retryable layout mismatches and preserve V3 layout
metadata, so column count/group mismatches do not loop forever and
recovery/balance can restore currentSplit for future layout-compatible
appends.
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2026-06-27 19:58:26 +08:00
656b600a46
fix: preserve text lob refs in compaction ( #50784 )
...
issue: #50770
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2026-06-26 10:30:27 +08:00
66bf1f168b
fix: avoid holding writebuffer lock during sync submit ( #50760 )
...
issue: #50750
Move sync task submission out of the writebuffer mutex. The writebuffer
still selects segments, yields buffered payload, records sync
checkpoints, and marks segment syncing while holding the lock, but
submits the already built tasks to syncmgr after releasing it.
This prevents ReleaseCollection from hanging in manual-flush preparation
when syncmgr dispatcher backpressure blocks SyncData submission, because
CheckReleaseManualFlushNeed can still acquire the writebuffer read lock.
Also move growing-source follow-up resync out of the callback critical
section and add a regression test that blocks SyncData submission while
verifying writebuffer readers are not blocked.
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2026-06-26 05:08:26 +08:00
b4f8684ce9
fix: preserve text lob metadata in merge-sort compaction ( #50754 )
...
issue: #50730
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2026-06-25 15:36:29 +08:00
7489fe45f4
fix: avoid compare fast path for misaligned chunks ( #50609 )
...
#49346
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2026-06-23 10:28:33 +08:00
fa441d8861
feat: add result cache function for expr ( #48873 )
...
#50225
Co-authored-by: luzhang <luzhang@zilliz.com >
2026-06-18 10:20:24 +08:00
85e6523f36
enhance: validate TEXT add field and optimize LOB text indexing ( #50494 )
...
issue: #50425
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2026-06-17 14:34:23 +08:00
80257b9f97
enhance: support text_match/bm25 for text type ( #50426 )
...
issue: #48783
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2026-06-11 16:28:21 +08:00
f756c52a22
enhance:refactor flush source ( #50300 )
...
issue: #50425
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2026-06-10 10:30:19 +08:00
81eac46f00
enhance: avoid count updates for delete snapshots ( #49718 )
...
issue: #47346
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2026-06-03 15:02:19 +08:00
cd8b3b11f0
fix: fill missing rows for JSON stats build ( #49590 )
...
issue: #49495
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2026-06-03 14:42:19 +08:00
36f4dd5a7f
fix: fold boolean literals in filter expressions ( #49446 )
...
issue: #48443
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2026-05-07 20:10:09 +08:00
8111fd65f7
fix: handle nullable compare offset input ( #49445 )
...
issue: #49346
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2026-05-07 20:08:13 +08:00
3bc0d7cdf1
fix: restore RowID and limit TEXT flush behavior changes ( #49529 )
...
issue: #49495
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2026-05-07 19:44:07 +08:00
2a09964041
fix: use configured parallel jobs for ninja install ( #49405 )
...
#48783
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2026-04-28 20:39:51 +08:00
5409a81f40
feat: support TEXT column with LOB storage for large text fields ( #47567 )
...
issue: #48783
design doc:
https://github.com/milvus-io/milvus-design-docs/blob/main/design_docs/20260407-text_lob_storage.md
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2026-04-24 14:25:45 +08:00
fe13e4cb06
fix: fix build bug for json stats with lack-of-binlog ( #49249 )
...
issue: #42533
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2026-04-24 11:01:45 +08:00
f79085e5ec
fix: fix wrong use of snapshot restore for jsonstats ( #48912 )
...
issue: #48713
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2026-04-23 15:19:45 +08:00
861466290d
fix:fix restore json stats bug ( #48597 )
...
#48579
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2026-04-01 14:49:35 +08:00
f6a568489f
fix: fix wrong bitset size when using textmatch index ( #48479 )
...
issue: #48388
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2026-04-01 14:47:41 +08:00
d221fce578
enhance: skip multi-batch execute when all indexed expr ( #48049 )
...
issue: #48257
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2026-03-17 19:49:28 +08:00
1997983cc0
fix:remove snapshot when DumpSnapshot Assert failure ( #48250 )
...
issue: #48258
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2026-03-17 12:07:26 +08:00
9be7ee6802
enhance: add json related query metric ( #47551 )
...
issue: #47572
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2026-03-05 15:13:25 +08:00
12f67efcec
fix: use {} placeholder in LOG_* macros to avoid treating error as format string ( #47483 )
...
issue: #47484
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2026-02-03 16:34:00 +08:00
2bcad90737
enhance: normalize constant-folded bool ValueExpr to AlwaysTrueExpr/A… ( #47467 )
...
issue: #47339
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2026-02-03 16:31:56 +08:00
2ac1ae4b7d
enhance: add latest delete snapshot optimization ( #47347 )
...
issue: #47346
Config: N=100000, Deleted=9000, Queries=10000
Optimization OFF (slow path): 201534 us (20 us/query)
Optimization ON (fast path): 5323 us (0 us/query)
Speedup: 37.86x faster with optimization ON
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2026-01-29 13:13:33 +08:00
24abf8581a
fix: avoid coredump and print more info for PhyRescoresNode ( #47340 )
...
issue: #47339
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2026-01-28 15:49:33 +08:00
92c2f83328
fix: azure precheck use a fixed bucket not belong to milvus ( #47170 )
...
issue: #47157
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2026-01-21 14:19:29 +08:00
925d5875fd
fix: remove unnecessary batching for avoiding OOM ( #47078 )
...
issue: #47001
different batch effect for same test:
| Batch Size | 峰值内存 | 构建时间 | Term Query |
|------------|------------|------------|--------------------|
| 1 | 107-118 MB | 10.4-10.7s | 0.11ms (~8900 QPS) |
| 64 | 330 MB | 10.6s | 0.12ms (~7900 QPS) |
| 148 | 610 MB | 10.7s | 0.12ms (~8000 QPS) |
| 304 | 1131 MB | 11.1s | 0.13ms (~8000 QPS) |
| 4096 | 13500 MB | 12.3s | 0.14ms (~8000 QPS) |
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2026-01-19 19:45:29 +08:00
635bead131
fix:fix incorrect rootPath for local storage mode ( #46692 )
...
#45959
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2025-12-31 12:23:22 +08:00
031acf5711
enhance: convert jsonstats translator to bson_index translator ( #45036 )
...
issue: #42533
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2025-12-31 10:39:21 +08:00
83ab90af93
enhance: modify bson thirdparty lib compile mode ( #45406 )
...
#42533
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2025-12-30 15:41:21 +08:00
9ba0c4e501
fix:add json stats version because previous change #46130 ( #46467 )
...
#42533
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2025-12-24 19:17:18 +08:00
52026cf07e
enhance: change jemalloc monitor from compile-time to runtime ( #46377 )
...
#46133
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2025-12-17 16:17:16 +08:00
a86b8b7a12
enhance: move jsonshredding meta from parquet to meta.json ( #46130 )
...
#42533
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2025-12-11 14:01:13 +08:00
8f0b7983ec
enhance: add jemalloc cached monitor ( #46041 )
...
#46133
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2025-12-09 19:53:13 +08:00
3901f112ae
enhance: make estimate json stats size more accurate ( #45875 )
...
#42533
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2025-12-01 15:31:10 +08:00
d5bd17315c
enhance: remove some meta cache for json shredding ( #45888 )
...
#42533
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2025-12-01 11:57:09 +08:00
1b58844319
enhance: support mmap for jsonstats shared key index ( #44914 )
...
#42533
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2025-11-27 16:01:08 +08:00
346449d87f
fix:fix undefined behavior for dump snapshot ( #45611 )
...
#45610
this fix add a little cost for execute:
=== Lower Bound Overhead (isolated) ===
Position 1 (list len = 90000): 39 ns per lower_bound
Position 2 (list len =180000): 45 ns per lower_bound
Position 3 (list len =270000): 46 ns per lower_bound
Position 4 (list len =360000): 38 ns per lower_bound
Position 5 (list len =450000): 42 ns per lower_bound
Position 6 (list len =540000): 55 ns per lower_bound
Position 7 (list len =630000): 56 ns per lower_bound
Position 8 (list len =720000): 49 ns per lower_bound
Position 9 (list len =810000): 48 ns per lower_bound
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2025-11-25 14:17:07 +08:00
59c64bee07
fix: not use json_shredding for json path is null ( #45310 )
...
#45284
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2025-11-06 11:43:33 +08:00
792e931fcb
enhance: rename jsonstats related user config params ( #45254 )
...
#44132
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2025-11-04 20:21:36 +08:00
653e95aaad
fix: fix bug for shredding json when empty json but not null ( #45221 )
...
#45157
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2025-11-04 11:11:33 +08:00
a38610cd5d
fix: disable build old version jsonstats from request ( #45101 )
...
#44132
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2025-10-27 19:50:10 +08:00
730308b1eb
fix: fix not equal not include None ( #44959 )
...
#44816
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2025-10-21 17:08:03 +08:00
05df48fbe4
fix:remove duplicated '/' in jsonstats path ( #44939 )
...
#44950
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2025-10-20 14:06:03 +08:00
b7935557e1
fix:unified json exists path semantic ( #44916 )
...
#44927
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2025-10-17 16:40:02 +08:00
ae19c93c14
enhance: remove timestamp filter for search_ids to optimize performance ( #44634 )
...
#44352
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2025-10-17 16:10:01 +08:00
2f178f810f
fix:fix json_contains(path, int) bug ( #44814 )
...
#44816
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2025-10-14 00:19:59 +08:00
3dd5deb70a
fix:disable using shredding for json_path contains digital ( #44724 )
...
#44132
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2025-10-13 17:25:59 +08:00
4c49295c3d
Revert "enhance: enable default json stats ( #44559 )" ( #44644 )
...
This reverts commit 1b5191974c#44132
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2025-09-30 12:11:53 +08:00
1b5191974c
enhance: enable default json stats ( #44559 )
...
#44132
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2025-09-28 10:45:04 +08:00
zhagnlu and GitHub
eac16a577c
enhance:support cachelayer for json stats ( #44446 )
...
#42533
Signed-off-by: zhagnlu <lu.zhang@zilliz.com >
2025-09-24 15:30:04 +08:00
9b6703626d
fix:fix unescaped bug for json stats ( #44421 )
...
#42533
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2025-09-17 20:54:01 +08:00
baa84e0b2b
fix: avoid mvcc when doing pk compare expr ( #44353 )
...
#44352
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2025-09-15 10:17:59 +08:00
e9bbb6aa9b
fix: fix json_contains bug for stats ( #44325 )
...
#42533
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2025-09-15 10:16:07 +08:00
16e6b6aa8a
fix:fix build json stats bug for nested object ( #44303 )
...
issue: #44132
Signed-off-by: luzhang <luzhang@zilliz.com >
Co-authored-by: luzhang <luzhang@zilliz.com >
2025-09-11 14:13:56 +08:00
zhagnlu and GitHub
77f7d19400
fix:avoid mmap rewrite by multi json fields ( #44299 )
...
issue: #44127
Signed-off-by: zhagnlu <lu.zhang@zilliz.com >
2025-09-11 10:13:57 +08:00
