furyhawk/milvus
1
0
Fork 0
mirror of https://github.com/milvus-io/milvus.git synced 2026-07-21 02:05:41 +00:00
Code Issues Packages Projects Releases Wiki Activity
25,145 Commits 208 Branches 226 Tags
master
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
yah01andGitHub d357139064 fix: the entities num metric may be contributed more than once (#29767) 
the growing segments contribute to this metric while inserting and
putting into the manager, but the current impl inserts data before
putting the segments into manager, which leads to double contributions

fix: #29766

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2024-01-10 10:00:51 +08:00
yah01andGitHub 97e4ec5a69 enhance: use random root path for minio unit tests (#29753) 
this avoids the conflicts while running multiple unit tests

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2024-01-08 15:58:48 +08:00
yah01andGitHub a8a0aa9357 fix: missing to support compact for Array type (#29505) 
the array type can't be compacted, the system could continue with the
inserted segments, but these segments can be never compacted

fix #29503

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-12-28 15:42:48 +08:00
yah01andGitHub aef483806d enhance: improve the segcore logs (#29372) 
- remove the streaming logging
- refine existing logs

fix #29366

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-12-23 21:52:43 +08:00
yah01andGitHub 9819090247 enhance: add more logs for target updating (#29090) 
- add more logs about the condition satisfying

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-12-12 14:06:43 +08:00
yah01andGitHub 0a87724f18 enhance: remove merger for load segments (#29062) 
remove merger as now QueryNode could load segments concurrently
fix #29063

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-12-12 10:48:37 +08:00
yah01andGitHub c4dda3c50f fix: panic caused by type assert LocalSegment on Segment (#29018) 
related #29017

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-12-08 01:50:38 +08:00
yah01andGitHub 9b3e06ae86 enhance: add more metrics for level zero segments (#29029) 
- Add SegmentNum metric for level zero segments
- Add level zero segments size metirc

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-12-07 14:34:35 +08:00
yah01andGitHub fab52d167b fix: may miss stream delta while loading (#28871) 
we consume the delta data from the lastest channel checkpoint while
loading segment,

this works well without level 0 segments, but now it may lead to miss
some delta data,

so we have to consume from the current target's channel checkpoint

related: #27349

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-12-05 17:34:45 +08:00
yah01andGitHub 342635ed61 enhance: enable assert method to format arguments (#28812) 
for now the assert method in segcore could accept a string information,
too many codes don't print the value they assert.

make it happy
related #28811

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-12-01 18:04:33 +08:00
yah01andGitHub bf633bb5d7 enhance: refine the retry error (#28573) 
return the last error but not combining all errors, to improve
readability and erorr handling

resolve: #28572

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-30 18:34:32 +08:00
yah01andGitHub d69440524b fix: bypass growing index if no index meta (#28791) 
we shouldn't panic if no index meta, just skip building it
fix #28022

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-11-30 14:10:27 +08:00
yah01andGitHub c0f6eccb7a fix: No LevelZero segment in target (#28803) 
the incorrect filter causes all LevelZero segment filtered, so the
deleted entities may be still visible
related: #27349

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-29 11:48:27 +08:00
yah01andGitHub 3ea0129eb3 enhance: improve the error messages and logs (#28684) 
- better name for log fields
- make the error and log consistent

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-24 15:08:24 +08:00
yah01andGitHub c96d07682e enhance: enhance the accuracy of memory usage (#28554) 
before this, Milvus use container/system's memory info to get the memory
usage, which could be inaccurate.

we allocates the memory by private anon mmap,
then `rss - shared` would be the accurate memory usage

resolve #28553

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-23 15:12:23 +08:00
yah01andGitHub d73dac52c7 enhance: validate JSON data while inserting (#28602) 
some SDKs doesn't check the JSON data validation,
add this in server.

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-21 23:30:22 +08:00
yah01andGitHub cc952e0486 enhance: optimize forwarding level0 deletions by respecting partition (#28456) 
- Cache the level 0 deletions after loading level0 segments
- Divide the level 0 deletions by partition
related: #27349

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-21 18:24:22 +08:00
yah01andGitHub bfccfcd0ca enhance: refine error messages (#28424) 
- Split the simple reason and full detail
- Refine existing error messages
related: #28422

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-21 17:02:24 +08:00
yah01andGitHub 02c5a649cf enhance: store system fields in segcore (#28524) 
we need the system fields info for some usacase
fix: #28523

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-21 09:28:22 +08:00
yah01andGitHub da03feb79a enhance: upgrade OpenDAL to v0.42.0 and optimize the building (#28556) 
- Upgrade OpenDAL to v0.42.0
- Clone OpenDAL with depth 1
- Add build log for OpenDAL

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-20 14:58:20 +08:00
yah01andGitHub d2f53aefa5 enhance: improve load speed (#28518) 
This check rejects load request if running out the pool workers, but
small segment would be loaded soon, another segments would been loading
again after a check interval, which leads to slow loading for collection

Block the request by go pool

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-17 19:56:21 +08:00
yah01andGitHub 1c3e763471 enhance: [skip e2e] remove missing related issue for small enhancement PR (#28503) 
remove the missing-related-issue if the PR size is not large and be with
`kind/enhancement`

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-16 17:34:19 +08:00
yah01andGitHub b4fa3b9a09 enhance: [skip e2e] add more types for PR classification (#28497) 
add 2 more types:
- test: for add tests to existing functionality.
- doc: for modifying documentation.

Also requires related issue for large PR

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-16 17:22:20 +08:00
yah01andGitHub f7d2ab6677 enhance: reduce 1x copy for variable length field while retrieving (#28345) 
- Reduce 1x copy for varchar/string/JSON/array types while retrieving
- Reduce 1x copy for int8/int16 while retrieving

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-15 18:08:20 +08:00
yah01andGitHub fd3ae4c4b8 enhance: improve the compile speed (#28444) 
improve the CI compile speed with mold

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-15 15:08:18 +08:00
yah01andGitHub e9ff7ed13d enhance: remove legacy error related code (#28385) 
/kind improvement

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-14 15:02:18 +08:00
yah01andGitHub 267c67dfee enhance: reduce 1x copy while retrieving data from growing segment (#28323) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-10 15:44:22 +08:00
yah01andGitHub 70995383bf enhance: modify log to avoid ambiguity and improve readability (#28331) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-10 14:32:20 +08:00
yah01andGitHub 6f59510723 Upgrade go to 1.20 (#24390) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-11-10 07:46:18 +08:00
yah01andGitHub 30847cad3e Handle exception while loading (#28304) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-09 17:59:12 +08:00
yah01andGitHub ecb3f585c3 Fix passing the wrong dropped list from current target (#28265) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-08 17:02:18 +08:00
yah01andGitHub d20ea061d6 Fix panic while forwarding empty deletions to growing segment (#28213) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-08 16:42:21 +08:00
yah01andGitHub f4341f254d Not convert legacy error code to new merr (#28232) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-08 14:42:20 +08:00
yah01andGitHub 1b90630633 Fix the target updated before version updated to cause data missing (#28250) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-08 11:36:22 +08:00
yah01andGitHub ece592a42f Deliver L0 segments delete records (#27722) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-07 01:44:18 +08:00
yah01andGitHub 863e26969a Reduce 1x memory copy for retrieving data (#28106) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-06 17:48:19 +08:00
yah01andGitHub 33f17ae5bd Fix mac code check (#28194) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-06 17:34:19 +08:00
yah01andGitHub dc89730a50 Support collection-level mmap control (#26901) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-02 23:52:16 +08:00
yah01andGitHub 1b9b8a31a8 Fix QueryNode panic while upgrading (#28034) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-31 15:50:13 +08:00
yah01andGitHub ab6dbf7659 Limit max thread num for pool (#28018) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-31 13:00:13 +08:00
yah01andGitHub 9658367a3c Refine chunk manager errors (#27590) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-31 12:18:15 +08:00
yah01andGitHub 2af46d7333 Increase the ChunkManager request timeout (#28015) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-31 09:06:13 +08:00
yah01andGitHub 9b6eeb46f1 Cache segment row num, size, and insert count to reduce CGO calls (#28007) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-30 17:54:14 +08:00
yah01andGitHub f212158d61 Fix delete records timestamp may be reordered (#27941) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-27 10:24:10 +08:00
yah01andGitHub a36a57bfec Enable mmap for IVF index (#27904) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-26 01:48:12 +08:00
yah01andGitHub f79c7370f4 Fix panic while flushing dropped/compacted segment (#27927) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-25 22:10:13 +08:00
yah01andGitHub 635efdf170 Schedule loading L0 segments first (#27593) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-19 11:14:06 +08:00
yah01andGitHub 9467de79c3 Use pool to block exceeded tasks (#27767) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-19 03:30:06 +08:00
yah01andGitHub 50672f0d60 Skip empty segment while query/search (#27711) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-15 16:10:04 +08:00
yah01andGitHub 15746b5b5f Refine KV errors (#27588) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-13 17:15:34 +08:00
yah01andGitHub be980fbc38 Refine state check (#27541) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-11 21:01:35 +08:00
yah01andGitHub 2df9908c6a Map old error code to new error (#27616) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-11 12:43:33 +08:00
yah01andGitHub 3759857bc5 Refine Proxy errors (#27499) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-09 10:09:33 +08:00
yah01andGitHub 41495ed266 Improve the error message for getting all indexes of collection (#27389) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-08 21:23:32 +08:00
yah01andGitHub a715165306 Set timeout for leader observer syncing (#27504) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-08 16:55:31 +08:00
yah01andGitHub 8394b3a1ec Block creating new error from status reason (#27426) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-07 11:29:32 +08:00
yah01andGitHub 63ac43a3b8 Refine errors for import (#27379) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-30 10:31:28 +08:00
yah01andGitHub a8ce1b6686 Refine QueryCoord stopping (#27371) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-27 16:27:27 +08:00
yah01andGitHub 2d6a968233 Refine QueryNode errors (#27380) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-26 22:09:28 +08:00
yah01andGitHub 6539a5ae2c Refine DataCoord status (#27262) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-26 17:15:27 +08:00
yah01andGitHub 24354b166c Fix unit test failed when run single test (#27348) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-26 09:23:25 +08:00
yah01andGitHub 61ed1c7de3 Fix code check passed even violates the rule (#27341) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-25 18:05:27 +08:00
yah01andGitHub 93e2eb78c9 Delete only if primary keys exist (#25292) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-09-20 19:03:25 +08:00
yah01andGitHub 773080df24 Fix miss to handle the error of allocating ts (#27264) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-20 18:43:23 +08:00
yah01andGitHub 9baff1b81c Rename error to match the nameing rules (#27214) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-20 10:59:24 +08:00
yah01andGitHub b4f86ea55e Construct all success status with merr (#27226) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-20 10:57:23 +08:00
yah01andGitHub 338848fbc3 Remove unused code and dup imports (#27228) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-20 10:55:23 +08:00
yah01andGitHub 0a750408d0 Skip delta logs have been applied (#26971) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-19 16:21:23 +08:00
yah01andGitHub b9ab18d692 Replace error code with merr for RateLimiter (#27176) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-19 10:37:23 +08:00
yah01andGitHub 0459a662e4 use MADV_WILLNEED for scalar column data (#27170) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-18 18:05:22 +08:00
yah01andGitHub a6b98740b7 Replace all status with only error string (#27125) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-18 14:05:28 +08:00
yah01andGitHub 168e82ee10 Fix panic while handling with the nil status (#27040) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-15 10:09:21 +08:00
yah01andGitHub 0f2883ef9d Optimize the generic zero value (#27102) 
allocate by declare,
this make it possible to allocate the value in stack,
faster and less presure for GC

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-14 19:05:20 +08:00
yah01andGitHub 7bb20fa394 Fix the double nil return values from RPC call (#27101) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-14 19:01:19 +08:00
yah01andGitHub 3d05ddf505 Reduce cpp test time (#27043) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-13 15:41:18 +08:00
yah01andGitHub 00c65fa0d7 Refine QueryNode errors (#27013) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-12 16:07:18 +08:00
yah01andGitHub 45d9fb5929 Fix wrong error codes & names (#26904) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-12 14:41:18 +08:00
yah01andGitHub a6c6d7301c Enable ASAN for ut (#27016) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-12 14:01:17 +08:00
yah01andGitHub 3203ce1654 Reduce copy while retrieving primary keys (#26616) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-09-11 21:31:18 +08:00
yah01andGitHub 22bb9c199b Forbid returning merr directly (#26918) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-08 10:31:15 +08:00
yah01andGitHub 4a2377e669 Fix retrieving may return more result than limit (#26670) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-06 17:47:15 +08:00
yah01andGitHub 3349db4aa7 Refine errors to remove changes breaking design (#26521) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-04 09:57:09 +08:00
yah01andGitHub 941a383019 Fix failed to load collection with more than 128 partitions (#26763) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-02 00:09:01 +08:00
yah01andGitHub 09218bfd3d Optimize loading by reduce 1x copy while reading data (#26746) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-01 10:15:01 +08:00
yah01andGitHub 9004601817 Enable vector index mmap (#26750) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-08-31 17:31:11 +08:00
yah01andGitHub bfcc691129 Fix segment leaked if task canceled (#26685) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-08-30 14:17:03 +08:00
yah01andGitHub 213db490bd Use pointer receiver for large struct (#26668) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-08-30 10:24:29 +08:00
yah01andGitHub b475f25042 Remove invalid offset check while filling data (#26666) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-08-30 09:52:27 +08:00
yah01andGitHub dd4bc5b6a0 Fix data race in gRPC client (#26574) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-08-28 18:26:28 +08:00
yah01andGitHub ba882b49b6 Optimize query/search on growing segment while output vector field (#26542) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-08-24 09:46:24 +08:00
yah01andGitHub c3f5856fbc Fix data race for config with FileSource (#26518) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-08-24 09:18:24 +08:00
yah01andGitHub 4a181c1635 Fix panic while stopping the cluster (#26580) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-08-23 19:08:23 +08:00
yah01andGitHub 74fb244bd7 Upgrade mockery (#26461) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-08-18 18:10:18 +08:00
yah01andGitHub 5dae6a654a Protect segment from being released while query/search (#26322) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-08-16 18:38:17 +08:00
yah01andGitHub 9723787141 Calculate memory usage without page cache (#26389) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-08-16 17:04:17 +08:00
yah01andGitHub 3d8871b30e Fix data race while updating segment version (#26387) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-08-16 16:30:18 +08:00
yah01andGitHub 9605c03c3c Fix the number of rows of column not correct (#26347) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-08-16 13:35:33 +08:00
yah01andGitHub 911406c822 Check the offsets of search result (#26383) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-08-16 11:43:34 +08:00
yah01andGitHub 732d92b642 Add -g flag to compile with debug info (#26354) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-08-15 14:13:32 +08:00
yah01andGitHub c0870d3c62 Set default thread pool size to be smaller (#26328) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-08-14 16:33:35 +08:00