furyhawk/milvus
1
0
Fork 0
mirror of https://github.com/milvus-io/milvus.git synced 2026-07-21 02:05:41 +00:00
Code Issues Packages Projects Releases Wiki Activity
25,145 Commits 208 Branches 226 Tags
master
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
yah01andGitHub 57397b1307 enhance: add new LRU cache impl (#30360) 
- remove  the unused LRU cache
- add new LRU cache impl which wraps github.com/karlseguin/ccache

related #30361

---------

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2024-02-27 20:58:40 +08:00
yah01andGitHub b74673c147 enhance: calculate the accuracy memory usage while loading segment (#30473) 
the old version Knowhere would copy the index data while loading, we
need to consider this to avoid OOM.

Knowhere provides a util function to indicate whether it will load the
index with disk, if not, we need to double the memory usage prediction
for index data

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2024-02-20 14:52:51 +08:00
yah01andGitHub ec688f5bf6 fix: failed to download OpenDAL (#30380) 
OpenDAL's url has been changed
fix #30379

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2024-02-02 14:35:04 +08:00
yah01andGitHub 2f778d9649 fix: proxy may never setup if the port binded (#30035) 
the proxy miss-returned nil while failed to listen the port, then the
server continues to run but we can't connect to service
resolve #30034

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2024-01-31 17:37:04 +08:00
yah01andGitHub 878c4c9463 enhance: limit the max pool size to 16 (#30371) 
according to our benchmark, concurrency level 16 is enough to fully
utilize the object storage network bandwidth

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2024-01-31 14:13:06 +08:00
yah01andGitHub a27c0e86fd enhance: reduce many I/O operations while loading disk index (#30189) 
before this, every time writting the index chunk data into the disk,
there are 4 I/O operations:
- open the file
- seek to the offset
- write the data
- close the file

this optimized this to open only once and continiously write all data.

This also makes it concurrent to load the files from object storage

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2024-01-25 15:23:02 +08:00
yah01andGitHub 51fe4743f1 enhance: hint the invalid metric type in error message (#30263) 
this hints the user passed invalid metric type in error message, so
users could know what's wrong

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2024-01-25 14:15:00 +08:00
yah01andGitHub a77693aa19 enhance: convert the GetObject util to async (#30166) 
This makes it much easier to use

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2024-01-22 19:20:57 +08:00
yah01andGitHub f542bdbf3c enhance: calc the accurate mem size of segment (#30093) 
this stats the real memory size of segment, also reduces the memory
usage in mmap mode
resolve #30095

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2024-01-19 12:32:53 +08:00
yah01andGitHub 0d4e781f69 fix: the system rejects all queries and never recovers if enabled read rate limit (#30061) 
fix #30060

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2024-01-17 23:30:55 +08:00
yah01andGitHub 1185e4dcd5 fix: written file size is over the int32 range and raises error (#30057) 
we sum the total data size in int32, which could lead to an overflow
error
related #30056

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2024-01-17 16:42:54 +08:00
yah01andGitHub 1c8ce33eea fix: report error if the altering index doesn't support mmap (#29832) 
this also checks the param value
fix #29909

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2024-01-17 16:40:54 +08:00
yah01andGitHub 9a3837212c enhance: add index after load succeeded (#30015) 
this avoids a corner case: after load index failed, this index can be
never loaded as it has been added into the segment's index map

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2024-01-17 15:06:53 +08:00
yah01andGitHub 8f083e45cb enhance: enable converting segcore error to merr (#29914) 
this converts the segcore error to merr if possible

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2024-01-17 11:28:53 +08:00
yah01andGitHub c68c128e47 fix: level 0 segments not loaded (#29908) 
the recent changes move the level 0 segments list to a new proto field,
which leads to the QueryCoord can't see the level 0 segments, handle the
new changes
fix #29907

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2024-01-16 14:40:53 +08:00
yah01andGitHub cda44c09f9 enhance: adjust the GPU pool size (#29937) 
according to benchmark, the GPU pool size with 6 performs best

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2024-01-16 11:54:52 +08:00
yah01andGitHub f2e36db488 enhance: optimize the loading index performance (#29894) 
this utilizes concurrent loading

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2024-01-12 17:44:51 +08:00
yah01andGitHub 6c477ce3a7 enhance: optimize the loading strategy (#29910) 
as we have the pool size limit so we don't need to limit the concurrency
manually

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2024-01-12 14:26:50 +08:00
yah01andGitHub aba2656e68 fix: missing field data after appending scalar index to loaded segment (#29912) 
related #29843

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2024-01-12 14:04:54 +08:00
yah01andGitHub 26e900180e fix: the insert count is zero after set the pointer to nil (#29870) 
this leads to the EntitiesNum metric would be never reduced

fix: #29766

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2024-01-11 14:56:50 +08:00
yah01andGitHub 44fe06f198 enhance: skip loading duplicated index (#29715) 
this protect the loading index from failure, and speed up the loading
progress

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2024-01-11 11:52:49 +08:00
yah01andGitHub 031243fee7 feat: support mmap for marisa trie (#29613) 
this supports mmap for marisa trie index
related https://github.com/milvus-io/milvus/issues/21866

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2024-01-11 10:22:50 +08:00
yah01andGitHub e8496d4d49 enhance: filter out the not needed collections while listing (#29690) 
this improves performance while many collections exist
resolve #29631

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2024-01-10 15:18:48 +08:00
yah01andGitHub f030f31d92 enhance: make the error of parsing expression to ParameterInvalid (#29681) 
before this, the error is unexpected error

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2024-01-09 15:36:47 +08:00
yah01andGitHub a0cec4047a fix: make the entity num metric accurate (#29643) 
fix #29642

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2024-01-05 18:24:47 +08:00
yah01andGitHub 0ae90443ba enhance: fill missed info for segcore error (#29610) 
- fill missed error info
- format the error message directly

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2024-01-04 17:54:46 +08:00
yah01andGitHub 9e0163e12f enhance: use GPU pool for gpu tasks (#29678) 
- this much improve the performance for GPU index

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2024-01-04 17:50:46 +08:00
yah01andGitHub 99e0f1e65a enhance: unable to compile C++ tests (#29616) 
The tests need to call a private method, Milvus uses `#define` to
replace private with public, the hack trick works but would be broken if
the including order changed.

This uses friend to make all things work well

Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2024-01-04 13:20:46 +08:00
yah01andGitHub b8318fcd7d enhance: improve the handling for segcore error (#29471) 
- fix lost exception details in segcore
- improve the logs of handling errors from segcore

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-12-26 14:06:46 +08:00
yah01andGitHub 1d6bcd1ded enhance: speed up loading with many deletions (#29455) 
the executor always fetches the latest segment info, so we could consume
from the latest checkpoint, which could save much time while deleted
many entities

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-12-25 20:58:45 +08:00
yah01andGitHub 1b7f1d7067 enhance: mmap data corrupted after seal the column (#29422) 
this bug was introduced in recent changes

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-12-23 15:20:43 +08:00
yah01andGitHub 1997fd150c enhance: use local path as mmap file path if no mmap path provided (#28992) 
the mmap mode isn't controlled by the config anymore, so we don't
require user to set it when they enabled mmap mode.

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-12-21 18:12:45 +08:00
yah01andGitHub a0e1a1eb31 feat: support enable/disable mmap for index (#29005) 
support enable/disable mmap for index, the user could alter the index's
mode by `AlterIndex` method
related: https://github.com/milvus-io/milvus/issues/21866

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-12-21 18:07:24 +08:00
yah01andGitHub 7a2374e698 enhance: reduce the memory usage of variable length data (#29387) 
add all loading data into a buffer and then copy them into the a
fit-in-size memory

---------

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-12-21 18:02:42 +08:00
yah01andGitHub 04b2518ae7 enhance: fix the incorrect init parameter (#29357) 
as the `driver_` field is not used so this doesn't matter for now

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-12-20 20:50:43 +08:00
yah01andGitHub 61fc822207 fix: creating growing segments may introduce many threads (#29306) 
many growing segments may be created in a short time and there is no
restriction to the process, the CGO call will leave many threads

related: #29282

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-12-20 10:06:43 +08:00
yah01andGitHub 8f89e9cf75 enhance: remove all unnecessary string formatting (#29323) 
done by two regex expressions:
- `PanicInfo\((.+),[. \n]+fmt::format\(([.\s\S]+?)\)\)`
- `AssertInfo\((.+),[. \n]+fmt::format\(([.\s\S]+?)\)\)`

related: #28811

---------

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-12-20 10:04:43 +08:00
yah01andGitHub 13beb5ccc0 fix: load gets stuck probably (#29191) 
we found the load got stuck probably, and reviewed the logs.

the target observer seems not working, the reason is the taskDispatcher
removes the task in a goroutine, and modifies the task status after
committing the task into the goroutine pool, but this may happen after
the task removed, which leads to the task will never be removed

related #29086

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-12-14 18:28:38 +08:00
yah01andGitHub b8674811cf fix: data race in ProxyClientManager (#29206) 
this PR changed the ProxyClientManager to thread-safe
fix #29205

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-12-14 18:22:39 +08:00
yah01andGitHub 58dbb7872a fix: forbid balancing level zero segments (#29168) 
related #29128
Signed-off-by: yah01 <yah2er0ne@outlook.com>

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-12-14 14:30:39 +08:00
yah01andGitHub 2f0c7a6544 fix: forbid balancing level zero segments (#29130) 
we can't balance the L0 segments
related #29128

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-12-12 20:38:38 +08:00
yah01andGitHub 90e2c63d9e Fix getting incorrect CPU num (#28146) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-11-06 06:02:16 +08:00
yah01andGitHub f9c060e0d2 Treat balance task with released source segment as stale (#26453) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-08-22 17:12:22 +08:00
yah01andGitHub bed034a44b Protect DataCoord from calculating segment lines by stale log entries num (#26523) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-08-22 14:16:22 +08:00
yah01andGitHub 07f08daf1a Fix failed to load index due to lost binary (#26135) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-08-07 14:53:07 +08:00
yah01andGitHub d9074bcb06 Remove unused code (#26100) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-08-03 09:45:05 +08:00
yah01andGitHub df26b909f0 Fix panic while saw the failed task status (#26082) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-08-02 13:47:05 +08:00
yah01andGitHub 2180ef180c Record only failed task error (#26033) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-08-01 10:11:05 +08:00
yah01andGitHub 39b00b97a6 Add more logs for committed resources (#26026) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-31 19:27:03 +08:00
yah01andGitHub 8245e078c0 Add max segment size log (#26015) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-31 14:37:04 +08:00
yah01andGitHub b986e3af81 Upgrade Knowhere to 4ba9091 (#25950) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-27 13:59:03 +08:00
yah01andGitHub 9618bd9b42 Set channel capacity before consuming it (#25895) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-26 17:35:01 +08:00
yah01andGitHub 3735a097e6 Refine the memeory usage prediction (#25889) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-25 17:41:01 +08:00
yah01andGitHub 45b89cfc71 Low the predicting memory usage factor to 1 (#25884) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-25 16:37:01 +08:00
yah01andGitHub dc37b4587e Fix panic if channel not watched while getting shard leaders (#25820) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-24 14:13:02 +08:00
yah01andGitHub 361db51337 Create dir if mmap enabled for validating path (#25750) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-19 18:48:58 +08:00
yah01andGitHub 224515eaa3 Add segment dist containing condition for loading segment (#25736) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-19 15:02:58 +08:00
yah01andGitHub 227d2c8b3a Reduce loading index memory usage (#25698) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-19 14:02:57 +08:00
yah01andGitHub 079cd9dc70 Add caller skip to improve the log for segcore error (#25710) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-18 19:54:57 +08:00
yah01andGitHub 228c0df3f7 Skip record ChannelNotFound error while loading segment (#25669) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-18 11:53:18 +08:00
yah01andGitHub 948d1f1f4a Handle errors by merr for QueryCoord (#24926) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-17 14:59:34 +08:00
yah01andGitHub d216f9abda Clear collection meta after all channels/segments released (#25486) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-14 10:28:30 +08:00
yah01andGitHub 205a7c430a Fix QueryNode panic while inserting to released segment (#25495) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-12 19:48:29 +08:00
yah01andGitHub cb721781f3 Improve error message throwed from knowhere (#25473) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-11 17:26:29 +08:00
yah01andGitHub 8b06941da3 Reduce the load memory usage predict factor to 2 (#25469) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-11 13:06:28 +08:00
yah01andGitHub a1a4b649e7 Set aio pool (#25450) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-11 10:04:27 +08:00
yah01andGitHub cd29b863d0 Fix data race in session (#25354) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-06 14:52:25 +08:00
yah01andGitHub 3d2283415f Fix delegator leaked (#25347) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-06 14:18:26 +08:00
yah01andGitHub dd5f896dc8 Load batch by batch (#25212) 
This will significantly reduce the memory usage while loading
- 1x memory usage and MBs overhead for buffer (memory mode)
- only MBs overhead for buffer (mmap mode)

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-06 13:58:27 +08:00
yah01andGitHub 4c0e36d28c Stop blocking until all channels & segments released (#25328) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-06 10:04:25 +08:00
yah01andGitHub 7b5013664d Support graceful stop (#25226) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-06-30 17:28:23 +08:00
yah01andGitHub cb4b88d5cf Refactor the column type (#25147) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-06-27 19:50:45 +08:00
yah01andGitHub 7eeb78726e Fix refresh unstable (#25027) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-06-20 21:50:43 +08:00
yah01andGitHub 8bc5282eb3 Fix datanode always retries to load stats even file corrupted (#25012) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-06-20 16:40:42 +08:00
yah01andGitHub 73b7cf5df4 Fix bool unary expression sometimes returns no data (#25016) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-06-20 16:38:41 +08:00
yah01andGitHub 247f117096 Refine segcore code style (#24999) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-06-20 14:16:46 +08:00
yah01andGitHub e962a8ba31 Limit the frequency of debug logs (#25009) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-06-20 14:14:41 +08:00
yah01andGitHub a413842e38 Fix deleted data is still visible (#24849) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-06-16 17:16:41 +08:00
yah01andGitHub 9250423288 Fix QueryNode metrics (#24916) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-06-16 16:02:39 +08:00
yah01andGitHub a9dccec03a Add go payload writer (#24656) (#24762) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-06-09 13:52:39 +08:00
yah01andGitHub ebd0279d3f Check error by Error() and NoError() for better report message (#24736) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-06-08 15:36:36 +08:00
yah01andGitHub fa69c71d35 Fix data race in MetaCache (#24739) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-06-08 13:58:35 +08:00
yah01andGitHub f40e22c78c Add array_at method to get array from JSON (#24671) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-06-06 14:10:36 +08:00
yah01andGitHub fb66924c90 Set metric type while init the collection (#24665) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-06-06 13:24:36 +08:00
yah01andGitHub 848ef7418b Fix the refresh may be notified finished early (#24438) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-05-26 18:43:26 +08:00
yah01andGitHub 71a7fef5c5 Consume threads of the number of nq (#24410) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-05-26 17:33:27 +08:00
yah01andGitHub 10c779d3d3 Upgrade image go version to 1.20.4 (#24393) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-05-25 14:51:26 +08:00
yah01andGitHub 7721cc73f0 Perf: Avoid copying string/json when filter on them (#24381) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-05-25 13:13:26 +08:00
yah01andGitHub abd250da09 Fix QueryNode is not able to recovery (#24300) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-05-23 17:17:26 +08:00
yah01andGitHub 6da8853756 Fix the JSON parser always parse all documents (#24290) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-05-22 15:49:27 +08:00
yah01andGitHub ceda0ed598 Optimize the performance of filter by JSON field (#24268) 
- Construct JSON pointer only once
- Avoid copying nested path for each row

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-05-22 00:47:25 +08:00
yah01andGitHub 99cc24b283 Fix compaction doesn't support JSON data (#24151) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-05-17 19:09:22 +08:00
yah01andGitHub 3ea4a39078 Fix insert node doesn't handle JSON data while merging (#24126) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-05-16 16:09:23 +08:00
yah01andGitHub 7d0c47dd65 Check the length of JSON (#24035) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-05-12 11:33:20 +08:00
yah01andGitHub 400364483d Improve predict the resource usage (#23968) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-05-11 15:33:24 +08:00
yah01andGitHub 67cf23d050 Fix panic while balancing releasing collection (#24003) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-05-11 12:47:20 +08:00
yah01andGitHub 62eea5286f Support to filter with json expr (#23739) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-04-30 20:36:39 +08:00
yah01andGitHub eab94489ba Refine the merge algorithm (#23767) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-04-27 18:26:35 +08:00
yah01andGitHub 0d3fcd5d16 Add method to access nested field at once (#23720) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-04-26 14:16:34 +08:00
yah01andGitHub 60fdd7e4f4 Introduce simdjson (#23644) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-04-26 10:30:34 +08:00