furyhawk/ik_llama.cpp
1
0
Fork 0
mirror of https://github.com/ikawrakow/ik_llama.cpp.git synced 2026-07-21 10:15:35 +00:00
Code Issues Packages Projects Releases Wiki Activity

Add --prefetch-experts to stream mmap'd MoE experts into page cache (#2101)

Browse Source 
* Add --prefetch-experts to stream mmap'd MoE experts into page cache

* Drop fds, fault experts in with MADV_POPULATE_READ instead of pread

* Remove stale note about pread workers

* Move MoE prefetch behind ggml_backend_prefetch_* wrappers

* Cleanup stale comments

* Add --prefetch-experts-threads, drop GGML_MOE_PREFETCH_THREADS env var
This commit is contained in:
dmaivel
2026-07-11 10:43:42 +03:00
committed by GitHub
parent 70ea15226f
commit 6a909f4ff6
13 changed files with 709 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
include/llama.h
+2
View File
@@ -497,6 +497,8 @@ extern "C" {
int min_experts;
float thresh_experts;
bool only_active_experts;
bool prefetch_experts; // if true, stream mmap'd MoE expert weights into the page cache (Linux only)
int prefetch_experts_threads; // number of expert prefetch workers (<=0 = auto)
bool k_cache_hadamard; // if true, apply Hadamard transfrom to K-cache
bool v_cache_hadamard; // if true, apply Hadamard transfrom to V-cache (needs FA)
bool split_mode_graph_scheduling; // if true, force split mode graph scheduling