Fused indexer top_k (CPU only) (#2098)

* Indexer topk op - CPU only

We do save memory, but it is somehow much slower than what we have on
main.

* Make it a command line option
This commit is contained in:
Kawrakow
2026-07-09 18:04:41 +03:00
committed by GitHub
parent c2b58f88dd
commit 6d30fa2fe6
10 changed files with 288 additions and 3 deletions
+1
View File
@@ -492,6 +492,7 @@ extern "C" {
bool rope_cache; // whether to use RoPE cache [EXPERIMENTAL]
bool graph_reuse; // whether to reuse graphs when possible [EXPERIMENTAL]
bool dsa; // enable GLM DSA sparse attention (off by default) [EXPERIMENTAL]
bool fused_idx_topk; // enable the fused indexer topk op (off by default) [EXPERIMENTAL]
int dsa_top_k; // DSA top-k override (<0 => model's configured indexer_top_k) [EXPERIMENTAL]
int min_experts;
float thresh_experts;