* WIP: indexer_topk on CUDA
* Forgot these
* WIP
* WIP
* This seems to work
* Minor
* Fix bug. Fix suggested by @sayap using GLM-5.2
* GLM-DSA: much better PP long context performance (CUDA)
* DSA: Better way to build the attention mask
* WIP: indexer_topk on CUDA
* Forgot these
* WIP
* WIP
* This seems to work
* Minor
* Fix bug. Fix suggested by @sayap using GLM-5.2
* GLM-DSA: much better PP long context performance (CUDA)
* GLM-DSA: improve TG performance even more
* Fix crash when using MTP
* GLM-DSA: much better PP performance (CPU-only)
* GLM-DSA: allow for quantized indexer cache
* Split mode graph for GLM4MoE MTP in split_mode_tensor_parallel
Analogous to what was done for Qwen35 dense MTP in PR 2027:
- ctx_for_layer_split(): return buft_matrix for GLM4_MOE MTP tail
layers instead of buft, so split tensor preparation uses the
correct split context.
- create_glm4_moe_tensors(): remove the ctx_split = ctx_layer
override for MTP tail layers. ctx_for_layer_split(i) now
returns buft_matrix for GLM4_MOE MTP tails, so regular layer
tensors (attn, ffn) are created in the split context. NextN
tensors (eh_proj, enorm, hnorm, shared_head_head,
shared_head_norm) stay monolithic via ctx_for_layer().
- create_tensors(): add LLM_ARCH_GLM4_MOE to the MTP tail
layer splitting exclusion so split processing visits them.
- build_glm4_moe_mtp(): pass inp_out_ids to build_std_attention
instead of post-processing with ggml_get_rows. Use build_output
for the output projection to properly handle split mode.
- build_output(): add LLM_ARCH_GLM4_MOE to the is_qwen_mtp
check to ensure MTP output is properly materialized.
* Skip loading shared_head_head for GLM4MoE MTP
Add TENSOR_SKIP flag to shared_head_head so it is never loaded, even
when present in the GGUF file. The graph code already falls back to
model.output when shared_head_head is nullptr (line 375-376), which
frees ~306 MiB on CUDA0 for models that include this tensor (e.g.,
GLM-Steam-106B). The 355B GLM-4.6 model does not have this tensor and
already uses the same fallback path.
* cuda-graph: GLM4_MOE - Don't load layer.nextn.embed_tokens
---------
Co-authored-by: Nexesenex <124105151+Nexesenex@users.noreply.github.com>
* WIP: Split mode graph for Gemma4 assistant
Something is not right - acceptance drops to nearly zero.
* Per model CUDA contexts
Still not working!?
* This works
The issue was that I was not correctly calculating the number
of KV heads for the split KV cache.
* Compiler warnings
* It is better to use llama_context pointers as keys
* Split mode graph for dense Qwen35 MTP
* WIP: Split mode graph for Gemma4 assistant
Something is not right - acceptance drops to nearly zero.
* Per model CUDA contexts
Still not working!?
* This works
The issue was that I was not correctly calculating the number
of KV heads for the split KV cache.
* Compiler warnings
* It is better to use llama_context pointers as keys
* Use hidden state from prev token from qwen mtp
* Fix Qwen35 MTP warmup
* Cleanup + remove unnecessary crippling performance by not using accept to sample draft token
* Provide API to gtet the model arch string
---------
Co-authored-by: SamuelOliveirads <samueloliveira32df@gmail.com>
* wip: port MTP architecture
Ports the Multi-Token Prediction (MTP) architecture to the older `llama.cpp` codebase used by `ikllama`.
Changes include:
- Updating `llama_batch` to support `mtp_params`.
- Modifying `llama_decode_internal` (and `encode`) to handle MTP operations (Warmup, Update, Draft).
- Adding public APIs for MTP state management (`llama_set_draft_input_hidden_state`).
- Adapting the embedding extraction logic to skip MTP update passes.
* Refactors `server_slot` to support generic speculative decoding (MTP or Draft Model).
* core: enable hybrid outputs (logits + embeddings) for MTP support
* fix(mtp): correct KV-cache slot finding for updates
* fix(mtp): persist hidden states to prevent context corruption during drafting
* refactor(mtp): clean unused code
* fix(mtp): update server to new functions name
* fix(mtp): fix graph and save hidden state
* mtp: refactor integration, context params and kv cache search
* mtp: fix hidden state extraction and speculative acceptance flow
* server: fix MTP warmup for long prompts and reset token buffer
* llama: refactor MTP operation state to context parameters
* server: fix n_past calculation in MTP acceptance
* llama: fix mtp enable flags
* speculative: refactor MTP to use common_speculative interface
* context: remove unused signatures
* clip: fix deprecated enum-enum conversion warning
* common: fix format string crash in help message
* context: fix mtp activation logic
* llamat: always use the extracted embedding
* llama: get all embeddings to kv cache
* llama: revert logit to not run mtp for not supported arch
* llama: allocate all the n_outputs for MTP
* wip
* server-context: get only the last embedding for hidden state
* ggml-backend: fix array of bounds in debug build
* server-context: run mt kv update to each prompt batch
* revert segmentation fault fixes
* glm-mtp(feat): optimize graph embedding and recursive drafting
* glm5-mtp(feat): add glm 5 mtp logic
* glm-mtp: standardize the MTP graph
* glm 5 mtp: apply post-layer cvec
* glm 5 mtp: mark head as mandatory
* get normed embeddings for glm 5
* Fix GLM5 MTP
* GLM5 MTP: just reuse the layer attention implementation
* Make MTP work with split mode graph
---------
Co-authored-by: samuel <samueloliveira32df@gmail.com>
* Disable K Hadamard transform if K-head size is not a power of 2
* Allow Hadamard transform for head sizes that are not power of 2
* Give more details why Hadamard is not possible
* Arghh
