furyhawk/ik_llama.cpp
1
0
Fork 0
mirror of https://github.com/ikawrakow/ik_llama.cpp.git synced 2026-07-20 09:45:35 +00:00
Code Issues Packages Projects Releases Wiki Activity

Fix uninitialized MROPE/IMROPE position sections in legacy-batch decode path (#2149)

Browse Source 
For IMROPE models (Qwen3.5/QWEN35MOE) the RoPE op reads 4 position sections per
token, but the legacy null-pos decode fallback (llama_batch_get_one path, used by
llama-perplexity/llama-cli/llama-eval-callback) sized the position vector to n_tokens.
llama_set_inputs then copies n_tokens*4 out of that n_tokens-sized vector, an
out-of-bounds read that feeds garbage into 3 of the 4 rope sections, giving wrong and
run-to-run non-deterministic RoPE. Build the sections like the explicit-pos path (text
t,t,t,0). Standard-rope (NEOX) unchanged; VL image input uses the explicit-pos builder.

Qwen3.5-4B self-vs-self same-top: 91.3% -> 100.0% (CUDA), single-thread CPU likewise.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
mb8565
2026-07-18 09:07:50 +03:00
committed by GitHub
co-authored by Claude Opus 4.8
parent 65891dcdb3
commit 27d6291222
1 changed files with 15 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/llama.cpp
+15 -2
View File
@@ -5953,9 +5953,22 @@ static int llama_decode_internal(
// helpers for smoother batch API transition
// after deprecating the llama_eval calls, these will be removed
if (u_batch.pos == nullptr) {
pos.resize(n_tokens);
// MROPE/IMROPE models (e.g. Qwen3.5) read 4 position sections per token; the RoPE op indexes
// pos[i], pos[i+n], pos[i+2n], pos[i+3n]. Sizing this legacy fallback to only n_tokens (as for
// standard rope) made llama_set_inputs copy n_tokens*4 out of a n_tokens-sized vector -> an
// out-of-bounds read (UB) that fed garbage into the extra rope sections -> wrong, run-to-run
// non-deterministic RoPE. Build the 4 sections like the explicit-pos path above (text: t,t,t,0).
const int rope_pt = (hparams.rope_type == LLAMA_ROPE_TYPE_MROPE ||
hparams.rope_type == LLAMA_ROPE_TYPE_IMROPE) ? 4 : 1;
pos.resize((size_t) n_tokens * rope_pt);
for (uint32_t i = 0; i < n_tokens; i++) {
pos[i] = u_batch.all_pos_0 + i*u_batch.all_pos_1;
const int32_t p = u_batch.all_pos_0 + i*u_batch.all_pos_1;
pos[i] = p;
if (rope_pt == 4) {
pos[ n_tokens + i] = p;
pos[2 * n_tokens + i] = p;
pos[3 * n_tokens + i] = 0;
}
}
u_batch.pos = pos.data();