furyhawk/ik_llama.cpp
1
0
Fork 0
mirror of https://github.com/ikawrakow/ik_llama.cpp.git synced 2026-07-21 02:05:35 +00:00
Code Issues Packages Projects Releases Wiki Activity

CUDA: route P100 (sm_60) decode flash-attention to fp32 vec kernel (#2144)

Browse Source 
On P100 (GP100, sm_60) the fp16 vec kernel used for decode (batch<=8)
accumulates the online-softmax denominator and the P*V product in fp16,
flipping ~3-4% of decode top-1 tokens vs an all-fp32 reference
(llama.cpp#25593). Decode is memory-bandwidth-bound on P100, so routing
sm_60 decode to the in-tree vec_f32 kernel is free (tg128 ~identical).

Gated on cc == CC_PASCAL && Q->ne[1] <= 8 (decode only) inside the
!fp16_mma_available block, so the prefill tile_f16 path, the D=256 prefill
vec path, and fast_fp16_available() are untouched, and the
is_pascal_mla_absorbed_decode early-return (MLA) is unaffected.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
mb8565
2026-07-17 17:40:06 +03:00
committed by GitHub
co-authored by Claude Opus 4.8
parent a0a10da5a9
commit 7174a124ca
1 changed files with 11 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ggml/src/ggml-cuda/fattn.cu
+11 -1
View File
@@ -96,7 +96,14 @@ void ggml_cuda_flash_attn_ext(ggml_backend_cuda_context & ctx, ggml_tensor * dst
}
if (precision == GGML_PREC_DEFAULT) {
if (Q->ne[1] <= 8 || Q->ne[0] == 256) {
ggml_cuda_flash_attn_ext_vec_f16(ctx, dst);
// P100 (sm_60/CC_PASCAL): the fp16 vec kernel accumulates the online-softmax
// denominator and the P*V product in fp16, flipping ~3-4% of decode top-1 tokens
// vs fp32 (llama.cpp#25593). Decode is bandwidth-bound on P100 so vec_f32 is free.
if (cc == CC_PASCAL && Q->ne[1] <= 8) { // decode only (batch<=8); D=256 prefill stays vec_f16
ggml_cuda_flash_attn_ext_vec_f32(ctx, dst);
} else {
ggml_cuda_flash_attn_ext_vec_f16(ctx, dst);
}
} else {
ggml_cuda_flash_attn_ext_tile_f16(ctx, dst);
}
@@ -194,6 +201,9 @@ bool ggml_cuda_fattn_is_supported(ggml_backend_cuda_context & ctx, const ggml_te
}
if (precision == GGML_PREC_DEFAULT) {
if (Q->ne[1] <= 8 || Q->ne[0] == 256) {
if (cc == CC_PASCAL && Q->ne[1] <= 8) { // decode only (batch<=8); D=256 prefill stays vec_f16
return ggml_cuda_fattn_vec_f32_is_supported(ctx, dst);
}
return ggml_cuda_fattn_vec_f16_is_supported(ctx, dst);
} else {
return ggml_cuda_fattn_tile_f16_is_supported(ctx, dst);