furyhawk/ik_llama.cpp
1
0
Fork 0
mirror of https://github.com/ikawrakow/ik_llama.cpp.git synced 2026-07-21 02:05:35 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'origin/main' into ik/glm_dsa_cuda

Browse Source
This commit is contained in:
Kawrakow
2026-07-12 05:10:28 +00:00
parent fdada9003e e913807605
commit b6cdcced6f
64 changed files with 4978 additions and 311 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+4 -1
View File
@@ -89,7 +89,7 @@ That's all! Open [http://127.0.0.1:8080](http://127.0.0.1:8080) in Browser start
### Model Support
LlaMA-3-Nemotron [PR 377](https://github.com/ikawrakow/ik_llama.cpp/pull/377), Qwen3 [PR 355](https://github.com/ikawrakow/ik_llama.cpp/pull/355), GLM-4 [PR 344](https://github.com/ikawrakow/ik_llama.cpp/pull/344), Command-A [PR 341](https://github.com/ikawrakow/ik_llama.cpp/pull/341), bitnet-b1.58-2B-4T [PR 337](https://github.com/ikawrakow/ik_llama.cpp/pull/337), LLaMA-4 [PR 321](https://github.com/ikawrakow/ik_llama.cpp/pull/321), Gemma3 [PR 276](https://github.com/ikawrakow/ik_llama.cpp/pull/276), DeepSeek-V3 [PR 176](https://github.com/ikawrakow/ik_llama.cpp/pull/176), Kimi-2 [PR 609](https://github.com/ikawrakow/ik_llama.cpp/pull/609), dots.llm1 [PR 573](https://github.com/ikawrakow/ik_llama.cpp/pull/573), Hunyuan [PR 565](https://github.com/ikawrakow/ik_llama.cpp/pull/565), GLM-4.5 [PR 668](https://github.com/ikawrakow/ik_llama.cpp/pull/668) (4.5/4.6/4.7/AIR), Ernie 4.5 MOE and 0.3B [PR 759](https://github.com/ikawrakow/ik_llama.cpp/pull/759), grok-2 [PR 782](https://github.com/ikawrakow/ik_llama.cpp/pull/782), Ling/Ring (Bailing-MoE2) [PR 833](https://github.com/ikawrakow/ik_llama.cpp/pull/833), Qwen3-VL [PR 883](https://github.com/ikawrakow/ik_llama.cpp/pull/883), SmolLM3 [PR 934](https://github.com/ikawrakow/ik_llama.cpp/pull/934), GigaChat3 [PR 995](https://github.com/ikawrakow/ik_llama.cpp/pull/995), ministral3 [PR 1030](https://github.com/ikawrakow/ik_llama.cpp/pull/1030), Mimo-V2-Flash [PR 1096](https://github.com/ikawrakow/ik_llama.cpp/pull/1096), GLM-4.7-Flash [PR 1168](https://github.com/ikawrakow/ik_llama.cpp/pull/1168), Seed-OSS [PR 1218](https://github.com/ikawrakow/ik_llama.cpp/pull/1218), Step-3.5-Flash [PR 1231](https://github.com/ikawrakow/ik_llama.cpp/pull/1231), GLM-5 [PR 1268](https://github.com/ikawrakow/ik_llama.cpp/pull/1268), Qwen3-Next [PR 1266](https://github.com/ikawrakow/ik_llama.cpp/pull/1266), Qwen3.5-MoE [PR 1288](https://github.com/ikawrakow/ik_llama.cpp/pull/1288) and dense Qwen-3.5 [1326](https://github.com/ikawrakow/ik_llama.cpp/pull/1326), Mistral 4 [PR 1450](https://github.com/ikawrakow/ik_llama.cpp/pull/1450), Bonsai 1-bit [PR 1570](https://github.com/ikawrakow/ik_llama.cpp/pull/1570), Gemma4 [PR 1581](https://github.com/ikawrakow/ik_llama.cpp/pull/1581), Mimo-2.5 [PR 1723](https://github.com/ikawrakow/ik_llama.cpp/pull/1723), JetBrains Mellum2 [PR 1919](https://github.com/ikawrakow/ik_llama.cpp/pull/1919), Poolside Laguna XS.2 [PR 1911](https://github.com/ikawrakow/ik_llama.cpp/pull/1911), Cohere2-MoE North Mini Code [PR 1945](https://github.com/ikawrakow/ik_llama.cpp/pull/1945)
LlaMA-3-Nemotron [PR 377](https://github.com/ikawrakow/ik_llama.cpp/pull/377), Qwen3 [PR 355](https://github.com/ikawrakow/ik_llama.cpp/pull/355), GLM-4 [PR 344](https://github.com/ikawrakow/ik_llama.cpp/pull/344), Command-A [PR 341](https://github.com/ikawrakow/ik_llama.cpp/pull/341), bitnet-b1.58-2B-4T [PR 337](https://github.com/ikawrakow/ik_llama.cpp/pull/337), LLaMA-4 [PR 321](https://github.com/ikawrakow/ik_llama.cpp/pull/321), Gemma3 [PR 276](https://github.com/ikawrakow/ik_llama.cpp/pull/276), DeepSeek-V3 [PR 176](https://github.com/ikawrakow/ik_llama.cpp/pull/176), Kimi-2 [PR 609](https://github.com/ikawrakow/ik_llama.cpp/pull/609), dots.llm1 [PR 573](https://github.com/ikawrakow/ik_llama.cpp/pull/573), Hunyuan [PR 565](https://github.com/ikawrakow/ik_llama.cpp/pull/565), GLM-4.5 [PR 668](https://github.com/ikawrakow/ik_llama.cpp/pull/668) (4.5/4.6/4.7/AIR), Ernie 4.5 MOE and 0.3B [PR 759](https://github.com/ikawrakow/ik_llama.cpp/pull/759), grok-2 [PR 782](https://github.com/ikawrakow/ik_llama.cpp/pull/782), Ling/Ring (Bailing-MoE2) [PR 833](https://github.com/ikawrakow/ik_llama.cpp/pull/833), Qwen3-VL [PR 883](https://github.com/ikawrakow/ik_llama.cpp/pull/883), SmolLM3 [PR 934](https://github.com/ikawrakow/ik_llama.cpp/pull/934), GigaChat3 [PR 995](https://github.com/ikawrakow/ik_llama.cpp/pull/995), ministral3 [PR 1030](https://github.com/ikawrakow/ik_llama.cpp/pull/1030), Mimo-V2-Flash [PR 1096](https://github.com/ikawrakow/ik_llama.cpp/pull/1096), GLM-4.7-Flash [PR 1168](https://github.com/ikawrakow/ik_llama.cpp/pull/1168), Seed-OSS [PR 1218](https://github.com/ikawrakow/ik_llama.cpp/pull/1218), Step-3.5-Flash [PR 1231](https://github.com/ikawrakow/ik_llama.cpp/pull/1231), GLM-5 [PR 1268](https://github.com/ikawrakow/ik_llama.cpp/pull/1268), Qwen3-Next [PR 1266](https://github.com/ikawrakow/ik_llama.cpp/pull/1266), Qwen3.5-MoE [PR 1288](https://github.com/ikawrakow/ik_llama.cpp/pull/1288) and dense Qwen-3.5 [1326](https://github.com/ikawrakow/ik_llama.cpp/pull/1326), Mistral 4 [PR 1450](https://github.com/ikawrakow/ik_llama.cpp/pull/1450), Bonsai 1-bit [PR 1570](https://github.com/ikawrakow/ik_llama.cpp/pull/1570), Gemma4 [PR 1581](https://github.com/ikawrakow/ik_llama.cpp/pull/1581) including assistant, Mimo-2.5 [PR 1723](https://github.com/ikawrakow/ik_llama.cpp/pull/1723), JetBrains Mellum2 [PR 1919](https://github.com/ikawrakow/ik_llama.cpp/pull/1919), Poolside Laguna XS.2 [PR 1911](https://github.com/ikawrakow/ik_llama.cpp/pull/1911), Cohere2-MoE North Mini Code [PR 1945](https://github.com/ikawrakow/ik_llama.cpp/pull/1945), MiniMax-M3 [PR 1963](https://github.com/ikawrakow/ik_llama.cpp/pull/1963), Laguna M.1 [PR 2003](https://github.com/ikawrakow/ik_llama.cpp/pull/2003)
### Quantization
@@ -148,12 +148,15 @@ Implemented for Zen4, AVX2, ARM_NEON, Metal, CUDA [PR 682](https://github.com/ik
* Checkpoints for recurrent models [PR 1310](https://github.com/ikawrakow/ik_llama.cpp/pull/1310) [PR 1398](https://github.com/ikawrakow/ik_llama.cpp/pull/1398)
* MTP decoding support for popular models like GLM-4.x MoE [1270](https://github.com/ikawrakow/ik_llama.cpp/pull/1270), Qwen 3.5/3.6 [1698](https://github.com/ikawrakow/ik_llama.cpp/pull/1698) [1745](https://github.com/ikawrakow/ik_llama.cpp/pull/1745), Gemma 4 [1744](https://github.com/ikawrakow/ik_llama.cpp/pull/1744), GLM 5 [1890](https://github.com/ikawrakow/ik_llama.cpp/pull/1890)
* Self speculative decoding, ngram [PR 1261](https://github.com/ikawrakow/ik_llama.cpp/pull/1261), suffix [PR 1646](https://github.com/ikawrakow/ik_llama.cpp/pull/1646)
* DFlash initial support [PR 1970](https://github.com/ikawrakow/ik_llama.cpp/pull/1970)
* GLM-DSA architecture indexer cache
* String ban function for all completions [PR 1185](https://github.com/ikawrakow/ik_llama.cpp/pull/1185) [PR 1243](https://github.com/ikawrakow/ik_llama.cpp/pull/1243)
* Expiring Logit Bias [PR 1731](https://github.com/ikawrakow/ik_llama.cpp/pull/1731)
* OpenAI `/v1/responses` API endpoint [PR 1184](https://github.com/ikawrakow/ik_llama.cpp/pull/1184)
* Function call support [PR 628](https://github.com/ikawrakow/ik_llama.cpp/pull/628)
* jinja template support [PR 677](https://github.com/ikawrakow/ik_llama.cpp/pull/677)
* Webui: New Features for Conversations, Settings, and Chat Messages [PR 618](https://github.com/ikawrakow/ik_llama.cpp/pull/618), MCP [PR 1904](https://github.com/ikawrakow/ik_llama.cpp/pull/1904)
* On-demand tensor reload [PR 1989](https://github.com/ikawrakow/ik_llama.cpp/pull/1989)
* Dynamic control vector management endpoints [PR 1223](https://github.com/ikawrakow/ik_llama.cpp/pull/1223)
* Legacy quants conversion schemes in `convert_hf_to_gguf.py` [PR 449](https://github.com/ikawrakow/ik_llama.cpp/pull/449), `Q6_0` in [PR 483](https://github.com/ikawrakow/ik_llama.cpp/pull/483)
* Adaptive-P Sampler [PR 1100](https://github.com/ikawrakow/ik_llama.cpp/pull/1100) implemented as designed by it's author; supported on Webui
common/chat-diff-analyzer.cpp
+21
View File
@@ -130,6 +130,27 @@ static std::vector<std::function<void(const common_chat_template & tmpl, autopar
analysis.tools.function.close = "```";
LOG_DBG(ANSI_ORANGE "[Patch: DeepSeek-R1-Distill-Qwen]\n" ANSI_RESET);
}
},
// openPangu-2.0 - prefills <think> in the generation prompt (like the Laguna case above),
// so the generated reasoning starts immediately and is delimited only by </think>. The
// <think> is concatenated into a larger literal ('...assistant\n<think>'), so the
// standalone-literal reasoning detector does not pick it up; set the markers explicitly.
// Tool calls (<|tool_call_start|>[{...}]<|tool_call_end|>) are already handled by the auto-parser.
[](const common_chat_template & tmpl, autoparser & analysis) -> void {
if (tmpl.src.find("<|pangu_text_start|>") != std::string::npos) {
// Force-set (do not gate on mode==NONE): the differential detector sees the
// assistant-history form <think>reasoning</think> and sets start="<think>", but at
// generation time <think> is prompt-prefilled, so the output is delimited only by
// </think> (start=""). Same shape as the Laguna patch above.
analysis.reasoning.mode = reasoning_mode::TAG_BASED;
analysis.reasoning.start = "";
analysis.reasoning.end = "</think>";
if (std::find(analysis.preserved_tokens.begin(), analysis.preserved_tokens.end(), "</think>") ==
analysis.preserved_tokens.end()) {
analysis.preserved_tokens.push_back("</think>");
}
LOG_DBG(ANSI_ORANGE "[Patch: openPangu-2.0 thinking template]\n" ANSI_RESET);
}
}
});
common/chat-peg-parser.cpp
+52
View File
@@ -75,6 +75,29 @@ static std::string escape_json_string_inner(const std::string & s) {
return escaped;
}
static bool truncate_json_strings_at_marker(ordered_json & value, const std::string & marker) {
bool changed = false;
if (value.is_string()) {
std::string s = value.get<std::string>();
size_t p = s.find(marker);
if (p != std::string::npos) {
value = s.substr(0, p);
changed = true;
}
} else if (value.is_array()) {
for (auto & item : value) {
changed = truncate_json_strings_at_marker(item, marker) || changed;
}
} else if (value.is_object()) {
for (auto & item : value.items()) {
changed = truncate_json_strings_at_marker(item.value(), marker) || changed;
}
}
return changed;
}
// Convert Python-style single-quoted strings to JSON double-quoted strings
// Only converts outer string delimiters, properly handling escape sequences:
// - {'key': 'value'} -> {"key": "value"}
@@ -249,6 +272,35 @@ void common_chat_peg_mapper::from_ast(const common_peg_ast_arena & arena,
}
}
void common_chat_peg_minimax_m3_mapper::from_ast(const common_peg_ast_arena & arena,
const common_peg_parse_result & parse_result) {
common_chat_peg_mapper::from_ast(arena, parse_result);
// MiniMax-M3 tool calls are emitted from the thinking/action phase.
if (!result.tool_calls.empty() && !result.content.empty()) {
result.reasoning_content += result.content;
result.content.clear();
}
// MiniMax-M3 sometimes starts emitting a new namespaced tool marker before
// closing a string argument. Keep that marker out of executable tool text.
static const std::string ns_marker = "]<]minimax[>[";
for (auto & tool_call : result.tool_calls) {
if (tool_call.arguments.find(ns_marker) == std::string::npos) {
continue;
}
try {
ordered_json args = ordered_json::parse(tool_call.arguments);
if (truncate_json_strings_at_marker(args, ns_marker)) {
tool_call.arguments = args.dump();
}
} catch (const ordered_json::exception &) {
// Leave malformed JSON to existing downstream validation.
}
}
}
void common_chat_peg_mapper::map(const common_peg_ast_node & node) {
// Handle reasoning/content tags
bool is_reasoning = node.tag == common_chat_peg_builder::REASONING;
common/chat-peg-parser.h
+6 -1
View File
@@ -40,6 +40,12 @@ class common_chat_peg_gemma4_mapper : public common_chat_peg_mapper {
void visit(const common_peg_ast_arena & arena, common_peg_ast_id id);
};
class common_chat_peg_minimax_m3_mapper : public common_chat_peg_mapper {
public:
common_chat_peg_minimax_m3_mapper(common_chat_msg & msg) : common_chat_peg_mapper(msg) {}
virtual void from_ast(const common_peg_ast_arena & arena, const common_peg_parse_result & result);
};
struct content_structure;
struct tool_call_structure;
@@ -195,4 +201,3 @@ struct tagged_peg_parser {
tagged_peg_parser build_tagged_peg_parser(
const std::function<common_peg_parser(common_peg_parser_builder & builder)> & fn);
common/chat.cpp
+69 -12
View File
@@ -162,12 +162,31 @@ std::vector<common_chat_msg_diff> common_chat_msg_diff::compute_diffs(const comm
// TODO: these can become expensive for long messages - how to optimize?
if (msg_prv.reasoning_content != msg_new.reasoning_content) {
auto & diff = diffs.emplace_back();
diff.reasoning_content_delta = string_diff(msg_prv.reasoning_content, msg_new.reasoning_content);
std::string reasoning_content_delta = string_diff(msg_prv.reasoning_content, msg_new.reasoning_content);
// Partial parsing can reclassify bytes that were previously streamed as
// content into reasoning_content once more context arrives. Streaming
// deltas are append-only, so remove that already-streamed content prefix
// from the reasoning delta instead of emitting the same bytes twice.
if (!msg_prv.content.empty() &&
string_starts_with(reasoning_content_delta, msg_prv.content) &&
string_starts_with(msg_prv.content, msg_new.content)) {
reasoning_content_delta = reasoning_content_delta.substr(msg_prv.content.size());
}
// string_diff() can return an empty delta for shrink/reclassification
// cases. Do not emit empty append-only chunks.
if (!reasoning_content_delta.empty()) {
auto & diff = diffs.emplace_back();
diff.reasoning_content_delta = std::move(reasoning_content_delta);
}
}
if (msg_prv.content != msg_new.content) {
auto & diff = diffs.emplace_back();
diff.content_delta = string_diff(msg_prv.content, msg_new.content);
std::string content_delta = string_diff(msg_prv.content, msg_new.content);
// Same empty-delta rule as above, without the reasoning/content overlap
// trim: there may be no new content bytes to append.
if (!content_delta.empty()) {
auto & diff = diffs.emplace_back();
diff.content_delta = std::move(content_delta);
}
}
if (msg_new.tool_calls.size() < msg_prv.tool_calls.size()) {
@@ -680,6 +699,8 @@ const char * common_chat_format_name(common_chat_format format) {
return "peg-native";
case COMMON_CHAT_FORMAT_PEG_GEMMA4:
return "peg-gemma4";
case COMMON_CHAT_FORMAT_PEG_MINIMAX_M3:
return "peg-minimax-m3";
default:
throw std::runtime_error("Unknown chat format");
}
@@ -760,6 +781,13 @@ static std::string common_chat_template_direct_apply_impl(
{"eos_token", tmpl.eos_token()},
{"enable_thinking", inputs.enable_thinking},
};
// openPangu's chat template gates reasoning on a `thinking` variable rather than the
// ecosystem-standard `enable_thinking`, so the normal toggle never reaches it. Bridge the
// standard control to it here so reasoning works on and off through `enable_thinking`. An
// explicit `thinking` chat_template_kwarg still wins (merged via extra_context below).
if (tmpl.source().find("<|pangu_text_start|>") != std::string::npos) {
inp["thinking"] = inputs.enable_thinking;
}
if (tools_override.has_value() || !inputs.tools.empty()) {
inp["tools"] = tools_override.has_value() ? *tools_override : inputs.tools;
}
@@ -1938,7 +1966,7 @@ static common_chat_params common_chat_params_init_minimax_m3(const common_chat_t
common_chat_params data;
data.prompt = common_chat_template_direct_apply_impl(tmpl, inputs);
data.format = COMMON_CHAT_FORMAT_PEG_NATIVE;
data.format = COMMON_CHAT_FORMAT_PEG_MINIMAX_M3;
data.supports_thinking = true;
data.thinking_start_tag = "<mm:think>";
data.thinking_end_tag = "</mm:think>";
@@ -1967,11 +1995,35 @@ static common_chat_params common_chat_params_init_minimax_m3(const common_chat_t
auto generation_prompt = p.prefix(inputs.generation_prompt, THINK_START);
auto end = p.end();
auto thinking_body = [&]() {
return has_tools && inputs.tool_choice != COMMON_CHAT_TOOL_CHOICE_NONE
? p.until_one_of({ THINK_END, FC_START })
: p.until(THINK_END);
};
auto reasoning = p.eps();
if (extract_reasoning && inputs.enable_thinking) {
reasoning = p.optional(p.optional(p.literal(THINK_START)) + p.reasoning(p.until(THINK_END)) + THINK_END);
} else if (extract_reasoning) {
reasoning = p.optional(p.optional(p.literal(THINK_START)) + p.until(THINK_END) + p.literal(THINK_END));
if (extract_reasoning) {
// The generation prompt usually opens <mm:think>, but MiniMax-M3 may
// also emit an unopened block ending in </mm:think>. Keep both forms
// quarantined as reasoning instead of leaking them into visible text.
auto opened = p.literal(THINK_START) +
p.reasoning(thinking_body()) +
p.optional(p.literal(THINK_END));
auto unopened = p.reasoning(thinking_body()) + p.literal(THINK_END);
reasoning = p.optional(p.choice({ opened, unopened }));
if (inputs.enable_thinking) {
// During streaming, the closing tag may not have arrived yet; if
// a tool call starts first, stop reasoning at the tool marker.
auto partial = p.reasoning(thinking_body());
reasoning = p.optional(p.choice({ opened, unopened, partial }));
}
} else if (inputs.enable_thinking) {
auto opened = p.content(p.literal(THINK_START) +
thinking_body() +
p.optional(p.literal(THINK_END)));
auto unopened = p.content(thinking_body()) + p.literal(THINK_END);
reasoning = p.optional(p.choice({ opened, unopened }));
}
if (has_response_format) {
@@ -2056,16 +2108,17 @@ static common_chat_params common_chat_params_init_minimax_m3(const common_chat_t
auto require_tools = inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED;
common_peg_parser tool_calls = p.eps();
common_peg_parser tool_call_envelope = p.eps();
if (inputs.parallel_tool_calls) {
tool_calls = p.trigger_rule("tool-call",
tool_call_envelope = p.trigger_rule("tool-call",
p.literal(FC_START) + p.space() + tool_choice +
p.zero_or_more(p.space() + tool_choice) + p.space() + p.literal(FC_END));
} else {
tool_calls = p.trigger_rule("tool-call",
tool_call_envelope = p.trigger_rule("tool-call",
p.literal(FC_START) + p.space() + tool_choice + p.space() + p.literal(FC_END));
}
auto tool_calls = tool_call_envelope;
if (!require_tools) {
tool_calls = p.optional(tool_calls);
}
@@ -2780,6 +2833,8 @@ common_chat_msg common_chat_peg_parse(const common_peg_arena & src_pars
std::unique_ptr<common_chat_peg_mapper> mapper;
if (params.format == COMMON_CHAT_FORMAT_PEG_GEMMA4) {
mapper = std::make_unique<common_chat_peg_gemma4_mapper>(msg);
} else if (params.format == COMMON_CHAT_FORMAT_PEG_MINIMAX_M3) {
mapper = std::make_unique<common_chat_peg_minimax_m3_mapper>(msg);
} else {
mapper = std::make_unique<common_chat_peg_mapper>(msg);
}
@@ -2801,6 +2856,8 @@ common_chat_msg common_chat_peg_parse(const common_peg_arena & src_pars
std::unique_ptr<common_chat_peg_mapper> mapper;
if (params.format == COMMON_CHAT_FORMAT_PEG_GEMMA4) {
mapper = std::make_unique<common_chat_peg_gemma4_mapper>(msg);
} else if (params.format == COMMON_CHAT_FORMAT_PEG_MINIMAX_M3) {
mapper = std::make_unique<common_chat_peg_minimax_m3_mapper>(msg);
} else {
mapper = std::make_unique<common_chat_peg_mapper>(msg);
}
common/chat.h
+1
View File
@@ -160,6 +160,7 @@ enum common_chat_format {
COMMON_CHAT_FORMAT_PEG_SIMPLE,
COMMON_CHAT_FORMAT_PEG_NATIVE,
COMMON_CHAT_FORMAT_PEG_GEMMA4,
COMMON_CHAT_FORMAT_PEG_MINIMAX_M3,
COMMON_CHAT_FORMAT_COUNT, // Not a format, just the # formats
};
common/common.cpp
+31 -3
View File
@@ -157,6 +157,9 @@ common_params_speculative common_params_speculative::with_stage_overrides(const
if (stage.has_p_min_override()) {
result.p_min = stage.p_min;
}
if (stage.has_mtp_heads_override()) {
result.mtp_heads = stage.mtp_heads;
}
if (stage.has_dflash_cross_ctx_override()) {
result.dflash_cross_ctx = stage.dflash_cross_ctx;
}
@@ -182,6 +185,7 @@ common_params_speculative common_params_speculative::with_stage_overrides(const
result.n_max = std::max(result.n_max, 0);
result.n_min = std::max(0, std::min(result.n_min, result.n_max));
result.mtp_heads = std::max(result.mtp_heads, 0);
result.stages.clear();
return result;
@@ -792,8 +796,8 @@ void gpt_params_parse_from_env(gpt_params & params) {
get_env("LLAMA_ARG_CONT_BATCHING", params.cont_batching);
get_env("LLAMA_ARG_HOST", params.hostname);
get_env("LLAMA_ARG_PORT", params.port);
get_env("LLAMA_ARG_CACHE_TYPE_K", params.cache_type_k);
get_env("LLAMA_ARG_CACHE_TYPE_V", params.cache_type_v);
get_env("LLAMA_ARG_CACHE_TYPE_K", params.cache_type_k);
get_env("LLAMA_ARG_CACHE_TYPE_V", params.cache_type_v);
get_env("LLAMA_ARG_MLOCK", params.use_mlock);
get_env("LLAMA_ARG_K_CACHE_HADAMARD", params.k_cache_hadamard);
get_env("LLAMA_ARG_V_CACHE_HADAMARD", params.v_cache_hadamard);
@@ -926,6 +930,13 @@ static void common_speculative_stage_apply_kv(
}
return;
}
if (key == "heads" || key == "mtp_heads") {
stage.mtp_heads = std::stoi(value_raw);
if (stage.mtp_heads < 0) {
throw std::invalid_argument("speculative stage mtp_heads must be >= 0");
}
return;
}
if (key == "cross_ctx" || key == "dflash_cross_ctx") {
stage.dflash_cross_ctx = std::stoi(value_raw);
if (stage.dflash_cross_ctx < 1) {
@@ -2141,6 +2152,15 @@ bool gpt_params_find_arg(int argc, char ** argv, const std::string & arg, gpt_pa
params.warmup = false;
return true;
}
if (arg == "--prefetch-experts") {
params.prefetch_experts = true;
return true;
}
if (arg == "--prefetch-experts-threads") {
CHECK_ARG;
params.prefetch_experts_threads = std::stoi(argv[i]);
return true;
}
if (arg == "--fit-margin") {
CHECK_ARG;
int32_t margin = std::stoi(argv[i]);
@@ -3247,6 +3267,9 @@ void gpt_params_print_usage(int /*argc*/, char ** argv, const gpt_params & param
options.push_back({ "*", " --cpu-moe", "keep all MoE weights in CPU memory"});
options.push_back({ "*", " --n-cpu-moe N", "keep MoE weights of the first N layers in CPU memory"});
options.push_back({ "*", " --defer-experts", "defer expert mmap residency on Linux to reduce model load time"});
options.push_back({ "*", " --prefetch-experts", "stream mmap'd MoE expert weights into the page cache on Linux"});
options.push_back({ "*", " --prefetch-experts-threads N",
"number of expert prefetch workers, tune to drive speed/type (default: auto)"});
options.push_back({ "*", " --fit-margin N", "safety margin in MiB when auto-fitting model offloading"});
options.push_back({ "*", "-wgt, --worst-graph-tokens N", "number of tokens to use for worst-case graph"});
options.push_back({ "*", " --fit", "automatically determine which tensors to offload to the GPU(s)"});
@@ -3308,7 +3331,8 @@ void gpt_params_print_usage(int /*argc*/, char ** argv, const gpt_params & param
" cpu serialise state via llama_state_seq; re-decode on rejection" });
options.push_back({ "*", "--spec-type SPEC[:k=v,...]", "canonical speculative stage entry; repeat for a supported two-stage chain.\n"
"types: none, draft, dflash, mtp, ngram-cache, ngram-simple, ngram-map-k, ngram-map-k4v, ngram-mod, suffix\n"
"canonical keys: n_max,n_min,p_min,cross_ctx,ngram_size_n,ngram_size_m,ngram_min_hits,suffix_min_match_len,suffix_max_depth,suffix_corpus\n"
"canonical keys: n_max,n_min,p_min,heads,cross_ctx,ngram_size_n,ngram_size_m,ngram_min_hits,suffix_min_match_len,suffix_max_depth,suffix_corpus\n"
"MTP heads: heads=1 is the default; heads>1 and heads=0 (all model heads) are experimental\n"
"for comma-bearing string values, quote the value inside the stage payload for normal shell use\n"
"if argv is passed directly without shell unescaping, the parser also accepts escaped commas as \\,\n"
"examples: --spec-type mtp:n_max=1,p_min=0.0\n"
@@ -4289,6 +4313,8 @@ struct llama_context_params common_context_params_to_llama(const gpt_params & pa
cparams.min_experts = params.min_experts;
cparams.thresh_experts = params.thresh_experts;
cparams.only_active_experts = params.only_active_exps;
cparams.prefetch_experts = params.prefetch_experts;
cparams.prefetch_experts_threads = params.prefetch_experts_threads;
cparams.max_extra_alloc = params.max_extra_alloc_MiB;
cparams.mtp = params.speculative.has_stage_type(COMMON_SPECULATIVE_TYPE_MTP);
cparams.mtp_op_type = MTP_OP_NONE;
@@ -5286,6 +5312,8 @@ void yaml_dump_non_result_info(FILE * stream, const gpt_params & params, const l
fprintf(stream, "merge_qkv: %s # default: false\n", params.merge_qkv ? "true" : "false");
fprintf(stream, "merge_up_gate_exps: %s # default: false\n", params.merge_up_gate_exps ? "true" : "false");
fprintf(stream, "defer_experts: %s # default: false\n", params.defer_experts ? "true" : "false");
fprintf(stream, "prefetch_experts: %s # default: false\n", params.prefetch_experts ? "true" : "false");
fprintf(stream, "prefetch_experts_threads: %d # default: 0 (auto)\n", params.prefetch_experts_threads);
fprintf(stream, "max_extra_alloc: %d # default: 256\n", params.max_extra_alloc_MiB);
fprintf(stream, "penalize_nl: %s # default: false\n", sparams.penalize_nl ? "true" : "false");
fprintf(stream, "ppl_output_type: %d # default: 0\n", params.ppl_output_type);
common/common.h
+5
View File
@@ -172,6 +172,7 @@ struct common_speculative_stage_params {
int32_t n_max = -1;
int32_t n_min = -1;
float p_min = -1.0f;
int32_t mtp_heads = -1;
int32_t dflash_cross_ctx = -1;
uint16_t ngram_size_n = 0;
@@ -185,6 +186,7 @@ struct common_speculative_stage_params {
bool has_n_max_override() const { return n_max >= 0; }
bool has_n_min_override() const { return n_min >= 0; }
bool has_p_min_override() const { return p_min >= 0.0f; }
bool has_mtp_heads_override() const { return mtp_heads >= 0; }
bool has_dflash_cross_ctx_override() const { return dflash_cross_ctx >= 0; }
bool has_ngram_size_n_override() const { return ngram_size_n > 0; }
bool has_ngram_size_m_override() const { return ngram_size_m > 0; }
@@ -218,6 +220,7 @@ struct common_params_speculative {
int32_t n_max = 16; // number of tokens to draft during speculative decoding
int32_t n_min = 0; // minimum number of tokens to draft during speculative decoding
std::vector<common_speculative_stage_params> stages; // explicit stage chain for single-spec or self-spec + model fallback
int32_t mtp_heads = 1; // MTP heads to use; 1 is the default, while >1 and 0 (all model heads) are experimental
int32_t dflash_cross_ctx = 512; // target-feature context window for DFlash
float p_split = 0.1f; // speculative decoding split probability
@@ -443,6 +446,8 @@ struct gpt_params {
bool merge_qkv = false; // if true, merge separate Q, K, V tensors into a single, contiguous tensor
bool merge_up_gate_exps= false; // if true, merge ffn_up_exps and ffn_gate_exps into a single, contiguous tensor
bool defer_experts = false; // if true, defer expert mmap residency to speed up model loading (Linux only)
bool prefetch_experts = false; // if true, stream mmap'd MoE expert weights into the page cache (Linux only)
int prefetch_experts_threads = 0; // number of expert prefetch workers (<=0 = auto)
bool k_cache_hadamard = false; // if true, use Hadamard transform for the K-cache (only makes sense with quantized cache)
bool v_cache_hadamard = false; // if true, use Hadamard transform for the V-cache (only makes sense with quantized cache, which requires FA)
bool split_mode_graph_scheduling = false; // if true, force split mode graph scheduling
common/spec-tuner.cpp
+168 -11
View File
@@ -98,6 +98,14 @@ void spec_tuner::reset_exploration() {
for (auto & coord : coords) {
coord.reset_scores();
}
dflash_quarantined.assign(dflash_quarantined.size(), false);
dflash_probe_cursor = 0;
dflash_last_exploratory = false;
dflash_last_recovery_probe = false;
if (has_dflash_target_only_arm() && !coords.empty() && !coords[0].arms.empty()) {
coords[0].current_idx = 0;
coords[0].best_idx = 0;
}
n_low = 0;
cooldown = cooldown_max;
step_ema = 0.0;
@@ -107,7 +115,11 @@ void spec_tuner::reset_exploration() {
void spec_tuner::write_best(common_params_speculative & params) const {
for (const auto & coord : coords) {
float val = coord.arms[coord.best_idx].value;
if (coord.name == "n_max") params.n_max = (int32_t)val;
if (coord.name == "n_max") {
params.n_max = (spec_type == COMMON_SPECULATIVE_TYPE_DFLASH && (int32_t)val == 0)
? configured_n_max
: (int32_t)val;
}
else if (coord.name == "p_min") params.p_min = val;
else if (coord.name == "n_min") params.n_min = (int32_t)val;
else if (coord.name == "ngram_size_n") params.ngram_size_n = (uint16_t)val;
@@ -120,6 +132,16 @@ void spec_tuner::write_best(common_params_speculative & params) const {
void spec_tuner::init(common_speculative_type type, const common_params_speculative & user_params, const llama_model * model_tgt) {
enabled = true;
spec_type = type;
configured_n_max = std::max(1, (int) user_params.n_max);
dflash_quarantined.clear();
dflash_probe_cursor = 0;
dflash_last_exploratory = false;
dflash_last_recovery_probe = false;
n_target_only_selections = 0;
n_dflash_selections = 0;
n_exploratory_selections = 0;
n_quarantines = 0;
n_recovery_probes = 0;
coords.clear();
n_calls = 0;
n_requests = 0;
@@ -137,12 +159,19 @@ void spec_tuner::init(common_speculative_type type, const common_params_speculat
spec_tuner_coord coord;
coord.name = "n_max";
const bool recurrent_target = model_tgt != nullptr && llama_model_has_recurrent(model_tgt);
int hi = recurrent_target ? std::max(1, (int) user_params.n_max)
: std::max(16, (int) user_params.n_max);
coord.build_grid_int(1, hi, 1, user_params.n_max);
int hi = type == COMMON_SPECULATIVE_TYPE_DFLASH
? configured_n_max
: (recurrent_target ? std::max(1, (int) user_params.n_max)
: std::max(16, (int) user_params.n_max));
const int lo = type == COMMON_SPECULATIVE_TYPE_DFLASH ? 0 : 1;
coord.build_grid_int(lo, hi, 1, user_params.n_max);
coords.push_back(std::move(coord));
}
if (type == COMMON_SPECULATIVE_TYPE_DFLASH) {
dflash_quarantined.assign(coords[0].arms.size(), false);
}
if (type == COMMON_SPECULATIVE_TYPE_DRAFT) {
{
spec_tuner_coord coord;
@@ -212,12 +241,111 @@ void spec_tuner::init(common_speculative_type type, const common_params_speculat
}
}
int spec_tuner::select_dflash_arm(spec_tuner_coord & coord) {
dflash_last_exploratory = false;
dflash_last_recovery_probe = false;
if (dflash_quarantined.size() != coord.arms.size()) {
dflash_quarantined.assign(coord.arms.size(), false);
}
for (int i = 0; i < (int) coord.arms.size(); ++i) {
if (coord.arms[i].N < dflash_min_samples_per_arm) {
dflash_last_exploratory = true;
return i;
}
}
if (dflash_recovery_probe_interval > 0 && n_calls > 0 &&
n_calls % dflash_recovery_probe_interval == 0) {
for (int offset = 0; offset < (int) coord.arms.size(); ++offset) {
const int i = (dflash_probe_cursor + offset) % (int) coord.arms.size();
if (dflash_quarantined[i]) {
dflash_probe_cursor = (i + 1) % (int) coord.arms.size();
dflash_last_exploratory = true;
dflash_last_recovery_probe = true;
n_recovery_probes++;
return i;
}
}
}
int best_idx = -1;
for (int i = 0; i < (int) coord.arms.size(); ++i) {
if (dflash_quarantined[i]) {
continue;
}
if (best_idx < 0 || coord.arms[i].Q > coord.arms[best_idx].Q) {
best_idx = i;
}
}
if (best_idx < 0) {
best_idx = 0;
}
coord.best_idx = best_idx;
return best_idx;
}
void spec_tuner::update_dflash_quarantine() {
if (!has_dflash_target_only_arm() || coords.empty() || coords[0].arms.empty()) {
return;
}
auto & coord = coords[0];
if (dflash_quarantined.size() != coord.arms.size()) {
dflash_quarantined.assign(coord.arms.size(), false);
}
const int zero_idx = coord.find_nearest_arm(0.0f);
const auto & zero = coord.arms[zero_idx];
if (zero.N < 3 || zero.Q <= 0.0) {
return;
}
for (int i = 0; i < (int) coord.arms.size(); ++i) {
if (i == zero_idx) {
continue;
}
const bool should_quarantine = coord.arms[i].N >= 3 &&
coord.arms[i].Q < zero.Q * dflash_quarantine_ratio;
if (should_quarantine && !dflash_quarantined[i]) {
n_quarantines++;
}
dflash_quarantined[i] = should_quarantine;
}
int best_idx = -1;
for (int i = 0; i < (int) coord.arms.size(); ++i) {
if (dflash_quarantined[i]) {
continue;
}
if (best_idx < 0 || coord.arms[i].Q > coord.arms[best_idx].Q) {
best_idx = i;
}
}
if (best_idx >= 0) {
coord.best_idx = best_idx;
}
}
void spec_tuner::propose(common_params_speculative & params) {
int64_t t_start = ggml_time_us();
// always select fresh arm for every draft call
for (auto & coord : coords) {
coord.current_idx = coord.select_epsilon_greedy(epsilon);
if (has_dflash_target_only_arm() && coord.name == "n_max") {
coord.current_idx = select_dflash_arm(coord);
if ((int) coord.arms[coord.current_idx].value == 0) {
n_target_only_selections++;
} else {
n_dflash_selections++;
}
if (dflash_last_exploratory) {
n_exploratory_selections++;
}
} else {
coord.current_idx = coord.select_epsilon_greedy(epsilon);
}
float val = coord.arms[coord.current_idx].value;
if (coord.name == "n_max") params.n_max = (int32_t)val;
@@ -235,7 +363,8 @@ void spec_tuner::propose(common_params_speculative & params) {
void spec_tuner::enforce_constraints(common_params_speculative & params) {
if (params.n_min < 0) params.n_min = 0;
if (params.n_max < 1) params.n_max = 1;
const int min_n_max = has_dflash_target_only_arm() ? 0 : 1;
if (params.n_max < min_n_max) params.n_max = min_n_max;
if (params.n_min > params.n_max) params.n_min = params.n_max;
if (params.p_min < 0.0f) params.p_min = 0.0f;
@@ -257,6 +386,8 @@ void spec_tuner::accept_feedback(int n_accepted, int n_drafted, double step_tps)
coord.update(reward);
}
update_dflash_quarantine();
if (cooldown > 0) {
cooldown--;
if (step_ema <= 0.0) {
@@ -287,11 +418,17 @@ void spec_tuner::accept_feedback(int n_accepted, int n_drafted, double step_tps)
<< " n_accepted=" << n_accepted
<< " step_tps=" << std::fixed << std::setprecision(1) << step_tps
<< " ema=" << std::fixed << std::setprecision(1) << step_ema;
if (has_dflash_target_only_arm()) {
oss << " configured_n_max=" << configured_n_max;
}
for (const auto & coord : coords) {
bool is_int = (coord.name != "p_min");
oss << " " << coord.name << "=";
if (is_int) oss << (int)coord.arms[coord.current_idx].value;
else oss << std::fixed << std::setprecision(2) << coord.arms[coord.current_idx].value;
if (coord.name == "n_max" && spec_type == COMMON_SPECULATIVE_TYPE_DFLASH) {
oss << "(target_only=" << ((int) coord.arms[coord.current_idx].value == 0 ? "true" : "false") << ")";
}
oss << "→best=";
if (is_int) oss << (int)coord.arms[coord.best_idx].value;
else oss << std::fixed << std::setprecision(2) << coord.arms[coord.best_idx].value;
@@ -332,12 +469,20 @@ void spec_tuner::print_best() const {
if (n_resets > 0) oss << " resets=" << n_resets;
if (n_low > 0) oss << " n_low=" << n_low;
if (has_dflash_target_only_arm()) {
oss << " configured_n_max=" << configured_n_max
<< " target_only=" << n_target_only_selections
<< " dflash=" << n_dflash_selections
<< " exploratory=" << n_exploratory_selections
<< " quarantines=" << n_quarantines
<< " probes=" << n_recovery_probes;
}
oss << " best:";
for (const auto & coord : coords) {
bool is_int = (coord.name != "p_min");
oss << " " << coord.name << "=";
if (is_int) oss << (int)coord.arms[coord.best_idx].value;
if (is_int) oss << (int) coord.arms[coord.best_idx].value;
else oss << std::fixed << std::setprecision(2) << coord.arms[coord.best_idx].value;
oss << "(Q=" << std::fixed << std::setprecision(2) << coord.arms[coord.best_idx].Q
<< ",N=" << coord.arms[coord.best_idx].N << ")";
@@ -345,14 +490,22 @@ void spec_tuner::print_best() const {
if (!coords.empty()) {
oss << " | n_max arms:";
for (const auto & arm : coords[0].arms) {
for (size_t arm_idx = 0; arm_idx < coords[0].arms.size(); ++arm_idx) {
const auto & arm = coords[0].arms[arm_idx];
const bool quarantined = has_dflash_target_only_arm() && arm_idx < dflash_quarantined.size()
? dflash_quarantined[arm_idx]
: false;
oss << " " << (int)arm.value << "(Q=" << std::fixed << std::setprecision(2) << arm.Q
<< ",N=" << arm.N << ")";
<< ",N=" << arm.N << ",quarantined=" << (quarantined ? "true" : "false") << ")";
}
}
oss << " tuner=" << std::fixed << std::setprecision(3) << t_tuner_us / 1000.0 << "ms";
LOG_DBG("%s\n", oss.str().c_str());
if (has_dflash_target_only_arm()) {
LOG_INF("%s\n", oss.str().c_str());
} else {
LOG_DBG("%s\n", oss.str().c_str());
}
}
{
@@ -364,7 +517,11 @@ void spec_tuner::print_best() const {
oss << (first_kv ? ':' : ',') << coord.name << '=';
first_kv = false;
if (is_int) oss << (int)coord.arms[coord.best_idx].value;
int reuse_value = is_int ? (int) coord.arms[coord.best_idx].value : 0;
if (coord.name == "n_max" && spec_type == COMMON_SPECULATIVE_TYPE_DFLASH && reuse_value == 0) {
reuse_value = configured_n_max;
}
if (is_int) oss << reuse_value;
else oss << std::fixed << std::setprecision(2) << coord.arms[coord.best_idx].value;
}
LOG_INF("%s\n", oss.str().c_str());
common/spec-tuner.h
+22
View File
@@ -33,6 +33,20 @@ struct spec_tuner {
double epsilon = 0.15; // 15% explore, 85% exploit
int configured_n_max = 0;
int dflash_min_samples_per_arm = 3;
int dflash_recovery_probe_interval = 128;
double dflash_quarantine_ratio = 0.90;
std::vector<bool> dflash_quarantined;
int dflash_probe_cursor = 0;
bool dflash_last_exploratory = false;
bool dflash_last_recovery_probe = false;
uint64_t n_target_only_selections = 0;
uint64_t n_dflash_selections = 0;
uint64_t n_exploratory_selections = 0;
uint64_t n_quarantines = 0;
uint64_t n_recovery_probes = 0;
// task-change detection (per-call)
// If tuner goes bad for 30 consecutive calls, reset the tuner.
double step_ema = 0.0;
@@ -66,4 +80,12 @@ struct spec_tuner {
void reset_exploration();
void write_best(common_params_speculative & params) const;
bool has_dflash_target_only_arm() const {
return enabled && spec_type == COMMON_SPECULATIVE_TYPE_DFLASH && configured_n_max > 0;
}
private:
int select_dflash_arm(spec_tuner_coord & coord);
void update_dflash_quarantine();
};
common/speculative.cpp
+139 -7
View File
@@ -24,6 +24,8 @@
#define SPEC_VOCAB_CHECK_START_TOKEN_ID 5
void llama_set_mtp_target_context(struct llama_context * ctx, struct llama_context * target_ctx);
void llama_set_mtp_step_idx(struct llama_context * ctx, int32_t mtp_step_idx);
void llama_set_mtp_n_heads(struct llama_context * ctx, int32_t mtp_n_heads);
const std::vector<enum common_speculative_type> common_speculative_types = {
COMMON_SPECULATIVE_TYPE_NONE,
@@ -212,12 +214,13 @@ static std::vector<llama_token> mtp_speculative_gen_draft(
struct llama_context * ctx,
int n_draft,
float p_min,
int32_t mtp_heads,
llama_token id_last,
llama_pos n_past,
llama_seq_id seq_id,
bool constant_draft_positions = false);
static int32_t mtp_update_kv_cache(struct llama_context * ctx, const llama_batch & batch, bool is_prompt_warmup);
static int32_t mtp_update_kv_cache(struct llama_context * ctx, const llama_batch & batch, bool is_prompt_warmup, int32_t mtp_heads);
struct mtp_last_embd {
std::vector<float> embd;
@@ -228,7 +231,19 @@ struct mtp_last_embd {
struct common_speculative_state_mtp : public common_speculative_state {
llama_context * ctx_tgt;
llama_context * ctx_mtp = nullptr;
int32_t mtp_heads_active = 0;
// number of NextN heads the model carries, and the minimum head count the committed
// context has been warmed with since position 0 (deeper heads' caches only hold valid
// rows for spans warmed with them; a request drafting with MORE heads than the cached
// prefix was warmed with must reprocess from scratch). Single-sequence by design, like
// the rest of the openPangu MTP state.
int32_t n_heads_model = 1;
int32_t mtp_warmed_heads = 0;
common_sampler * smpl;
int32_t resolved_heads() const {
return mtp_heads_active > 0 ? std::min(mtp_heads_active, n_heads_model) : n_heads_model;
}
// For Gemma 4 external MTP assistant: draft positions are held constant
bool constant_draft_positions = false;
int n_embd = 0;
@@ -252,9 +267,15 @@ struct common_speculative_state_mtp : public common_speculative_state {
smpl = common_sampler_init(llama_get_model(ctx_mtp), sparams);
llama_set_mtp_target_context(ctx_mtp, ctx_tgt);
n_embd = llama_mtp_state_n_embd(ctx_mtp);
n_heads_model = std::max(1, llama_model_n_nextn_layer(llama_get_model(ctx_mtp)));
LOG_INF("%s: MTP context ready (n_ctx=%d, constant_draft_positions=%s)\n", __func__,
llama_n_ctx(ctx_mtp), constant_draft_positions ? "true" : "false");
if (n_heads_model > 1) {
LOG_INF("%s: model carries %d NextN/MTP heads; heads=1 is the default, while "
"heads>1 and heads=0 (all model heads) are experimental\n",
__func__, n_heads_model);
}
}
~common_speculative_state_mtp() override {
@@ -304,6 +325,7 @@ struct common_speculative_state_mtp : public common_speculative_state {
}
llama_context * ctx = ctx_mtp;
mtp_heads_active = std::max<int32_t>(0, params.mtp_heads);
const auto hidden_it = target_hidden_by_seq.find(seq_id);
if (hidden_it == target_hidden_by_seq.end() || (int) hidden_it->second.size() != n_embd) {
@@ -323,6 +345,7 @@ struct common_speculative_state_mtp : public common_speculative_state {
ctx,
params.n_max,
params.p_min,
params.mtp_heads,
id_last,
n_past,
seq_id,
@@ -1044,6 +1067,7 @@ struct common_speculative {
std::unique_ptr<spec_tuner> tuner;
int last_n_drafted = 0;
int64_t t_step_start_us = 0;
bool last_step_target_only = false;
};
static bool common_speculative_stage_chain_matches(
@@ -1072,6 +1096,7 @@ static common_params_speculative common_speculative_get_runtime_params(
result.n_max = stage.has_n_max_override() ? stage.n_max : params.n_max;
result.n_min = stage.has_n_min_override() ? stage.n_min : params.n_min;
result.p_min = stage.has_p_min_override() ? stage.p_min : params.p_min;
result.mtp_heads = stage.has_mtp_heads_override() ? stage.mtp_heads : params.mtp_heads;
if (config.type == COMMON_SPECULATIVE_TYPE_SUFFIX) {
result.suffix_min_match_len = stage.has_suffix_min_match_len_override()
@@ -1081,11 +1106,43 @@ static common_params_speculative common_speculative_get_runtime_params(
result.n_max = std::max(result.n_max, 0);
result.n_min = std::max(0, std::min(result.n_min, result.n_max));
result.mtp_heads = std::max(result.mtp_heads, 0);
result.stages.clear();
return result;
}
bool common_speculative_mtp_requires_fresh_warmup(const common_speculative * spec) {
const auto * mtp_state = common_speculative_get_mtp_state(spec);
if (mtp_state == nullptr || mtp_state->n_heads_model <= 1) {
return false;
}
// drafting with more heads than the cached prefix was warmed with would read
// never-written deeper-head cache rows; the caller must reprocess from position 0
return mtp_state->resolved_heads() > mtp_state->mtp_warmed_heads && mtp_state->mtp_warmed_heads > 0;
}
void common_speculative_prepare_request(common_speculative * spec, common_params_speculative & params) {
if (spec == nullptr) {
return;
}
const auto runtime_stages = params.get_resolved_stages();
const bool use_runtime_stage_overrides = common_speculative_stage_chain_matches(runtime_stages, spec->configs);
for (size_t i = 0; i < spec->impls.size(); ++i) {
auto * mtp_state = dynamic_cast<common_speculative_state_mtp *>(spec->impls[i].get());
if (mtp_state == nullptr) {
continue;
}
const auto & runtime_stage = use_runtime_stage_overrides ? runtime_stages[i] : spec->configs[i].stage;
common_params_speculative impl_params = common_speculative_get_runtime_params(spec->configs[i], params, runtime_stage);
mtp_state->mtp_heads_active = std::max<int32_t>(0, impl_params.mtp_heads);
}
}
static common_ngram_map get_common_ngram_map(const common_speculative_config & config) {
uint16_t size_key = config.params.ngram_size_n;
uint16_t size_value = config.params.ngram_size_m;
@@ -1281,7 +1338,8 @@ common_speculative * common_speculative_init(
configs.push_back(common_speculative_config(stage, stage_params));
}
if (!configs.empty() && llama_model_has_recurrent(llama_get_model(ctx_tgt))) {
if (!configs.empty() && (llama_model_has_recurrent(llama_get_model(ctx_tgt)) ||
llama_model_is_openpangu(llama_get_model(ctx_tgt)))) {
const int ckpt_tokens = std::max(1, params.get_max_stage_n_max() + 1);
const int actual_mode = llama_spec_ckpt_init(ctx_tgt, params.recurrent_ckpt_mode, ckpt_tokens);
if (actual_mode == LLAMA_SPEC_CKPT_NONE) {
@@ -1407,6 +1465,7 @@ common_speculative * common_speculative_init(
/* .configs = */ std::move(configs),
/* .impls = */ std::move(impls)
};
common_speculative_prepare_request(result, params);
// initialize autotune if requested
if (params.autotune && params.has_composite_stage_chain()) {
@@ -1416,7 +1475,7 @@ common_speculative * common_speculative_init(
if (actual_type != COMMON_SPECULATIVE_TYPE_NONE &&
actual_type != COMMON_SPECULATIVE_TYPE_EAGLE3) {
result->tuner = std::make_unique<spec_tuner>();
result->tuner->init(actual_type, params, llama_get_model(ctx_tgt));
result->tuner->init(actual_type, result->configs[0].params, llama_get_model(ctx_tgt));
LOG_DBG("Autotune initialized for %s, tuning %zu parameters\n",
common_speculative_type_to_str(actual_type).c_str(),
result->tuner->coords.size());
@@ -1460,6 +1519,7 @@ llama_tokens common_speculative_draft(
llama_tokens result;
spec->t_step_start_us = ggml_time_us();
spec->last_step_target_only = false;
// apply autotune proposal if enabled
if (spec->tuner && spec->tuner->enabled) {
@@ -1475,8 +1535,19 @@ llama_tokens common_speculative_draft(
auto & impl = spec->impls[i];
const auto & runtime_stage = use_runtime_stage_overrides ? runtime_stages[i] : spec->configs[i].stage;
common_params_speculative impl_params = common_speculative_get_runtime_params(spec->configs[i], params, runtime_stage);
if (spec->tuner && spec->tuner->enabled && impl->type == COMMON_SPECULATIVE_TYPE_DFLASH) {
impl_params.n_max = params.n_max;
}
result.clear();
if (spec->tuner && spec->tuner->has_dflash_target_only_arm() &&
impl->type == COMMON_SPECULATIVE_TYPE_DFLASH && impl_params.n_max == 0) {
spec->curr_impl = impl.get();
spec->last_step_target_only = true;
LOG_DBG("%s: selected DFlash target-only arm\n", __func__);
break;
}
{
common_time_meas tm(impl->t_draft_us, !impl->gen_perf);
impl->draft(impl_params, prompt_tgt, id_last, draft_base_pos, draft_seq_id, result);
@@ -1789,10 +1860,18 @@ common_speculative_draft_result common_speculative_draft_ex(
result.type = spec != nullptr && spec->curr_impl != nullptr
? spec->curr_impl->type
: COMMON_SPECULATIVE_TYPE_NONE;
result.target_only = spec != nullptr && spec->last_step_target_only;
return result;
}
int common_speculative_get_configured_n_max(const common_speculative * spec) {
if (spec == nullptr || spec->tuner == nullptr || !spec->tuner->has_dflash_target_only_arm()) {
return 0;
}
return spec->tuner->configured_n_max;
}
static bool common_speculative_has_target_features(const common_speculative * spec) {
return common_speculative_has_type(spec, COMMON_SPECULATIVE_TYPE_MTP) ||
common_speculative_has_type(spec, COMMON_SPECULATIVE_TYPE_DFLASH);
@@ -2623,6 +2702,7 @@ void common_speculative_clear_sequence(
spec->curr_impl = nullptr;
spec->last_n_drafted = 0;
spec->t_step_start_us = 0;
spec->last_step_target_only = false;
}
common_speculative_clear_sequence_hidden(spec, seq_id);
@@ -2683,10 +2763,20 @@ static int32_t mtp_accept_batch(
if (!llama_set_draft_input_hidden_state_copy(state.ctx_mtp, hidden_rows, hidden_rows_floats)) {
return -1;
}
if (mtp_update_kv_cache(state.ctx_mtp, accepted_batch, false) != 0) {
if (mtp_update_kv_cache(state.ctx_mtp, accepted_batch, false, state.mtp_heads_active) != 0) {
return -1;
}
if (llama_model_is_openpangu(llama_get_model(state.ctx_mtp))) {
// The one-token draft shortcut re-seeded below would skip re-decoding the last
// sampled token next round, leaving a hole at its position. openPangu's KV cache
// is position-addressed append-only (cell == position), so draft decodes must be
// position-contiguous; decline the shortcut and let the next round decode the
// sampled token normally.
mtp_invalidate_cached_draft(state, seq_id);
return 0;
}
auto & last = mtp_get_last_embd(state, seq_id);
const float * embd = llama_get_embeddings_ith(state.ctx_mtp, accepted_batch.n_tokens - 1);
if (embd != nullptr) {
@@ -2767,6 +2857,18 @@ int32_t common_speculative_on_target_batch(
const float * last_hidden = hidden_rows_storage.data() + (size_t) (batch.n_tokens - 1) * features.width;
mtp_store_target_hidden(*mtp_state, seq_id, last_hidden, features.width);
// track the minimum head count the committed context has been warmed with: a fresh
// position-0 warmup resets it, everything after can only narrow it
{
const int32_t resolved = mtp_state->resolved_heads();
if (is_prompt_warmup && batch.pos != nullptr && batch.n_tokens > 0 && batch.pos[0] == 0) {
mtp_state->mtp_warmed_heads = resolved;
} else {
mtp_state->mtp_warmed_heads = mtp_state->mtp_warmed_heads > 0
? std::min(mtp_state->mtp_warmed_heads, resolved) : resolved;
}
}
if (mtp_state->constant_draft_positions) {
mtp_invalidate_cached_draft(*mtp_state, seq_id);
return 0;
@@ -2804,7 +2906,7 @@ int32_t common_speculative_on_target_batch(
if (!llama_set_draft_input_hidden_state_copy(mtp_state->ctx_mtp, conditioned_hidden_rows, hidden_rows_storage.size())) {
return -1;
}
const int32_t ret = mtp_update_kv_cache(mtp_state->ctx_mtp, batch, true);
const int32_t ret = mtp_update_kv_cache(mtp_state->ctx_mtp, batch, true, mtp_state->mtp_heads_active);
mtp_invalidate_cached_draft(*mtp_state, seq_id);
return ret;
}
@@ -2823,6 +2925,10 @@ void common_speculative_context_shift(
llama_pos kv_keep,
llama_pos kv_discard,
llama_pos kv_past) {
if (spec != nullptr) {
spec->last_step_target_only = false;
spec->t_step_start_us = 0;
}
if (auto * ctx_mtp = common_speculative_get_companion_ctx(spec); ctx_mtp != nullptr) {
llama_kv_cache_seq_rm (ctx_mtp, seq_id, kv_keep, kv_keep + kv_discard);
llama_kv_cache_seq_add(ctx_mtp, seq_id, kv_keep + kv_discard, kv_past, -kv_discard);
@@ -2839,6 +2945,7 @@ std::vector<llama_token> mtp_speculative_gen_draft(
struct llama_context * ctx,
int n_draft,
float p_min,
int32_t mtp_heads,
llama_token id_last,
llama_pos n_past,
llama_seq_id seq_id,
@@ -2856,7 +2963,22 @@ std::vector<llama_token> mtp_speculative_gen_draft(
common_sampler_reset(smpl);
if (llama_model_is_openpangu(llama_get_model(ctx)) &&
llama_kv_cache_seq_pos_max(ctx, seq_id) >= n_past) {
// Position-addressed cache: drafting restarts at n_past, so any rows at or beyond
// it (the accepted-update writes one row past the accepted prefix) must be dropped
// first to keep the draft decode position-contiguous with the cache head.
llama_kv_cache_seq_rm(ctx, seq_id, n_past, -1);
}
const int n_embd = llama_mtp_state_n_embd(ctx);
const int n_mtp_heads_model = std::max(1, llama_model_n_nextn_layer(llama_get_model(ctx)));
const int n_mtp_heads = mtp_heads > 0
? std::max(1, std::min((int) mtp_heads, n_mtp_heads_model))
: n_mtp_heads_model;
llama_batch mtp_batch = llama_batch_init(1, 0, 1);
llama_set_mtp_n_heads(ctx, n_mtp_heads);
llama_set_mtp_op_type(ctx, MTP_OP_DRAFT_GEN);
float prob;
@@ -2864,7 +2986,6 @@ std::vector<llama_token> mtp_speculative_gen_draft(
llama_token current_input_id = id_last;
llama_pos current_n_past = n_past;
const int n_embd = llama_mtp_state_n_embd(ctx);
auto & last = mtp_get_last_embd(state, seq_id);
int i0 = 0;
@@ -2878,6 +2999,8 @@ std::vector<llama_token> mtp_speculative_gen_draft(
current_n_past++;
if (!llama_set_draft_input_hidden_state_copy(ctx, last.embd.data(), last.embd.size())) {
llama_batch_free(mtp_batch);
llama_set_mtp_step_idx(ctx, 0);
llama_set_mtp_n_heads(ctx, 0);
llama_set_mtp_op_type(ctx, MTP_OP_NONE);
return drafts;
}
@@ -2889,6 +3012,7 @@ std::vector<llama_token> mtp_speculative_gen_draft(
mtp_batch.n_tokens = 0;
const llama_pos draft_pos = constant_draft_positions ? n_past : current_n_past;
common_batch_add(mtp_batch, current_input_id, draft_pos, {seq_id}, true);
llama_set_mtp_step_idx(ctx, std::min(i, n_mtp_heads - 1));
++n_decode;
if (llama_decode(ctx, mtp_batch) != 0) {
@@ -2922,6 +3046,8 @@ std::vector<llama_token> mtp_speculative_gen_draft(
}
}
llama_batch_free(mtp_batch);
llama_set_mtp_step_idx(ctx, 0);
llama_set_mtp_n_heads(ctx, 0);
llama_set_mtp_op_type(ctx, MTP_OP_NONE);
// Purge the metadata for the draft tokens.
@@ -2938,7 +3064,7 @@ std::vector<llama_token> mtp_speculative_gen_draft(
}
int32_t mtp_update_kv_cache(struct llama_context * ctx, const llama_batch& batch, bool is_prompt_warmup) {
int32_t mtp_update_kv_cache(struct llama_context * ctx, const llama_batch& batch, bool is_prompt_warmup, int32_t mtp_heads) {
if (batch.n_tokens == 0) {
return 0;
}
@@ -2962,12 +3088,18 @@ int32_t mtp_update_kv_cache(struct llama_context * ctx, const llama_batch& batch
}
mtp_batch.logits[mtp_batch.n_tokens-1] = true;
if (is_prompt_warmup) {
llama_set_mtp_n_heads(ctx, mtp_heads);
llama_set_mtp_step_idx(ctx, 0);
llama_set_mtp_op_type(ctx, MTP_OP_WARMUP);
} else {
llama_set_mtp_n_heads(ctx, mtp_heads);
llama_set_mtp_step_idx(ctx, 0);
llama_set_mtp_op_type(ctx, MTP_OP_UPDATE_ACCEPTED);
}
const int32_t ret = llama_decode(ctx, mtp_batch);
llama_set_mtp_step_idx(ctx, 0);
llama_set_mtp_n_heads(ctx, 0);
llama_set_mtp_op_type(ctx, MTP_OP_NONE);
return ret;
}
common/speculative.h
+10
View File
@@ -35,6 +35,7 @@ struct common_speculative_checkpoint {
struct common_speculative_draft_result {
llama_tokens tokens;
common_speculative_type type = COMMON_SPECULATIVE_TYPE_NONE;
bool target_only = false;
};
// comma separated list of all types
@@ -82,6 +83,13 @@ void common_speculative_free(common_speculative * spec);
// optionally call once at the beginning of a new generation
void common_speculative_begin(common_speculative * spec, const llama_tokens & prompt);
// apply per-request runtime parameters before prompt warmup can touch companion state
void common_speculative_prepare_request(common_speculative * spec, common_params_speculative & params);
// true when the active request drafts with more MTP heads than the cached prefix was
// warmed with; the caller must then reprocess the prompt from position 0
bool common_speculative_mtp_requires_fresh_warmup(const common_speculative * spec);
// sample up to n_draft tokens and add them to the batch using the draft model
// draft_base_pos/draft_seq_id override the MTP position for id_last
llama_tokens common_speculative_draft(
@@ -101,6 +109,8 @@ common_speculative_draft_result common_speculative_draft_ex(
llama_pos draft_base_pos = -1,
llama_seq_id draft_seq_id = 0);
int common_speculative_get_configured_n_max(const common_speculative * spec);
// informs the speculative decoder that n_accepted tokens were accepted by the target model
void common_speculative_accept(common_speculative * spec, uint16_t n_accepted);
convert_hf_to_gguf.py
+145
View File
@@ -592,6 +592,9 @@ class Model:
if chkhsh == "66b8d4e19ab16c3bfd89bce5d785fb7e0155e8648708a1f42077cb9fe002c273":
# ref: https://huggingface.co/alvarobartt/grok-2-tokenizer
res = "grok-2"
if chkhsh == "65df2fe396b537a53433301848c0a739f56d56f67ad3d35eba27961ac33c12bb":
# ref: https://huggingface.co/openpangu/openPangu-2.0-Flash
res = "openpangu"
if chkhsh == "972da7b59cec44d1f0a490a86c96df53859e486e481563e5dddac155013d87ac":
# ref: https://huggingface.co/poolside/Laguna-XS.2
res = "laguna"
@@ -4588,6 +4591,148 @@ class DeepseekV2Model(Model):
raise ValueError(f"Unprocessed experts: {experts}")
@Model.register("OpenPanguV2ForCausalLM")
class OpenPanguV2Model(DeepseekV2Model):
# openPangu-2.0-Flash: MLA + DSA/SWA hybrid + MoE + mHC(Hyper-Connections) + MoME convs.
# Emits a complete, self-contained GGUF: weights (incl. pre-split attn_k_b/attn_v_b for
# the latent-attention graph) plus the DSA/SWA schedule and mHC/MoME/sink metadata.
model_arch = gguf.MODEL_ARCH.OPENPANGU
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
# MTP/NextN layers (num_hidden_layers .. +num_nextn_predict_layers-1) are real blocks
# in ik_llama's layout (n_layer includes NextN; n_layer_kv_from_start excludes them).
self._nextn = int(self.hparams.get("num_nextn_predict_layers", 0) or 0)
self.block_count = int(self.hparams["num_hidden_layers"]) + self._nextn
self.tensor_map = gguf.get_tensor_name_map(self.model_arch, self.block_count)
def set_vocab(self):
# OpenPanguV2Tokenizer is a GPT2/BPE-style tokenizer. The pre-tokenizer hash is
# registered in Model.get_vocab_base_pre (see the openpangu entry).
self._set_vocab_gpt2()
# HF prepends <|pangu_text_start|> via the tokenizer post-processor; tokenizer_config
# has no add_bos_token key, so state it explicitly for the GGUF.
self.gguf_writer.add_add_bos_token(True)
def set_gguf_parameters(self):
# Base transformer params (block_count now includes NextN layers).
Model.set_gguf_parameters(self)
hparams = self.hparams
arch = gguf.MODEL_ARCH_NAMES[self.model_arch]
self.gguf_writer.add_leading_dense_block_count(hparams["first_k_dense_replace"])
self.gguf_writer.add_vocab_size(hparams["vocab_size"])
self.gguf_writer.add_q_lora_rank(hparams["q_lora_rank"])
self.gguf_writer.add_kv_lora_rank(hparams["kv_lora_rank"])
self.gguf_writer.add_key_length(hparams["qk_nope_head_dim"] + hparams["qk_rope_head_dim"])
self.gguf_writer.add_value_length(hparams["v_head_dim"])
# MoE
self.gguf_writer.add_expert_feed_forward_length(hparams["moe_intermediate_size"])
self.gguf_writer.add_expert_count(hparams["n_routed_experts"])
self.gguf_writer.add_expert_used_count(hparams["num_experts_per_tok"])
self.gguf_writer.add_expert_shared_count(hparams["n_shared_experts"])
self.gguf_writer.add_expert_weights_scale(hparams["routed_scaling_factor"])
self.gguf_writer.add_expert_weights_norm(hparams["norm_topk_prob"])
# router_enable_expert_bias => sigmoid gating with e_score_correction bias
self.gguf_writer.add_expert_gating_func(gguf.ExpertGatingFuncType.SIGMOID)
# RoPE
self.gguf_writer.add_rope_dimension_count(hparams["qk_rope_head_dim"])
self.gguf_writer.add_rope_freq_base(hparams["rope_theta"])
# NextN / MTP
self.gguf_writer.add_uint32(
gguf.Keys.LLM.NEXTN_PREDICT_LAYERS.format(arch=arch), self._nextn
)
# DSA lightning indexer
self.gguf_writer.add_uint32(
gguf.Keys.Attention.INDEXER_HEAD_COUNT.format(arch=arch), hparams["index_n_heads"]
)
self.gguf_writer.add_uint32(
gguf.Keys.Attention.INDEXER_KEY_LENGTH.format(arch=arch), hparams["index_head_dim"]
)
self.gguf_writer.add_uint32(
gguf.Keys.Attention.INDEXER_TOP_K.format(arch=arch), hparams["index_topk"]
)
# SWA (window; last few layers widen to 2048 per sliding_window_list)
if hparams.get("sliding_window") is not None:
self.gguf_writer.add_sliding_window(hparams["sliding_window"])
# Pangu-specific structural metadata (consumed by the OPENPANGU graph). The runtime
# derives everything else from these plus tensor presence: DSA layers are the
# windowless base layers (dsa_layers is redundant with swa_layers, so it is not
# written), and block_post_norm placement follows the tensors themselves.
self.gguf_writer.add_uint32(f"{arch}.mhc_num_stream", hparams["mhc_num_stream"])
self.gguf_writer.add_uint32(f"{arch}.mhc_recur_norm", hparams["mhc_recur_norm"])
self.gguf_writer.add_uint32(f"{arch}.param_sink_number", hparams["param_sink_number"])
self.gguf_writer.add_array(f"{arch}.swa_layers", hparams["swa_layers"])
if hparams.get("sliding_window_list") is not None:
self.gguf_writer.add_array(f"{arch}.sliding_window_list", hparams["sliding_window_list"])
_experts: list[dict[str, Tensor]] | None = None
def modify_tensors(self, data_torch: Tensor, name: str, bid: int | None) -> Iterable[tuple[str, Tensor]]:
# Sigmoid router bias: rename to the deepseek-style ".bias" the tensor map expects.
if name.endswith("e_score_correction_bias"):
name = name.replace("e_score_correction_bias", "e_score_correction.bias")
# NOTE: unlike DeepseekV2, we do NOT skip MTP layers — they are real blocks here.
# Merge routed experts into stacked 3D tensors (same layout as deepseek/glm4moe).
if name.find("mlp.experts") != -1:
n_experts = self.hparams["n_routed_experts"]
assert bid is not None
if self._experts is None:
self._experts = [{} for _ in range(self.block_count)]
self._experts[bid][name] = data_torch
if len(self._experts[bid]) >= n_experts * 3:
tensors: list[tuple[str, Tensor]] = []
for w_name in ["down_proj", "gate_proj", "up_proj"]:
datas: list[Tensor] = []
for xid in range(n_experts):
ename = f"model.layers.{bid}.mlp.experts.{xid}.{w_name}.weight"
datas.append(self._experts[bid][ename])
del self._experts[bid][ename]
data_torch = torch.stack(datas, dim=0)
merged_name = f"model.layers.{bid}.mlp.experts.{w_name}.weight"
tensors.append((self.map_tensor_name(merged_name), data_torch))
return tensors
else:
return []
# Split the fused MLA kv_b into k_b / v_b (deepseek MLA layout). OpenPangu's
# graph consumes the pre-split tensors directly, so do not emit the fused copy.
if name.endswith("kv_b_proj.weight"):
name_kb = name.replace("kv_b_proj", "k_b_proj")
name_vb = name.replace("kv_b_proj", "v_b_proj")
n_head_kv = self.hparams["num_attention_heads"]
v_head_dim = self.hparams["v_head_dim"]
qk_nope_head_dim = self.hparams["qk_nope_head_dim"]
assert data_torch.shape[0] == n_head_kv * (v_head_dim + qk_nope_head_dim)
kv_b = data_torch.view(n_head_kv, v_head_dim + qk_nope_head_dim, data_torch.shape[-1])
k_b, v_b = torch.split(kv_b, [qk_nope_head_dim, v_head_dim], dim=1)
k_b = k_b.transpose(1, 2)
k_b = k_b.reshape(n_head_kv * data_torch.shape[-1], qk_nope_head_dim)
v_b = v_b.reshape(n_head_kv * v_head_dim, data_torch.shape[-1])
return [
(self.map_tensor_name(name_kb), k_b),
(self.map_tensor_name(name_vb), v_b),
]
# Everything else (attn/norms/mHC/MoME conv/param-sink/indexer/nextn) maps by name.
return [(self.map_tensor_name(name), data_torch)]
@Model.register("T5WithLMHeadModel")
@Model.register("T5ForConditionalGeneration")
@Model.register("MT5ForConditionalGeneration")
convert_hf_to_gguf_update.py
+1
View File
@@ -107,6 +107,7 @@ models = [
{"name": "minimax-m2", "tokt": TOKENIZER_TYPE.BPE, "repo": "https://huggingface.co/MiniMaxAI/MiniMax-M2", },
{"name": "mellum2", "tokt": TOKENIZER_TYPE.BPE, "repo": "https://huggingface.co/JetBrains/Mellum2-12B-A2.5B-Base", },
{"name": "gpt-4o", "tokt": TOKENIZER_TYPE.BPE, "repo": "https://huggingface.co/openai/gpt-oss-20b", "chkhsh": "ccc2ef013c104be7bae2965776d611e1d7a8a2a9c547dd93a682c9a9fc80352e", }, # o200k_harmony shares the GPT-4o pre-tokenizer
{"name": "openpangu", "tokt": TOKENIZER_TYPE.BPE, "repo": "https://huggingface.co/openpangu/openPangu-2.0-Flash", "chkhsh": "65df2fe396b537a53433301848c0a739f56d56f67ad3d35eba27961ac33c12bb", }, # primary source: https://gitcode.com/ascend-tribe/openPangu-2.0-Flash
]
docs/parameters.md
+18 -10
View File
@@ -59,9 +59,9 @@ Some often used terms.
| full gpu | All processes offloaded to the GPU. |
| hybrid cpu/gpu | Partial offload to the GPU. |
| RAG | Retrieval Augmented Generation. Provide external documents to the LLM for information lookup. |
| MCP | Model Context Protocol ), an [open standard](https://en.wikipedia.org/wiki/Model_Context_Protocol) for the way artificial intelligence (AI) systems like large language models (LLMs) integrate and share data with external tools, systems, and data sources |
| MCP | Model Context Protocol, an [open standard](https://en.wikipedia.org/wiki/Model_Context_Protocol) for the way artificial intelligence (AI) systems like large language models (LLMs) integrate and share data with external tools, systems, and data sources |
| AI agent | Tool/program that uses LLM to achieve a goal/task via a series of planning/steps/actions/tool-calling/etc. `Coding agents` are specialized in software goals. |
| Agent harness | The tools and the infrastructure around the LLM in an AI Agent. `AI Agent = LLM+ Agent harness` |
| Agent harness | The tools and the infrastructure around the LLM in an AI Agent. `AI Agent = LLM + Agent harness` |
## General Parameters
@@ -84,8 +84,12 @@ Some often used terms.
| `-dr, --dry-run` | Skip loading tensors in the files | - | Skips loading files, yet still report OOM error and print memory usage correctly, which is helpful for manually tuning of very large models. |
| `--minilog` | Print important information | - | For `llama-server`, log request message for completions/response/anthropic and response. The prompt in the json format and the text response are saved in the log file and printed to the console. [PR 1477](https://github.com/ikawrakow/ik_llama.cpp/pull/1477) |
| `-fa, --flash-attn` | Enables Flash Attention | on | auto / on / off Improves t/s and reduces memory usage. |
| `--no-fa, --no-flash-attn` | Disable Flash Attention | | Alternative parameter to turn of FA. See `--flash-attn` |
| `--no-fa, --no-flash-attn` | Disable Flash Attention | | Alternative parameter to turn off FA. See `--flash-attn` |
| `-mla, --mla-use` | Enable MLA | 3 | 0 / 1 / 2 / 3 For DeepSeek models, and other recent models that are using MLA. [PR 188](https://github.com/ikawrakow/ik_llama.cpp/pull/188) [PR 205](https://github.com/ikawrakow/ik_llama.cpp/pull/205) [PR 235](https://github.com/ikawrakow/ik_llama.cpp/pull/235) [PR 243](https://github.com/ikawrakow/ik_llama.cpp/pull/243) [PR 252](https://github.com/ikawrakow/ik_llama.cpp/pull/252) [PR 253](https://github.com/ikawrakow/ik_llama.cpp/pull/253) [PR 273](https://github.com/ikawrakow/ik_llama.cpp/pull/273) [PR 386](https://github.com/ikawrakow/ik_llama.cpp/pull/386) [PR 497](https://github.com/ikawrakow/ik_llama.cpp/pull/497) [PR 943](https://github.com/ikawrakow/ik_llama.cpp/pull/943) [PR 1821](https://github.com/ikawrakow/ik_llama.cpp/pull/1821) |
| `--dsa, -dsa` | | off | Enable GLM DSA sparse attention [PR 2045](https://github.com/ikawrakow/ik_llama.cpp/pull/2045) |
| `--dsa-top-k, -dsatk` | DSA top-k override | -1 | `<0` uses the model's configured `indexer_top_k` [PR 2045](https://github.com/ikawrakow/ik_llama.cpp/pull/2045) |
| `--indexer-cache-type-k type, -ictk` | Indexer K-cache data type | off | Use quantized indexer cache [PR 2075](https://github.com/ikawrakow/ik_llama.cpp/pull/2075) |
| `--fused-indexer-topk, -fidx` | Enable the fused indexer topk op | disabled | Use a dedicated op for computing the DSA indexer top_k KV cache entries [PR 2098](https://github.com/ikawrakow/ik_llama.cpp/pull/2098) |
| `-amb, --attention-max-batch` | Max batch size for attention computations | 0 | Specifies the maximum K*Q size in MB we want to tolerate. [PR 237](https://github.com/ikawrakow/ik_llama.cpp/pull/237) |
| `-fmoe or --fused-moe` | Fused MoE ffn_up and ffn_gate | - | Speedup for MoE models. [PR 229](https://github.com/ikawrakow/ik_llama.cpp/pull/229) |
| `--no-fmoe, --no-fused-moe` | Disable fused MoE | Enabled | See `--fused-moe` |
@@ -108,8 +112,10 @@ Some often used terms.
| `--ui-mcp-proxy, --webui-mcp-proxy` | Experimental: whether to enable MCP CORS proxy - do not enable in untrusted environments | disabled | Support CORS Proxy on llama-server backend side. It is required to make external mcp server work on llamacpp webui. [PR 1904](https://github.com/ikawrakow/ik_llama.cpp/pull/1904) |
| `--defer-experts` | Defer expert mmap residency on Linux to reduce model load time | false | Using this flag, expert tensor pages are faulted in on demand rather than being eagerly loaded during initialization. This allows us to reduce cold-start latency, thus improving the load time of MoE models, particularly on systems where users are running models off of storage. [PR 1634](https://github.com/ikawrakow/ik_llama.cpp/pull/1634) |
| `-rtr, --run-time-repack` | Repack tensors if interleaved variant is available | - | May improve performance on some systems. [PR 147](https://github.com/ikawrakow/ik_llama.cpp/pull/147) |
| `--ctx-checkpoints` | set the number of checkpoints per slot | - | enable checkpoint for recurrent models Qwen3-Next and Qwen3.5-MoE. [PR 1310](https://github.com/ikawrakow/ik_llama.cpp/pull/1310) |
| `--ctx-checkpoints-interval` | minimum number of tokens between each context checkpoint. | - | If you want to create the checkpoint more frequently, set it to a small value. If it's set to positive number, it saves checkpoints during TG at this interval. During PP, it can only save checkpoint every batch size, so it becomes minimum number of tokens between each context checkpoint. [PR 1310](https://github.com/ikawrakow/ik_llama.cpp/pull/1310) |
| `--ctx-checkpoints N` | Set the number of checkpoints per slot | 32 | Enable checkpoint for recurrent models Qwen3-Next and Qwen3.5-MoE. [PR 1310](https://github.com/ikawrakow/ik_llama.cpp/pull/1310) |
| `--ctx-checkpoints-interval N` | Minimum number of tokens between each context checkpoint. | 512 | If you want to create the checkpoint more frequently, set it to a small value. If it's set to positive number, it saves checkpoints during TG at this interval. During PP, it can only save checkpoint every batch size, so it becomes minimum number of tokens between each context checkpoint. [PR 1310](https://github.com/ikawrakow/ik_llama.cpp/pull/1310) |
| `--ctx-checkpoints-tolerance N` | The number of tokens before the full prompt to create the checkpoint | 5 | Creates the checkpoint N tokens before the prompt is fully processed to reduce prompt process for Qwen 3.5 thinking models. [PR 1346](https://github.com/ikawrakow/ik_llama.cpp/pull/1346) |
| `--ctx-checkpoints-eviction NAME` | Eviction strategy for checkpoint. | `variance` | Accepts `fifo`, `variance` and `auto`. `Variance` preserves coverage and maintains uniform interval. [PR 2020](https://github.com/ikawrakow/ik_llama.cpp/pull/2020) |
## Speculative Decoding
@@ -126,7 +132,7 @@ Check the details [here](./speculative.md).
| `-ctkd, --cache-type-k-draft TYPE` | KV cache data type for K for the draft model | - | For draft model, see: `-ctk` |
| `-ctvd, --cache-type-v-draft TYPE` | KV cache data type for V for the draft model | - | For draft model, see: `-ctk` |
| `-draft, --draft-params` | Comma-separated list of draft model parameters | - | |
| `--spec-type SPEC[:k=v,...]` | Canonical speculative stage entry; repeat to configure the supported two-stage chain | - | Types: `none`, `draft`, `mtp`, `ngram-cache`, `ngram-simple`, `ngram-map-k`, `ngram-map-k4v`, `ngram-mod`, `suffix`. Canonical keys: `n_max`, `n_min`, `p_min`, `ngram_size_n`, `ngram_size_m`, `ngram_min_hits`, `suffix_min_match_len`, `suffix_max_depth`, `suffix_corpus`. String values may escape commas as `\,` or quote the value inside the stage payload. Example: `--spec-type ngram-mod:n_max=64,n_min=2,ngram_size_n=8 --spec-type mtp:n_max=1,p_min=0.0` |
| `--spec-type SPEC[:k=v,...]` | Canonical speculative stage entry; repeat to configure the supported two-stage chain | - | Types: `none`, `draft`, `dflash`, `mtp`, `ngram-cache`, `ngram-simple`, `ngram-map-k`, `ngram-map-k4v`, `ngram-mod`, `suffix`. Canonical keys include `n_max`, `n_min`, `p_min`, `heads`, `cross_ctx`, `ngram_size_n`, `ngram_size_m`, `ngram_min_hits`, `suffix_min_match_len`, `suffix_max_depth`, `suffix_corpus`. For MTP, `heads=1` is the default; values above `1` and `heads=0` (all model heads) are experimental. String values may escape commas as `\,` or quote the value inside the stage payload. Examples: `--spec-type ngram-mod:n_max=64,n_min=2,ngram_size_n=8 --spec-type mtp:n_max=1,p_min=0.0`, `--model-draft draft.gguf --spec-type dflash:n_max=4,cross_ctx=512` |
| `--spec-autotune` | Automatically tune speculative params to maximize tokens/sec | - | Automatically determines the near-optimal arguments for the type of speculation being performed [PR 1595](https://github.com/ikawrakow/ik_llama.cpp/pull/1595) |
| `--recurrent-ckpt-mode MODE` | Checkpoint strategy for recurrent/hybrid speculative decoding | auto | One of: - `auto` auto-select: per-step if CUDA full-GPU, gpu-fallback otherwise - `per-step` save SSM state per draft step in VRAM; no re-decode on rejection - `gpu-fallback` copy state to GPU buffer; re-decode on rejection - `cpu` serialise state via llama_state_seq; re-decode on rejection [PR 1669](https://github.com/ikawrakow/ik_llama.cpp/pull/1669) [PR 1774](https://github.com/ikawrakow/ik_llama.cpp/pull/1774) |
@@ -171,7 +177,7 @@ Incorrect prompt template or it's format may break the model output.
| - | - | - | - |
| `--jinja` | Set custom jinja chat template | Template taken from model's metadata | Mandatory for Tool Calling. |
| `--chat-template JINJA_TEMPLATE` | Use jinja template for chat | Disabled | If there is no official `tool_use` Jinja template, you may want to set `--chat-template chatml` to use a default that works with many models |
| `--chat-template-file file_with_JINJA_TEMPLATE` | Load jinja template for chat from the file | - | Sometimes the model producer or community fixes the template after the GGUF files are released, therefore it’ metadata contains buggy version. To avoid re-downloading the entire model file, download only the .jinja file the use it (`--chat-template-file /models/Qwen_Qwen3-Coder-30B-A3B-Instruct-fixed.jinja`). |
| `--chat-template-file file_with_JINJA_TEMPLATE` | Load jinja template for chat from the file | - | Sometimes the model producer or community fixes the template after the GGUF files are released, therefore its metadata contains buggy version. To avoid re-downloading the entire model file, download only the .jinja file then use it (`--chat-template-file /models/Qwen_Qwen3-Coder-30B-A3B-Instruct-fixed.jinja`). |
| `--reasoning-format FORMAT` | Controls whether thought tags are allowed and/or extracted from the response | none | One of: - `none` leaves thoughts unparsed in `message.content` - `deepseek` puts thoughts in `message.reasoning_content` (except in streaming mode, which behaves as `none`) - `deepseek-legacy` keeps `<think>` tags in `message.content` while also populating `message.reasoning_content`. This is useful when the frontend (including agents) is hardcoded to use just a specific format. |
| `--chat-template-kwargs JSON` | Sets additional params for the json template parser | - | Example for gpt-oss: `--chat-template-kwargs '{"reasoning_effort": "medium"}'` |
| `--reasoning-budget N` | Controls the amount of thinking allowed | -1 (unrestricted) | 0 (disable thinking) |
@@ -233,7 +239,7 @@ Therefore, the "offloading" term is used when sending some processing to another
As the GPUs (including their VRAM) are more powerful for LLM specific processing than CPU+RAM, the aim is to offload as much as possible to the GPU.
Beside the improved quants (better quality and performance at the same size; usable low BPW), superior performance (faster PP ang TG), ik_llama.cpp really shines at providing:
Besides the improved quants (better quality and performance at the same size; usable low BPW), superior performance (faster PP and TG), ik_llama.cpp really shines at providing:
- Detailed output log which e.g. includes layers and buffers sizes to support offload calculations.
- A big collection of parameters to tweak offloading (what/where runs: processing, tensors, KV cache, operations, etc.).
- Split mode `graph` when multiple GPUs are available, including mixes of different GPU types, various VRAM sizes.
@@ -328,7 +334,7 @@ python3 gguf-py/scripts/gguf_dump.py /models/Qwen_Qwen3-0.6B-IQ4_NL.gguf
`-ngl 999` To put all layers in VRAM by default
`-ot "blk.(?:[0-9]|[1-7][0-9]|[8][0-7]).ffn._exps.=CPU"` To create exceptions and put back in ram anything that has "ffn" and "_exps" in its name, and that sits in layers called "blk.n", where "n" (the lawyer number) is any match between 0 and 9, or between 1 to 7 + 0 to 9 (aka a number between 10 and 79), or 8 + 0 to 7 (aka a number between 80 and 87).
`-ot "blk.(?:[0-9]|[1-7][0-9]|[8][0-7]).ffn._exps.=CPU"` To create exceptions and put back in ram anything that has "ffn" and "_exps" in its name, and that sits in layers called "blk.n", where "n" (the layer number) is any match between 0 and 9, or between 1 to 7 + 0 to 9 (aka a number between 10 and 79), or 8 + 0 to 7 (aka a number between 80 and 87).
Basically a complicated way of saying put all experts from layer 0 to 87 in ram. Experts from layer 88 to 93 (there's 93 layers in qwen3vl 235b) can sit in VRAM still. (Thats all I can load on a 5090).
C. Other tips
@@ -377,7 +383,7 @@ WIP
| `--override-kv KEY=TYPE:VALUE` | Override model metadata by key | - | Advanced option to override model metadata by key. May be specified multiple times. types: int, float, bool, str. Example: `--override-kv tokenizer.ggml.add_bos_token=bool:false` |
| `-m, --model FNAME` | Model path | models/$filename | Mandatory, the GGUF model file to be served. |
| `-md, --model-draft FNAME` | Draft model for speculative decoding | unused | Required when an explicit `draft` stage is used. |
| `--spec-type SPEC[:k=v,...]` | Canonical speculative stage entry; repeat for the supported two-stage chain | none | Use stage-local keys like `n_max`, `n_min`, `p_min`, `ngram_size_n`, `ngram_size_m`, `ngram_min_hits`, `suffix_min_match_len`, `suffix_max_depth`, and `suffix_corpus`. |
| `--spec-type SPEC[:k=v,...]` | Canonical speculative stage entry; repeat for the supported two-stage chain | none | Use stage-local keys like `n_max`, `n_min`, `p_min`, `heads`, `ngram_size_n`, `ngram_size_m`, `ngram_min_hits`, `suffix_min_match_len`, `suffix_max_depth`, and `suffix_corpus`. For MTP, `heads=1` is the default; values above `1` and `heads=0` (all model heads) are experimental. |
### Request-Level Speculative Overrides
@@ -506,6 +512,7 @@ cmake --build build --config Release -j$(nproc)
| `-DGGML_NATIVE=ON` | Turn it off when cross-compiling. |
| `-DGGML_NCCL=OFF` | To disable usage of NCCL. |
| `-DGGML_MAX_CONTEXTS=2048` | Only need this if you are planning to use quants generated with the Thireus quantization suite |
| `-DGGML_MAX_SRC=N` | The maximum number of GPUs. |
### Environment variables
@@ -519,6 +526,7 @@ CUDA_VISIBLE_DEVICES=0,2 llama-server -m /models/model-bf16.gguf
| - | - |
| CUDA_VISIBLE_DEVICES | Use only specified GPUs. Example: Use first and 3rd `CUDA_VISIBLE_DEVICES=0,2` |
| GGML_CUDA_NO_PINNED | Do not use pinned memory |
| GGML_CUDA_HOST_MALLOC_THP | Use THP for host allocations with GGML_CUDA_HOST_MALLOC_THP [PR 2010](https://github.com/ikawrakow/ik_llama.cpp/pull/2010) |
## Unique parameters
docs/speculative.md
+1
View File
@@ -128,6 +128,7 @@ Canonical stage keys:
| `n_max` | Maximum drafted tokens for that stage |
| `n_min` | Minimum usable drafted tokens for that stage |
| `p_min` | Minimum speculative probability threshold |
| `heads` | MTP heads to use; `1` is the default, while values above `1` and `0` (all model heads) are experimental |
| `ngram_size_n` | Lookup n-gram size |
| `ngram_size_m` | Draft m-gram size |
| `ngram_min_hits` | Minimum matching hits for n-gram map stages |
examples/main/main.cpp
+43 -21
View File
@@ -941,17 +941,20 @@ int main(int argc, char ** argv) {
const int n_predict_budget = n_remain < 0 ? std::numeric_limits<int>::max() : n_remain;
bool used_speculative = false;
const bool sampled_before_from_carry = have_speculative_sampled;
llama_token sampled_before = LLAMA_TOKEN_NULL;
if (sampled_before_from_carry) {
sampled_before = speculative_sampled;
have_speculative_sampled = false;
speculative_sampled = LLAMA_TOKEN_NULL;
}
bool sampled_before_ready = sampled_before_from_carry;
if (spec != nullptr && n_predict_budget != 1) {
const bool sampled_before_from_carry = have_speculative_sampled;
llama_token sampled_before = LLAMA_TOKEN_NULL;
if (sampled_before_from_carry) {
sampled_before = speculative_sampled;
have_speculative_sampled = false;
speculative_sampled = LLAMA_TOKEN_NULL;
} else {
if (!sampled_before_ready) {
sampled_before = common_sampler_sample_legacy(ctx_sampling, ctx, ctx_guidance);
common_sampler_accept(ctx_sampling, ctx, sampled_before, /* apply_grammar= */ true);
sampled_before_ready = true;
}
static const llama_tokens empty_speculative_tokens;
const llama_tokens & draft_history =
@@ -981,7 +984,7 @@ int main(int argc, char ** argv) {
const int min_usable_draft = params.speculative.get_min_usable_stage_n_min();
if ((int) draft.size() >= min_usable_draft && (!draft.empty() || n_predict_budget > 1)) {
if (llama_model_has_recurrent(model)) {
if (llama_model_has_recurrent(model) || llama_model_is_openpangu(model)) {
if (!common_speculative_before_draft(
spec,
model,
@@ -998,7 +1001,7 @@ int main(int argc, char ** argv) {
}
}
if (!draft.empty()) {
if (!draft.empty() || draft_result.target_only) {
llama_batch verify_batch = llama_batch_init((int) draft.size() + 1, 0, 1);
std::vector<int> verify_indices;
verify_indices.reserve(draft.size() + 1);
@@ -1066,23 +1069,42 @@ int main(int argc, char ** argv) {
}
if (!used_speculative) {
const llama_token id = common_sampler_sample_legacy(ctx_sampling, ctx, ctx_guidance);
common_sampler_accept(ctx_sampling, ctx, id, /* apply_grammar= */ true);
if (sampled_before_ready) {
// A carried token was already displayed by the previous speculative step but
// still needs to be decoded. If a new draft is unusable, stage that token instead
// of sampling the unchanged logits again and emitting it twice.
embd.push_back(sampled_before);
embd_is_prompt = false;
input_echo = true;
LOG("last: %s\n", LOG_TOKENS_TOSTR_PRETTY(ctx, ctx_sampling->prev).c_str());
if (sampled_before_from_carry) {
emitted_generated = false;
} else {
emitted = embd;
emitted_generated = true;
--n_remain;
}
embd.push_back(id);
emitted = embd;
embd_is_prompt = false;
emitted_generated = true;
LOG("n_remain: %d\n", n_remain);
} else {
const llama_token id = common_sampler_sample_legacy(ctx_sampling, ctx, ctx_guidance);
common_sampler_accept(ctx_sampling, ctx, id, /* apply_grammar= */ true);
// echo this to console
input_echo = true;
LOG("last: %s\n", LOG_TOKENS_TOSTR_PRETTY(ctx, ctx_sampling->prev).c_str());
// decrement remaining sampling budget
--n_remain;
embd.push_back(id);
emitted = embd;
embd_is_prompt = false;
emitted_generated = true;
LOG("n_remain: %d\n", n_remain);
// echo this to console
input_echo = true;
// decrement remaining sampling budget
--n_remain;
LOG("n_remain: %d\n", n_remain);
}
} else {
input_echo = true;
}
examples/mtmd/clip-impl.h
+3
View File
@@ -40,6 +40,7 @@
#define KEY_FEATURE_LAYER "clip.vision.feature_layer"
#define KEY_PROJ_SCALE_FACTOR "clip.vision.projector.scale_factor"
#define KEY_SPATIAL_MERGE_SIZE "clip.vision.spatial_merge_size"
#define KEY_TEMPORAL_PATCH_SIZE "clip.vision.temporal_patch_size"
#define KEY_IS_DEEPSTACK_LAYERS "clip.vision.is_deepstack_layers"
#define KEY_MM_PATCH_MERGE_TYPE "clip.vision.mm_patch_merge_type"
@@ -167,6 +168,7 @@ enum projector_type {
PROJECTOR_TYPE_LIGHTONOCR,
PROJECTOR_TYPE_COGVLM,
PROJECTOR_TYPE_JANUS_PRO,
PROJECTOR_TYPE_MINIMAX_M3_VL,
PROJECTOR_TYPE_UNKNOWN,
};
@@ -197,6 +199,7 @@ static std::map<projector_type, std::string> PROJECTOR_TYPE_NAMES = {
{ PROJECTOR_TYPE_LIGHTONOCR,"lightonocr"},
{ PROJECTOR_TYPE_COGVLM, "cogvlm"},
{ PROJECTOR_TYPE_JANUS_PRO, "janus_pro"},
{ PROJECTOR_TYPE_MINIMAX_M3_VL, "minimax_m3_vl"},
};
static projector_type clip_projector_type_from_string(const std::string & str) {
examples/mtmd/clip.cpp
+194 -4
View File
@@ -192,6 +192,7 @@ struct clip_hparams {
int32_t image_min_pixels = -1;
int32_t image_max_pixels = -1;
int32_t n_merge = 0; // number of patch merges **per-side**
int32_t temporal_patch_size = 1;
float image_mean[3];
float image_std[3];
@@ -758,6 +759,101 @@ struct clip_graph {
return gf;
}
ggml_cgraph * build_minimax_m3_vl() {
GGML_ASSERT(model.patch_bias == nullptr);
GGML_ASSERT(model.class_embedding == nullptr);
GGML_ASSERT(model.patch_embeddings_0 != nullptr);
GGML_ASSERT(model.patch_embeddings_1 != nullptr);
GGML_ASSERT(hparams.n_merge == 2);
GGML_ASSERT(img.nx % (patch_size * hparams.n_merge) == 0);
GGML_ASSERT(img.ny % (patch_size * hparams.n_merge) == 0);
const int batch_size = 1;
// MiniMax-M3 uses 3-axis NEOX RoPE. Each axis gets an even slice of the
// head dim; any remainder is left unrotated.
const int rope_dims = 2 * (d_head / 2);
const int axis_dim = 2 * ((rope_dims / 3) / 2);
const int rot_dim = 3 * axis_dim;
ggml_tensor * rope_cos = ggml_new_tensor_3d(ctx0, GGML_TYPE_F32, d_head, 1, n_patches);
ggml_set_name(rope_cos, "rope_cos");
ggml_set_input(rope_cos);
ggml_tensor * rope_sin = ggml_new_tensor_3d(ctx0, GGML_TYPE_F32, d_head, 1, n_patches);
ggml_set_name(rope_sin, "rope_sin");
ggml_set_input(rope_sin);
ggml_tensor * inp_raw = build_inp_raw();
// The generic convolution path converts im2col output to f16. MiniMax-M3's
// patch embedding is sensitive to that loss, so keep the accumulation in f32.
auto conv_2d_f32 = [&](ggml_tensor * kernel) {
ggml_tensor * kernel_f32 = ggml_cast(ctx0, kernel, GGML_TYPE_F32);
ggml_tensor * col = ggml_im2col(
ctx0, kernel_f32, inp_raw,
patch_size, patch_size, 0, 0, 1, 1,
true, GGML_TYPE_F32);
ggml_tensor * cur = ggml_mul_mat(
ctx0,
ggml_reshape_2d(ctx0, col, col->ne[0], col->ne[3] * col->ne[2] * col->ne[1]),
ggml_reshape_2d(ctx0, kernel_f32, kernel_f32->ne[0] * kernel_f32->ne[1] * kernel_f32->ne[2], kernel_f32->ne[3]));
cur = ggml_reshape_4d(ctx0, cur, col->ne[1], col->ne[2], col->ne[3], kernel->ne[3]);
return ggml_cont(ctx0, ggml_permute(ctx0, cur, 0, 1, 3, 2));
};
ggml_tensor * inp = conv_2d_f32(model.patch_embeddings_0);
inp = ggml_add(ctx0, inp, conv_2d_f32(model.patch_embeddings_1));
inp = ggml_permute(ctx0, inp, 1, 2, 0, 3);
inp = ggml_cont_3d(ctx0, inp, n_embd, n_patches, batch_size);
cb(inp, "patch_embd", -1);
auto add_pos = [&](ggml_tensor * cur, const clip_layer &) {
return build_rope_vision_neox(ctx0, cur, rope_cos, rope_sin, rot_dim);
};
ggml_tensor * cur = build_vit(
inp,
n_patches,
NORM_TYPE_NORMAL,
hparams.ffn_op,
nullptr,
add_pos);
cur = ggml_mul_mat(ctx0, model.mm_0_w, cur);
cur = ggml_add(ctx0, cur, model.mm_0_b);
cur = ggml_gelu_erf(ctx0, cur);
cur = ggml_mul_mat(ctx0, model.mm_1_w, cur);
cur = ggml_add(ctx0, cur, model.mm_1_b);
// The reference processor groups patches by 2x2 merge blocks. We keep the
// ViT tokens in raster order, then reorder here before concatenating each
// merge block for patch_merge_mlp.
cur = ggml_reshape_4d(ctx0, cur,
hparams.projection_dim * hparams.n_merge,
n_patches_x / hparams.n_merge,
hparams.n_merge,
batch_size * (n_patches_y / hparams.n_merge));
cur = ggml_permute(ctx0, cur, 0, 2, 1, 3);
cur = ggml_cont(ctx0, cur);
cur = ggml_reshape_3d(ctx0, cur, hparams.projection_dim * hparams.n_merge * hparams.n_merge,
n_patches / (hparams.n_merge * hparams.n_merge), batch_size);
cur = ggml_mul_mat(ctx0, model.mm_2_w, cur);
cur = ggml_add(ctx0, cur, model.mm_2_b);
cur = ggml_gelu_erf(ctx0, cur);
cur = ggml_mul_mat(ctx0, model.mm_3_w, cur);
cur = ggml_add(ctx0, cur, model.mm_3_b);
cur = ggml_reshape_3d(ctx0, cur, hparams.projection_dim, n_patches / (hparams.n_merge * hparams.n_merge), batch_size);
ggml_build_forward_expand(gf, cur);
return gf;
}
// Qwen2VL and Qwen2.5VL use M-RoPE
ggml_cgraph * build_qwen2vl() {
GGML_ASSERT(model.patch_bias == nullptr);
@@ -2788,6 +2884,29 @@ private:
return cur;
}
// MiniMax-M3 vision RoPE uses rotate_half semantics over host-computed
// cos/sin tables. cur is [d_head, n_head, n_pos]; cos/sin broadcast over heads.
static ggml_tensor * build_rope_vision_neox(
ggml_context * ctx0, ggml_tensor * cur,
ggml_tensor * cos, ggml_tensor * sin, int rot_dim) {
const int64_t d_head = cur->ne[0];
const int64_t n_head = cur->ne[1];
const int64_t n_pos = cur->ne[2];
const int64_t half = rot_dim / 2;
const size_t es = ggml_element_size(cur);
ggml_tensor * first = ggml_cont(ctx0, ggml_view_3d(ctx0, cur, half, n_head, n_pos, cur->nb[1], cur->nb[2], 0));
ggml_tensor * second = ggml_cont(ctx0, ggml_view_3d(ctx0, cur, half, n_head, n_pos, cur->nb[1], cur->nb[2], half * es));
ggml_tensor * rotated = ggml_concat(ctx0, ggml_neg(ctx0, second), first, 0);
if (rot_dim < d_head) {
ggml_tensor * tail = ggml_cont(ctx0, ggml_view_3d(ctx0, cur, d_head - rot_dim, n_head, n_pos, cur->nb[1], cur->nb[2], rot_dim * es));
rotated = ggml_concat(ctx0, rotated, tail, 0);
}
return ggml_add(ctx0, ggml_mul(ctx0, cur, cos), ggml_mul(ctx0, rotated, sin));
}
// aka pixel_shuffle / pixel_unshuffle / patch_merger (Kimi-VL)
// support dynamic resolution
ggml_tensor * build_patch_merge_permute(ggml_tensor * cur, int scale_factor) {
@@ -2878,6 +2997,10 @@ static ggml_cgraph * clip_image_build_graph(clip_ctx * ctx, const clip_image_f32
{
res = graph.build_qwen3vl();
} break;
case PROJECTOR_TYPE_MINIMAX_M3_VL:
{
res = graph.build_minimax_m3_vl();
} break;
case PROJECTOR_TYPE_GEMMA4V:
{
res = graph.build_gemma4();
@@ -3242,6 +3365,24 @@ struct clip_model_loader {
LOG_WRN("%s: more info: https://github.com/ggml-org/llama.cpp/issues/16842\n\n", __func__);
}
} break;
case PROJECTOR_TYPE_MINIMAX_M3_VL:
{
hparams.rope_theta = 10000.0f;
hparams.n_merge = 2;
hparams.temporal_patch_size = 2;
hparams.ffn_op = FFN_GELU_ERF;
log_ffn_op = "gelu_erf";
get_u32(KEY_SPATIAL_MERGE_SIZE, hparams.n_merge, false);
get_u32(KEY_TEMPORAL_PATCH_SIZE, hparams.temporal_patch_size, false);
get_u32(KEY_IMAGE_MIN_PIXELS, hparams.image_min_pixels, false);
get_u32(KEY_IMAGE_MAX_PIXELS, hparams.image_max_pixels, false);
if (hparams.image_min_pixels <= 0 || hparams.image_max_pixels <= 0) {
hparams.set_limit_image_tokens(8, 576);
} else {
hparams.warmup_image_size = static_cast<int>(std::sqrt(hparams.image_max_pixels));
}
hparams.set_warmup_n_tokens(256);
} break;
case PROJECTOR_TYPE_GEMMA4V:
{
hparams.rope_theta = 100.0f;
@@ -3296,6 +3437,7 @@ struct clip_model_loader {
LOG_INF("%s: has_llava_proj: %d\n", __func__, hparams.has_llava_projector);
LOG_INF("%s: minicpmv_version: %d\n", __func__, hparams.minicpmv_version);
LOG_INF("%s: n_merge: %d\n", __func__, hparams.n_merge);
LOG_INF("%s: temporal_patch_size:%d\n", __func__, hparams.temporal_patch_size);
LOG_INF("%s: n_wa_pattern: %d\n", __func__, hparams.n_wa_pattern);
if (hparams.image_min_pixels > 0) {
LOG_INF("%s: image_min_pixels: %d%s\n", __func__, hparams.image_min_pixels, hparams.custom_image_min_tokens > 0 ? " (custom value)" : "");
@@ -3577,6 +3719,17 @@ struct clip_model_loader {
model.mm_1_w = get_tensor(string_format(TN_LLAVA_PROJ, 2, "weight"));
model.mm_1_b = get_tensor(string_format(TN_LLAVA_PROJ, 2, "bias"));
} break;
case PROJECTOR_TYPE_MINIMAX_M3_VL:
{
model.mm_0_w = get_tensor(string_format(TN_LLAVA_PROJ, 0, "weight"));
model.mm_0_b = get_tensor(string_format(TN_LLAVA_PROJ, 0, "bias"));
model.mm_1_w = get_tensor(string_format(TN_LLAVA_PROJ, 2, "weight"));
model.mm_1_b = get_tensor(string_format(TN_LLAVA_PROJ, 2, "bias"));
model.mm_2_w = get_tensor(string_format(TN_LLAVA_PROJ, 4, "weight"));
model.mm_2_b = get_tensor(string_format(TN_LLAVA_PROJ, 4, "bias"));
model.mm_3_w = get_tensor(string_format(TN_LLAVA_PROJ, 6, "weight"));
model.mm_3_b = get_tensor(string_format(TN_LLAVA_PROJ, 6, "bias"));
} break;
case PROJECTOR_TYPE_GEMMA3:
{
model.mm_input_proj_w = get_tensor(TN_MM_INP_PROJ);
@@ -4590,8 +4743,6 @@ private:
}
};
// returns the normalized float tensor for llava-1.5, for spatial_unpad with anyres processing for llava-1.6 it returns the normalized image patch tensors as a vector
// res_imgs memory is being allocated here, previous allocations will be freed if found
bool clip_image_preprocess(struct clip_ctx * ctx, const clip_image_u8 * img, struct clip_image_f32_batch * res_imgs) {
clip_image_size original_size{img->nx, img->ny};
auto & params = ctx->model.hparams;
@@ -4617,6 +4768,7 @@ bool clip_image_preprocess(struct clip_ctx * ctx, const clip_image_u8 * img, str
case PROJECTOR_TYPE_QWEN25VL:
case PROJECTOR_TYPE_QWEN3VL:
case PROJECTOR_TYPE_GEMMA4V:
case PROJECTOR_TYPE_MINIMAX_M3_VL:
{
GGML_ASSERT(params.image_min_pixels > 0 && params.image_max_pixels > 0);
clip_image_u8 resized;
@@ -4876,7 +5028,7 @@ const char * clip_patch_merge_type(const struct clip_ctx * ctx) {
int clip_n_output_tokens_x(const struct clip_ctx * ctx, struct clip_image_f32 * img) {
const auto & params = ctx->model.hparams;
const int n_total = clip_n_output_tokens(ctx, img);
if (ctx->proj_type() == PROJECTOR_TYPE_QWEN2VL || ctx->proj_type() == PROJECTOR_TYPE_QWEN25VL || ctx->proj_type() == PROJECTOR_TYPE_QWEN3VL) {
if (ctx->proj_type() == PROJECTOR_TYPE_QWEN2VL || ctx->proj_type() == PROJECTOR_TYPE_QWEN25VL || ctx->proj_type() == PROJECTOR_TYPE_QWEN3VL || ctx->proj_type() == PROJECTOR_TYPE_MINIMAX_M3_VL) {
return img->nx / (params.patch_size * 2);
}
return n_total;
@@ -4884,7 +5036,7 @@ int clip_n_output_tokens_x(const struct clip_ctx * ctx, struct clip_image_f32 *
int clip_n_output_tokens_y(const struct clip_ctx * ctx, struct clip_image_f32 * img) {
const auto & params = ctx->model.hparams;
if (ctx->proj_type() == PROJECTOR_TYPE_QWEN2VL || ctx->proj_type() == PROJECTOR_TYPE_QWEN25VL || ctx->proj_type() == PROJECTOR_TYPE_QWEN3VL) {
if (ctx->proj_type() == PROJECTOR_TYPE_QWEN2VL || ctx->proj_type() == PROJECTOR_TYPE_QWEN25VL || ctx->proj_type() == PROJECTOR_TYPE_QWEN3VL || ctx->proj_type() == PROJECTOR_TYPE_MINIMAX_M3_VL) {
return img->ny / (params.patch_size * 2);
}
return 1;
@@ -4942,6 +5094,7 @@ int clip_n_output_tokens(const struct clip_ctx * ctx, struct clip_image_f32 * im
case PROJECTOR_TYPE_QWEN2VL:
case PROJECTOR_TYPE_QWEN25VL:
case PROJECTOR_TYPE_QWEN3VL:
case PROJECTOR_TYPE_MINIMAX_M3_VL:
{
// dynamic size (2 conv, so double patch size)
int x_patch = img->nx / (params.patch_size * 2);
@@ -5390,6 +5543,41 @@ bool clip_image_batch_encode(clip_ctx * ctx, const int n_threads, const clip_ima
}
set_input_i32("pos_w", pos_data);
} break;
case PROJECTOR_TYPE_MINIMAX_M3_VL:
{
// 3-axis (T|H|W) NEOX RoPE. Still images use t=0, so the T
// band is identity; trailing dims keep the identity defaults.
const int n_head = hparams.n_head;
const int d_head = hparams.n_embd / n_head;
const int rope_dims = 2 * (d_head / 2);
const int axis_dim = 2 * ((rope_dims / 3) / 2);
const int n_freq = axis_dim / 2;
const int rot_dim = 3 * axis_dim;
const int half = rot_dim / 2;
const int grid_w = image_size_width / patch_size;
std::vector<float> inv_freq(n_freq);
for (int i = 0; i < n_freq; i++) {
inv_freq[i] = std::pow(hparams.rope_theta, -2.0f * i / (float) axis_dim);
}
std::vector<float> cos_data((size_t) d_head * n_pos, 1.0f);
std::vector<float> sin_data((size_t) d_head * n_pos, 0.0f);
for (int p = 0; p < n_pos; p++) {
const int pos_axis[3] = { 0, p / grid_w, p % grid_w };
float * cptr = cos_data.data() + (size_t) p * d_head;
float * sptr = sin_data.data() + (size_t) p * d_head;
for (int d = 0; d < half; d++) {
const float ang = pos_axis[d / n_freq] * inv_freq[d % n_freq];
const float c = std::cos(ang);
const float s = std::sin(ang);
cptr[d] = cptr[d + half] = c;
sptr[d] = sptr[d + half] = s;
}
}
set_input_f32("rope_cos", cos_data);
set_input_f32("rope_sin", sin_data);
} break;
case PROJECTOR_TYPE_GLM_EDGE:
{
// llava and other models
@@ -5512,6 +5700,8 @@ int clip_n_mmproj_embd(const struct clip_ctx * ctx) {
case PROJECTOR_TYPE_QWEN25VL:
case PROJECTOR_TYPE_JANUS_PRO:
return ctx->model.mm_1_b->ne[0];
case PROJECTOR_TYPE_MINIMAX_M3_VL:
return ctx->model.mm_3_b->ne[0];
case PROJECTOR_TYPE_QWEN3VL:
// main path + deepstack paths
return ctx->model.mm_1_b->ne[0] * (1 + ctx->model.n_deepstack_layers);
examples/mtmd/legacy-models/minimaxm3-convert-image-encoder-to-gguf.py
+157
View File
@@ -0,0 +1,157 @@
#!/usr/bin/env python3
import argparse
import json
import os
import re
from pathlib import Path
import torch
from gguf import GGUFEndian, GGUFWriter
from safetensors.torch import load_file
VISION_PREFIX = "vision_tower.vision_model."
def load_index(model_dir: Path) -> dict[str, str]:
index_path = model_dir / "model.safetensors.index.json"
if index_path.exists():
with index_path.open("r", encoding="utf-8") as f:
return json.load(f)["weight_map"]
shards = sorted(model_dir.glob("*.safetensors"))
if len(shards) == 1:
tensors = load_file(str(shards[0]), device="cpu")
return {name: shards[0].name for name in tensors}
raise FileNotFoundError(f"unable to find safetensors index in {model_dir}")
def rename_tensor(name: str) -> str | None:
if name == "vision_tower.vision_model.embeddings.patch_embedding.weight":
return "v.patch_embd.weight"
if name in (
"vision_tower.vision_model.pre_layrnorm.weight",
"vision_tower.vision_model.pre_layrnorm.bias",
):
return name.replace("vision_tower.vision_model.pre_layrnorm", "v.pre_ln")
if name.startswith("multi_modal_projector."):
name = name.replace("multi_modal_projector.linear_1", "mm.0")
name = name.replace("multi_modal_projector.linear_2", "mm.2")
return name
if name.startswith("patch_merge_mlp."):
name = name.replace("patch_merge_mlp.linear_1", "mm.4")
name = name.replace("patch_merge_mlp.linear_2", "mm.6")
return name
if not name.startswith(VISION_PREFIX + "encoder.layers."):
return None
name = name[len(VISION_PREFIX):]
name = name.replace("encoder.layers", "blk")
name = name.replace("layer_norm1", "ln1")
name = name.replace("layer_norm2", "ln2")
name = name.replace("self_attn.q_proj", "attn_q")
name = name.replace("self_attn.k_proj", "attn_k")
name = name.replace("self_attn.v_proj", "attn_v")
name = name.replace("self_attn.out_proj", "attn_out")
name = name.replace("mlp.fc1", "ffn_up")
name = name.replace("mlp.fc2", "ffn_down")
return "v." + name
def read_json(path: Path) -> dict:
with path.open("r", encoding="utf-8") as f:
return json.load(f)
def main() -> None:
parser = argparse.ArgumentParser(description="Convert MiniMax-M3 vision encoder/projector to GGUF")
parser.add_argument("-m", "--model-dir", required=True, help="Path to MiniMax-M3 HF model directory")
parser.add_argument("-o", "--output", default=None, help="Output GGUF path")
parser.add_argument("--use-f32", action="store_true", help="Write tensors as f32 instead of f16")
parser.add_argument("--bigendian", action="store_true", help="Write big-endian GGUF")
args = parser.parse_args()
model_dir = Path(args.model_dir)
config = read_json(model_dir / "config.json")
vision_config = config["vision_config"]
preprocessor_path = model_dir / "preprocessor_config.json"
preprocessor = read_json(preprocessor_path) if preprocessor_path.exists() else {}
compression = vision_config.get("img_token_compression_config", {})
output = Path(args.output) if args.output else model_dir / "mmproj-minimax-m3-vl.gguf"
ftype = 0 if args.use_f32 else 1
writer = GGUFWriter(
path=str(output),
arch="clip",
endianess=GGUFEndian.BIG if args.bigendian else GGUFEndian.LITTLE,
)
writer.add_bool("clip.has_text_encoder", False)
writer.add_bool("clip.has_vision_encoder", True)
writer.add_bool("clip.has_audio_encoder", False)
writer.add_string("clip.projector_type", "minimax_m3_vl")
writer.add_string("general.name", "MiniMax-M3 vision projector")
writer.add_uint32("general.file_type", ftype)
writer.add_uint32("clip.vision.image_size", vision_config["image_size"])
writer.add_uint32("clip.vision.patch_size", vision_config["patch_size"])
writer.add_uint32("clip.vision.embedding_length", vision_config["hidden_size"])
writer.add_uint32("clip.vision.feed_forward_length", vision_config["intermediate_size"])
writer.add_uint32("clip.vision.projection_dim", vision_config["projection_dim"])
writer.add_uint32("clip.vision.attention.head_count", vision_config["num_attention_heads"])
writer.add_uint32("clip.vision.block_count", vision_config["num_hidden_layers"])
writer.add_float32("clip.vision.attention.layer_norm_epsilon", vision_config.get("layer_norm_eps", 1e-5))
writer.add_uint32("clip.vision.spatial_merge_size", compression.get("spatial_merge_size", 2))
writer.add_uint32("clip.vision.temporal_patch_size", compression.get("temporal_patch_size", 2))
writer.add_uint32("clip.vision.image_min_pixels", preprocessor.get("min_pixels", 4 * 28 * 28))
writer.add_uint32("clip.vision.image_max_pixels", preprocessor.get("max_pixels", 451584))
writer.add_array("clip.vision.image_mean", preprocessor.get("image_mean", [0.48145466, 0.4578275, 0.40821073]))
writer.add_array("clip.vision.image_std", preprocessor.get("image_std", [0.26862954, 0.26130258, 0.27577711]))
writer.add_bool("clip.use_gelu", True)
weight_map = load_index(model_dir)
shard_cache: dict[str, dict[str, torch.Tensor]] = {}
for src_name in sorted(weight_map):
dst_name = rename_tensor(src_name)
if dst_name is None:
continue
shard_name = weight_map[src_name]
if shard_name not in shard_cache:
shard_cache[shard_name] = load_file(str(model_dir / shard_name), device="cpu")
data = shard_cache[shard_name][src_name]
if src_name.endswith("patch_embedding.weight") and data.ndim == 5:
if data.shape[2] != 2:
raise ValueError(f"expected temporal_patch_size 2, got {data.shape[2]}")
for i in range(data.shape[2]):
patch_name = dst_name if i == 0 else f"{dst_name}.{i}"
patch_data = data[:, :, i]
if args.use_f32:
patch_data = patch_data.float()
else:
patch_data = patch_data.half()
writer.add_tensor(patch_name, patch_data.numpy())
continue
if args.use_f32:
data = data.float()
elif data.ndim == 2 and dst_name.endswith(".weight"):
data = data.half()
else:
data = data.float()
writer.add_tensor(dst_name, data.numpy())
writer.write_header_to_file()
writer.write_kv_data_to_file()
writer.write_tensors_to_file()
writer.close()
print(f"Wrote {output}")
if __name__ == "__main__":
main()
examples/mtmd/mtmd.cpp
+5 -1
View File
@@ -283,6 +283,11 @@ struct mtmd_context {
img_beg = "<|vision_start|>";
img_end = "<|vision_end|>";
} else if (proj == PROJECTOR_TYPE_MINIMAX_M3_VL) {
// ]<]start of image[>[ ... (image embeddings) ... ]<]end of image[>[
img_beg = "]<]start of image[>[";
img_end = "]<]end of image[>[";
} else if (proj == PROJECTOR_TYPE_LLAMA4) {
// (more details in mtmd_context constructor)
img_beg = "<|image_start|>";
@@ -1208,4 +1213,3 @@ void mtmd_input_chunk_to_json(mtmd_input_chunk * chunk, json & j) {
examples/server/server-common.cpp
+3 -1
View File
@@ -1199,7 +1199,9 @@ size_t server_tokens::size_up_to_pos(llama_pos max_pos) const {
size_t idx = 0;
llama_pos pos = 0;
if (pos >= max_pos) {
return idx;
}
while (idx < tokens.size()) {
const auto media_it = map_idx_to_media.find(idx);
if (media_it != map_idx_to_media.end()) {
examples/server/server-context.cpp
+102 -21
View File
@@ -164,7 +164,15 @@ static common_speculative_stage_params server_parse_speculative_stage_json(const
continue;
}
throw std::runtime_error("Error: per-request speculative.stages only support type, n_max, n_min, and p_min; structural stage overrides are startup-only");
if (item.key() == "heads" || item.key() == "mtp_heads") {
stage.mtp_heads = item.value().get<int32_t>();
if (stage.mtp_heads < 0) {
throw std::runtime_error("Error: speculative.stages[].heads must be >= 0");
}
continue;
}
throw std::runtime_error("Error: per-request speculative.stages only support type, n_max, n_min, p_min, and heads; structural stage overrides are startup-only");
}
return stage;
@@ -449,8 +457,8 @@ void server_slot::prompt_save(server_prompt_cache& prompt_cache) const {
llama_state_seq_get_data(ctx, cur->data.data(), cur_size, id, 0);
}
void server_slot::prompt_load(server_prompt_cache& prompt_cache, const server_tokens& tokens) {
bool res = prompt_cache.load(server_cached_prompt, tokens, ctx, id);
void server_slot::prompt_load(server_prompt_cache& prompt_cache, const server_tokens& tokens, float min_reusable_fraction) {
bool res = prompt_cache.load(server_cached_prompt, tokens, ctx, id, min_reusable_fraction);
if (!res) {
LLAMA_LOG_INFO("failed to load prompt from cache\n");
}
@@ -473,6 +481,7 @@ void server_slot::reset() {
prompt_batch_i1 = -1;
n_sent_text = 0;
drafted.clear();
spec_target_only = false;
i_batch_dft.clear();
spec_prompt_warmup_failed = false;
n_sent_token_probs = 0;
@@ -509,6 +518,8 @@ void server_slot::reset() {
// Reset speculative decoding stats
n_draft_total = 0;
n_draft_accepted = 0;
n_draft_by_depth.clear();
n_draft_accepted_by_depth.clear();
chat_msg = {};
json_schema = json();
generated_tool_call_ids.clear();
@@ -577,6 +588,10 @@ int server_slot::get_n_draft_max() const {
// determine the max draft that fits the current slot state
int n_draft_max = params.speculative.get_max_stage_n_max();
const int configured_dflash_n_max = common_speculative_get_configured_n_max(spec);
if (configured_dflash_n_max > 0) {
n_draft_max = configured_dflash_n_max;
}
// note: slot.prompt is not yet expanded with the `id` token sampled above
// also, need to leave space for 1 extra token to allow context shifts
@@ -603,12 +618,13 @@ void server_slot::release() {
state = SLOT_STATE_IDLE;
task.reset();
}
spec_target_only = false;
llama_decode_reset();
}
json server_slot::get_formated_timings() const {
return json{
json timings = json{
{"prompt_n", n_prompt_tokens_processed},
{"prompt_ms", t_prompt_processing},
{"prompt_per_token_ms", t_prompt_processing / n_prompt_tokens_processed},
@@ -622,6 +638,27 @@ json server_slot::get_formated_timings() const {
{"n_ctx", n_ctx},
{"n_past", n_past},
};
if (n_draft_total > 0) {
timings["draft_n"] = n_draft_total;
timings["draft_n_accepted"] = n_draft_accepted;
json by_depth = json::array();
for (size_t i = 0; i < n_draft_by_depth.size(); ++i) {
if (n_draft_by_depth[i] <= 0) {
continue;
}
const int32_t accepted = i < n_draft_accepted_by_depth.size()
? n_draft_accepted_by_depth[i] : 0;
by_depth.push_back({
{"depth", (int32_t) i + 1},
{"draft_n", n_draft_by_depth[i]},
{"draft_n_accepted", accepted},
});
}
if (!by_depth.empty()) {
timings["draft_by_depth"] = by_depth;
}
}
return timings;
}
result_timings server_slot::get_timings() const {
@@ -644,6 +681,8 @@ result_timings server_slot::get_timings() const {
if (n_draft_total > 0) {
timings.draft_n = n_draft_total;
timings.draft_n_accepted = n_draft_accepted;
timings.draft_n_by_depth = n_draft_by_depth;
timings.draft_n_accepted_by_depth = n_draft_accepted_by_depth;
}
return timings;
@@ -1010,7 +1049,7 @@ server_slot* server_context::get_available_slot(const server_task& task) {
const int64_t t_start = ggml_time_us();
copy_data_to_cached_prompt(tokens, *ret);
ret->prompt_load(*prompt_cache, task.tokens);
ret->prompt_load(*prompt_cache, task.tokens, cache_ram_similarity);
prompt_cache->update();
ret->cache_tokens = ret->server_cached_prompt.tokens.clone(); // recover cache tokens
@@ -1158,6 +1197,9 @@ bool server_context::launch_slot_with_task(server_slot& slot, server_task& task)
if (stage_override.has_p_min_override()) {
slot.params.speculative.stages[i].p_min = stage_override.p_min;
}
if (stage_override.has_mtp_heads_override()) {
slot.params.speculative.stages[i].mtp_heads = stage_override.mtp_heads;
}
}
const auto resolved = slot.params.speculative.get_resolved_stages();
@@ -1195,6 +1237,7 @@ bool server_context::launch_slot_with_task(server_slot& slot, server_task& task)
if (!common_speculative_validate_chain(slot.params.speculative, &spec_error)) {
throw std::runtime_error("Error: invalid speculative request configuration: " + spec_error);
}
common_speculative_prepare_request(slot.spec, slot.params.speculative);
} catch (const std::exception & e) {
send_error(task, e.what(), ERROR_TYPE_INVALID_REQUEST);
return false;
@@ -1762,6 +1805,12 @@ bool server_context::launch_slot_with_task(server_slot& slot, server_task& task)
LOG_WARNING("%s\n", "ctx_shift is not implemented for split mode graph, it will be disabled");
}
}
if (!llama_model_supports_ctx_shift(llama_get_model(slot.ctx))) {
if (params_base.ctx_shift) {
params_base.ctx_shift = false;
LOG_WARNING("%s\n", "ctx_shift is not supported by this model's KV cache, it will be disabled");
}
}
{
const auto& stop = data.find("stop");
if (stop != data.end() && stop->is_array()) {
@@ -3433,6 +3482,7 @@ void server_context::add_sampled_tokens() {
if (slot.state == SLOT_STATE_IDLE) {
continue;
}
slot.spec_target_only = false;
// generate draft tokens in speculative decoding mode
// TODO: rework to have a single draft llama_context shared across all slots [TAG_SERVER_SPEC_REWORK]
@@ -3460,6 +3510,7 @@ void server_context::add_sampled_tokens() {
draft_base_pos,
slot.id);
llama_tokens & draft = draft_result.tokens;
slot.spec_target_only = draft_result.target_only;
const int n_draft_max = slot.get_n_draft_max();
@@ -3479,15 +3530,24 @@ void server_context::add_sampled_tokens() {
slot.cache_tokens.push_back(slot.sampled);
const int min_usable_draft = slot.params.speculative.get_min_usable_stage_n_min();
if (min_usable_draft > (int)draft.size()) {
if (!slot.spec_target_only && min_usable_draft > (int)draft.size()) {
SLT_DBG(slot, "ignoring small draft: %d < %d\n", (int)draft.size(), min_usable_draft);
// fallback to normal decoding
slot.i_batch = slot.i_batch_dft[0];
slot.drafted.clear();
slot.i_batch_dft.clear();
} else {
if (slot.spec_target_only) {
SLT_DBG(slot, "%s\n", "selected DFlash target-only arm: root-only target batch");
}
// keep track of total number of drafted tokens tested
slot.n_draft_total += draft.size();
if (slot.n_draft_by_depth.size() < draft.size()) {
slot.n_draft_by_depth.resize(draft.size(), 0);
}
for (size_t i = 0; i < draft.size(); ++i) {
slot.n_draft_by_depth[i]++;
}
// add all drafted tokens to the batch
for (size_t i = 0; i < draft.size(); i++) {
@@ -3816,25 +3876,38 @@ void server_context::batch_pending_prompt(const int32_t n_ubatch, const int32_t
GGML_ASSERT(slot.ga_n == 1);
// reuse any previously computed tokens that are common with the new prompt
common_prefix prefix = slot.cache_tokens.get_common_prefix(ctx, prompt_tokens, true); // string level match
common_prefix prefix_nonexact = slot.cache_tokens.get_common_prefix(ctx, prompt_tokens, false);
auto n_past0 = slot.cache_tokens.get_common_prefix_exact(prompt_tokens); // token level match
LLAMA_LOG_INFO("======== Cache: cache_size = %d, n_past0 = %d, n_past1 = %d, n_past_prompt1 = %d, n_past2 = %d, n_past_prompt2 = %d\n", (int32_t)slot.cache_tokens.size(), (int32_t)n_past0, (int32_t)prefix.first, (int32_t)prefix.second, (int32_t)prefix_nonexact.first, (int32_t)prefix_nonexact.second);
common_prefix prefix = slot.cache_tokens.get_common_prefix(ctx, prompt_tokens);
LLAMA_LOG_INFO("======== Cache: cache_size = %d, n_past = %d, n_past_prompt = %d\n", (int32_t)slot.cache_tokens.size(), (int32_t)prefix.first, (int32_t)prefix.second);
int32_t size_threshold = 20;
if (prefix.first + size_threshold < prefix_nonexact.first) {
// LLAMA_LOG_WARN("Common part contains missing or extra space and new line\n");
prefix = prefix_nonexact;
}
slot.n_past = prefix.first;
slot.n_past_prompt = prefix.second;
slot.n_past_offset = slot.n_past_prompt - slot.n_past;
if (!llama_model_supports_partial_kv_reuse(model) &&
slot.n_past < (int32_t) slot.cache_tokens.size()) {
// the cache diverges from the new prompt mid-sequence; this
// model can only extend or reset a cached sequence (per-position
// side state past the divergence point is already lost)
LLAMA_LOG_INFO("%s: cached sequence diverges at %d/%d and this model does not support partial KV reuse - reprocessing from scratch\n",
__func__, (int) slot.n_past, (int) slot.cache_tokens.size());
slot.n_past = 0;
slot.n_past_prompt = 0;
slot.n_past_offset = 0;
}
if (slot.n_past > 0 && slot.spec != nullptr &&
common_speculative_mtp_requires_fresh_warmup(slot.spec)) {
// the request drafts with more MTP heads than the cached
// prefix was warmed with; deeper-head cache rows for the
// reused span were never written
LLAMA_LOG_INFO("%s: request drafts with more MTP heads than the cached prefix was warmed with - reprocessing from scratch\n",
__func__);
slot.n_past = 0;
slot.n_past_prompt = 0;
slot.n_past_offset = 0;
}
//if (slot.n_past != slot.n_past_prompt) {
// LLAMA_LOG_INFO("Mistokenization found and handled successfully.\n");
//}
if ((slot.n_past + size_threshold < slot.cache_tokens.size()))
{
LLAMA_LOG_WARN("Common part does not match fully\n");
int32_t back = 4;
if (prefix.second >= back && prefix.first >= back) {
print_tokens(slot.prompt_tokens, slot.cache_tokens, prefix.second - back, prefix.first - back, 30);
@@ -3847,7 +3920,6 @@ void server_context::batch_pending_prompt(const int32_t n_ubatch, const int32_t
}
}
}
apply_checkpoint(slot);
if (slot.n_past_prompt == slot.n_prompt_tokens && slot.n_past_prompt > 0) {
// we have to evaluate at least 1 token to generate logits.
LOG_INFO("we have to evaluate at least 1 token to generate logits", {
@@ -3861,6 +3933,7 @@ void server_context::batch_pending_prompt(const int32_t n_ubatch, const int32_t
slot.n_past_se--;
}
}
apply_checkpoint(slot);
slot.n_prompt_tokens_cache = slot.n_past_prompt;
slot.n_prompt_tokens_processed = 0;
}
@@ -4129,7 +4202,14 @@ void server_context::speculative_decoding_accept() {
slot.t_token_generation = std::max<int64_t>(1, t_current - slot.t_start_generation) / 1e3;
// update how many tokens out of those tested were accepted
slot.n_draft_accepted += ids.size() - 1;
const size_t n_draft_accepted = ids.size() - 1;
slot.n_draft_accepted += n_draft_accepted;
if (slot.n_draft_accepted_by_depth.size() < n_draft_accepted) {
slot.n_draft_accepted_by_depth.resize(n_draft_accepted, 0);
}
for (size_t i = 0; i < n_draft_accepted; ++i) {
slot.n_draft_accepted_by_depth[i]++;
}
// rollback to the state before sampling the draft tokens
slot.cache_tokens.keep_first(slot.cache_tokens.n_tokens() - n_draft);
@@ -4152,6 +4232,7 @@ void server_context::speculative_decoding_accept() {
n_draft,
spec_pos_base,
accepted_output_indices);
slot.spec_target_only = false;
for (size_t i = 0; i < ids.size(); ++i) {
completion_token_output result;
@@ -4733,7 +4814,7 @@ void server_context::update_slots() {
// make sure we're in the right embedding mode
llama_set_embeddings(ctx, batch_type == 1);
if (llama_model_has_recurrent(model)) {
if (llama_model_has_recurrent(model) || llama_model_is_openpangu(model)) {
const int ckpt_mode = params_base.speculative.recurrent_ckpt_mode;
for (auto & slot : slots) {
examples/server/server-context.h
+5 -1
View File
@@ -6,6 +6,7 @@
#include <cstddef>
#include <memory>
#include <vector>
@@ -116,7 +117,7 @@ struct server_slot {
void prompt_save(server_prompt_cache& prompt_cache) const;
void prompt_load(server_prompt_cache& prompt_cache, const server_tokens& tokens);
void prompt_load(server_prompt_cache& prompt_cache, const server_tokens& tokens, float min_reusable_fraction);
size_t checkpoint_pos = 0;
bool do_checkpoint = false;
@@ -125,6 +126,7 @@ struct server_slot {
// sampling
llama_token sampled; // in speculative mode, this is the last accepted token
llama_tokens drafted;
bool spec_target_only = false;
json json_schema;
@@ -163,6 +165,8 @@ struct server_slot {
// speculative decoding stats
int32_t n_draft_total = 0; // Total draft tokens generated
int32_t n_draft_accepted = 0; // Draft tokens actually accepted
std::vector<int32_t> n_draft_by_depth;
std::vector<int32_t> n_draft_accepted_by_depth;
int32_t n_past_se = 0; // self-extend
examples/server/server-task.cpp
+22 -1
View File
@@ -20,6 +20,24 @@ json result_timings::to_json() const {
if (draft_n > 0) {
base["draft_n"] = draft_n;
base["draft_n_accepted"] = draft_n_accepted;
if (!draft_n_by_depth.empty()) {
json by_depth = json::array();
for (size_t i = 0; i < draft_n_by_depth.size(); ++i) {
if (draft_n_by_depth[i] <= 0) {
continue;
}
const int32_t accepted = i < draft_n_accepted_by_depth.size()
? draft_n_accepted_by_depth[i] : 0;
by_depth.push_back({
{"depth", (int32_t) i + 1},
{"draft_n", draft_n_by_depth[i]},
{"draft_n_accepted", accepted},
});
}
if (!by_depth.empty()) {
base["draft_by_depth"] = by_depth;
}
}
}
return base;
@@ -1073,7 +1091,7 @@ size_t server_prompt_cache::n_tokens() const {
}
bool server_prompt_cache::load(server_prompt& prompt, const server_tokens& tokens_new, llama_context* ctx, int32_t id_slot) {
bool server_prompt_cache::load(server_prompt& prompt, const server_tokens& tokens_new, llama_context* ctx, int32_t id_slot, float min_reusable_fraction) {
thinking_tokens think_tokens;
for (auto it = states.begin(); it != states.end(); ++it) {
think_tokens = it->think_tokens;
@@ -1107,6 +1125,9 @@ bool server_prompt_cache::load(server_prompt& prompt, const server_tokens& token
}
const auto lcp_cur = tokens.get_common_prefix(ctx, tokens_new_ex);
const float f_keep_cur = float(lcp_cur.first) / tokens.size();
if (f_keep_cur < min_reusable_fraction) {
continue;
}
const float sim_cur = tokens.get_tokens_similarity(ctx, tokens_new_ex, it->n_kept_prompt, it->n_discarded_prompt);
if (sim_best < sim_cur) {
f_keep_best = f_keep_cur;
examples/server/server-task.h
+5 -1
View File
@@ -8,6 +8,8 @@
// TODO: prevent including the whole server-common.h as we only use server_tokens
#include "server-common.h"
#include <vector>
using json = nlohmann::ordered_json;
enum stop_type {
@@ -130,6 +132,8 @@ struct result_timings {
// Optional speculative metrics - only included when > 0
int32_t draft_n = 0;
int32_t draft_n_accepted = 0;
std::vector<int32_t> draft_n_by_depth;
std::vector<int32_t> draft_n_accepted_by_depth;
json to_json() const;
};
@@ -446,7 +450,7 @@ struct server_prompt_cache {
server_prompt* alloc(const server_prompt& prompt, size_t state_size);
bool load(server_prompt& prompt, const server_tokens& tokens_new, llama_context* ctx, int32_t id_slot);
bool load(server_prompt& prompt, const server_tokens& tokens_new, llama_context* ctx, int32_t id_slot, float min_reusable_fraction);
void update();
};
ggml/include/ggml-backend.h
+6
View File
@@ -102,6 +102,7 @@ extern "C" {
GGML_API GGML_CALL bool ggml_backend_is_cpu (ggml_backend_t backend);
GGML_API void ggml_backend_cpu_set_n_threads (ggml_backend_t backend_cpu, int n_threads);
GGML_API void ggml_backend_cpu_set_moe_expert_prefetch(ggml_backend_t backend_cpu, bool enable);
GGML_API void ggml_backend_cpu_set_abort_callback(ggml_backend_t backend_cpu, ggml_abort_callback abort_callback, void * abort_callback_data);
// Create a backend buffer from an existing pointer
@@ -215,6 +216,11 @@ extern "C" {
GGML_API void ggml_backend_sched_set_split_mode_graph(ggml_backend_sched_t sched, bool on_or_off, bool async);
GGML_API void ggml_backend_sched_set_max_extra_alloc(ggml_backend_sched_t sched, int extra_alloc_MiB);
// prefetch mmap'd MoE expert weights into the page cache
GGML_API bool ggml_backend_prefetch_init(int n_threads);
GGML_API void ggml_backend_prefetch_register_mapping(const void * addr, size_t size);
GGML_API void ggml_backend_prefetch_unregister_mapping(const void * addr);
//
// Utils
//
ggml/include/ggml.h
+3
View File
@@ -831,6 +831,9 @@ extern "C" {
// abort ggml_graph_compute when true
ggml_abort_callback abort_callback;
void * abort_callback_data;
// read-ahead selected MoE expert weights in the CPU matmul-id kernels
bool moe_expert_prefetch;
};
enum ggml_cgraph_eval_order {
ggml/src/CMakeLists.txt
+2
View File
@@ -1297,6 +1297,8 @@ add_library(ggml
ggml.c
ggml-alloc.c
ggml-backend.cpp
ggml-moe-prefetch.cpp
ggml-moe-prefetch.h
ggml-quants.c
ggml-quants.h
${GGML_SOURCES_CUDA} ${GGML_HEADERS_CUDA}
ggml/src/ggml-backend.cpp
+129
View File
@@ -2,6 +2,7 @@
#include "ggml-alloc.h"
#include "ggml-impl.h"
#include "ggml-rpc.h"
#include "ggml-moe-prefetch.h"
#include <cassert>
#include <climits>
@@ -803,6 +804,8 @@ struct ggml_backend_cpu_context {
ggml_abort_callback abort_callback;
void * abort_callback_data;
bool moe_expert_prefetch;
};
GGML_CALL static const char * ggml_backend_cpu_name(ggml_backend_t backend) {
@@ -847,6 +850,7 @@ GGML_CALL static ggml_backend_graph_plan_t ggml_backend_cpu_graph_plan_create(gg
cpu_plan->cplan.abort_callback = cpu_ctx->abort_callback;
cpu_plan->cplan.abort_callback_data = cpu_ctx->abort_callback_data;
cpu_plan->cplan.moe_expert_prefetch = cpu_ctx->moe_expert_prefetch;
return cpu_plan;
}
@@ -886,6 +890,7 @@ GGML_CALL static enum ggml_status ggml_backend_cpu_graph_compute(ggml_backend_t
cplan.abort_callback = cpu_ctx->abort_callback;
cplan.abort_callback_data = cpu_ctx->abort_callback_data;
cplan.moe_expert_prefetch = cpu_ctx->moe_expert_prefetch;
return ggml_graph_compute(cgraph, &cplan);
}
@@ -953,6 +958,7 @@ ggml_backend_t ggml_backend_cpu_init(void) {
ctx->work_size = 0;
ctx->abort_callback = NULL;
ctx->abort_callback_data = NULL;
ctx->moe_expert_prefetch = false;
ggml_backend_t cpu_backend = (ggml_backend_t)malloc(sizeof(struct ggml_backend));
if (cpu_backend == NULL) {
@@ -979,6 +985,13 @@ void ggml_backend_cpu_set_n_threads(ggml_backend_t backend_cpu, int n_threads) {
ctx->n_threads = n_threads;
}
void ggml_backend_cpu_set_moe_expert_prefetch(ggml_backend_t backend_cpu, bool enable) {
GGML_ASSERT(ggml_backend_is_cpu(backend_cpu));
struct ggml_backend_cpu_context * ctx = (struct ggml_backend_cpu_context *)backend_cpu->context;
ctx->moe_expert_prefetch = enable;
}
void ggml_backend_cpu_set_abort_callback(ggml_backend_t backend_cpu, ggml_abort_callback abort_callback, void * abort_callback_data) {
GGML_ASSERT(ggml_backend_is_cpu(backend_cpu));
@@ -1216,6 +1229,22 @@ void ggml_backend_sched_set_max_extra_alloc(ggml_backend_sched_t sched, int extr
}
}
bool ggml_backend_prefetch_init(int n_threads) {
if (n_threads <= 0) {
n_threads = std::max(1, std::min(8, (int) std::thread::hardware_concurrency()));
}
ggml_moe_prefetch_set_n_threads(n_threads);
return ggml_moe_prefetch_enabled();
}
void ggml_backend_prefetch_register_mapping(const void * addr, size_t size) {
ggml_moe_prefetch_register_mapping(addr, size);
}
void ggml_backend_prefetch_unregister_mapping(const void * addr) {
ggml_moe_prefetch_unregister_mapping(addr);
}
static inline bool ggml_backend_sched_offload_enabled(ggml_backend_sched_t sched, enum ggml_op op) {
int int_op = (int)op;
if (!sched || op < 0 || op >= GGML_OP_COUNT) return false;
@@ -2049,6 +2078,11 @@ static void ggml_backend_sched_copy_inputs(ggml_backend_sched_t sched, ggml_back
last_ids_tensor = ids_tensor;
}
// when the expert prefetch engine streamed this tensor ahead
// (see the lookahead in compute_splits), wait for it so the
// host-side reads below hit warm page cache instead of faulting
ggml_moe_prefetch_wait(input);
const size_t expert_size = input->ne[2] > 1 ? input->nb[2] : input->nb[1];
if (input->ne[2] > 1) {
@@ -2365,6 +2399,67 @@ static enum ggml_status ggml_backend_sched_compute_splits(ggml_backend_sched_t s
std::vector<uint32_t> unique_ids;
ggml_tensor * last_ids_tensor = nullptr;
// MoE expert prefetch; pre-scan the splits for expert-weight matmuls whose
// weights live in a host mmap. Two behaviors, both page-cache warmers.
// - lookahead streams the next splits' expert tensors in full while split
// i computes (batch/PP graphs only, where most experts are hit)
// - selective enqueues just the selected expert slices of a split's
// up/gate/down tensors once its ids have been copied to the host
struct moe_split_info {
int split;
int64_t n_tokens;
std::vector<ggml_tensor *> host_weights; // originals (host mmap)
std::vector<ggml_tensor *> nodes; // MoE nodes computed on a host buffer
};
std::vector<moe_split_info> moe_infos;
const bool moe_prefetch = ggml_moe_prefetch_enabled();
static const size_t moe_ahead = [] {
const char * env = getenv("GGML_MOE_PREFETCH_AHEAD");
return env ? (size_t) std::max(0, atoi(env)) : (size_t) 3;
}();
if (moe_prefetch) {
ggml_moe_prefetch_new_epoch();
for (int i = 0; i < sched->n_splits; i++) {
moe_split_info info;
info.split = i;
info.n_tokens = 0;
for (int n = 0; n < splits[i].graph.n_nodes; ++n) {
ggml_tensor * node = splits[i].graph.nodes[n];
if (node->op != GGML_OP_MUL_MAT_ID && node->op != GGML_OP_MOE_FUSED_UP_GATE) {
continue;
}
ggml_tensor * node_ids = node->op == GGML_OP_MUL_MAT_ID ? node->src[2] : node->src[3];
info.n_tokens = std::max(info.n_tokens, node_ids ? node_ids->ne[1] : 0);
ggml_tensor * ws[2] = { node->src[0], node->op == GGML_OP_MOE_FUSED_UP_GATE ? node->src[1] : nullptr };
bool node_on_host = false;
for (ggml_tensor * w : ws) {
if (w && w->buffer && ggml_backend_buffer_is_host(w->buffer) &&
ggml_backend_buffer_get_usage(w->buffer) == GGML_BACKEND_BUFFER_USAGE_WEIGHTS) {
info.host_weights.push_back(w);
node_on_host = true;
}
}
if (node_on_host) {
info.nodes.push_back(node);
}
}
// in the offloaded case the node's weight srcs were rewritten to
// device copies; the host originals arrive as split inputs
for (int j = 0; j < splits[i].n_inputs; ++j) {
ggml_tensor * input = splits[i].inputs[j];
if (input->ne[2] > 1 && input->buffer && ggml_backend_buffer_is_host(input->buffer) &&
ggml_backend_buffer_get_usage(input->buffer) == GGML_BACKEND_BUFFER_USAGE_WEIGHTS) {
info.host_weights.push_back(input);
}
}
if (!info.host_weights.empty()) {
moe_infos.push_back(std::move(info));
}
}
}
size_t moe_next = 0; // first moe_infos entry with split >= current split
size_t moe_enq = 0; // moe_infos entries already enqueued for lookahead
for (int i = 0; i < sched->n_splits; i++) {
#if IK_PRINT_TIMING
int64_t tim1 = ggml_time_us();
@@ -2373,9 +2468,34 @@ static enum ggml_status ggml_backend_sched_compute_splits(ggml_backend_sched_t s
int split_backend_id = split->backend_id;
ggml_backend_t split_backend = sched->backends[split_backend_id];
if (moe_prefetch && !moe_infos.empty()) {
while (moe_next < moe_infos.size() && moe_infos[moe_next].split < i) {
moe_next++;
}
// keep the next `moe_ahead` MoE-bearing splits streaming in
const size_t want_end = std::min(moe_next + moe_ahead, moe_infos.size());
for (size_t k = std::max(moe_enq, moe_next); k < want_end; ++k) {
if (moe_infos[k].n_tokens >= 32) { // batch graphs touch most experts (min batch offload)
for (ggml_tensor * w : moe_infos[k].host_weights) {
ggml_moe_prefetch_tensor(w);
}
}
moe_enq = k + 1;
}
}
// copy the input tensors to the split backend
ggml_backend_sched_copy_inputs(sched, split, sched->needs_sync, ids, unique_ids, last_ids_tensor);
// ids are now final and host-visible; enqueue the selected expert
// slices of this split's host-computed MoE matmuls (up/gate and down
// share one ids tensor, so the down weights stream in during up/gate)
if (moe_prefetch && moe_next < moe_infos.size() && moe_infos[moe_next].split == i) {
for (ggml_tensor * node : moe_infos[moe_next].nodes) {
ggml_moe_prefetch_node(node);
}
}
if (split->n_inputs > 0 && !sched->own_cpy[split_backend_id]) {
sched->needs_sync[split_backend_id] = true;
} else {
@@ -2390,6 +2510,15 @@ static enum ggml_status ggml_backend_sched_compute_splits(ggml_backend_sched_t s
return ec;
}
// the pages the lookahead streamer just read for this split are one-shot
// streaming traffic; MADV_COLD them so the decode working set survives
if (moe_prefetch && moe_next < moe_infos.size() && moe_infos[moe_next].split == i &&
moe_infos[moe_next].n_tokens >= 32) {
for (ggml_tensor * w : moe_infos[moe_next].host_weights) {
ggml_moe_prefetch_cold(w);
}
}
// record the event of this copy
if (split->n_inputs > 0) {
if (sched->events[split_backend_id][sched->cur_copy] != NULL) {
ggml/src/ggml-cuda.cu
+4
View File
@@ -3773,6 +3773,9 @@ static bool ggml_cuda_compute_forward(ggml_backend_cuda_context & ctx, struct gg
case GGML_UNARY_OP_GELU:
ggml_cuda_op_gelu(ctx, dst);
break;
case GGML_UNARY_OP_GELU_ERF:
ggml_cuda_op_gelu_erf(ctx, dst);
break;
case GGML_UNARY_OP_SILU:
ggml_cuda_op_silu(ctx, dst);
break;
@@ -4696,6 +4699,7 @@ GGML_CALL static bool ggml_backend_cuda_supports_op(ggml_backend_t backend, cons
case GGML_OP_UNARY:
switch (ggml_get_unary_op(op)) {
case GGML_UNARY_OP_GELU:
case GGML_UNARY_OP_GELU_ERF:
case GGML_UNARY_OP_SILU:
case GGML_UNARY_OP_SWIGLU:
case GGML_UNARY_OP_SWIGLU_OAI:
ggml/src/ggml-moe-prefetch.cpp
+470
View File
@@ -0,0 +1,470 @@
#include "ggml-moe-prefetch.h"
#if defined(__linux__)
#include <sys/mman.h>
#include <unistd.h>
#include <algorithm>
#include <atomic>
#include <condition_variable>
#include <cstdlib>
#include <cstdio>
#include <deque>
#include <memory>
#include <mutex>
#include <thread>
#include <unordered_map>
#include <vector>
// chunk granularity; big enough to amortize syscall cost,
// small enough to spread one expert (~MBs) across several workers.
static constexpr size_t GGML_MOE_PREFETCH_CHUNK = 2u*1024u*1024u;
// lookahead jobs are dropped beyond this queue depth so a stalled consumer
// cannot accumulate unbounded work.
static constexpr size_t GGML_MOE_PREFETCH_MAX_QUEUE = 65536;
// experts are faulted in with MADV_POPULATE_READ, which brings pages into the
// page cache AND this mm's page tables, so consumers take no minor faults.
// Unsupported kernels leave the engine off (the callers fall back to
// madvise(MADV_WILLNEED) hints)
#ifndef MADV_POPULATE_READ
#define MADV_POPULATE_READ 22
#endif
namespace {
struct mapping_entry {
uintptr_t base;
size_t size;
};
struct ticket {
std::atomic<int> pending{0};
uint64_t epoch = 0;
bool track_reads = false;
// chunks the workers actually read from storage this epoch (i.e. pages
// that were cold before the sweep); guarded by prefetch_pool::mtx.
// Used by ggml_moe_prefetch_cold() to deactivate streaming traffic.
std::vector<std::pair<uintptr_t, uint32_t>> read_chunks;
};
struct job {
uintptr_t addr;
uint32_t len;
std::shared_ptr<ticket> tk;
};
struct prefetch_pool {
std::mutex mtx;
std::condition_variable cv_work; // workers sleep here
std::condition_variable cv_done; // waiters sleep here
std::deque<job> queue;
std::vector<std::thread> workers;
bool shutdown = false;
std::unordered_map<const void *, std::shared_ptr<ticket>> tickets;
// cumulative observability counters (GGML_MOE_PREFETCH_DEBUG)
std::atomic<uint64_t> n_jobs{0};
std::atomic<uint64_t> n_skipped{0};
std::atomic<uint64_t> bytes_populated{0};
std::atomic<uint64_t> n_colded{0};
~prefetch_pool() { stop(); }
void stop() {
{
std::lock_guard<std::mutex> lock(mtx);
shutdown = true;
// complete outstanding tickets so no waiter blocks forever
for (auto & j : queue) {
if (j.tk) {
j.tk->pending.fetch_sub(1, std::memory_order_acq_rel);
}
}
queue.clear();
cv_work.notify_all();
cv_done.notify_all();
}
for (auto & w : workers) {
if (w.joinable()) w.join();
}
workers.clear();
if (getenv("GGML_MOE_PREFETCH_DEBUG") && n_jobs.load() > 0) {
fprintf(stderr, "%s: jobs=%llu skipped_resident=%llu bytes_populated=%.2f GiB colded=%llu\n", __func__,
(unsigned long long) n_jobs.load(), (unsigned long long) n_skipped.load(),
(double) bytes_populated.load()/(1024.0*1024.0*1024.0),
(unsigned long long) n_colded.load());
}
}
void start(int n_threads) {
stop();
std::lock_guard<std::mutex> lock(mtx);
shutdown = false;
workers.reserve(n_threads);
for (int i = 0; i < n_threads; ++i) {
workers.emplace_back([this] { run(); });
}
}
static bool chunk_resident(uintptr_t addr, size_t len) {
const long page = sysconf(_SC_PAGESIZE);
const uintptr_t astart = addr & ~(uintptr_t)(page - 1);
const size_t alen = (addr + len) - astart;
const size_t npages = (alen + page - 1)/page;
if (npages > GGML_MOE_PREFETCH_CHUNK/4096 + 2) {
return false;
}
unsigned char vec[GGML_MOE_PREFETCH_CHUNK/4096 + 2];
if (mincore((void *)astart, alen, vec) != 0) {
return false;
}
for (size_t i = 0; i < npages; ++i) {
if (!(vec[i] & 1)) return false;
}
return true;
}
void run() {
const long page = sysconf(_SC_PAGESIZE);
for (;;) {
job j;
{
std::unique_lock<std::mutex> lock(mtx);
cv_work.wait(lock, [this] { return shutdown || !queue.empty(); });
if (shutdown) return;
j = std::move(queue.front());
queue.pop_front();
}
n_jobs.fetch_add(1, std::memory_order_relaxed);
bool done_read = false;
// skip when every page is already resident; keeps the drain rate
// high when the cache is warm
if (chunk_resident(j.addr, j.len)) {
n_skipped.fetch_add(1, std::memory_order_relaxed);
} else {
const uintptr_t astart = j.addr & ~(uintptr_t)(page - 1);
const size_t alen = ((j.addr + j.len + page - 1) & ~(uintptr_t)(page - 1)) - astart;
if (madvise((void *)astart, alen, MADV_POPULATE_READ) == 0) {
bytes_populated.fetch_add(alen, std::memory_order_relaxed);
done_read = true;
} // on failure the fault path takes over
}
if (j.tk) {
const int left = j.tk->pending.fetch_sub(1, std::memory_order_acq_rel);
std::lock_guard<std::mutex> lock(mtx);
if (done_read && j.tk->track_reads) {
j.tk->read_chunks.emplace_back(j.addr, j.len);
}
if (left == 1) {
cv_done.notify_all();
}
}
}
}
};
struct prefetch_state {
std::mutex reg_mtx;
std::vector<mapping_entry> mappings;
std::atomic<uint64_t> epoch{1};
std::mutex pool_mtx;
std::shared_ptr<prefetch_pool> pool;
};
static prefetch_state & state() {
static prefetch_state s;
return s;
}
// true when [p, p+len) lies inside a registered mmap
static bool is_mapped(const void * p, size_t len) {
auto & s = state();
std::lock_guard<std::mutex> lock(s.reg_mtx);
const uintptr_t a = (uintptr_t)p;
for (const auto & m : s.mappings) {
if (a >= m.base && a + len <= m.base + m.size) return true;
}
return false;
}
// collect [offset, offset+len) ranges of the selected experts of weight tensor w
static void collect_selected_ranges(const ggml_tensor * w, const ggml_tensor * ids,
std::vector<std::pair<size_t, size_t>> & ranges) {
const int64_t n_as = w->ne[2];
const size_t stride = w->nb[2];
const size_t wbytes = ggml_nbytes(w);
if (n_as <= 1 || stride == 0) {
ranges.emplace_back(0, wbytes);
return;
}
std::vector<uint32_t> seen((n_as + 31)/32, 0);
for (int64_t i1 = 0; i1 < ids->ne[1]; ++i1) {
for (int64_t i0 = 0; i0 < ids->ne[0]; ++i0) {
const int32_t id = *(const int32_t *)((const char *)ids->data + i1*ids->nb[1] + i0*ids->nb[0]);
if (id < 0 || id >= n_as) continue; // ids may hold -1 sentinels (ggml_top_k_thresh)
seen[id >> 5] |= 1u << (id & 31);
}
}
// coalesce consecutive experts into single ranges
int64_t id = 0;
while (id < n_as) {
while (id < n_as && !(seen[id >> 5] & (1u << (id & 31)))) ++id;
if (id >= n_as) break;
int64_t first = id;
while (id < n_as && (seen[id >> 5] & (1u << (id & 31)))) ++id;
const size_t off = (size_t)first*stride;
const size_t len = std::min<size_t>((size_t)(id - first)*stride, wbytes - off);
if (off < wbytes && len > 0) {
ranges.emplace_back(off, len);
}
}
}
// enqueue ranges of tensor w; returns false when the engine is off or w is not mmap-registered
static bool enqueue_ranges(const ggml_tensor * w, const std::vector<std::pair<size_t, size_t>> & ranges, bool urgent, bool track_reads) {
auto & s = state();
std::shared_ptr<prefetch_pool> pool_sp;
{
std::lock_guard<std::mutex> plock(s.pool_mtx);
pool_sp = s.pool;
}
if (!pool_sp) return false;
if (!is_mapped(w->data, ggml_nbytes(w))) return false;
const uint64_t cur_epoch = s.epoch.load(std::memory_order_relaxed);
prefetch_pool & pool = *pool_sp;
std::lock_guard<std::mutex> lock(pool.mtx);
if (pool.shutdown) return false;
auto & tk = pool.tickets[w];
if (!tk) tk = std::make_shared<ticket>();
if (tk->epoch == cur_epoch) {
return true; // already enqueued for this scheduler pass
}
if (!urgent && pool.queue.size() > GGML_MOE_PREFETCH_MAX_QUEUE) {
return true; // drop lookahead under backpressure; do not mark the epoch
}
tk->epoch = cur_epoch;
tk->track_reads = track_reads;
tk->read_chunks.clear();
std::vector<job> jobs;
for (const auto & r : ranges) {
for (size_t o = r.first; o < r.first + r.second; o += GGML_MOE_PREFETCH_CHUNK) {
const size_t len = std::min(GGML_MOE_PREFETCH_CHUNK, r.first + r.second - o);
jobs.push_back({(uintptr_t)w->data + o, (uint32_t)len, tk});
}
}
if (jobs.empty()) return true;
tk->pending.fetch_add((int)jobs.size(), std::memory_order_acq_rel);
if (urgent) {
pool.queue.insert(pool.queue.begin(), std::make_move_iterator(jobs.begin()), std::make_move_iterator(jobs.end()));
} else {
pool.queue.insert(pool.queue.end(), std::make_move_iterator(jobs.begin()), std::make_move_iterator(jobs.end()));
}
pool.cv_work.notify_all();
return true;
}
static void node_weights_and_ids(const ggml_tensor * node, const ggml_tensor * & w0, const ggml_tensor * & w1, const ggml_tensor * & ids) {
w0 = nullptr; w1 = nullptr; ids = nullptr;
if (node->op == GGML_OP_MUL_MAT_ID) {
w0 = node->src[0];
ids = node->src[2];
} else if (node->op == GGML_OP_MOE_FUSED_UP_GATE) {
w0 = node->src[0];
w1 = node->src[1]; // NULL when up/gate are packed into one tensor
ids = node->src[3];
}
}
// legacy fallback; hints kernel readahead for the selected experts when the
// read pool is unavailable
static void legacy_madvise(const ggml_tensor * w, const ggml_tensor * ids) {
if (!w || !w->data || !ids || !ids->data) return;
std::vector<std::pair<size_t, size_t>> ranges;
collect_selected_ranges(w, ids, ranges);
const uintptr_t page_mask = (uintptr_t)sysconf(_SC_PAGESIZE) - 1;
const char * base = (const char *)w->data;
for (const auto & r : ranges) {
const uintptr_t start = (uintptr_t)(base + r.first);
const uintptr_t astart = start & ~page_mask;
(void) madvise((void *)astart, (size_t)(start + r.second - astart), MADV_WILLNEED);
}
}
} // namespace
void ggml_moe_prefetch_register_mapping(const void * base, size_t size) {
if (!base || size == 0) return;
auto & s = state();
std::lock_guard<std::mutex> lock(s.reg_mtx);
for (const auto & m : s.mappings) {
if (m.base == (uintptr_t)base) return; // contexts may re-register the same model
}
s.mappings.push_back({(uintptr_t)base, size});
}
void ggml_moe_prefetch_unregister_mapping(const void * base) {
auto & s = state();
std::lock_guard<std::mutex> lock(s.reg_mtx);
// a queued job may still point into this range; its madvise then fails
// (ENOMEM once unmapped) and the worker skips the chunk
s.mappings.erase(std::remove_if(s.mappings.begin(), s.mappings.end(),
[base](const mapping_entry & m) { return m.base == (uintptr_t)base; }),
s.mappings.end());
}
static bool populate_read_supported() {
const long page = sysconf(_SC_PAGESIZE);
void * p = mmap(nullptr, (size_t)page, PROT_READ, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
if (p == MAP_FAILED) return false;
const bool ok = madvise(p, (size_t)page, MADV_POPULATE_READ) == 0;
munmap(p, (size_t)page);
return ok;
}
void ggml_moe_prefetch_set_n_threads(int n_threads) {
auto & s = state();
std::lock_guard<std::mutex> lock(s.pool_mtx);
if (n_threads <= 0) {
s.pool.reset();
return;
}
if (!populate_read_supported()) {
fprintf(stderr, "%s: MADV_POPULATE_READ not supported; MoE prefetch disabled\n", __func__);
s.pool.reset();
return;
}
if (s.pool && s.pool->workers.size() == (size_t)n_threads) {
return;
}
s.pool.reset();
s.pool = std::make_shared<prefetch_pool>();
s.pool->start(n_threads);
}
bool ggml_moe_prefetch_enabled(void) {
auto & s = state();
std::lock_guard<std::mutex> lock(s.pool_mtx);
return s.pool != nullptr;
}
void ggml_moe_prefetch_new_epoch(void) {
state().epoch.fetch_add(1, std::memory_order_relaxed);
}
void ggml_moe_prefetch_node(const struct ggml_tensor * node) {
const ggml_tensor * w0; const ggml_tensor * w1; const ggml_tensor * ids;
node_weights_and_ids(node, w0, w1, ids);
if (!w0 || !ids || !ids->data) return;
std::vector<std::pair<size_t, size_t>> ranges;
collect_selected_ranges(w0, ids, ranges);
enqueue_ranges(w0, ranges, /*urgent =*/ true, /*track_reads =*/ false);
if (w1) {
ranges.clear();
collect_selected_ranges(w1, ids, ranges);
enqueue_ranges(w1, ranges, /*urgent =*/ true, /*track_reads =*/ false);
}
}
void ggml_moe_prefetch_tensor(const struct ggml_tensor * w) {
if (!w || !w->data) return;
std::vector<std::pair<size_t, size_t>> ranges;
ranges.emplace_back(0, ggml_nbytes(w));
enqueue_ranges(w, ranges, /*urgent =*/ false, /*track_reads =*/ true);
}
#ifndef MADV_COLD
#define MADV_COLD 20
#endif
void ggml_moe_prefetch_cold(const struct ggml_tensor * w) {
auto & s = state();
std::shared_ptr<prefetch_pool> pool;
{
std::lock_guard<std::mutex> plock(s.pool_mtx);
pool = s.pool;
}
if (!pool) return;
std::vector<std::pair<uintptr_t, uint32_t>> chunks;
{
std::lock_guard<std::mutex> lock(pool->mtx);
auto it = pool->tickets.find(w);
if (it == pool->tickets.end() || !it->second->track_reads) return;
chunks.swap(it->second->read_chunks);
}
const uintptr_t page_mask = (uintptr_t)sysconf(_SC_PAGESIZE) - 1;
for (const auto & c : chunks) {
const uintptr_t astart = c.first & ~page_mask;
(void) madvise((void *)astart, (size_t)(c.first + c.second - astart), MADV_COLD);
}
pool->n_colded.fetch_add(chunks.size(), std::memory_order_relaxed);
}
void ggml_moe_prefetch_wait(const struct ggml_tensor * w) {
auto & s = state();
std::shared_ptr<prefetch_pool> pool;
{
std::lock_guard<std::mutex> plock(s.pool_mtx);
pool = s.pool;
}
if (!pool) return;
std::shared_ptr<ticket> tk;
{
std::lock_guard<std::mutex> lock(pool->mtx);
auto it = pool->tickets.find(w);
if (it == pool->tickets.end()) return;
tk = it->second;
}
if (tk->pending.load(std::memory_order_acquire) <= 0) return;
// the shared_ptr keeps the pool alive; shutdown wakes cv_done
std::unique_lock<std::mutex> lock(pool->mtx);
pool->cv_done.wait(lock, [&] {
return pool->shutdown || tk->pending.load(std::memory_order_acquire) <= 0;
});
}
void ggml_moe_prefetch_kernel_hook(const struct ggml_tensor * node, int ith) {
if (ith != 0) return;
const ggml_tensor * w0; const ggml_tensor * w1; const ggml_tensor * ids;
node_weights_and_ids(node, w0, w1, ids);
if (!w0 || !ids || !ids->data) return;
if (!ggml_moe_prefetch_enabled()) {
legacy_madvise(w0, ids);
if (w1) legacy_madvise(w1, ids);
return;
}
// fire-and-forget enqueue, idempotent within the current epoch; a no-op
// when the scheduler hook already covered this node (the self-enqueue
// handles pure-CPU graphs). We deliberately do not wait; the compute
// threads' demand faults overlap with the workers' populates, which run ahead
// in the same expert-index order.
ggml_moe_prefetch_node(node);
}
#else // !__linux__
void ggml_moe_prefetch_register_mapping(const void *, size_t) {}
void ggml_moe_prefetch_unregister_mapping(const void *) {}
void ggml_moe_prefetch_set_n_threads(int) {}
bool ggml_moe_prefetch_enabled(void) { return false; }
void ggml_moe_prefetch_new_epoch(void) {}
void ggml_moe_prefetch_node(const struct ggml_tensor *) {}
void ggml_moe_prefetch_tensor(const struct ggml_tensor *) {}
void ggml_moe_prefetch_wait(const struct ggml_tensor *) {}
void ggml_moe_prefetch_kernel_hook(const struct ggml_tensor *, int) {}
void ggml_moe_prefetch_cold(const struct ggml_tensor *) {}
#endif
ggml/src/ggml-moe-prefetch.h
+52
View File
@@ -0,0 +1,52 @@
#pragma once
#include "ggml.h"
#include <stddef.h>
#include <stdbool.h>
#ifdef __cplusplus
extern "C" {
#endif
// mapping registry; a registered range marks tensors as mmap-backed and
// eligible for prefetch. Nothing is opened or retained beyond (base, size).
void ggml_moe_prefetch_register_mapping (const void * base, size_t size);
void ggml_moe_prefetch_unregister_mapping(const void * base);
// worker pool control; n_threads > 0 (re)creates the pool, <= 0 shuts it down
void ggml_moe_prefetch_set_n_threads(int n_threads);
bool ggml_moe_prefetch_enabled(void);
// the epoch is bumped once per scheduler pass so per-tensor enqueues are
// idempotent within one graph execution but re-issued on the next.
void ggml_moe_prefetch_new_epoch(void);
// selective enqueue for one MoE node (GGML_OP_MUL_MAT_ID or
// GGML_OP_MOE_FUSED_UP_GATE); reads the ids tensor bytes (must be final and
// host-visible) and enqueues the selected expert ranges of src weights.
void ggml_moe_prefetch_node(const struct ggml_tensor * node);
// full-tensor lookahead enqueue (low priority), e.g. for prompt processing
// where the next layer's experts are needed in bulk.
void ggml_moe_prefetch_tensor(const struct ggml_tensor * w);
// block until all pending prefetch jobs for tensor w are complete.
// Returns immediately when nothing is pending.
void ggml_moe_prefetch_wait(const struct ggml_tensor * w);
// MADV_COLD the pages the streamer had to read from storage for tensor w,
// so prompt-processing streaming traffic is reclaimed ahead of the decode
// working set. Pages already resident before the sweep are left alone.
// Only applies to tensors enqueued via ggml_moe_prefetch_tensor.
void ggml_moe_prefetch_cold(const struct ggml_tensor * w);
// kernel-entry hook, called at MoE matmul start (gated by
// cplan->moe_expert_prefetch). Thread 0 fire-and-forget enqueues when the
// scheduler hook did not run for this epoch (pure-CPU graphs); falls back to
// madvise(MADV_WILLNEED) when the engine is off.
void ggml_moe_prefetch_kernel_hook(const struct ggml_tensor * node, int ith);
#ifdef __cplusplus
}
#endif
ggml/src/ggml.c
+88 -2
View File
@@ -11,6 +11,7 @@
#include "ggml-quants.h"
#include "ggml.h"
#include "ggml-aarch64.h"
#include "ggml-moe-prefetch.h"
#include "iqk/iqk_quantize.h"
#include "iqk/iqk_cpu_ops.h"
#if GGML_USE_IQK_MULMAT
@@ -3316,11 +3317,16 @@ inline static void ggml_vec_hardsigmoid_f32 (const int n, float * y, const float
static const float GELU_QUICK_COEF = -1.702f;
static const float GELU_COEF_A = 0.044715f;
static const float SQRT_2_OVER_PI = 0.79788456080286535587989211986876f;
static const float SQRT_2_INV = 0.70710678118654752440084436210484f;
inline static float ggml_gelu_f32(float x) {
return 0.5f*x*(1.0f + tanhf(SQRT_2_OVER_PI*x*(1.0f + GELU_COEF_A*x*x)));
}
inline static float ggml_gelu_erf_f32(float x) {
return 0.5f*x*(1.0f + erff(x*SQRT_2_INV));
}
inline static float ggml_gelu_quick_f32(float x) {
return x*(1.0f/(1.0f+expf(GELU_QUICK_COEF*x)));
}
@@ -3942,6 +3948,13 @@ inline static void ggml_vec_gelu_f32(const int n, float * y, const float * x) {
}
#endif
}
inline static void ggml_vec_gelu_erf_f32(const int n, float * y, const float * x) {
for (int i = 0; i < n; ++i) {
y[i] = ggml_gelu_erf_f32(x[i]);
}
}
inline static void ggml_vec_mul_gelu_f32(const int n, float * z, const float * x, const float * y) {
int i = 0;
#if defined(__AVX512F__) && defined(__AVX512DQ__)
@@ -4158,8 +4171,6 @@ inline static void ggml_vec_geglu_f16(const int n, ggml_fp16_t * y, const ggml_f
}
}
static const float SQRT_2_INV = 0.70710678118654752440084436210484f;
inline static void ggml_vec_geglu_erf_f32(const int n, float * y, const float * x, const float * g) {
for (int i = 0; i < n; ++i) {
float xi = x[i];
@@ -15608,6 +15619,65 @@ static void ggml_compute_forward_gelu(
}
}
// ggml_compute_forward_gelu_erf
static void ggml_compute_forward_gelu_erf_f32(
const struct ggml_compute_params * params,
struct ggml_tensor * dst) {
const struct ggml_tensor * src0 = dst->src[0];
assert(ggml_is_contiguous_1(src0));
assert(ggml_is_contiguous_1(dst));
assert(ggml_are_same_shape(src0, dst));
const int ith = params->ith;
const int nth = params->nth;
const int nc = src0->ne[0];
const int nr = ggml_nrows(src0);
// rows per thread
const int dr = (nr + nth - 1)/nth;
// row range for this thread
const int ir0 = dr*ith;
const int ir1 = MIN(ir0 + dr, nr);
for (int i1 = ir0; i1 < ir1; i1++) {
ggml_vec_gelu_erf_f32(nc,
(float *) ((char *) dst->data + i1*( dst->nb[1])),
(float *) ((char *) src0->data + i1*(src0->nb[1])));
#ifndef NDEBUG
for (int k = 0; k < nc; k++) {
const float x = ((float *) ((char *) dst->data + i1*( dst->nb[1])))[k];
UNUSED(x);
assert(!isnan(x));
assert(!isinf(x));
}
#endif
}
}
static void ggml_compute_forward_gelu_erf(
const struct ggml_compute_params * params,
struct ggml_tensor * dst) {
const struct ggml_tensor * src0 = dst->src[0];
switch (src0->type) {
case GGML_TYPE_F32:
{
ggml_compute_forward_gelu_erf_f32(params, dst);
} break;
default:
{
GGML_ABORT("fatal error");
}
}
}
// ggml_compute_forward_fill
static void ggml_compute_forward_fill_f32(const struct ggml_compute_params * params, struct ggml_tensor * dst) {
@@ -17354,6 +17424,11 @@ static void ggml_compute_forward_mul_mat_id(
const int n_ids = ids->ne[0]; // n_expert_used
const int n_as = ne02; // n_expert
// kick off read-ahead of the selected experts before src1 quantization, so storage reads overlap that work
if (params->shared->cplan && params->shared->cplan->moe_expert_prefetch) {
ggml_moe_prefetch_kernel_hook(dst, ith);
}
char * wdata_src1_end = (src1->type == vec_dot_type) ?
(char *) params->wdata :
(char *) params->wdata + GGML_PAD(ggml_row_size(vec_dot_type, src1->ne[0])*ggml_nrows(src1), sizeof(int64_t));
@@ -17621,6 +17696,12 @@ static void ggml_compute_forward_mul_mat_id_up_gate(
const int n_ids = ids->ne[0]; // n_expert_used
const int n_as = ne02; // n_expert
// read-ahead of the selected experts for both the up and gate weight tensors
// (gate is null when up/gate are merged into a single tensor)
if (params->shared->cplan && params->shared->cplan->moe_expert_prefetch) {
ggml_moe_prefetch_kernel_hook(dst, ith);
}
char * wdata_src1_end = (src1->type == vec_dot_type) ?
(char *) params->wdata :
(char *) params->wdata + GGML_PAD(ggml_row_size(vec_dot_type, src1->ne[0])*ggml_nrows(src1), sizeof(int64_t));
@@ -23174,6 +23255,10 @@ static void ggml_compute_forward_unary(
{
ggml_compute_forward_gelu(params, dst);
} break;
case GGML_UNARY_OP_GELU_ERF:
{
ggml_compute_forward_gelu_erf(params, dst);
} break;
case GGML_UNARY_OP_GELU_QUICK:
{
ggml_compute_forward_gelu_quick(params, dst);
@@ -26493,6 +26578,7 @@ static int ggml_get_n_tasks(struct ggml_tensor * node, int n_threads) {
case GGML_UNARY_OP_SIGMOID:
case GGML_UNARY_OP_NEG:
case GGML_UNARY_OP_GELU:
case GGML_UNARY_OP_GELU_ERF:
case GGML_UNARY_OP_GELU_QUICK:
case GGML_UNARY_OP_SILU:
case GGML_UNARY_OP_EXP:
ggml/src/iqk/iqk_common.h
+3 -2
View File
@@ -80,6 +80,7 @@ struct Perf {
#ifdef __AVX2__
#define MM256_SET_M128I(a, b) _mm256_insertf128_si256(_mm256_castsi128_si256(b), (a), 1)
#define MM256_SET1_M128I(x) _mm256_broadcastsi128_si256(x)
#endif
typedef struct {
@@ -281,7 +282,7 @@ static inline __m128i load_iq4nl_values_128() {
static inline __m256i load_iq4nl_values_256() {
auto val128 = load_iq4nl_values_128();
return MM256_SET_M128I(val128, val128);
return MM256_SET1_M128I(val128);
}
#ifdef HAVE_FANCY_SIMD
@@ -297,7 +298,7 @@ static inline __m128i load_iq4k_values_128() {
static inline __m256i load_iq4k_values_256() {
auto val128 = load_iq4k_values_128();
return MM256_SET_M128I(val128, val128);
return MM256_SET1_M128I(val128);
}
template <int nrc, typename block_q8 = block_q8_K> struct Q8 {
ggml/src/iqk/iqk_gemm_1bit.cpp
+6 -6
View File
@@ -817,7 +817,7 @@ void mul_mat_iq1_s_q8_K(int n, const void * vx, size_t bx, const DataInfo& info,
auto deltas_l = _mm_unpacklo_epi16(deltas128, deltas128);
auto deltas_h = _mm_unpackhi_epi16(deltas128, deltas128);
auto deltas = MM256_SET_M128I(deltas_h, deltas_l); // blocks 0,0, 1,1, 2,2, ..., 7,7
auto all_scales = MM256_SET_M128I(scales128, scales128);
auto all_scales = MM256_SET1_M128I(scales128);
auto shuffle = shuffle0;
for (int ib64 = 0; ib64 < QK_K/64; ++ib64) {
scales[ib64] = _mm256_shuffle_epi8(all_scales, shuffle);
@@ -883,7 +883,7 @@ void mul_mat_iq1_m_q8_K(int n, const void * vx, size_t bx, const DataInfo& info,
auto qs = iq1m[ibl].qs;
auto qh = iq1m[ibl].qh;
auto aux = _mm_loadl_epi64((const __m128i *)iq1m[ibl].scales);
auto sc16 = _mm256_shuffle_epi8(MM256_SET_M128I(aux, aux), scale_shuffle);
auto sc16 = _mm256_shuffle_epi8(MM256_SET1_M128I(aux), scale_shuffle);
sc16 = _mm256_and_si256(sc16, _mm256_set1_epi64x(0x0e0001c000380007));
sc16 = _mm256_mullo_epi16(sc16, _mm256_set1_epi64x(0x0001000800400200));
helper.vec = _mm256_add_epi8(_mm256_srli_epi16(sc16, 8), _mm256_set1_epi16(1));
@@ -1036,7 +1036,7 @@ static void mul_mat_iq1_s_r4_q8_1(int n, const void * vx, size_t bx, const DataI
auto delta4 = _mm_mul_ps(_mm_set1_ps(0.0625f), _mm_cvtepi32_ps(_mm_cvtepi16_epi32(signs)));
auto delta = _mm256_set_m128(delta4, delta4);
scales4 = _mm_unpacklo_epi16(scales4, scales4); // 0,0, 1,1, 2,2, 3,3
auto scales = MM256_SET_M128I(scales4, scales4);
auto scales = MM256_SET1_M128I(scales4);
auto idxl = _mm256_cvtepu8_epi16(_mm_loadu_si128((const __m128i *)x[4*ib+k].qs));
idxh = _mm256_sllv_epi64(idxh, _mm256_set_epi64x(0, 2, 5, 8));
idxh = _mm256_srlv_epi64(idxh, _mm256_set_epi64x(1, 0, 0, 0));
@@ -1118,7 +1118,7 @@ static void mul_mat_iq1_m_r4_q8_0(int n, const void * vx, size_t bx, const DataI
auto signs128 = _mm_or_si128(_mm_cmpeq_epi8(_mm_and_si128(idxh, ms), ms), _mm_set1_epi8(1));
signs128 = _mm_add_epi8(_mm_set1_epi8(-8), signs128);
auto signs = MM256_SET_M128I(signs128, signs128);
auto signs = MM256_SET1_M128I(signs128);
auto idxl = _mm256_cvtepu8_epi16(_mm_loadu_si128((const __m128i *)x[4*ib+k].qs));
idxh = _mm_and_si128(idxh, _mm_set1_epi8(0x07));
helper.vec = _mm256_or_si256(idxl, _mm256_slli_epi16(_mm256_cvtepu8_epi16(idxh), 8));
@@ -1228,7 +1228,7 @@ struct DequantizerIQ1BN {
IQK_ALWAYS_INLINE void prepare_iq1bn_quants(const block_iq1_bn * x, __m256i& v1, __m256i& v2) const {
auto data128 = _mm_loadu_si128((const __m128i *)x); // Note: we load 16 instead of 13 bytes!
auto data = MM256_SET_M128I(data128, data128);
auto data = MM256_SET1_M128I(data128);
auto val1 = _mm256_mulhi_epu16(_mm256_mullo_epi16(_mm256_shuffle_epi8(data, shuff[0]), mult[0]), m3);
auto val2 = _mm256_mulhi_epu16(_mm256_mullo_epi16(_mm256_shuffle_epi8(data, shuff[1]), mult[1]), m3);
auto val3 = _mm256_mulhi_epu16(_mm256_mullo_epi16(_mm256_shuffle_epi8(data, shuff[2]), mult[2]), m3);
@@ -1476,7 +1476,7 @@ static void mul_mat_q1_0_g128_q8_0(int n, const void * vx, size_t bx, const Data
}
#else
auto bits128 = _mm_loadu_si128((const __m128i *)x[ib].qs);
auto bits = MM256_SET_M128I(bits128, bits128);
auto bits = MM256_SET1_M128I(bits128);
for (int k = 0; k < 4; ++k) {
qx[k] = _mm256_shuffle_epi8(bits, shuffle[k]);
qx[k] = _mm256_cmpeq_epi8(_mm256_and_si256(qx[k], mask), mask);
ggml/src/iqk/iqk_gemm_floats.cpp
+2 -2
View File
@@ -352,7 +352,7 @@ static void mul_mat_bf16_r16_bf16(int n, const void * vx, size_t bx, const DataI
static_for<nrc_y>([&](const int iy) {
auto y128 = _mm_loadu_si128((const __m128i*)y[iy]+ib);
//auto y = _mm512_broadcast_i32x4(y128);
auto y256 = MM256_SET_M128I(y128, y128);
auto y256 = MM256_SET1_M128I(y128);
auto y = _mm512_inserti32x8(_mm512_castsi256_si512(y256), y256, 1);
acc[2*iy+0] = _mm512_dpbf16_ps(acc[2*iy+0], qx[0], (__m512bh)_mm512_shuffle_epi32(y, _MM_PERM_ENUM(0x00)));
acc[2*iy+0] = _mm512_dpbf16_ps(acc[2*iy+0], qx[1], (__m512bh)_mm512_shuffle_epi32(y, _MM_PERM_ENUM(0x55)));
@@ -380,7 +380,7 @@ static void mul_mat_bf16_r16_bf16(int n, const void * vx, size_t bx, const DataI
qx[3] = (__m512bh)_mm512_loadu_si512((const __m512i *)b8+4*ib+3);
static_for<nrc_y>([&](const int iy) {
auto y128 = _mm_loadu_si128((const __m128i*)y[iy]+ib);
auto y256 = MM256_SET_M128I(y128, y128);
auto y256 = MM256_SET1_M128I(y128);
auto y = _mm512_inserti32x8(_mm512_castsi256_si512(y256), y256, 1);
acc[iy] = _mm512_dpbf16_ps(acc[iy], qx[0], (__m512bh)_mm512_shuffle_epi32(y, _MM_PERM_ENUM(0x00)));
acc[iy] = _mm512_dpbf16_ps(acc[iy], qx[1], (__m512bh)_mm512_shuffle_epi32(y, _MM_PERM_ENUM(0x55)));
ggml/src/iqk/iqk_gemm_iqk_quants.cpp
+68 -68
View File
@@ -24,7 +24,7 @@ struct IQXKScales {
const __m256i prod = _mm256_madd_epi16(scales16, q8.load_bsums(iy, i));
accm[iy] = _mm256_fmadd_ps(_mm256_set1_ps(d * q8.scale(iy, i)), _mm256_cvtepi32_ps(prod), accm[iy]);
}
scales16 = MM256_SET_M128I(scales8, scales8);
scales16 = MM256_SET1_M128I(scales8);
scales[0] = _mm512_cvtepi8_epi16(_mm256_shuffle_epi8(scales16, shuffle1));
scales[1] = _mm512_cvtepi8_epi16(_mm256_shuffle_epi8(scales16, shuffle2));
}
@@ -50,8 +50,8 @@ struct IQXKScales2 {
const __m256i prod = _mm256_madd_epi16(scales_s, q8.load_bsums(iy, i));
accm[iy] = _mm256_fmadd_ps(_mm256_set1_ps(d * q8.scale(iy, i)), _mm256_cvtepi32_ps(prod), accm[iy]);
}
auto aux_1 = MM256_SET_M128I(_mm256_castsi256_si128(scales16), _mm256_castsi256_si128(scales16));
auto aux_2 = MM256_SET_M128I(_mm256_extracti128_si256(scales16, 1), _mm256_extracti128_si256(scales16, 1));
auto aux_1 = MM256_SET1_M128I(_mm256_castsi256_si128(scales16));
auto aux_2 = MM256_SET1_M128I(_mm256_extracti128_si256(scales16, 1));
auto scales16_1 = _mm512_inserti32x8(_mm512_castsi256_si512(aux_1), aux_1, 1);
auto scales16_2 = _mm512_inserti32x8(_mm512_castsi256_si512(aux_2), aux_2, 1);
scales[0] = _mm512_shuffle_epi8(scales16_1, shuffles[0]);
@@ -81,7 +81,7 @@ struct DequantizerIQ2KS final : public BaseDequantizer<block_iq2_ks, true, true>
auto shifts = _mm_and_si128(_mm_cmpeq_epi8(_mm_and_si128(_mm_set1_epi8(x[i].extra), hmask), hmask), m5);
auto mins128 = _mm_mullo_epi16(scales128, _mm_cvtepi8_epi16(_mm_add_epi8(m32, shifts)));
auto mins = MM256_SET_M128I(_mm_shuffle_epi8(mins128, s8k.shuffles[1]), _mm_shuffle_epi8(mins128, s8k.shuffles[0]));
auto scales256 = MM256_SET_M128I(scales128, scales128);
auto scales256 = MM256_SET1_M128I(scales128);
auto all_scales = _mm512_inserti32x8(_mm512_castsi256_si512(scales256), scales256, 1);
__m512i scales[4];
for (int k = 0; k < 4; ++k) scales[k] = _mm512_shuffle_epi8(all_scales, shuffles[k]);
@@ -106,7 +106,7 @@ struct DequantizerIQ2KS final : public BaseDequantizer<block_iq2_ks, true, true>
static inline __m512i load_values() {
static const uint8_t kvalues_iq2nl[16] = {1, 19, 33, 49, 0, 0, 0, 0, 6, 24, 38, 54, 0, 0, 0, 0};
auto val128 = _mm_loadu_si128((const __m128i *)kvalues_iq2nl);
auto val256 = MM256_SET_M128I(val128, val128);
auto val256 = MM256_SET1_M128I(val128);
return _mm512_inserti32x8(_mm512_castsi256_si512(val256), val256, 1);
}
inline __m128i make_scales(const uint8_t * scales_l, uint8_t scales_h) const {
@@ -154,7 +154,7 @@ struct DequantizerIQ2K final : public BaseDequantizer<block_iq2_k> {
static inline __m512i load_values() {
static const uint8_t kvalues_iq2nl[16] = {1, 19, 33, 49, 0, 0, 0, 0, 6, 24, 38, 54, 0, 0, 0, 0};
auto val128 = _mm_loadu_si128((const __m128i *)kvalues_iq2nl);
auto val256 = MM256_SET_M128I(val128, val128);
auto val256 = MM256_SET1_M128I(val128);
return _mm512_inserti32x8(_mm512_castsi256_si512(val256), val256, 1);
}
inline __m128i make_scales(const uint8_t * scales_l) const {
@@ -193,7 +193,7 @@ struct DequantizerIQ3K final : public BaseDequantizer<block_iq3_k> {
static inline __m512i load_values() {
static const uint8_t kvalues_iq3nl[16] = {1, 24, 41, 54, 65, 77, 92, 111, 5, 28, 45, 58, 69, 81, 96, 115};
auto val128 = _mm_loadu_si128((const __m128i *)kvalues_iq3nl);
auto val256 = MM256_SET_M128I(val128, val128);
auto val256 = MM256_SET1_M128I(val128);
return _mm512_inserti32x8(_mm512_castsi256_si512(val256), val256, 1);
}
inline __m128i make_scales(uint16_t signs, const uint8_t * scales_l) const {
@@ -227,7 +227,7 @@ struct DequantizerIQ3KS final : public BaseDequantizer<block_iq3_ks, true, true>
auto shifts = _mm_mask_add_epi16(m64, __mmask8(x[i].extra >> 8), m64, _mm_set1_epi16(4));
auto mins128 = _mm_mullo_epi16(scales128, shifts);
auto mins = MM256_SET_M128I(_mm_shuffle_epi8(mins128, s8k.shuffles[1]), _mm_shuffle_epi8(mins128, s8k.shuffles[0]));
auto scales256 = MM256_SET_M128I(scales128, scales128);
auto scales256 = MM256_SET1_M128I(scales128);
auto all_scales = _mm512_inserti32x8(_mm512_castsi256_si512(scales256), scales256, 1);
__m512i scales[4];
for (int k = 0; k < 4; ++k) scales[k] = _mm512_shuffle_epi8(all_scales, shuffles[k]);
@@ -259,7 +259,7 @@ struct DequantizerIQ3KS final : public BaseDequantizer<block_iq3_ks, true, true>
static inline __m512i load_values() {
static const uint8_t kvalues_iq3nl[16] = {1, 24, 41, 54, 65, 77, 92, 111, 5, 28, 45, 58, 69, 81, 96, 115};
auto val128 = _mm_loadu_si128((const __m128i *)kvalues_iq3nl);
auto val256 = MM256_SET_M128I(val128, val128);
auto val256 = MM256_SET1_M128I(val128);
return _mm512_inserti32x8(_mm512_castsi256_si512(val256), val256, 1);
}
@@ -310,7 +310,7 @@ struct DequantizerIQ4KSS final : public BaseDequantizer<block_iq4_kss, true> {
scales128 = _mm_add_epi16(_mm_and_si128(scales128, mask), m127);
auto scales_s = _mm_mullo_epi16(scales128, _mm_add_epi16(m128, shifts));
s8k.accum_mins(scales_s, q8, i, d, accm);
auto scales256 = MM256_SET_M128I(scales128, scales128);
auto scales256 = MM256_SET1_M128I(scales128);
auto all_scales = _mm512_inserti32x8(_mm512_castsi256_si512(scales256), scales256, 1);
scales[0] = _mm512_shuffle_epi8(all_scales, shuffles[0]);
scales[1] = _mm512_shuffle_epi8(all_scales, shuffles[1]);
@@ -351,7 +351,7 @@ struct DequantizerIQ2KL final : public BaseDequantizer<block_iq2_kl, true, true>
auto scales128 = make_scales(i);
auto mins128 = _mm_mullo_epi16(scales128, _mm_set1_epi16(-64));
auto mins = MM256_SET_M128I(_mm_shuffle_epi8(mins128, s8k.shuffles[1]), _mm_shuffle_epi8(mins128, s8k.shuffles[0]));
auto scales256 = MM256_SET_M128I(scales128, scales128);
auto scales256 = MM256_SET1_M128I(scales128);
auto all_scales = _mm512_inserti32x8(_mm512_castsi256_si512(scales256), scales256, 1);
__m512i scales[4];
for (int k = 0; k < 4; ++k) scales[k] = _mm512_shuffle_epi8(all_scales, shuffles[k]);
@@ -406,7 +406,7 @@ struct DequantizerIQ2KL final : public BaseDequantizer<block_iq2_kl, true, true>
};
for (int k = 0; k < 4; ++k) {
auto v128 = _mm_loadu_si128((const __m128i *)k_values + k);
auto v256 = MM256_SET_M128I(v128, v128);
auto v256 = MM256_SET1_M128I(v128);
values[k] = _mm512_inserti32x8(_mm512_castsi256_si512(v256), v256, 1);
}
}
@@ -437,7 +437,7 @@ struct DequantizerIQ4KS final : public BaseDequantizer<block_iq4_ks, true> {
scales128 = _mm_add_epi16(_mm_and_si128(scales128, mask), m127);
auto scales_s = _mm_mullo_epi16(scales128, _mm_add_epi16(m128, shifts));
s8k.accum_mins(scales_s, q8, i, d, accm);
auto scales256 = MM256_SET_M128I(scales128, scales128);
auto scales256 = MM256_SET1_M128I(scales128);
auto all_scales = _mm512_inserti32x8(_mm512_castsi256_si512(scales256), scales256, 1);
scales[0] = _mm512_shuffle_epi8(all_scales, shuffles[0]);
scales[1] = _mm512_shuffle_epi8(all_scales, shuffles[1]);
@@ -452,7 +452,7 @@ struct DequantizerIQ4KS final : public BaseDequantizer<block_iq4_ks, true> {
scales128 = _mm_add_epi16(_mm_and_si128(scales128, mask), m127);
auto mins128 = _mm_mullo_epi16(scales128, _mm_add_epi16(m128, shifts));
auto mins = MM256_SET_M128I(_mm_shuffle_epi8(mins128, s8k.shuffles[1]), _mm_shuffle_epi8(mins128, s8k.shuffles[0]));
auto scales256 = MM256_SET_M128I(scales128, scales128);
auto scales256 = MM256_SET1_M128I(scales128);
auto all_scales = _mm512_inserti32x8(_mm512_castsi256_si512(scales256), scales256, 1);
__m512i scales[4];
for (int k = 0; k < 4; ++k) scales[k] = _mm512_shuffle_epi8(all_scales, shuffles[k]);
@@ -548,7 +548,7 @@ struct DequantizerIQ5KS final : public BaseDequantizer<block_iq5_ks, true> {
scales128 = _mm_add_epi16(_mm_and_si128(scales128, mask), m127);
auto scales_s = _mm_mullo_epi16(scales128, _mm_add_epi16(m128, shifts));
s8k.accum_mins(scales_s, q8, i, d, accm);
auto scales256 = MM256_SET_M128I(scales128, scales128);
auto scales256 = MM256_SET1_M128I(scales128);
auto all_scales = _mm512_inserti32x8(_mm512_castsi256_si512(scales256), scales256, 1);
scales[0] = _mm512_shuffle_epi8(all_scales, shuffles[0]);
scales[1] = _mm512_shuffle_epi8(all_scales, shuffles[1]);
@@ -563,7 +563,7 @@ struct DequantizerIQ5KS final : public BaseDequantizer<block_iq5_ks, true> {
scales128 = _mm_add_epi16(_mm_and_si128(scales128, mask), m127);
auto mins128 = _mm_mullo_epi16(scales128, _mm_add_epi16(m128, shifts));
auto mins = MM256_SET_M128I(_mm_shuffle_epi8(mins128, s8k.shuffles[1]), _mm_shuffle_epi8(mins128, s8k.shuffles[0]));
auto scales256 = MM256_SET_M128I(scales128, scales128);
auto scales256 = MM256_SET1_M128I(scales128);
auto all_scales = _mm512_inserti32x8(_mm512_castsi256_si512(scales256), scales256, 1);
__m512i scales[4];
for (int k = 0; k < 4; ++k) scales[k] = _mm512_shuffle_epi8(all_scales, shuffles[k]);
@@ -600,8 +600,8 @@ struct DequantizerIQ5KS final : public BaseDequantizer<block_iq5_ks, true> {
};
auto values128_1 = _mm_loadu_si128((const __m128i *)kvalues_iq5nl + 0);
auto values128_2 = _mm_loadu_si128((const __m128i *)kvalues_iq5nl + 1);
auto values256_1 = MM256_SET_M128I(values128_1, values128_1);
auto values256_2 = MM256_SET_M128I(values128_2, values128_2);
auto values256_1 = MM256_SET1_M128I(values128_1);
auto values256_2 = MM256_SET1_M128I(values128_2);
values[0] = _mm512_inserti32x8(_mm512_castsi256_si512(values256_1), values256_1, 1);
values[1] = _mm512_inserti32x8(_mm512_castsi256_si512(values256_2), values256_2, 1);
}
@@ -671,8 +671,8 @@ struct DequantizerIQ5K final : public BaseDequantizer<block_iq5_k> {
};
auto values128_1 = _mm_loadu_si128((const __m128i *)kvalues_iq5nl + 0);
auto values128_2 = _mm_loadu_si128((const __m128i *)kvalues_iq5nl + 1);
auto values256_1 = MM256_SET_M128I(values128_1, values128_1);
auto values256_2 = MM256_SET_M128I(values128_2, values128_2);
auto values256_1 = MM256_SET1_M128I(values128_1);
auto values256_2 = MM256_SET1_M128I(values128_2);
values[0] = _mm512_inserti32x8(_mm512_castsi256_si512(values256_1), values256_1, 1);
values[1] = _mm512_inserti32x8(_mm512_castsi256_si512(values256_2), values256_2, 1);
}
@@ -734,7 +734,7 @@ struct DequantizerIQ6K final : public BaseDequantizer<block_iq6_k> {
};
for (int k = 0; k < 4; ++k) {
auto values128 = _mm_loadu_si128((const __m128i *)kvalues_iq6nl + k);
auto values256 = MM256_SET_M128I(values128, values128);
auto values256 = MM256_SET1_M128I(values128);
values[k] = _mm512_inserti32x8(_mm512_castsi256_si512(values256), values256, 1);
}
}
@@ -924,8 +924,8 @@ static void mul_mat_qX_K_q8_K_AVX512(int n, const void * vx, size_t bx, const Da
inline void prepare_scales_16(const __m256i& all_scales, __m256i * scales) {
const __m128i l_scales = _mm256_extracti128_si256(all_scales, 0);
const __m128i h_scales = _mm256_extracti128_si256(all_scales, 1);
scales[0] = MM256_SET_M128I(l_scales, l_scales);
scales[1] = MM256_SET_M128I(h_scales, h_scales);
scales[0] = MM256_SET1_M128I(l_scales);
scales[1] = MM256_SET1_M128I(h_scales);
}
struct IQXKScales {
@@ -964,7 +964,7 @@ struct DequantizerIQ2KS final : public BaseDequantizer<block_iq2_ks, true, true>
auto shifts = _mm_and_si128(_mm_cmpeq_epi8(_mm_and_si128(_mm_set1_epi8(x[i].extra), hmask), hmask), m5);
auto scales_s = _mm_mullo_epi16(scales128, _mm_cvtepi8_epi16(_mm_add_epi8(m32, shifts)));
s8k.accum_mins(scales_s, q8, i, d, accm);
return MM256_SET_M128I(scales128, scales128);
return MM256_SET1_M128I(scales128);
}
inline void prepare(int i, int j) {
bits.prepare(x[i].qs, j);
@@ -976,7 +976,7 @@ struct DequantizerIQ2KS final : public BaseDequantizer<block_iq2_ks, true, true>
static inline __m256i load_values() {
static const uint8_t kvalues_iq2nl[16] = {1, 19, 33, 49, 0, 0, 0, 0, 6, 24, 38, 54, 0, 0, 0, 0};
auto val128 = _mm_loadu_si128((const __m128i *)kvalues_iq2nl);
return MM256_SET_M128I(val128, val128);
return MM256_SET1_M128I(val128);
}
inline __m128i make_scales(const uint8_t * scales_l, uint8_t scales_h) const {
const uint16_t * scales = (const uint16_t *)scales_l;
@@ -1008,7 +1008,7 @@ struct DequantizerIQ2KL final : public BaseDequantizer<block_iq2_kl, true, true>
auto scales128 = make_scales(i);
auto scales_s = _mm_mullo_epi16(scales128, _mm_set1_epi16(-64));
s8k.accum_mins(scales_s, q8, i, d, accm);
return MM256_SET_M128I(scales128, scales128);
return MM256_SET1_M128I(scales128);
}
inline void prepare(int i, int j) {
__m256i ql[2], mask[2];
@@ -1050,7 +1050,7 @@ struct DequantizerIQ2KL final : public BaseDequantizer<block_iq2_kl, true, true>
};
for (int k = 0; k < 4; ++k) {
auto v128 = _mm_loadu_si128((const __m128i *)k_values + k);
values[k] = MM256_SET_M128I(v128, v128);
values[k] = MM256_SET1_M128I(v128);
}
}
struct { __m256i values[4]; } bits;
@@ -1078,7 +1078,7 @@ struct DequantizerIQ2K final : public BaseDequantizer<block_iq2_k> {
static inline __m256i load_values() {
static const uint8_t kvalues_iq2nl[16] = {1, 19, 33, 49, 0, 0, 0, 0, 6, 24, 38, 54, 0, 0, 0, 0};
auto val128 = _mm_loadu_si128((const __m128i *)kvalues_iq2nl);
return MM256_SET_M128I(val128, val128);
return MM256_SET1_M128I(val128);
}
inline __m128i make_scales(const uint8_t * scales_l) const {
uint64_t aux64; std::memcpy(&aux64, scales_l, 8);
@@ -1116,7 +1116,7 @@ struct DequantizerIQ3K final : public BaseDequantizer<block_iq3_k> {
static inline __m256i load_values() {
static const uint8_t kvalues_iq3nl[16] = {1, 24, 41, 54, 65, 77, 92, 111, 5, 28, 45, 58, 69, 81, 96, 115};
auto val128 = _mm_loadu_si128((const __m128i *)kvalues_iq3nl);
return MM256_SET_M128I(val128, val128);
return MM256_SET1_M128I(val128);
}
inline __m128i make_scales(uint16_t signs, const uint8_t * scales_l) const {
uint64_t aux64; std::memcpy(&aux64, scales_l, 8);
@@ -1147,7 +1147,7 @@ struct DequantizerIQ3KS final : public BaseDequantizer<block_iq3_ks, true, true>
auto sch = _mm_cmpeq_epi16(_mm_and_si128(_mm_set1_epi16(x[i].extra), mask), mask);
auto scales128 = _mm_add_epi16(scl, _mm_and_si128(sch, _mm_set1_epi16(16)));
scales128 = _mm_sub_epi16(scales128, _mm_set1_epi16(16));
return MM256_SET_M128I(scales128, scales128);
return MM256_SET1_M128I(scales128);
}
inline void prepare(int i, int j) {
uint8_t extra = x[i].extra >> (8 + 4*j);
@@ -1161,7 +1161,7 @@ struct DequantizerIQ3KS final : public BaseDequantizer<block_iq3_ks, true, true>
}
inline __m256i load_values() {
auto v = _mm_loadu_si128((const __m128i *)iq3nl_values);
return MM256_SET_M128I(v, v);
return MM256_SET1_M128I(v);
}
@@ -1193,7 +1193,7 @@ struct DequantizerIQ4KSS final : public BaseDequantizer<block_iq4_kss, true> {
scales128 = _mm_add_epi16(_mm_and_si128(scales128, mask), m127);
auto scales_s = _mm_mullo_epi16(scales128, _mm_add_epi16(m128, shifts));
s8k.accum_mins(scales_s, q8, i, d, accd);
return MM256_SET_M128I(scales128, scales128);
return MM256_SET1_M128I(scales128);
}
inline void prepare(int, int j) {
for (int k = 0; k < 2; ++k) {
@@ -1226,7 +1226,7 @@ struct DequantizerIQ4KS final : public BaseDequantizer<block_iq4_ks, true> {
inline __m256i new_block(int i, [[maybe_unused]] const Q8& q8, [[maybe_unused]] __m256 * accd) {
auto scales128 = _mm_cvtepu8_epi16(_mm_loadl_epi64((const __m128i *)x[i].scales));
scales128 = _mm_add_epi16(_mm_and_si128(scales128, mask), m127);
return MM256_SET_M128I(scales128, scales128);
return MM256_SET1_M128I(scales128);
}
inline void prepare(int i, int j) {
bits.prepare16(x[i].qs, j);
@@ -1238,8 +1238,8 @@ struct DequantizerIQ4KS final : public BaseDequantizer<block_iq4_ks, true> {
void load_values() {
auto v1 = _mm_loadu_si128((const __m128i *)iq4k_values+0);
auto v2 = _mm_loadu_si128((const __m128i *)iq4k_values+1);
values[0] = MM256_SET_M128I(v1, v1);
values[1] = MM256_SET_M128I(v2, v2);
values[0] = MM256_SET1_M128I(v1);
values[1] = MM256_SET1_M128I(v2);
}
@@ -1278,10 +1278,10 @@ struct DequantizerIQ4K final : public BaseDequantizer<block_iq4_k> {
void load_values() {
auto v1 = _mm_loadu_si128((const __m128i *)iq4k_values+0);
auto v2 = _mm_loadu_si128((const __m128i *)iq4k_values+1);
values[0] = MM256_SET_M128I(v1, v1);
values[0] = MM256_SET1_M128I(v1);
values[1] = MM256_SET_M128I(v1, v2);
values[2] = MM256_SET_M128I(v2, v1);
values[3] = MM256_SET_M128I(v2, v2);
values[3] = MM256_SET1_M128I(v2);
}
Q4Bits bits;
@@ -1302,7 +1302,7 @@ struct DequantizerIQ5KS final : public BaseDequantizer<block_iq5_ks, true> {
scales128 = _mm_add_epi16(_mm_and_si128(scales128, mask), m127);
auto scales_s = _mm_mullo_epi16(scales128, _mm_add_epi16(m128, shifts));
s8k.accum_mins(scales_s, q8, i, d, accd);
return MM256_SET_M128I(scales128, scales128);
return MM256_SET1_M128I(scales128);
}
inline void prepare(int i, int j) {
bits.prepare(x[i].qs, j);
@@ -1321,8 +1321,8 @@ struct DequantizerIQ5KS final : public BaseDequantizer<block_iq5_ks, true> {
};
auto values128_1 = _mm_loadu_si128((const __m128i *)kvalues_iq5nl + 0);
auto values128_2 = _mm_loadu_si128((const __m128i *)kvalues_iq5nl + 1);
values[0] = MM256_SET_M128I(values128_1, values128_1);
values[1] = MM256_SET_M128I(values128_2, values128_2);
values[0] = MM256_SET1_M128I(values128_1);
values[1] = MM256_SET1_M128I(values128_2);
}
Q4Bits bits;
@@ -1369,8 +1369,8 @@ struct DequantizerIQ5K final : public BaseDequantizer<block_iq5_k> {
static void load_values(__m256i * values) {
auto values128_1 = _mm_loadu_si128((const __m128i *)iq5nl_values + 0);
auto values128_2 = _mm_loadu_si128((const __m128i *)iq5nl_values + 1);
values[0] = MM256_SET_M128I(values128_1, values128_1);
values[1] = MM256_SET_M128I(values128_2, values128_2);
values[0] = MM256_SET1_M128I(values128_1);
values[1] = MM256_SET1_M128I(values128_2);
}
Q4Bits bits;
@@ -1414,7 +1414,7 @@ struct DequantizerIQ6K final : public BaseDequantizer<block_iq6_k> {
static void load_values(__m256i * values) {
for (int k = 0; k < 4; ++k) {
auto values128 = _mm_loadu_si128((const __m128i *)iq6nl_values + k);
values[k] = MM256_SET_M128I(values128, values128);
values[k] = MM256_SET1_M128I(values128);
}
}
@@ -1740,7 +1740,7 @@ static void mul_mat_iq3_k_r4_q8_k(int n, const void * vx, size_t bx, const DataI
auto smask = _mm256_set_epi64x(0x0808080808080808, 0x0404040404040404, 0x0202020202020202, 0x0101010101010101);
auto shift_shuffle = _mm256_set_epi64x(0x0707070706060606, 0x0505050504040404, 0x0303030302020202, 0x0101010100000000);
auto values128 = _mm_loadu_si128((const __m128i *)iq3nl_values);
auto values = MM256_SET_M128I(values128, values128);
auto values = MM256_SET1_M128I(values128);
values = _mm256_add_epi8(values, _mm256_set1_epi8(64));
static const uint8_t k_shuff[32] = {0, 1, 8, 9, 2, 3, 10, 11, 4, 5, 12, 13, 6, 7, 14, 15, 0, 1, 8, 9, 2, 3, 10, 11, 4, 5, 12, 13, 6, 7, 14, 15};
auto shuff = _mm256_loadu_si256((const __m256i *)k_shuff);
@@ -1835,7 +1835,7 @@ static void mul_mat_iq4_k_r4_q8_k(int n, const void * vx, size_t bx, const DataI
#else
auto s_shuffle = _mm256_set_epi64x(0x0f0e0f0e0d0c0d0c, 0x0b0a0b0a09080908, 0x0706070605040504, 0x0302030201000100);
auto values128 = _mm_loadu_si128((const __m128i *)iq4k_values);
auto values = MM256_SET_M128I(values128, values128);
auto values = MM256_SET1_M128I(values128);
#endif
int nbl = n / QK_K;
__m256 acc[nrc_y] = {};
@@ -1934,8 +1934,8 @@ static void mul_mat_iq5_k_r4_q8_k(int n, const void * vx, size_t bx, const DataI
{
auto val1 = _mm_loadu_si128((const __m128i *)iq5nl_values+0);
auto val2 = _mm_loadu_si128((const __m128i *)iq5nl_values+1);
values[0] = MM256_SET_M128I(val1, val1);
values[1] = MM256_SET_M128I(val2, val2);
values[0] = MM256_SET1_M128I(val1);
values[1] = MM256_SET1_M128I(val2);
#ifdef HAVE_FANCY_SIMD
values[0] = _mm256_sub_epi8(values[0], _mm256_set1_epi8(-128));
values[1] = _mm256_sub_epi8(values[1], _mm256_set1_epi8(-128));
@@ -2053,7 +2053,7 @@ static void mul_mat_iq4_ks_r4_q8_k(int n, const void * vx, size_t bx, const Data
#ifndef HAVE_FANCY_SIMD
auto s_shuffle = _mm256_set_epi64x(0x0f0e0f0e0d0c0d0c, 0x0b0a0b0a09080908, 0x0706070605040504, 0x0302030201000100);
auto values128 = _mm_loadu_si128((const __m128i *)iq4k_values);
auto values = MM256_SET_M128I(values128, values128);
auto values = MM256_SET1_M128I(values128);
#else
auto values = load_iq4nl_values_256();
#endif
@@ -2163,8 +2163,8 @@ static void mul_mat_iq5_ks_r4_q8_k(int n, const void * vx, size_t bx, const Data
{
auto val1 = _mm_loadu_si128((const __m128i *)iq5nl_values+0);
auto val2 = _mm_loadu_si128((const __m128i *)iq5nl_values+1);
values[0] = MM256_SET_M128I(val1, val1);
values[1] = MM256_SET_M128I(val2, val2);
values[0] = MM256_SET1_M128I(val1);
values[1] = MM256_SET1_M128I(val2);
#ifdef HAVE_FANCY_SIMD
values[0] = _mm256_sub_epi8(values[0], _mm256_set1_epi8(-128));
values[1] = _mm256_sub_epi8(values[1], _mm256_set1_epi8(-128));
@@ -2325,7 +2325,7 @@ void iqk_convert_iq2_ks_q8_k_r8(int n, const void * vx, size_t bx, void * vy, in
__m256i values;
{
auto v = _mm_loadl_epi64((const __m128i *)iq2nl_values);
values = MM256_SET_M128I(v, v);
values = MM256_SET1_M128I(v);
}
ggml_half dh[k_nr];
@@ -2404,7 +2404,7 @@ void iqk_convert_iq2_k_q8_k_r8(int n, const void * vx, size_t bx, void * vy, int
__m256i values;
{
auto v = _mm_loadl_epi64((const __m128i *)iq2nl_values);
values = MM256_SET_M128I(v, v);
values = MM256_SET1_M128I(v);
}
__m256i xv[8];
@@ -2485,7 +2485,7 @@ void iqk_convert_iq2_kl_q8_k_r8(int n, const void * vx, size_t bx, void * vy, in
};
for (int k = 0; k < 4; ++k) {
auto v = _mm_loadu_si128((const __m128i *)k_values + k);
values[k] = MM256_SET_M128I(v, v);
values[k] = MM256_SET1_M128I(v);
}
}
@@ -2584,7 +2584,7 @@ void iqk_convert_iq3_ks_q8_k_r8(int n, const void * vx, size_t bx, void * vy, in
__m256i values;
{
auto v = _mm_loadu_si128((const __m128i *)iq3nl_values);
values = MM256_SET_M128I(v, v);
values = MM256_SET1_M128I(v);
}
ggml_half drow[k_nr];
@@ -2662,7 +2662,7 @@ void iqk_convert_iq3_k_q8_k_r8(int n, const void * vx, size_t bx, void * vy, int
__m256i values;
{
auto v = _mm_loadu_si128((const __m128i *)iq3nl_values);
values = MM256_SET_M128I(v, v);
values = MM256_SET1_M128I(v);
}
__m256i xv[8];
@@ -2751,8 +2751,8 @@ void iqk_convert_iq4_kss_q8_k_r8(int n, const void * vx, size_t bx, void * vy, i
{
auto v1 = _mm_loadu_si128((const __m128i *)iq4k_values+0);
auto v2 = _mm_loadu_si128((const __m128i *)iq4k_values+1);
values[0] = MM256_SET_M128I(v1, v1);
values[1] = MM256_SET_M128I(v2, v2);
values[0] = MM256_SET1_M128I(v1);
values[1] = MM256_SET1_M128I(v2);
}
float drow[k_nr];
@@ -2824,8 +2824,8 @@ void iqk_convert_iq4_ks_q8_k_r8(int n, const void * vx, size_t bx, void * vy, in
{
auto v1 = _mm_loadu_si128((const __m128i *)iq4k_values+0);
auto v2 = _mm_loadu_si128((const __m128i *)iq4k_values+1);
values[0] = MM256_SET_M128I(v1, v1);
values[1] = MM256_SET_M128I(v2, v2);
values[0] = MM256_SET1_M128I(v1);
values[1] = MM256_SET1_M128I(v2);
}
float drow[k_nr];
@@ -2892,10 +2892,10 @@ void iqk_convert_iq4_k_q8_k_r8(int n, const void * vx, size_t bx, void * vy, int
{
auto v1 = _mm_loadu_si128((const __m128i *)iq4k_values+0);
auto v2 = _mm_loadu_si128((const __m128i *)iq4k_values+1);
values[0] = MM256_SET_M128I(v1, v1);
values[0] = MM256_SET1_M128I(v1);
values[1] = MM256_SET_M128I(v1, v2);
values[2] = MM256_SET_M128I(v2, v1);
values[3] = MM256_SET_M128I(v2, v2);
values[3] = MM256_SET1_M128I(v2);
}
__m256i xv[8];
@@ -2964,8 +2964,8 @@ void iqk_convert_iq5_ks_q8_k_r8(int n, const void * vx, size_t bx, void * vy, in
{
auto v1 = _mm_loadu_si128((const __m128i *)iq5nl_values+0);
auto v2 = _mm_loadu_si128((const __m128i *)iq5nl_values+1);
values[0] = MM256_SET_M128I(v1, v1);
values[1] = MM256_SET_M128I(v2, v2);
values[0] = MM256_SET1_M128I(v1);
values[1] = MM256_SET1_M128I(v2);
}
float drow[k_nr];
@@ -3050,8 +3050,8 @@ void iqk_convert_iq5_k_q8_k_r8(int n, const void * vx, size_t bx, void * vy, int
{
auto v1 = _mm_loadu_si128((const __m128i *)iq5nl_values+0);
auto v2 = _mm_loadu_si128((const __m128i *)iq5nl_values+1);
values[0] = MM256_SET_M128I(v1, v1);
values[1] = MM256_SET_M128I(v2, v2);
values[0] = MM256_SET1_M128I(v1);
values[1] = MM256_SET1_M128I(v2);
}
__m256i xv[8];
@@ -3118,8 +3118,8 @@ void iqk_convert_iq5_k_q8_0_r8(int n, const void * vx, size_t bx, void * vy, int
{
auto v1 = _mm_loadu_si128((const __m128i *)iq5nl_values+0);
auto v2 = _mm_loadu_si128((const __m128i *)iq5nl_values+1);
values[0] = MM256_SET_M128I(v1, v1);
values[1] = MM256_SET_M128I(v2, v2);
values[0] = MM256_SET1_M128I(v1);
values[1] = MM256_SET1_M128I(v2);
}
__m256i xv[8];
@@ -3229,7 +3229,7 @@ void iqk_convert_iq6_k_q8_k_r8(int n, const void * vx, size_t bx, void * vy, int
__m256i values[4];
for (int k = 0; k < 4; ++k) {
auto values128 = _mm_loadu_si128((const __m128i *)iq6nl_values + k);
values[k] = MM256_SET_M128I(values128, values128);
values[k] = MM256_SET1_M128I(values128);
}
__m256i xv[8];
ggml/src/iqk/iqk_gemm_iquants.cpp
+21 -21
View File
@@ -126,7 +126,7 @@ struct SignHelper {
values[3] = _mm256_mask_sub_epi8(values[3], mask[3], _mm256_setzero_si256(), values[3]);
#else
auto s128 = _mm_loadu_si128((const __m128i *)sign_bits);
auto s256 = MM256_SET_M128I(s128, s128);
auto s256 = MM256_SET1_M128I(s128);
__m256i aux256;
auto shuffle = mask1;
auto step = _mm256_set1_epi8(4);
@@ -164,7 +164,7 @@ struct DequantizerIQ2XXS final : public BaseDequantizer<block_iq2_xxs> {
inline void new_block(int i, __m256i * scales) {
auto sc16 = load_scales(i);
scales[0] = MM256_SET_M128I(sc16, sc16);
scales[0] = MM256_SET1_M128I(sc16);
}
inline void new_block_f(int i, __m256 * scales) {
auto sc16 = load_scales(i);
@@ -179,7 +179,7 @@ struct DequantizerIQ2XXS final : public BaseDequantizer<block_iq2_xxs> {
inline float new_block(int i, __m256i * scales, __m256i& mins) {
auto sc16 = load_scales(i);
mins = scb.shuffle(sc16);
scales[0] = MM256_SET_M128I(sc16, sc16);
scales[0] = MM256_SET1_M128I(sc16);
return -d*minv;
}
@@ -252,8 +252,8 @@ struct DequantizerIQ2XS final : public BaseDequantizer<block_iq2_xs> {
inline static void prepare_scales(const __m256i& all, __m256i * scales) {
auto scales_l = _mm256_castsi256_si128(all);
auto scales_h = _mm256_extractf128_si256(all, 1);
scales[0] = MM256_SET_M128I(scales_l, scales_l);
scales[1] = MM256_SET_M128I(scales_h, scales_h);
scales[0] = MM256_SET1_M128I(scales_l);
scales[1] = MM256_SET1_M128I(scales_h);
}
inline void new_block(int i, __m256i * scales) {
@@ -321,8 +321,8 @@ struct DequantizerIQ2XS final : public BaseDequantizer<block_iq2_xs> {
auto full = _mm256_or_si256(psb1, oddb);
auto full_l = _mm256_castsi256_si128(full);
auto full_h = _mm256_extractf128_si256(full, 1);
auto full_1 = MM256_SET_M128I(full_l, full_l);
auto full_2 = MM256_SET_M128I(full_h, full_h);
auto full_1 = MM256_SET1_M128I(full_l);
auto full_2 = MM256_SET1_M128I(full_h);
sign_value(full_1, helper.shuff1, helper.mask, helper.mone, values[0]);
sign_value(full_1, helper.shuff2, helper.mask, helper.mone, values[1]);
sign_value(full_2, helper.shuff1, helper.mask, helper.mone, values[2]);
@@ -397,8 +397,8 @@ struct DequantizerIQ2S final : public BaseDequantizer<block_iq2_s> {
inline static void prepare_scales(const __m256i& all, __m256i * scales) {
auto scales_l = _mm256_castsi256_si128(all);
auto scales_h = _mm256_extractf128_si256(all, 1);
scales[0] = MM256_SET_M128I(scales_l, scales_l);
scales[1] = MM256_SET_M128I(scales_h, scales_h);
scales[0] = MM256_SET1_M128I(scales_l);
scales[1] = MM256_SET1_M128I(scales_h);
}
inline void new_block(int i, __m256i * scales) {
@@ -506,7 +506,7 @@ struct DequantizerIQ3XXS final : public BaseDequantizer<block_iq3_xxs> {
inline void new_block(int i, __m256i * scales) {
auto scales16 = prepare_scales(i);
scales[0] = MM256_SET_M128I(scales16, scales16);
scales[0] = MM256_SET1_M128I(scales16);
}
inline void new_block_f(int i, __m256 * scales) {
auto sc16 = prepare_scales(i);
@@ -520,7 +520,7 @@ struct DequantizerIQ3XXS final : public BaseDequantizer<block_iq3_xxs> {
inline float new_block(int i, __m256i * scales, __m256i& mins) {
auto scales16 = prepare_scales(i);
mins = scb.shuffle(scales16);
scales[0] = MM256_SET_M128I(scales16, scales16);
scales[0] = MM256_SET1_M128I(scales16);
return -d*minv;
}
@@ -638,7 +638,7 @@ struct DequantizerIQ3S final : public BaseDequantizer<block_iq3_s> {
}
inline void new_block(int i, __m256i * scales) {
auto scales16 = make_scales(i, d);
scales[0] = MM256_SET_M128I(scales16, scales16);
scales[0] = MM256_SET1_M128I(scales16);
}
inline void new_block_f(int i, __m256 * scales) {
auto sc16 = make_scales(i, d);
@@ -652,7 +652,7 @@ struct DequantizerIQ3S final : public BaseDequantizer<block_iq3_s> {
inline float new_block(int i, __m256i * scales, __m256i& mins) {
auto scales16 = make_scales(i, d);
mins = scb.shuffle(scales16);
scales[0] = MM256_SET_M128I(scales16, scales16);
scales[0] = MM256_SET1_M128I(scales16);
return -minv*d;
}
@@ -1056,7 +1056,7 @@ static void mul_mat_iq2_xxs_r4_q8_k(int n, const void * vx, size_t bx, const Dat
scales = _mm_maddubs_epi16(scales, _mm_set1_epi32(0x10080402));
scales = _mm_add_epi32(_mm_madd_epi16(_mm_set1_epi16(1), scales), _mm_set1_epi32(1));
#endif
auto scales32 = MM256_SET_M128I(scales, scales);
auto scales32 = MM256_SET1_M128I(scales);
auto signs128 = _mm_and_si128(sas, _mm_set1_epi8(-2)); // 0xfe = -2 as signed. Needed to shutup compiler warning.
signs128 = _mm_xor_si128(signs128, _mm_srli_epi16(signs128, 1));
#ifdef HAVE_FANCY_SIMD
@@ -1073,7 +1073,7 @@ static void mul_mat_iq2_xxs_r4_q8_k(int n, const void * vx, size_t bx, const Dat
isum[iy] = _mm256_add_epi32(isum[iy], _mm256_mullo_epi32(scales32, sumi));
}
#else
auto signs = MM256_SET_M128I(signs128, signs128);
auto signs = MM256_SET1_M128I(signs128);
auto shuffle = sign_shuffle;
auto s1 = _mm256_or_si256(_mm256_cmpeq_epi8(_mm256_and_si256(_mm256_shuffle_epi8(signs, shuffle), smask), smask), _mm256_set1_epi8(1));
shuffle = _mm256_add_epi8(shuffle, m4);
@@ -1173,7 +1173,7 @@ static void mul_mat_iq2_xs_r4_q8_k(int n, const void * vx, size_t bx, const Data
isum[2*iy+1] = _mm256_add_epi32(isum[2*iy+1], _mm256_madd_epi16(scs[1], s34));
}
#else
auto signs = MM256_SET_M128I(signs128, signs128);
auto signs = MM256_SET1_M128I(signs128);
auto shuffle = sign_shuffle;
auto s1 = _mm256_or_si256(_mm256_cmpeq_epi8(_mm256_and_si256(_mm256_shuffle_epi8(signs, shuffle), smask), smask), _mm256_set1_epi8(1));
shuffle = _mm256_add_epi8(shuffle, m4);
@@ -1335,7 +1335,7 @@ static void mul_mat_iq2_xs_r4_q8_k_16(int n, const void * vx, size_t bx, const D
isum[2*iy+1] = _mm256_add_epi32(isum[2*iy+1], _mm256_madd_epi16(scs[1], s34));
}
#else
auto signs = MM256_SET_M128I(signs128, signs128);
auto signs = MM256_SET1_M128I(signs128);
auto shuffle = sign_shuffle;
auto s = _mm256_or_si256(_mm256_cmpeq_epi8(_mm256_and_si256(_mm256_shuffle_epi8(signs, shuffle), smask), smask), _mm256_set1_epi8(1));
shuffle = _mm256_add_epi8(shuffle, m4);
@@ -1454,7 +1454,7 @@ static void mul_mat_iq2_s_r4_q8_k(int n, const void * vx, size_t bx, const DataI
isum[2*iy+1] = _mm256_add_epi32(isum[2*iy+1], _mm256_madd_epi16(scs[1], s34));
}
#else
auto signs = MM256_SET_M128I(signs128, signs128);
auto signs = MM256_SET1_M128I(signs128);
auto shuffle = sign_shuffle;
auto s1 = _mm256_or_si256(_mm256_cmpeq_epi8(_mm256_and_si256(_mm256_shuffle_epi8(signs, shuffle), smask), smask), _mm256_set1_epi8(1));
shuffle = _mm256_add_epi8(shuffle, m4);
@@ -1613,7 +1613,7 @@ static void mul_mat_iq2_s_r4_q8_k_16(int n, const void * vx, size_t bx, const Da
isum[2*iy+1] = _mm256_add_epi32(isum[2*iy+1], _mm256_madd_epi16(scs[1], s34));
}
#else
auto signs = MM256_SET_M128I(signs128, signs128);
auto signs = MM256_SET1_M128I(signs128);
auto shuffle = sign_shuffle;
auto s = _mm256_or_si256(_mm256_cmpeq_epi8(_mm256_and_si256(_mm256_shuffle_epi8(signs, shuffle), smask), smask), _mm256_set1_epi8(1));
shuffle = _mm256_add_epi8(shuffle, m4);
@@ -1703,7 +1703,7 @@ static void mul_mat_iq3_xxs_r4_q8_k(int n, const void * vx, size_t bx, const Dat
//auto t2 = _mm_or_si128(_mm_srli_epi32(_mm_and_si128(scales, _mm_set1_epi32(0x00010000)), 14), _mm_srli_epi32(_mm_and_si128(scales, _mm_set1_epi32(0x01000000)), 21));
//scales = _mm_or_si128(_mm_slli_epi32(_mm_or_si128(t1, t2), 1), _mm_set1_epi32(1));
#endif
auto scales32 = MM256_SET_M128I(scales, scales);
auto scales32 = MM256_SET1_M128I(scales);
auto signs128 = _mm_and_si128(sas, _mm_set1_epi8(-2)); // 0xfe = -2 as signed. Needed to shutup compiler warning.
signs128 = _mm_xor_si128(signs128, _mm_srli_epi16(signs128, 1));
#ifdef HAVE_FANCY_SIMD
@@ -1720,7 +1720,7 @@ static void mul_mat_iq3_xxs_r4_q8_k(int n, const void * vx, size_t bx, const Dat
isum[iy] = _mm256_add_epi32(isum[iy], _mm256_mullo_epi32(scales32, sumi));
}
#else
auto signs = MM256_SET_M128I(signs128, signs128);
auto signs = MM256_SET1_M128I(signs128);
auto shuffle = sign_shuffle;
auto s1 = _mm256_or_si256(_mm256_cmpeq_epi8(_mm256_and_si256(_mm256_shuffle_epi8(signs, shuffle), smask), smask), _mm256_set1_epi8(1));
shuffle = _mm256_add_epi8(shuffle, m4);
ggml/src/iqk/iqk_gemm_kquants.cpp
+16 -16
View File
@@ -22,7 +22,7 @@ struct Scales8K {
const __m128i mins128 = _mm256_extracti128_si256(mins_and_scales, 1);
accum_mins(mins128, q8, i, c, accd);
const __m128i sc128 = _mm256_extracti128_si256(mins_and_scales, 0);
return MM256_SET_M128I(sc128, sc128);
return MM256_SET1_M128I(sc128);
}
#ifdef HAVE_FANCY_SIMD
template <typename Q8>
@@ -58,8 +58,8 @@ inline void process_mins_16(const __m256i& all_scales, const Q8& q8, int i, floa
inline void prepare_scales_16(const __m256i& all_scales, __m256i * scales) {
const __m128i l_scales = _mm256_extracti128_si256(all_scales, 0);
const __m128i h_scales = _mm256_extracti128_si256(all_scales, 1);
scales[0] = MM256_SET_M128I(l_scales, l_scales);
scales[1] = MM256_SET_M128I(h_scales, h_scales);
scales[0] = MM256_SET1_M128I(l_scales);
scales[1] = MM256_SET1_M128I(h_scales);
}
// Handles q3_K scales
@@ -81,7 +81,7 @@ struct ScaleQ3 {
struct Scale16 {
inline void make_scales(const __m128i& scales8, __m512i * scales) const {
auto all_scales8 = MM256_SET_M128I(scales8, scales8);
auto all_scales8 = MM256_SET1_M128I(scales8);
auto scales1 = _mm256_shuffle_epi8(all_scales8, shuffle1);
auto scales2 = _mm256_shuffle_epi8(all_scales8, shuffle2);
scales[0] = _mm512_cvtepi8_epi16(scales1);
@@ -297,7 +297,7 @@ struct DequantizerIQ4XS final : public BaseDequantizer<block_iq4_xs> {
prepare(x[i].qs);
auto scales128 = siq4.make_scales(*(const uint32_t *)x[i].scales_l, x[i].scales_h);
s8k.accum_mins(scales128, q8, i, -128.f*d, accd);
auto scales256 = MM256_SET_M128I(scales128, scales128);
auto scales256 = MM256_SET1_M128I(scales128);
auto all_scales = _mm512_inserti32x8(_mm512_castsi256_si512(scales256), scales256, 1);
scales[0] = _mm512_shuffle_epi8(all_scales, shuffles[0]);
scales[1] = _mm512_shuffle_epi8(all_scales, shuffles[1]);
@@ -610,7 +610,7 @@ struct DequantizerIQ4XS final : public BaseDequantizer<block_iq4_xs> {
d = GGML_FP16_TO_FP32(x[i].d);
auto scales128 = siq4.make_scales(*(const uint32_t *)x[i].scales_l, x[i].scales_h);
s8k.accum_mins(scales128, q8, i, -128.f*d, accd);
return MM256_SET_M128I(scales128, scales128);
return MM256_SET1_M128I(scales128);
}
inline void prepare(int i, int j) {
bits.prepare16(x[i].qs, j);
@@ -727,7 +727,7 @@ static void mul_mat_qY_K_q8_K_T(int n, const void * vx, size_t bx, const DataInf
// const __m128i mins128 = _mm256_extracti128_si256(mins_and_scales, 1);
// accum_mins(mins128, q8, i, c, accd);
// const __m128i sc128 = _mm256_extracti128_si256(mins_and_scales, 0);
// return MM256_SET_M128I(sc128, sc128);
// return MM256_SET1_M128I(sc128);
// }
//
// inline void new_block(int i, const Q8& q8, __m256 * accd, __m512i * scales) {
@@ -1042,7 +1042,7 @@ static void mul_mat_iq4_xs_r8_q8_k_avx2(int n, const void * vx, size_t bx, const
#ifndef HAVE_VNNI256
auto s_shuffle = _mm256_set_epi64x(0x0f0e0f0e0d0c0d0c, 0x0b0a0b0a09080908, 0x0706070605040504, 0x0302030201000100);
auto values128 = _mm_loadu_si128((const __m128i *)iq4k_values);
auto values = MM256_SET_M128I(values128, values128);
auto values = MM256_SET1_M128I(values128);
#else
auto values = load_iq4nl_values_256();
#endif
@@ -1089,7 +1089,7 @@ static void mul_mat_iq4_xs_r8_q8_k_avx2(int n, const void * vx, size_t bx, const
#endif
for (int iy = 0; iy < nrc_y; ++iy) {
auto y128 = _mm_loadu_si128((const __m128i*)q8.y[iy][ibl].qs+2*ib+0);
auto y = MM256_SET_M128I(y128, y128);
auto y = MM256_SET1_M128I(y128);
#ifdef HAVE_VNNI256
auto sumi = _mm256_setzero_si256();
sumi = ggml_mm256_dpbusd_epi32(sumi, qx[0], _mm256_shuffle_epi32(y, 0x00));
@@ -1121,7 +1121,7 @@ static void mul_mat_iq4_xs_r8_q8_k_avx2(int n, const void * vx, size_t bx, const
#endif
for (int iy = 0; iy < nrc_y; ++iy) {
auto y128 = _mm_loadu_si128((const __m128i*)q8.y[iy][ibl].qs+2*ib+1);
auto y = MM256_SET_M128I(y128, y128);
auto y = MM256_SET1_M128I(y128);
#ifdef HAVE_VNNI256
auto sumi = _mm256_setzero_si256();
sumi = ggml_mm256_dpbusd_epi32(sumi, qx[0], _mm256_shuffle_epi32(y, 0x00));
@@ -1562,7 +1562,7 @@ static void mul_mat_q4_k_r4_q8_k(int n, const void * vx, size_t bx, const DataIn
#else
auto aux = _mm_set1_epi32(hd.val[ib]);
aux = _mm_cvtepu8_epi16(_mm_unpacklo_epi8(aux, aux));
auto scales_d = MM256_SET_M128I(aux, aux);
auto scales_d = MM256_SET1_M128I(aux);
#endif
auto bits1 = _mm256_loadu_si256((const __m256i *)iq4[ibl].qs+2*ib+0);
auto bits2 = _mm256_loadu_si256((const __m256i *)iq4[ibl].qs+2*ib+1);
@@ -1631,7 +1631,7 @@ static void mul_mat_q5_k_r4_q8_k(int n, const void * vx, size_t bx, const DataIn
#else
auto aux = _mm_set1_epi32(hd.val[ib]);
aux = _mm_cvtepu8_epi16(_mm_unpacklo_epi8(aux, aux));
auto scales_d = MM256_SET_M128I(aux, aux);
auto scales_d = MM256_SET1_M128I(aux);
#endif
auto lbits1 = _mm256_loadu_si256((const __m256i *)iq5[ibl].qs+2*ib+0);
auto lbits2 = _mm256_loadu_si256((const __m256i *)iq5[ibl].qs+2*ib+1);
@@ -1859,7 +1859,7 @@ static void mul_mat_q8_k_r8_q8_k(int n, const void * vx, size_t bx, const DataIn
auto s3 = _mm256_sign_epi8(qx[3], qx[3]);
for (int iy = 0; iy < nrc_y; ++iy) {
auto y128 = _mm_loadu_si128((const __m128i*)q8.y[iy][ibl].qs+ib);
auto y = MM256_SET_M128I(y128, y128);
auto y = MM256_SET1_M128I(y128);
#ifdef HAVE_VNNI256
isum[iy] = ggml_mm256_dpbusd_epi32(isum[iy], s0, _mm256_sign_epi8(_mm256_shuffle_epi32(y, 0x00), qx[0]));
isum[iy] = ggml_mm256_dpbusd_epi32(isum[iy], s1, _mm256_sign_epi8(_mm256_shuffle_epi32(y, 0x55), qx[1]));
@@ -1908,7 +1908,7 @@ static void mul_mat_q8_k_r16_q8_k(int n, const void * vx, size_t bx, const DataI
qx[3] = _mm512_loadu_si512((const __m512i *)iq16[ibl].qs+4*ib+3);
for (int iy = 0; iy < nrc_y; ++iy) {
auto y128 = _mm_loadu_si128((const __m128i*)q8.y[iy][ibl].qs+ib);
auto y256 = MM256_SET_M128I(y128, y128);
auto y256 = MM256_SET1_M128I(y128);
auto y = _mm512_inserti32x8(_mm512_castsi256_si512(y256), y256, 1);
isum[iy] = _mm512_dpbusd_epi32(isum[iy], qx[0], _mm512_shuffle_epi32(y, _MM_PERM_ENUM(0x00)));
isum[iy] = _mm512_dpbusd_epi32(isum[iy], qx[1], _mm512_shuffle_epi32(y, _MM_PERM_ENUM(0x55)));
@@ -2059,7 +2059,7 @@ static void mul_mat_q8_KV_r8_q8_KV(int n, const void * vx, size_t bx, const Data
#endif
for (int iy = 0; iy < nrc_y; ++iy) {
auto y128 = _mm_loadu_si128((const __m128i*)q8y[iy]+ib);
auto y = MM256_SET_M128I(y128, y128);
auto y = MM256_SET1_M128I(y128);
#ifdef HAVE_FANCY_SIMD
acc[iy] = _mm256_dpbusd_epi32(acc[iy], qx[0], _mm256_shuffle_epi32(y, 0x00));
acc[iy] = _mm256_dpbusd_epi32(acc[iy], qx[1], _mm256_shuffle_epi32(y, 0x55));
@@ -2634,7 +2634,7 @@ void iqk_convert_iq4_xs_q8_k_r8(int n, const void * vx, size_t bx, void * vy, in
block_q8_k_r * y = (block_q8_k_r *)vy;
auto values128 = _mm_loadu_si128((const __m128i *)iq4k_values);
auto values = MM256_SET_M128I(values128, values128);
auto values = MM256_SET1_M128I(values128);
int16_t ls[16];
float dnew[k_nr];
ggml/src/iqk/iqk_gemm_ktquants.cpp
+3 -3
View File
@@ -220,13 +220,13 @@ struct Trellis3 {
tmp[1] = _mm256_cvtepu16_epi32(_mm256_extracti128_si256(val, 1));
for (int k = 0; k < 2; ++k) {
auto vl = _mm256_castsi256_si128(tmp[k]);
auto v = MM256_SET_M128I(vl, vl);
auto v = MM256_SET1_M128I(vl);
aux[8*k+0] = _mm256_shuffle_epi32(v, 0x00);
aux[8*k+1] = _mm256_shuffle_epi32(v, 0x55);
aux[8*k+2] = _mm256_shuffle_epi32(v, 0xaa);
aux[8*k+3] = _mm256_shuffle_epi32(v, 0xff);
auto vh = _mm256_extracti128_si256(tmp[k], 1);
v = MM256_SET_M128I(vh, vh);
v = MM256_SET1_M128I(vh);
aux[8*k+4] = _mm256_shuffle_epi32(v, 0x00);
aux[8*k+5] = _mm256_shuffle_epi32(v, 0x55);
aux[8*k+6] = _mm256_shuffle_epi32(v, 0xaa);
@@ -267,7 +267,7 @@ struct Trellis3 {
__m256i aux[16];
for (int k = 0; k < 4; ++k) {
auto v128 = _mm_add_epi32(_mm_cvtepu16_epi32(_mm_loadl_epi64((const __m128i *)(val + 4*k))), _mm_set1_epi32(v0));
auto v = MM256_SET_M128I(v128, v128);
auto v = MM256_SET1_M128I(v128);
aux[4*k+0] = _mm256_shuffle_epi32(v, 0x00);
aux[4*k+1] = _mm256_shuffle_epi32(v, 0x55);
aux[4*k+2] = _mm256_shuffle_epi32(v, 0xaa);
ggml/src/iqk/iqk_gemm_legacy_quants.cpp
+14 -14
View File
@@ -619,7 +619,7 @@ static inline __m128i load_unsigned_mxfp4_values_128() {
static inline __m256i load_unsigned_mxfp4_values_256() {
auto val128 = load_unsigned_mxfp4_values_128();
return MM256_SET_M128I(val128, val128);
return MM256_SET1_M128I(val128);
}
#ifdef HAVE_FANCY_SIMD
@@ -635,7 +635,7 @@ static inline __m128i load_mxfp4_values_128() {
static inline __m256i load_mxfp4_values_256() {
auto val128 = load_mxfp4_values_128();
return MM256_SET_M128I(val128, val128);
return MM256_SET1_M128I(val128);
}
struct MXFP4_Dequantizer {
@@ -896,7 +896,7 @@ static void mul_mat_iq4_nl_r4_q8_2(int n, const void * vx, size_t bx, const Data
auto m1 = _mm256_set1_epi16(1);
#endif
auto values128 = _mm_loadu_si128((const __m128i *)iq4k_values);
auto values = MM256_SET_M128I(values128, values128);
auto values = MM256_SET1_M128I(values128);
int nb = n / QK4_NL;
__m256 acc[nrc_y] = {};
__m256i qs[4];
@@ -989,8 +989,8 @@ inline void prepare_q4_0_quants_avx2(const uint8_t * qs, __m256i * v, const __m2
inline __m256i accum_q4_0_quants(const __m256i * v, const int8_t * qs) {
auto y4l = _mm_loadu_si128((const __m128i*)qs+0);
auto y4h = _mm_loadu_si128((const __m128i*)qs+1);
auto yl = MM256_SET_M128I(y4l, y4l);
auto yh = MM256_SET_M128I(y4h, y4h);
auto yl = MM256_SET1_M128I(y4l);
auto yh = MM256_SET1_M128I(y4h);
#ifdef HAVE_VNNI256
auto sumi = _mm256_setzero_si256();
sumi = ggml_mm256_dpbusd_epi32(sumi, v[0], _mm256_shuffle_epi32(yl, 0x00));
@@ -1135,8 +1135,8 @@ static void mul_mat_q4_0_r8_q8_2(int n, const void * vx, size_t bx, const DataIn
auto dot = [&qx] (const int8_t * qy) {
auto y4l = _mm_loadu_si128((const __m128i*)qy+0);
auto y4h = _mm_loadu_si128((const __m128i*)qy+1);
auto y8l = MM256_SET_M128I(y4l, y4l);
auto y8h = MM256_SET_M128I(y4h, y4h);
auto y8l = MM256_SET1_M128I(y4l);
auto y8h = MM256_SET1_M128I(y4h);
auto yl = _mm512_inserti32x8(_mm512_castsi256_si512(y8l), y8l, 1);
auto yh = _mm512_inserti32x8(_mm512_castsi256_si512(y8h), y8h, 1);
auto sumi = _mm512_setzero_si512();
@@ -1539,8 +1539,8 @@ static void mul_mat_q6_0_r4_q8_2(int n, const void * vx, size_t bx, const DataIn
inline __m512i qx_r8_q8_dot_product(const __m512i * qx, const int8_t * y) {
auto y4l = _mm_loadu_si128((const __m128i*)y+0);
auto y4h = _mm_loadu_si128((const __m128i*)y+1);
auto y8l = MM256_SET_M128I(y4l, y4l);
auto y8h = MM256_SET_M128I(y4h, y4h);
auto y8l = MM256_SET1_M128I(y4l);
auto y8h = MM256_SET1_M128I(y4h);
auto yl = _mm512_inserti32x8(_mm512_castsi256_si512(y8l), y8l, 1);
auto yh = _mm512_inserti32x8(_mm512_castsi256_si512(y8h), y8h, 1);
auto sumi = _mm512_setzero_si512();
@@ -1557,8 +1557,8 @@ inline __m512i qx_r8_q8_dot_product(const __m512i * qx, const int8_t * y) {
inline __m256i qx_r8_q8_dot_product(const __m256i * qx, const int8_t * y) {
auto y4l = _mm_loadu_si128((const __m128i*)y+0);
auto y4h = _mm_loadu_si128((const __m128i*)y+1);
auto yl = MM256_SET_M128I(y4l, y4l);
auto yh = MM256_SET_M128I(y4h, y4h);
auto yl = MM256_SET1_M128I(y4l);
auto yh = MM256_SET1_M128I(y4h);
auto sumi = _mm256_setzero_si256();
sumi = _mm256_dpbusd_epi32(sumi, qx[0], _mm256_shuffle_epi32(yl, 0x00));
sumi = _mm256_dpbusd_epi32(sumi, qx[1], _mm256_shuffle_epi32(yl, 0x55));
@@ -1677,7 +1677,7 @@ static void mul_mat_q8_0_r8_q8_2(int n, const void * vx, size_t bx, const DataIn
__m256i qx[4], sx[4];
auto dot = [&qx, &sx, &m1] (const int8_t * qy) {
auto y128 = _mm_loadu_si128((const __m128i*)qy);
auto y = MM256_SET_M128I(y128, y128);
auto y = MM256_SET1_M128I(y128);
#ifdef HAVE_VNNI256
auto sumi = _mm256_setzero_si256();
sumi = ggml_mm256_dpbusd_epi32(sumi, sx[0], _mm256_sign_epi8(_mm256_shuffle_epi32(y, 0x00), qx[0]));
@@ -1772,7 +1772,7 @@ static void mul_mat_q8_1_r8_q8_2(int n, const void * vx, size_t bx, const DataIn
__m256i qx[4];
auto dot = [&qx] (const int8_t * qy) {
auto y128 = _mm_loadu_si128((const __m128i*)qy);
auto y = MM256_SET_M128I(y128, y128);
auto y = MM256_SET1_M128I(y128);
auto sumi = _mm256_setzero_si256();
sumi = _mm256_dpbusd_epi32(sumi, qx[0], _mm256_shuffle_epi32(y, 0x00));
sumi = _mm256_dpbusd_epi32(sumi, qx[1], _mm256_shuffle_epi32(y, 0x55));
@@ -1876,7 +1876,7 @@ static void mul_mat_q8_1_r8_q8_2(int n, const void * vx, size_t bx, const DataIn
__m256i qx[4];
auto dot = [&qx] (const int8_t * qy) {
auto y128 = _mm_loadu_si128((const __m128i*)qy);
auto y = MM256_SET_M128I(y128, y128);
auto y = MM256_SET1_M128I(y128);
#ifdef HAVE_VNNI256
auto sumi = _mm256_setzero_si256();
sumi = ggml_mm256_dpbusd_epi32(sumi, qx[0], _mm256_shuffle_epi32(y, 0x00));
gguf-py/gguf/constants.py
+119
View File
@@ -113,6 +113,9 @@ class Keys:
CAUSAL = "{arch}.attention.causal"
Q_LORA_RANK = "{arch}.attention.q_lora_rank"
KV_LORA_RANK = "{arch}.attention.kv_lora_rank"
INDEXER_HEAD_COUNT = "{arch}.attention.indexer.head_count"
INDEXER_KEY_LENGTH = "{arch}.attention.indexer.key_length"
INDEXER_TOP_K = "{arch}.attention.indexer.top_k"
REL_BUCKETS_COUNT = "{arch}.attention.relative_buckets_count"
SLIDING_WINDOW = "{arch}.attention.sliding_window"
SLIDING_WINDOW_PATTERN = "{arch}.attention.sliding_window_pattern"
@@ -260,6 +263,7 @@ class MODEL_ARCH(IntEnum):
ARCTIC = auto()
DEEPSEEK2 = auto()
GLM4_MOE = auto()
OPENPANGU = auto()
CHATGLM = auto()
BITNET = auto()
BITNET_25 = auto()
@@ -386,6 +390,33 @@ class MODEL_TENSOR(IntEnum):
MTP_CENTROIDS = auto()
DFLASH_FC = auto()
DFLASH_HIDDEN_NORM = auto()
# openPangu-2.0 (DSA lightning indexer)
INDEXER_K_NORM = auto()
INDEXER_PROJ = auto() # weights_proj
INDEXER_ATTN_K = auto() # wk
INDEXER_ATTN_Q_B = auto() # wq_b
# openPangu-2.0 (MoME causal-conv on MLA latents)
ATTN_QA_CONV = auto()
ATTN_KV_CONV = auto() # compresskv_conv
ATTN_O_CONV = auto()
# openPangu-2.0 (learned static param sink)
ATTN_PARAM_SINK_KV = auto() # param_sink_compressed_kv
ATTN_PARAM_SINK_K_PE = auto() # param_sink_k_pe
# openPangu-2.0 (mHC / Hyper-Connections: per-attn, per-mlp, global merge)
MHC_ATTN_PHI = auto()
MHC_ATTN_ALPHA = auto()
MHC_ATTN_BETA = auto()
MHC_ATTN_GAMMA = auto()
MHC_MLP_PHI = auto()
MHC_MLP_ALPHA = auto()
MHC_MLP_BETA = auto()
MHC_MLP_GAMMA = auto()
MHC_MERGE_PHI = auto()
MHC_MERGE_ALPHA = auto() # branch_alpha_pre
MHC_MERGE_BETA = auto() # branch_beta_pre
MHC_MERGE_GAMMA = auto()
# openPangu-2.0 (sandwich norm: extra whole-block post-norm on a layer subset)
BLOCK_POST_NORM = auto()
MODEL_ARCH_NAMES: dict[MODEL_ARCH, str] = {
@@ -437,6 +468,7 @@ MODEL_ARCH_NAMES: dict[MODEL_ARCH, str] = {
MODEL_ARCH.DEEPSEEK2: "deepseek2",
MODEL_ARCH.CHATGLM: "chatglm",
MODEL_ARCH.GLM4_MOE: "glm4moe",
MODEL_ARCH.OPENPANGU: "openpangu",
MODEL_ARCH.BITNET: "bitnet",
MODEL_ARCH.BITNET_25: "bitnet-25",
MODEL_ARCH.T5: "t5",
@@ -563,6 +595,29 @@ TENSOR_NAMES: dict[MODEL_TENSOR, str] = {
MODEL_TENSOR.MTP_CENTROIDS: "mtp_centroids",
MODEL_TENSOR.DFLASH_FC: "dflash_fc",
MODEL_TENSOR.DFLASH_HIDDEN_NORM: "dflash_hidden_norm",
# openPangu-2.0
MODEL_TENSOR.INDEXER_K_NORM: "blk.{bid}.attn_indexer_k_norm",
MODEL_TENSOR.INDEXER_PROJ: "blk.{bid}.attn_indexer_weights_proj",
MODEL_TENSOR.INDEXER_ATTN_K: "blk.{bid}.attn_indexer_k",
MODEL_TENSOR.INDEXER_ATTN_Q_B: "blk.{bid}.attn_indexer_q_b",
MODEL_TENSOR.ATTN_QA_CONV: "blk.{bid}.attn_qa_conv",
MODEL_TENSOR.ATTN_KV_CONV: "blk.{bid}.attn_compresskv_conv",
MODEL_TENSOR.ATTN_O_CONV: "blk.{bid}.attn_o_conv",
MODEL_TENSOR.ATTN_PARAM_SINK_KV: "blk.{bid}.attn_param_sink_kv",
MODEL_TENSOR.ATTN_PARAM_SINK_K_PE: "blk.{bid}.attn_param_sink_k_pe",
MODEL_TENSOR.MHC_ATTN_PHI: "blk.{bid}.attn_mhc_phi",
MODEL_TENSOR.MHC_ATTN_ALPHA: "blk.{bid}.attn_mhc_alpha",
MODEL_TENSOR.MHC_ATTN_BETA: "blk.{bid}.attn_mhc_beta",
MODEL_TENSOR.MHC_ATTN_GAMMA: "blk.{bid}.attn_mhc_gamma",
MODEL_TENSOR.MHC_MLP_PHI: "blk.{bid}.mlp_mhc_phi",
MODEL_TENSOR.MHC_MLP_ALPHA: "blk.{bid}.mlp_mhc_alpha",
MODEL_TENSOR.MHC_MLP_BETA: "blk.{bid}.mlp_mhc_beta",
MODEL_TENSOR.MHC_MLP_GAMMA: "blk.{bid}.mlp_mhc_gamma",
MODEL_TENSOR.MHC_MERGE_PHI: "merge_mhc_phi",
MODEL_TENSOR.MHC_MERGE_ALPHA: "merge_mhc_alpha",
MODEL_TENSOR.MHC_MERGE_BETA: "merge_mhc_beta",
MODEL_TENSOR.MHC_MERGE_GAMMA: "merge_mhc_gamma",
MODEL_TENSOR.BLOCK_POST_NORM: "blk.{bid}.block_post_norm",
}
MODEL_TENSORS: dict[MODEL_ARCH, list[MODEL_TENSOR]] = {
@@ -1298,6 +1353,70 @@ MODEL_TENSORS: dict[MODEL_ARCH, list[MODEL_TENSOR]] = {
MODEL_TENSOR.NEXTN_SHARED_HEAD_HEAD,
MODEL_TENSOR.NEXTN_SHARED_HEAD_NORM,
],
MODEL_ARCH.OPENPANGU: [
MODEL_TENSOR.TOKEN_EMBD,
MODEL_TENSOR.OUTPUT_NORM,
MODEL_TENSOR.OUTPUT,
# MLA attention (deepseek-style)
MODEL_TENSOR.ATTN_NORM,
MODEL_TENSOR.ATTN_Q_A,
MODEL_TENSOR.ATTN_Q_B,
MODEL_TENSOR.ATTN_KV_A_MQA,
MODEL_TENSOR.ATTN_KV_B,
MODEL_TENSOR.ATTN_K_B,
MODEL_TENSOR.ATTN_V_B,
MODEL_TENSOR.ATTN_Q_A_NORM,
MODEL_TENSOR.ATTN_KV_A_NORM,
MODEL_TENSOR.ATTN_OUT,
MODEL_TENSOR.ATTN_POST_NORM, # post_attention_layernorm (sandwich)
# DSA lightning indexer
MODEL_TENSOR.INDEXER_K_NORM,
MODEL_TENSOR.INDEXER_PROJ,
MODEL_TENSOR.INDEXER_ATTN_K,
MODEL_TENSOR.INDEXER_ATTN_Q_B,
# MoME causal convs + param sink
MODEL_TENSOR.ATTN_QA_CONV,
MODEL_TENSOR.ATTN_KV_CONV,
MODEL_TENSOR.ATTN_O_CONV,
MODEL_TENSOR.ATTN_PARAM_SINK_KV,
MODEL_TENSOR.ATTN_PARAM_SINK_K_PE,
# MoE (routed + shared + sigmoid bias) and dense-lead FFN
MODEL_TENSOR.FFN_NORM, # pre_mlp_layernorm
MODEL_TENSOR.FFN_POST_NORM, # post_mlp_layernorm (sandwich)
MODEL_TENSOR.FFN_GATE,
MODEL_TENSOR.FFN_DOWN,
MODEL_TENSOR.FFN_UP,
MODEL_TENSOR.FFN_GATE_INP,
MODEL_TENSOR.FFN_GATE_EXP,
MODEL_TENSOR.FFN_DOWN_EXP,
MODEL_TENSOR.FFN_UP_EXP,
MODEL_TENSOR.FFN_GATE_SHEXP,
MODEL_TENSOR.FFN_DOWN_SHEXP,
MODEL_TENSOR.FFN_UP_SHEXP,
MODEL_TENSOR.FFN_EXP_PROBS_B,
# mHC / Hyper-Connections
MODEL_TENSOR.MHC_ATTN_PHI,
MODEL_TENSOR.MHC_ATTN_ALPHA,
MODEL_TENSOR.MHC_ATTN_BETA,
MODEL_TENSOR.MHC_ATTN_GAMMA,
MODEL_TENSOR.MHC_MLP_PHI,
MODEL_TENSOR.MHC_MLP_ALPHA,
MODEL_TENSOR.MHC_MLP_BETA,
MODEL_TENSOR.MHC_MLP_GAMMA,
MODEL_TENSOR.MHC_MERGE_PHI,
MODEL_TENSOR.MHC_MERGE_ALPHA,
MODEL_TENSOR.MHC_MERGE_BETA,
MODEL_TENSOR.MHC_MERGE_GAMMA,
# sandwich extra block post-norm (layer subset)
MODEL_TENSOR.BLOCK_POST_NORM,
# NextN / MTP tail (3 layers)
MODEL_TENSOR.NEXTN_EH_PROJ,
MODEL_TENSOR.NEXTN_EMBED_TOKENS,
MODEL_TENSOR.NEXTN_ENORM,
MODEL_TENSOR.NEXTN_HNORM,
MODEL_TENSOR.NEXTN_SHARED_HEAD_HEAD,
MODEL_TENSOR.NEXTN_SHARED_HEAD_NORM,
],
MODEL_ARCH.DFLASH: [
MODEL_TENSOR.OUTPUT_NORM,
MODEL_TENSOR.ATTN_NORM,
gguf-py/gguf/tensor_mapping.py
+88
View File
@@ -87,6 +87,20 @@ class TensorNameMap:
MODEL_TENSOR.ROPE_FACTORS_LONG: (),
MODEL_TENSOR.ROPE_FACTORS_SHORT: (),
# openPangu-2.0 global mHC stream-merge module (non-block)
MODEL_TENSOR.MHC_MERGE_PHI: (
"model.merge_mhc_module.phi",
),
MODEL_TENSOR.MHC_MERGE_ALPHA: (
"model.merge_mhc_module.branch_alpha_pre",
),
MODEL_TENSOR.MHC_MERGE_BETA: (
"model.merge_mhc_module.branch_beta_pre",
),
MODEL_TENSOR.MHC_MERGE_GAMMA: (
"model.merge_mhc_module.norm_gamma",
),
}
block_mappings_cfg: dict[MODEL_TENSOR, tuple[str, ...]] = {
@@ -697,6 +711,80 @@ class TensorNameMap:
"model.layers.{bid}.post_attention_layernorm",
),
},
# openPangu-2.0: disambiguate the sandwich norms (post_attention/pre_mlp/post_mlp)
# from the generic collisions, and pin all Pangu-only block tensors.
MODEL_ARCH.OPENPANGU: {
MODEL_TENSOR.ATTN_POST_NORM: (
"model.layers.{bid}.post_attention_layernorm",
),
MODEL_TENSOR.FFN_NORM: (
"model.layers.{bid}.pre_mlp_layernorm",
),
MODEL_TENSOR.FFN_POST_NORM: (
"model.layers.{bid}.post_mlp_layernorm",
),
MODEL_TENSOR.BLOCK_POST_NORM: (
"model.layers.{bid}.block_post_layernorm",
),
MODEL_TENSOR.FFN_EXP_PROBS_B: (
"model.layers.{bid}.mlp.e_score_correction",
),
# DSA lightning indexer
MODEL_TENSOR.INDEXER_K_NORM: (
"model.layers.{bid}.self_attn.indexer.k_norm",
),
MODEL_TENSOR.INDEXER_PROJ: (
"model.layers.{bid}.self_attn.indexer.weights_proj",
),
MODEL_TENSOR.INDEXER_ATTN_K: (
"model.layers.{bid}.self_attn.indexer.wk",
),
MODEL_TENSOR.INDEXER_ATTN_Q_B: (
"model.layers.{bid}.self_attn.indexer.wq_b",
),
# MoME causal convs
MODEL_TENSOR.ATTN_QA_CONV: (
"model.layers.{bid}.self_attn.qa_conv",
),
MODEL_TENSOR.ATTN_KV_CONV: (
"model.layers.{bid}.self_attn.compresskv_conv",
),
MODEL_TENSOR.ATTN_O_CONV: (
"model.layers.{bid}.self_attn.o_conv",
),
# learned static param sink
MODEL_TENSOR.ATTN_PARAM_SINK_KV: (
"model.layers.{bid}.self_attn.param_sink_compressed_kv",
),
MODEL_TENSOR.ATTN_PARAM_SINK_K_PE: (
"model.layers.{bid}.self_attn.param_sink_k_pe",
),
# mHC / Hyper-Connections (per attn + per mlp)
MODEL_TENSOR.MHC_ATTN_PHI: (
"model.layers.{bid}.attn_mhc_module.phi",
),
MODEL_TENSOR.MHC_ATTN_ALPHA: (
"model.layers.{bid}.attn_mhc_module.branch_alpha",
),
MODEL_TENSOR.MHC_ATTN_BETA: (
"model.layers.{bid}.attn_mhc_module.branch_beta",
),
MODEL_TENSOR.MHC_ATTN_GAMMA: (
"model.layers.{bid}.attn_mhc_module.norm_gamma",
),
MODEL_TENSOR.MHC_MLP_PHI: (
"model.layers.{bid}.mlp_mhc_module.phi",
),
MODEL_TENSOR.MHC_MLP_ALPHA: (
"model.layers.{bid}.mlp_mhc_module.branch_alpha",
),
MODEL_TENSOR.MHC_MLP_BETA: (
"model.layers.{bid}.mlp_mhc_module.branch_beta",
),
MODEL_TENSOR.MHC_MLP_GAMMA: (
"model.layers.{bid}.mlp_mhc_module.norm_gamma",
),
},
}
mapping: dict[str, tuple[MODEL_TENSOR, str]]
include/llama.h
+14
View File
@@ -497,6 +497,8 @@ extern "C" {
int min_experts;
float thresh_experts;
bool only_active_experts;
bool prefetch_experts; // if true, stream mmap'd MoE expert weights into the page cache (Linux only)
int prefetch_experts_threads; // number of expert prefetch workers (<=0 = auto)
bool k_cache_hadamard; // if true, apply Hadamard transfrom to K-cache
bool v_cache_hadamard; // if true, apply Hadamard transfrom to V-cache (needs FA)
bool split_mode_graph_scheduling; // if true, force split mode graph scheduling
@@ -696,6 +698,9 @@ extern "C" {
LLAMA_API bool llama_model_has_recurrent(const struct llama_model * model);
// Returns true if the model is openPangu (conv-only recurrent state that rides the spec-rollback checkpoint)
LLAMA_API bool llama_model_is_openpangu(const struct llama_model * model);
// Returns true if the model is a Gemma 4 MTP assistant (external frozen-KV speculative drafter)
LLAMA_API bool llama_model_is_gemma4_mtp_assistant(const struct llama_model * model);
@@ -703,6 +708,15 @@ extern "C" {
LLAMA_API bool llama_model_is_split_mode_graph(const struct llama_model * model);
// Returns false for models whose KV cache cannot be re-positioned after the fact
// (K-shift / context shift / self-extend), e.g. openPangu's latent cache.
LLAMA_API bool llama_model_supports_ctx_shift(const struct llama_model * model);
// Returns false for models that can only reuse a cached sequence as a pure extension:
// rewinding into the middle of a decoded sequence loses per-position side state
// (e.g. openPangu keeps only the current recurrent conv state).
LLAMA_API bool llama_model_supports_partial_kv_reuse(const struct llama_model * model);
LLAMA_API const char * llama_model_arch_string(const struct llama_model * model);
// Returns 0 on success
src/CMakeLists.txt
+1
View File
@@ -111,6 +111,7 @@ add_library(llama
graphs/build_gptneox.cpp
graphs/build_arctic.cpp
graphs/build_deepseek2.cpp
graphs/build_openpangu.cpp
graphs/build_glm4.cpp
graphs/build_bitnet.cpp
graphs/build_cohere2.cpp
src/graphs/build_openpangu.cpp
+1222
View File
File diff suppressed because it is too large Load Diff
src/llama-arch.cpp
+5
View File
@@ -86,6 +86,7 @@ static const std::map<llm_arch, const char *> LLM_ARCH_NAMES = {
{ LLM_ARCH_GEMMA4_MTP, "gemma4_mtp" },
{ LLM_ARCH_DFLASH_DRAFT, "dflash-draft" },
{ LLM_ARCH_GEMMA4_ASSISTANT,"gemma4-assistant" },
{ LLM_ARCH_OPENPANGU, "openpangu" },
{ LLM_ARCH_UNKNOWN, "(unknown)" },
};
@@ -264,6 +265,10 @@ static const std::map<llm_kv, const char *> LLM_KV_NAMES = {
{ LLM_KV_ADAPTER_TYPE, "adapter.type" },
{ LLM_KV_ADAPTER_LORA_ALPHA, "adapter.lora.alpha" },
{ LLM_KV_OPENPANGU_MHC_NUM_STREAM, "%s.mhc_num_stream" },
{ LLM_KV_OPENPANGU_MHC_RECUR_NORM, "%s.mhc_recur_norm" },
{ LLM_KV_OPENPANGU_PARAM_SINK_NUMBER, "%s.param_sink_number" },
};
LLM_KV::LLM_KV(llm_arch arch, const char* suffix) : arch(arch), suffix(suffix) {}
src/llama-arch.h
+26
View File
@@ -84,6 +84,7 @@ enum llm_arch {
LLM_ARCH_GEMMA4_MTP,
LLM_ARCH_DFLASH_DRAFT,
LLM_ARCH_GEMMA4_ASSISTANT,
LLM_ARCH_OPENPANGU,
LLM_ARCH_UNKNOWN,
};
@@ -247,6 +248,11 @@ enum llm_kv {
LLM_KV_ADAPTER_TYPE,
LLM_KV_ADAPTER_LORA_ALPHA,
// openPangu-2.0 (mHC / Hyper-Connections + param sink)
LLM_KV_OPENPANGU_MHC_NUM_STREAM,
LLM_KV_OPENPANGU_MHC_RECUR_NORM,
LLM_KV_OPENPANGU_PARAM_SINK_NUMBER,
};
struct LLM_KV {
@@ -381,6 +387,26 @@ enum llm_tensor {
LLM_TENSOR_DFLASH_FC,
LLM_TENSOR_DFLASH_HIDDEN_NORM,
// openPangu-2.0
LLM_TENSOR_ATTN_QA_CONV, // MoME causal conv on q-lora latent
LLM_TENSOR_ATTN_KV_CONV, // MoME causal conv on compressed-kv latent
LLM_TENSOR_ATTN_O_CONV, // MoME causal conv on attn output
LLM_TENSOR_ATTN_PARAM_SINK_KV, // learned static latent-kv sink
LLM_TENSOR_ATTN_PARAM_SINK_K_PE,
LLM_TENSOR_BLOCK_POST_NORM, // sandwich whole-block post-norm (layer subset)
LLM_TENSOR_MHC_ATTN_PHI, // mHC / Hyper-Connections (per attn sublayer)
LLM_TENSOR_MHC_ATTN_ALPHA,
LLM_TENSOR_MHC_ATTN_BETA,
LLM_TENSOR_MHC_ATTN_GAMMA,
LLM_TENSOR_MHC_MLP_PHI, // mHC (per mlp sublayer)
LLM_TENSOR_MHC_MLP_ALPHA,
LLM_TENSOR_MHC_MLP_BETA,
LLM_TENSOR_MHC_MLP_GAMMA,
LLM_TENSOR_MHC_MERGE_PHI, // mHC global stream-merge (non-block)
LLM_TENSOR_MHC_MERGE_ALPHA,
LLM_TENSOR_MHC_MERGE_BETA,
LLM_TENSOR_MHC_MERGE_GAMMA,
LLM_TENSOR_UNKNOWN,
};
src/llama-build-context.cpp
+26
View File
@@ -106,6 +106,8 @@ void llm_build_context::init() {
lctx.inp_out_ids = nullptr;
lctx.inp_KQ_mask = nullptr;
lctx.inp_KQ_mask_swa = nullptr;
lctx.inp_KQ_mask_swa_win = nullptr;
lctx.openpangu_swa_window_view = {};
lctx.inp_K_shift = nullptr;
lctx.inp_mean = nullptr;
lctx.inp_cls = nullptr;
@@ -117,6 +119,7 @@ void llm_build_context::init() {
lctx.inp_embd_enc = nullptr;
lctx.inp_KQ_mask_cross = nullptr;
lctx.inp_dsa_sink = nullptr;
lctx.inp_mtp_carry = nullptr;
lctx.dflash.inputs.target_features = nullptr;
lctx.dflash.inputs.pos_ctx = nullptr;
lctx.dflash.inputs.kq_mask = nullptr;
@@ -521,6 +524,25 @@ ggml_tensor * llm_build_context::build_inp_KQ_mask_swa(bool causal) {
return flash_attn ? ggml_cast(ctx0, lctx.inp_KQ_mask_swa, GGML_TYPE_F16) : lctx.inp_KQ_mask_swa;
}
ggml_tensor * llm_build_context::build_inp_KQ_mask_swa_win(int64_t n_kv_win, bool causal) {
GGML_ASSERT(hparams.n_swa > 0);
GGML_ASSERT(n_kv_win > 0);
if (causal && flash_attn) {
lctx.inp_KQ_mask_swa_win = ggml_new_tensor_2d(ctx0, GGML_TYPE_F16, n_kv_win, GGML_PAD(n_tokens, GGML_KQ_MASK_PAD));
cb(lctx.inp_KQ_mask_swa_win, "KQ_mask_swa_win", -1);
ggml_set_input(lctx.inp_KQ_mask_swa_win);
return lctx.inp_KQ_mask_swa_win;
}
lctx.inp_KQ_mask_swa_win = causal
? ggml_new_tensor_2d(ctx0, GGML_TYPE_F32, n_kv_win, GGML_PAD(n_tokens, GGML_KQ_MASK_PAD))
: ggml_new_tensor_2d(ctx0, GGML_TYPE_F32, n_tokens, GGML_PAD(n_tokens, GGML_KQ_MASK_PAD));
cb(lctx.inp_KQ_mask_swa_win, "KQ_mask_swa_win", -1);
ggml_set_input(lctx.inp_KQ_mask_swa_win);
return flash_attn ? ggml_cast(ctx0, lctx.inp_KQ_mask_swa_win, GGML_TYPE_F16) : lctx.inp_KQ_mask_swa_win;
}
ggml_tensor * llm_build_context::build_inp_mean() {
lctx.inp_mean = ggml_new_tensor_2d(ctx0, GGML_TYPE_F32, n_tokens, n_tokens);
cb(lctx.inp_mean, "inp_mean", -1);
@@ -2623,6 +2645,10 @@ ggml_cgraph * llm_build_context::llama_build_graph(
{
result = llm.build_deepseek2();
} break;
case LLM_ARCH_OPENPANGU:
{
result = llm.build_openpangu();
} break;
case LLM_ARCH_CHATGLM:
{
result = llm.build_chatglm();
src/llama-build-context.h
+35
View File
@@ -139,6 +139,8 @@ struct llm_build_context {
ggml_tensor * build_inp_KQ_mask_swa(bool causal = true);
ggml_tensor * build_inp_KQ_mask_swa_win(int64_t n_kv_win, bool causal = true);
ggml_tensor * build_inp_mean();
ggml_tensor * build_inp_cls();
@@ -273,6 +275,39 @@ struct llm_build_context {
ggml_cgraph * build_arctic();
ggml_cgraph * build_deepseek2();
ggml_cgraph * build_openpangu();
// openPangu attention sublayer body (shared by base layers and the NextN/MTP head):
// input is the already-input-normed hidden; returns the post-o_proj attention output.
ggml_tensor * build_openpangu_attention(
ggml_cgraph * gf,
const struct llama_layer & layer,
int il,
ggml_tensor * x_normed,
ggml_tensor * KQ_mask,
ggml_tensor * inp_pos,
ggml_tensor * conv_state,
ggml_tensor * seq_qnext,
float kq_scale,
bool KQ_mask_swa_windowed = false);
// openPangu NextN/MTP head (plain-residual block, no mHC): eh_proj stitching ->
// attention -> MoE -> shared head. Returns the draft logits tensor.
ggml_tensor * build_openpangu_mtp(
const struct llama_layer & mtp_layer,
ggml_tensor * prev_embeddings,
ggml_cgraph * gf,
int il,
ggml_tensor * inp_pos,
ggml_tensor * KQ_mask,
ggml_tensor * inp_out_ids,
ggml_tensor * inp_tokens,
ggml_tensor * seq_qnext,
ggml_tensor ** full_hidden_out = nullptr,
bool select_outputs = true,
bool build_logits = true,
bool cache_writes_only = false,
bool KQ_mask_swa_windowed = false);
ggml_tensor * build_deepseek2_tp_attention(
ggml_cgraph * gf, int il,
src/llama-context.h
+57 -4
View File
@@ -13,6 +13,28 @@ struct llama_model;
#include <set>
#include <memory>
struct llama_openpangu_swa_window_view {
int64_t w_view = 0;
int64_t win_off = 0;
bool engaged = false;
};
static inline llama_openpangu_swa_window_view llama_openpangu_calc_swa_window_view(
int64_t n_kv, int64_t n_tokens, int64_t window, int64_t pad) {
llama_openpangu_swa_window_view result;
if (window <= 0 || n_kv <= 0) {
result.w_view = n_kv;
return result;
}
const int64_t unpadded = window + pad + n_tokens;
const int64_t overcovered = pad > 1 ? ((unpadded + pad - 1) / pad) * pad : unpadded;
result.w_view = overcovered < n_kv ? overcovered : n_kv;
result.win_off = n_kv - result.w_view;
result.engaged = result.w_view < n_kv;
return result;
}
struct llama_kv_cell {
llama_pos pos = -1;
llama_pos delta = 0;
@@ -42,6 +64,11 @@ struct llama_kv_cache {
bool hybrid = false;
bool v_trans = true; // the value tensor is transposed
// openPangu s_l holds position-strict MoME conv state, not per-sequence recurrent
// slots; Qwen3Next-style s_l handling (seq ops, state serialization, s_copy) must
// skip it. Speculative rollback snapshots/restores it via the whole-slot spec checkpoint.
bool s_l_position_strict = false;
// Note: The value of head isn't only used to optimize searching
// for a free KV slot. llama_decode_internal also uses it, so it
// cannot be freely changed after a slot has been allocated.
@@ -61,10 +88,10 @@ struct llama_kv_cache {
std::vector<struct ggml_tensor *> v_l;
std::vector<struct ggml_tensor *> s_l; // per layer recurrent state storage (Qwen3Next)
// DSA lightning-indexer key cache (GLM-5.2 / DeepSeek-V3.2). One per layer, MQA single
// head: [indexer_head_size, kv_size]. Mirrors k_l but stores the (Hadamard-rotated)
// indexer keys so a decoded token scores against ALL past indexer keys, not just the
// current batch. Empty unless the model has the DSA indexer.
// Persistent DSA indexer-key cache. One per indexer layer, MQA single head:
// [indexer_head_size, kv_size]. Stores architecture-specific indexer keys in their
// scoring representation so a decoded token scores against all past indexer keys.
// Empty unless the model has the DSA indexer.
std::vector<struct ggml_tensor *> kr_l;
// When true, the delta_net graph builder will enable per-step SSM state saves
@@ -372,6 +399,7 @@ struct llama_context {
struct ggml_tensor * inp_out_ids; // I32 [n_outputs]
struct ggml_tensor * inp_KQ_mask; // F32 [kv_size, n_batch]
struct ggml_tensor * inp_KQ_mask_swa; // F32 [kv_size, n_batch]
struct ggml_tensor * inp_KQ_mask_swa_win = nullptr; // F32 [openPangu SWA W_view, n_batch]
struct ggml_tensor * inp_K_shift; // I32 [kv_size]
struct ggml_tensor * inp_mean; // F32 [n_batch, n_batch]
struct ggml_tensor * inp_cls; // I32 [n_batch]
@@ -384,14 +412,35 @@ struct llama_context {
struct ggml_tensor * inp_KQ_mask_cross; // F32 [n_outputs_enc, n_batch]
struct ggml_tensor * inp_scale = nullptr; // F32 [n_tokens]
struct ggml_tensor * inp_mtp_states = nullptr;
struct ggml_tensor * inp_mtp_carry = nullptr; // F32 [n_embd, nextn-1] per-head hidden at the last committed position
struct ggml_tensor * inp_dsa_sink = nullptr; // F32 [n_kv, n_tokens] per-sequence attention-sink boost for DSA indexer top-k
struct ggml_tensor * inp_mask_inf = nullptr;
struct openpangu_swa_window_view_state {
bool active = false;
int32_t n_kv = 0;
int32_t n_tokens = 0;
uint32_t window = 0;
uint32_t pad = 0;
int64_t w_view = 0;
int64_t win_off = 0;
} openpangu_swa_window_view;
// multi-head MTP chaining state: head k's output row at the last committed position,
// written back after each warmup/update decode and fed into the next MTP graph through
// inp_mtp_carry (zeroed when a prompt warmup restarts from position 0). The readback is
// issued async after compute; mtp_carry_pending marks a copy that must be synchronized
// before the host buffer is read or resized.
std::vector<float> mtp_carry;
bool mtp_carry_pending = false;
ggml_backend_t ggml_backend_by_name(const char * name);
struct Prev;
std::unique_ptr<Prev> prev;
std::unique_ptr<Prev> prev_mtp;
int32_t mtp_step_idx = 0;
int32_t mtp_n_heads = 0;
void reset_scheduler();
bool can_reuse_graph(const llama_batch & u_batch);
@@ -409,6 +458,8 @@ struct llama_context {
// uninitialized -> wrong block-max-pool/top-k -> degraded/NaN sparse-FA decode). Register the
// kr_l cpy per layer here and patch its offset in update_cache_copies(), exactly like K/V.
std::vector<CacheCopy> dsa_cache_copies;
std::vector<CacheCopy> openpangu_cache_copies;
std::vector<CacheCopy> openpangu_cache_copies_mtp;
bool update_cache_copies();
@@ -418,6 +469,8 @@ struct llama_context {
bool prepare_mtp_graph_inputs(
struct llama_context & lctx);
void set_mtp_op_type(llama_mtp_op_type value);
void set_mtp_step_idx(int32_t value);
void set_mtp_n_heads(int32_t value);
int max_nodes(int n_tokens, int n_kv) const;
};
src/llama-cparams.h
+1
View File
@@ -39,6 +39,7 @@ struct llama_cparams {
bool fused_mmad;
bool rope_cache;
bool graph_reuse;
bool prefetch_experts;
bool k_cache_hadamard;
bool v_cache_hadamard;
bool dsa_indexer_hadamard = true; // apply Walsh-Hadamard rotation to DSA indexer q/k (precision)
src/llama-hparams.cpp
+72
View File
@@ -1131,6 +1131,78 @@ void llm_load_hparams(
default: model.type = e_model::MODEL_UNKNOWN;
}
} break;
case LLM_ARCH_OPENPANGU:
{
ml.get_key(LLM_KV_ATTENTION_LAYERNORM_RMS_EPS, hparams.f_norm_rms_eps);
ml.get_key(LLM_KV_LEADING_DENSE_BLOCK_COUNT, hparams.n_layer_dense_lead);
ml.get_key(LLM_KV_ATTENTION_Q_LORA_RANK, hparams.n_lora_q);
ml.get_key(LLM_KV_ATTENTION_KV_LORA_RANK, hparams.n_lora_kv);
ml.get_key(LLM_KV_EXPERT_FEED_FORWARD_LENGTH, hparams.n_ff_exp);
ml.get_key(LLM_KV_EXPERT_SHARED_COUNT, hparams.n_expert_shared);
ml.get_key(LLM_KV_EXPERT_WEIGHTS_SCALE, hparams.expert_weights_scale);
ml.get_key(LLM_KV_EXPERT_WEIGHTS_NORM, hparams.expert_weights_norm, false);
// openPangu routes with a sigmoid gate + e_score_correction bias
hparams.expert_gating_func = LLM_EXPERT_GATING_FUNC_TYPE_NONE;
ml.get_key(LLM_KV_EXPERT_GATING_FUNC, hparams.expert_gating_func, false);
if (hparams.expert_gating_func == LLM_EXPERT_GATING_FUNC_TYPE_NONE) {
hparams.expert_gating_func = LLM_EXPERT_GATING_FUNC_SIGMOID;
}
// DSA lightning indexer
ml.get_key(LLM_KV_ATTENTION_INDEXER_HEAD_COUNT, hparams.indexer_n_head, false);
ml.get_key(LLM_KV_ATTENTION_INDEXER_KEY_LENGTH, hparams.indexer_head_size, false);
ml.get_key(LLM_KV_ATTENTION_INDEXER_TOP_K, hparams.indexer_top_k, false);
// mHC / Hyper-Connections + learned param sink
ml.get_key(LLM_KV_OPENPANGU_MHC_NUM_STREAM, hparams.mhc_num_stream);
ml.get_key(LLM_KV_OPENPANGU_MHC_RECUR_NORM, hparams.mhc_recur_norm);
ml.get_key(LLM_KV_OPENPANGU_PARAM_SINK_NUMBER, hparams.param_sink_number);
// NextN / MTP layers are appended at the end and skipped for base generation
ml.get_key(LLM_KV_NEXTN_PREDICT_LAYERS, hparams.nextn_predict_layers, false);
if (hparams.nextn_predict_layers > 0 && hparams.nextn_predict_layers < hparams.n_layer) {
hparams.n_layer_kv_from_start = hparams.n_layer - hparams.nextn_predict_layers;
}
// DSA/SWA schedule: openpangu.swa_layers lists the sliding-window layer ids and
// openpangu.sliding_window_list the per-entry window; the remaining base layers
// are DSA (indexer + top-k, no window). The NextN/MTP layers appear in the SWA
// list with their own (larger) window, used by the MTP graphs. Absent keys keep
// every window at 0 = dense fallback (pre-DSA GGUFs keep working).
{
std::vector<uint32_t> swa_ids, swa_windows;
const bool have_ids = ml.get_arr("openpangu.swa_layers", swa_ids, false);
const bool have_win = ml.get_arr("openpangu.sliding_window_list", swa_windows, false);
if (have_ids && have_win && swa_ids.size() == swa_windows.size()) {
const uint32_t n_base = hparams.n_layer > hparams.nextn_predict_layers
? hparams.n_layer - hparams.nextn_predict_layers : hparams.n_layer;
for (size_t i = 0; i < swa_ids.size(); ++i) {
const uint32_t il = swa_ids[i];
if (il >= hparams.n_layer) {
throw std::runtime_error(format("openpangu.swa_layers contains out-of-range layer %u", il));
}
hparams.openpangu_window[il] = swa_windows[i];
if (il < n_base) {
if (hparams.n_swa != 0 && hparams.n_swa != swa_windows[i]) {
throw std::runtime_error("openpangu: non-uniform base sliding windows are not supported");
}
hparams.n_swa = swa_windows[i];
} else {
if (hparams.n_swa_mtp != 0 && hparams.n_swa_mtp != swa_windows[i]) {
throw std::runtime_error("openpangu: non-uniform MTP sliding windows are not supported");
}
hparams.n_swa_mtp = swa_windows[i];
}
}
} else if (have_ids || have_win) {
LLAMA_LOG_WARN("%s: openpangu SWA schedule keys are inconsistent - keeping dense fallback\n", __func__);
}
}
model.type = e_model::MODEL_UNKNOWN; // 92B-A6B (46 + 3 MTP layers)
} break;
case LLM_ARCH_CHATGLM:
{
ml.get_key(LLM_KV_ATTENTION_LAYERNORM_RMS_EPS, hparams.f_norm_rms_eps);
src/llama-hparams.h
+10
View File
@@ -131,6 +131,16 @@ struct llama_hparams {
// indexer_types metadata if present, else derived from the GLM-5.2 config rule at load time.
std::array<bool, LLAMA_MAX_LAYERS> indexer_is_full = {};
// openPangu-2.0 (mHC / Hyper-Connections + learned param sink)
uint32_t mhc_num_stream = 1;
uint32_t mhc_recur_norm = 0;
uint32_t param_sink_number = 0;
// openPangu DSA/SWA schedule: per-layer sliding window (0 = DSA layer, full causal
// attention over the indexer's top-k selection). The NextN/MTP layers carry their own
// (larger) window, applied when the graph is built with an MTP op type.
uint32_t n_swa_mtp = 0;
std::array<uint32_t, LLAMA_MAX_LAYERS> openpangu_window = {};
// qwen3vl deepstack
uint32_t n_deepstack_layers = 0;
src/llama-load-tensors.cpp
+148
View File
@@ -119,6 +119,7 @@ struct create_tensors_helper : public create_tensors_helper_interface {
bool create_arctix_tensors(const LLM_TN & tn);
bool create_deepseek2_tensors(const LLM_TN & tn);
bool create_openpangu_tensors(const LLM_TN & tn);
bool create_glm_dsa_tensors(const LLM_TN & tn);
@@ -2870,6 +2871,151 @@ bool create_tensors_helper::create_glm_dsa_tensors(const LLM_TN & tn) {
return use_mmap_buffer;
}
// openPangu-2.0-Flash: GLM-DSA-style MLA + MoE base, plus mHC / MoME conv / param-sink /
// sandwich norms / DSA lightning indexer. The graph runs absorbed MLA over a latent KV
// cache from the converter's pre-split attn_k_b/attn_v_b; the indexer tensors feed the
// DSA top-k selection on windowless layers when the GGUF carries a DSA/SWA schedule.
// Conv weights keep their torch [C,1,kernel] layout (ggml ne = {kernel,1,C}).
bool create_tensors_helper::create_openpangu_tensors(const LLM_TN & tn) {
LOADING_PRELUDE
const int64_t n_embd_head_qk_rope = hparams.n_rot;
const int64_t n_embd_head_qk_nope = hparams.n_embd_head_k(0) - hparams.n_rot;
const int64_t q_lora_rank = hparams.n_lora_q;
const int64_t kv_lora_rank = hparams.n_lora_kv;
const int64_t n_ff_exp = hparams.n_ff_exp;
const int64_t n_expert_shared = hparams.n_expert_shared;
const int64_t S = hparams.mhc_num_stream; // 4
const int64_t SH = S * n_embd; // concatenated multi-stream width
const int64_t phi_out = (S + 2) * S; // mHC phi output width
const int64_t beta_len = S * (S + 2); // mHC beta length
const int64_t sink_n = hparams.param_sink_number; // 128
const int64_t kernel = 3; // MoME causal conv width
model.tok_embd = create_tensor(ctx_input, tn(LLM_TENSOR_TOKEN_EMBD, "weight"), {n_embd, n_vocab});
// output
{
model.output_norm = create_tensor(ctx_output, tn(LLM_TENSOR_OUTPUT_NORM, "weight"), {n_embd});
model.output = create_tensor(ctx_output, tn(LLM_TENSOR_OUTPUT, "weight"), {n_embd, n_vocab});
}
// global mHC stream-merge module (non-block)
model.mhc_merge_phi = create_tensor(ctx_output, tn(LLM_TENSOR_MHC_MERGE_PHI, "weight"), {SH, S});
model.mhc_merge_alpha = create_tensor(ctx_output, tn(LLM_TENSOR_MHC_MERGE_ALPHA), {1});
model.mhc_merge_beta = create_tensor(ctx_output, tn(LLM_TENSOR_MHC_MERGE_BETA), {S});
model.mhc_merge_gamma = create_tensor(ctx_output, tn(LLM_TENSOR_MHC_MERGE_GAMMA), {SH});
for (int i = 0; i < n_layer; ++i) {
const bool is_mtp_layer = hparams.nextn_predict_layers > 0 &&
static_cast<uint32_t>(i) >= n_layer - hparams.nextn_predict_layers;
int flags = 0;
if (!model.mtp && is_mtp_layer) {
flags |= llama_model_loader::TENSOR_SKIP | llama_model_loader::TENSOR_NOT_REQUIRED;
}
ggml_context * ctx_layer = ctx_for_layer(i);
ggml_context * ctx_split = ctx_for_layer_split(i);
auto & layer = model.layers[i];
const auto graph_or_attn = (model.split_mode == LLAMA_SPLIT_MODE_GRAPH ||
model.split_mode == LLAMA_SPLIT_MODE_ATTN);
auto norm_ctx = graph_or_attn ? ctx_split : ctx_layer;
auto moe_ctx = graph_or_attn ? ctx_split : ctx_layer;
// --- norms (sandwich): input / post-attn / pre-mlp(ffn) / post-mlp ---
layer.attn_norm = create_tensor(norm_ctx, tn(LLM_TENSOR_ATTN_NORM, "weight", i), {n_embd}, flags);
layer.attn_post_norm = create_tensor(norm_ctx, tn(LLM_TENSOR_ATTN_POST_NORM, "weight", i), {n_embd}, flags);
layer.attn_q_a_norm = create_tensor(norm_ctx, tn(LLM_TENSOR_ATTN_Q_A_NORM, "weight", i), {q_lora_rank}, flags);
layer.attn_kv_a_norm = create_tensor(norm_ctx, tn(LLM_TENSOR_ATTN_KV_A_NORM, "weight", i), {kv_lora_rank}, flags);
// block_post_norm only present on a layer subset -> optional
// block_post_layernorm is RMSNorm over the concatenated S*H (mhc_num_stream * hidden)
layer.block_post_norm = create_tensor(norm_ctx, tn(LLM_TENSOR_BLOCK_POST_NORM, "weight", i), {SH}, flags | llama_model_loader::TENSOR_NOT_REQUIRED);
// --- MLA projections (ik-native, pre-split k_b/v_b) ---
layer.wq_a = create_tensor(ctx_split, tn(LLM_TENSOR_ATTN_Q_A, "weight", i), {n_embd, q_lora_rank}, flags);
layer.wq_b = create_tensor(ctx_split, tn(LLM_TENSOR_ATTN_Q_B, "weight", i), {q_lora_rank, n_head * n_embd_head_k}, flags);
layer.wkv_a_mqa = create_tensor(ctx_split, tn(LLM_TENSOR_ATTN_KV_A_MQA, "weight", i), {n_embd, kv_lora_rank + n_embd_head_qk_rope}, flags);
// older GGUFs include the fused kv_b_proj, but the absorbed-MLA graph has no
// consumer for it (attention runs entirely on pre-split k_b/v_b below) -> skip
// when present and allow it to be absent from new conversions
layer.wkv_b = create_tensor(ctx_split, tn(LLM_TENSOR_ATTN_KV_B, "weight", i), {kv_lora_rank, n_head * (n_embd_head_qk_nope + n_embd_head_v)}, flags | llama_model_loader::TENSOR_SKIP | llama_model_loader::TENSOR_NOT_REQUIRED);
// converter-emitted pre-split k_b/v_b, loaded 2D as written (head-major rows) and
// reshaped in-graph: k_b absorbs q_nope into latent space, v_b up-projects the output
layer.wk_b = create_tensor(ctx_split, tn(LLM_TENSOR_ATTN_K_B, "weight", i), {n_embd_head_qk_nope, n_head * kv_lora_rank}, flags);
layer.wv_b = create_tensor(ctx_split, tn(LLM_TENSOR_ATTN_V_B, "weight", i), {kv_lora_rank, n_head * n_embd_head_v}, flags);
layer.wo = create_tensor(ctx_split, tn(LLM_TENSOR_ATTN_OUT, "weight", i), {n_head * n_embd_head_v, n_embd}, flags);
// --- MoME causal convs (torch [C,1,3] -> gguf squeezes to 2D {3, C}) ---
layer.qa_conv = create_tensor(ctx_split, tn(LLM_TENSOR_ATTN_QA_CONV, "weight", i), {kernel, q_lora_rank}, flags);
// compresskv_conv acts on the compressed-kv latent only (kv_lora_rank), NOT the k_pe part
layer.kv_conv = create_tensor(ctx_split, tn(LLM_TENSOR_ATTN_KV_CONV, "weight", i), {kernel, kv_lora_rank}, flags);
layer.o_conv = create_tensor(ctx_split, tn(LLM_TENSOR_ATTN_O_CONV, "weight", i), {kernel, n_head * n_embd_head_v}, flags);
// --- learned static param sink (latent-kv prepended to attention) ---
layer.param_sink_kv = create_tensor(ctx_split, tn(LLM_TENSOR_ATTN_PARAM_SINK_KV, i), {kv_lora_rank, sink_n}, flags);
layer.param_sink_k_pe = create_tensor(ctx_split, tn(LLM_TENSOR_ATTN_PARAM_SINK_K_PE, i), {n_embd_head_qk_rope, sink_n}, flags);
// --- DSA indexer (loaded-but-unused in the dense-fallback graph) ---
layer.indexer_k_norm = create_tensor(ctx_split, tn(LLM_TENSOR_INDEXER_K_NORM, "weight", i), {hparams.indexer_head_size}, flags | llama_model_loader::TENSOR_NOT_REQUIRED);
layer.indexer_proj = create_tensor(ctx_split, tn(LLM_TENSOR_INDEXER_PROJ, "weight", i), {n_embd, hparams.indexer_n_head}, flags | llama_model_loader::TENSOR_NOT_REQUIRED);
layer.indexer_attn_k = create_tensor(ctx_split, tn(LLM_TENSOR_INDEXER_ATTN_K, "weight", i), {n_embd, hparams.indexer_head_size}, flags | llama_model_loader::TENSOR_NOT_REQUIRED);
layer.indexer_attn_q_b = create_tensor(ctx_split, tn(LLM_TENSOR_INDEXER_ATTN_Q_B, "weight", i), {q_lora_rank, hparams.indexer_n_head * hparams.indexer_head_size}, flags | llama_model_loader::TENSOR_NOT_REQUIRED);
// --- mHC / Hyper-Connections (per attn + per mlp sublayer, float32) ---
// MTP/NextN layers run WITHOUT mHC (tail_use_mhc=false in the reference; the
// checkpoint has no mhc tensors for them), so only create these for base layers.
if (!is_mtp_layer) {
layer.mhc_attn_phi = create_tensor(norm_ctx, tn(LLM_TENSOR_MHC_ATTN_PHI, "weight", i), {SH, phi_out}, flags);
layer.mhc_attn_alpha = create_tensor(norm_ctx, tn(LLM_TENSOR_MHC_ATTN_ALPHA, i), {3}, flags);
layer.mhc_attn_beta = create_tensor(norm_ctx, tn(LLM_TENSOR_MHC_ATTN_BETA, i), {beta_len}, flags);
layer.mhc_attn_gamma = create_tensor(norm_ctx, tn(LLM_TENSOR_MHC_ATTN_GAMMA, i), {SH}, flags);
layer.mhc_mlp_phi = create_tensor(norm_ctx, tn(LLM_TENSOR_MHC_MLP_PHI, "weight", i), {SH, phi_out}, flags);
layer.mhc_mlp_alpha = create_tensor(norm_ctx, tn(LLM_TENSOR_MHC_MLP_ALPHA, i), {3}, flags);
layer.mhc_mlp_beta = create_tensor(norm_ctx, tn(LLM_TENSOR_MHC_MLP_BETA, i), {beta_len}, flags);
layer.mhc_mlp_gamma = create_tensor(norm_ctx, tn(LLM_TENSOR_MHC_MLP_GAMMA, i), {SH}, flags);
}
// --- FFN: dense-lead then MoE (routed + shared, sigmoid bias) ---
layer.ffn_norm = create_tensor(norm_ctx, tn(LLM_TENSOR_FFN_NORM, "weight", i), {n_embd}, flags);
layer.ffn_post_norm = create_tensor(norm_ctx, tn(LLM_TENSOR_FFN_POST_NORM, "weight", i), {n_embd}, flags);
if (i < (int) hparams.n_layer_dense_lead) {
layer.ffn_gate = create_tensor(ctx_split, tn(LLM_TENSOR_FFN_GATE, "weight", i), {n_embd, n_ff}, flags);
layer.ffn_down = create_tensor(ctx_split, tn(LLM_TENSOR_FFN_DOWN, "weight", i), { n_ff, n_embd}, flags);
layer.ffn_up = create_tensor(ctx_split, tn(LLM_TENSOR_FFN_UP, "weight", i), {n_embd, n_ff}, flags);
} else {
layer.ffn_gate_inp = create_tensor(moe_ctx, tn(LLM_TENSOR_FFN_GATE_INP, "weight", i), {n_embd, n_expert}, flags);
layer.ffn_exp_probs_b = create_tensor(moe_ctx, tn(LLM_TENSOR_FFN_EXP_PROBS_B, "bias", i), {n_expert}, flags);
GGML_ASSERT(n_expert > 0);
GGML_ASSERT(n_expert_used > 0);
layer.ffn_gate_exps = create_tensor(ctx_split, tn(LLM_TENSOR_FFN_GATE_EXPS, "weight", i), { n_embd, n_ff_exp, n_expert}, flags);
layer.ffn_down_exps = create_tensor(ctx_split, tn(LLM_TENSOR_FFN_DOWN_EXPS, "weight", i), {n_ff_exp, n_embd, n_expert}, flags);
layer.ffn_up_exps = create_tensor(ctx_split, tn(LLM_TENSOR_FFN_UP_EXPS, "weight", i), { n_embd, n_ff_exp, n_expert}, flags);
layer.ffn_gate_shexp = create_tensor(ctx_split, tn(LLM_TENSOR_FFN_GATE_SHEXP, "weight", i), {n_embd, n_ff_exp * n_expert_shared}, flags);
layer.ffn_down_shexp = create_tensor(ctx_split, tn(LLM_TENSOR_FFN_DOWN_SHEXP, "weight", i), { n_ff_exp * n_expert_shared, n_embd}, flags);
layer.ffn_up_shexp = create_tensor(ctx_split, tn(LLM_TENSOR_FFN_UP_SHEXP, "weight", i), {n_embd, n_ff_exp * n_expert_shared}, flags);
}
if (is_mtp_layer) {
layer.nextn.eh_proj = create_tensor(ctx_split, tn(LLM_TENSOR_NEXTN_EH_PROJ, "weight", i), { 2 * n_embd, n_embd }, flags);
layer.nextn.enorm = create_tensor(ctx_split, tn(LLM_TENSOR_NEXTN_ENORM, "weight", i), { n_embd }, flags);
layer.nextn.hnorm = create_tensor(ctx_split, tn(LLM_TENSOR_NEXTN_HNORM, "weight", i), { n_embd }, flags);
layer.nextn.embed_tokens = create_tensor(ctx_split, tn(LLM_TENSOR_NEXTN_EMBED_TOKENS, "weight", i), { n_embd, n_vocab }, flags | llama_model_loader::TENSOR_NOT_REQUIRED);
layer.nextn.shared_head_head = create_tensor(ctx_split, tn(LLM_TENSOR_NEXTN_SHARED_HEAD_HEAD, "weight", i), { n_embd, n_vocab }, flags | llama_model_loader::TENSOR_NOT_REQUIRED);
layer.nextn.shared_head_norm = create_tensor(ctx_split, tn(LLM_TENSOR_NEXTN_SHARED_HEAD_NORM, "weight", i), { n_embd }, flags);
}
}
return use_mmap_buffer;
}
bool create_tensors_helper::create_glm4_moe_tensors(const LLM_TN & tn) {
LOADING_PRELUDE
@@ -4514,6 +4660,8 @@ bool create_tensors_helper::create_tensors() {
use_mmap_buffer = create_deepseek2_tensors(tn); break;
case LLM_ARCH_GLM_DSA:
use_mmap_buffer = create_glm_dsa_tensors(tn); break;
case LLM_ARCH_OPENPANGU:
use_mmap_buffer = create_openpangu_tensors(tn); break;
case LLM_ARCH_GLM4_MOE:
use_mmap_buffer = create_glm4_moe_tensors(tn); break;
case LLM_ARCH_BITNET:
src/llama-mmap.cpp
+2
View File
@@ -3,6 +3,7 @@
#include "llama-impl.h"
#include "ggml.h"
#include "ggml-backend.h"
#include <cstring>
#include <climits>
@@ -435,6 +436,7 @@ struct llama_mmap::impl {
}
~impl() {
ggml_backend_prefetch_unregister_mapping(addr);
for (const auto & frag : mapped_fragments) {
if (munmap((char *) addr + frag.first, frag.second - frag.first)) {
LLAMA_LOG_WARN("warning: munmap failed: %s\n", strerror(errno));
src/llama-model-loader.cpp
+15 -8
View File
@@ -1078,6 +1078,8 @@ bool llama_model_loader::load_all_data(
std::vector<void*> host_ptrs;
std::vector<ggml_backend_event_t> events;
std::vector<std::unique_ptr<llama_mmap>> split_mappings(files.size());
ggml_backend_t cuda_backend = nullptr;
if (!use_mmap && !check_tensors) {
// When not using mmaped io use async uploads from pinned memory to GPU memory.
@@ -1197,17 +1199,21 @@ bool llama_model_loader::load_all_data(
const char * buffer_name = ggml_backend_buffer_name(cur->buffer);
const bool is_probably_split_mode_graph = std::strncmp(buffer_name, GGML_CUDA_NAME, strlen(GGML_CUDA_NAME)) == 0;
if (is_probably_split_mode_graph) {
auto & read_buf = read_bufs[thread_idx];
if (read_buf.capacity() > n_size) {
read_buf = std::vector<no_init<uint8_t>>();
llama_mmap * mapping;
{
std::lock_guard<std::mutex> lock(load_mutex);
auto & m = split_mappings[weight->idx];
if (!m) {
m.reset(new llama_mmap(files.at(weight->idx).get(), 0, ggml_is_numa()));
}
mapping = m.get();
}
read_buf.resize(n_size);
file->seek(weight->offs, SEEK_SET);
file->read_raw(read_buf.data(), n_size);
ggml_backend_tensor_set(cur, read_buf.data(), 0, n_size);
if (check_tensors && !ggml_validate_row_data(cur->type, read_buf.data(), n_size)) {
uint8_t * data = (uint8_t *) mapping->addr() + weight->offs;
ggml_backend_tensor_set(cur, data, 0, n_size);
if (check_tensors && !ggml_validate_row_data(cur->type, data, n_size)) {
throw std::runtime_error(format("tensor '%s' has invalid data", ggml_get_name(cur)));
}
mapping->dontneed_fragment(weight->offs, weight->offs + n_size);
return n_size;
}
#endif
@@ -1370,3 +1376,4 @@ template std::enable_if<std::is_integral<unsigned int>::value, bool>::type llama
template std::enable_if<std::is_integral<unsigned int>::value, bool>::type llama_model_loader::get_arr_n<unsigned int>(enum llm_kv, unsigned int&, bool);
template bool llama_model_loader::get_arr<int32_t, 8>(const std::string &, std::array<int32_t, 8> &, bool);
template bool llama_model_loader::get_arr<uint32_t, 8>(const std::string &, std::array<uint32_t, 8> &, bool);
template bool llama_model_loader::get_arr<uint32_t>(const std::string &, std::vector<uint32_t> &, bool);
src/llama-model.cpp
+94 -1
View File
@@ -1775,6 +1775,71 @@ static const std::map<llm_arch, std::map<llm_tensor, std::string>> LLM_TENSOR_NA
},
},
{
LLM_ARCH_OPENPANGU,
{
{ LLM_TENSOR_TOKEN_EMBD, "token_embd" },
{ LLM_TENSOR_OUTPUT_NORM, "output_norm" },
{ LLM_TENSOR_OUTPUT, "output" },
{ LLM_TENSOR_ATTN_NORM, "blk.%d.attn_norm" },
{ LLM_TENSOR_ATTN_Q_A, "blk.%d.attn_q_a" },
{ LLM_TENSOR_ATTN_Q_A_NORM, "blk.%d.attn_q_a_norm" },
{ LLM_TENSOR_ATTN_Q_B, "blk.%d.attn_q_b" },
{ LLM_TENSOR_ATTN_KV_A_MQA, "blk.%d.attn_kv_a_mqa" },
{ LLM_TENSOR_ATTN_KV_A_NORM, "blk.%d.attn_kv_a_norm" },
{ LLM_TENSOR_ATTN_KV_B, "blk.%d.attn_kv_b" },
{ LLM_TENSOR_ATTN_K_B, "blk.%d.attn_k_b" },
{ LLM_TENSOR_ATTN_V_B, "blk.%d.attn_v_b" },
{ LLM_TENSOR_ATTN_OUT, "blk.%d.attn_output" },
{ LLM_TENSOR_ATTN_POST_NORM, "blk.%d.post_attention_norm" },
// DSA lightning indexer (loaded; unused in the dense-fallback graph)
{ LLM_TENSOR_INDEXER_K_NORM, "blk.%d.attn_indexer_k_norm" },
{ LLM_TENSOR_INDEXER_PROJ, "blk.%d.attn_indexer_weights_proj" },
{ LLM_TENSOR_INDEXER_ATTN_K, "blk.%d.attn_indexer_k" },
{ LLM_TENSOR_INDEXER_ATTN_Q_B, "blk.%d.attn_indexer_q_b" },
// MoME causal convs + param sink
{ LLM_TENSOR_ATTN_QA_CONV, "blk.%d.attn_qa_conv" },
{ LLM_TENSOR_ATTN_KV_CONV, "blk.%d.attn_compresskv_conv" },
{ LLM_TENSOR_ATTN_O_CONV, "blk.%d.attn_o_conv" },
{ LLM_TENSOR_ATTN_PARAM_SINK_KV, "blk.%d.attn_param_sink_kv" },
{ LLM_TENSOR_ATTN_PARAM_SINK_K_PE, "blk.%d.attn_param_sink_k_pe" },
// FFN (dense-lead + MoE, sandwich norms)
{ LLM_TENSOR_FFN_NORM, "blk.%d.ffn_norm" },
{ LLM_TENSOR_FFN_POST_NORM, "blk.%d.post_ffw_norm" },
{ LLM_TENSOR_FFN_GATE, "blk.%d.ffn_gate" },
{ LLM_TENSOR_FFN_UP, "blk.%d.ffn_up" },
{ LLM_TENSOR_FFN_DOWN, "blk.%d.ffn_down" },
{ LLM_TENSOR_FFN_GATE_INP, "blk.%d.ffn_gate_inp" },
{ LLM_TENSOR_FFN_GATE_EXPS, "blk.%d.ffn_gate_exps" },
{ LLM_TENSOR_FFN_DOWN_EXPS, "blk.%d.ffn_down_exps" },
{ LLM_TENSOR_FFN_UP_EXPS, "blk.%d.ffn_up_exps" },
{ LLM_TENSOR_FFN_GATE_SHEXP, "blk.%d.ffn_gate_shexp" },
{ LLM_TENSOR_FFN_DOWN_SHEXP, "blk.%d.ffn_down_shexp" },
{ LLM_TENSOR_FFN_UP_SHEXP, "blk.%d.ffn_up_shexp" },
{ LLM_TENSOR_FFN_EXP_PROBS_B, "blk.%d.exp_probs_b" },
// mHC / Hyper-Connections
{ LLM_TENSOR_MHC_ATTN_PHI, "blk.%d.attn_mhc_phi" },
{ LLM_TENSOR_MHC_ATTN_ALPHA, "blk.%d.attn_mhc_alpha" },
{ LLM_TENSOR_MHC_ATTN_BETA, "blk.%d.attn_mhc_beta" },
{ LLM_TENSOR_MHC_ATTN_GAMMA, "blk.%d.attn_mhc_gamma" },
{ LLM_TENSOR_MHC_MLP_PHI, "blk.%d.mlp_mhc_phi" },
{ LLM_TENSOR_MHC_MLP_ALPHA, "blk.%d.mlp_mhc_alpha" },
{ LLM_TENSOR_MHC_MLP_BETA, "blk.%d.mlp_mhc_beta" },
{ LLM_TENSOR_MHC_MLP_GAMMA, "blk.%d.mlp_mhc_gamma" },
{ LLM_TENSOR_MHC_MERGE_PHI, "merge_mhc_phi" },
{ LLM_TENSOR_MHC_MERGE_ALPHA, "merge_mhc_alpha" },
{ LLM_TENSOR_MHC_MERGE_BETA, "merge_mhc_beta" },
{ LLM_TENSOR_MHC_MERGE_GAMMA, "merge_mhc_gamma" },
{ LLM_TENSOR_BLOCK_POST_NORM, "blk.%d.block_post_norm" },
// NextN / MTP tail
{ LLM_TENSOR_NEXTN_EH_PROJ, "blk.%d.nextn.eh_proj" },
{ LLM_TENSOR_NEXTN_EMBED_TOKENS, "blk.%d.nextn.embed_tokens" },
{ LLM_TENSOR_NEXTN_ENORM, "blk.%d.nextn.enorm" },
{ LLM_TENSOR_NEXTN_HNORM, "blk.%d.nextn.hnorm" },
{ LLM_TENSOR_NEXTN_SHARED_HEAD_HEAD, "blk.%d.nextn.shared_head_head" },
{ LLM_TENSOR_NEXTN_SHARED_HEAD_NORM, "blk.%d.nextn.shared_head_norm" },
},
},
{
LLM_ARCH_UNKNOWN,
{
@@ -2050,6 +2115,10 @@ bool llama_model_has_recurrent(const llama_model * model) {
return llm_arch_is_hybrid(model->arch) || llm_arch_is_recurrent(model->arch);
}
bool llama_model_is_openpangu(const llama_model * model) {
return model && model->arch == LLM_ARCH_OPENPANGU;
}
bool llama_model_is_gemma4_mtp_assistant(const llama_model * model) {
return model && (model->arch == LLM_ARCH_GEMMA4_MTP || model->arch == LLM_ARCH_GEMMA4_ASSISTANT);
}
@@ -2075,6 +2144,18 @@ bool llama_model_is_split_mode_graph(const struct llama_model * model) {
return model && (model->split_mode == LLAMA_SPLIT_MODE_GRAPH || model->split_mode == LLAMA_SPLIT_MODE_ATTN);
}
bool llama_model_supports_ctx_shift(const struct llama_model * model) {
// openPangu's latent K rows carry baked-in rope (k_pe) and the DSA indexer cache is
// keyed by absolute position; neither survives K-shift/defrag-style repositioning.
return model && model->arch != LLM_ARCH_OPENPANGU;
}
bool llama_model_supports_partial_kv_reuse(const struct llama_model * model) {
// openPangu keeps only the current MoME conv state, so a sequence can be extended or
// reset, but rewinding into its decoded middle cannot reconstruct the state at that point.
return model && model->arch != LLM_ARCH_OPENPANGU;
}
llm_tensor llm_tensor_type(llm_arch arch, const std::string & tensor_name, int il) {
auto it = LLM_TENSOR_NAMES.find(arch);
if (it == LLM_TENSOR_NAMES.end()) {
@@ -2106,12 +2187,24 @@ llm_tensor llm_tensor_type(llm_arch arch, const std::string & tensor_name, int i
return LLM_TENSOR_UNKNOWN;
}
size_t llama_model::cache_size(int il, ggml_type type_k, ggml_type type_v, uint32_t kv_size, int mla_attn, int n_seq_max, bool flash_attn) const {
size_t llama_model::cache_size(int il, ggml_type type_k, ggml_type type_v, ggml_type idx_type_k, uint32_t kv_size, int mla_attn, int n_seq_max, bool flash_attn) const {
if (il < 0 || il >= hparams.n_layer) return 0;
if (hparams.recurrent_layer_arr[il]) {
auto state_sots = std::min<uint32_t>(std::max<uint32_t>(1, n_seq_max), kv_size);
return hparams.n_embd_v_s() * state_sots * sizeof(float);
}
if (arch == LLM_ARCH_OPENPANGU) {
// MLA-latent cache: K row [ckv | roped k_pe]. The value-side latent is
// rederived from K per graph. DSA layers also cache one indexer key per
// position. The recurrent conv slot is constant-size and negligible here.
size_t size = ggml_row_size(type_k, hparams.n_lora_kv + hparams.n_rot) * kv_size;
if (hparams.indexer_head_size > 0 && hparams.n_swa > 0 &&
il < (int) hparams.n_layer - (int) hparams.nextn_predict_layers &&
hparams.openpangu_window[il] == 0) {
size += ggml_row_size(idx_type_k, hparams.indexer_head_size) * kv_size;
}
return size;
}
bool is_mla_attn = arch == LLM_ARCH_DEEPSEEK2 || arch == LLM_ARCH_GLM_DSA || arch == LLM_ARCH_MISTRAL4;
if (is_mla_attn && mla_attn) {
auto n_embd_head_qk_rope = hparams.n_rot;
src/llama-model.h
+27 -1
View File
@@ -388,10 +388,30 @@ struct llama_layer {
struct llama_layer_nextn nextn;
// openPangu-2.0: MoME causal convs + learned static param sink + mHC + block post-norm
struct ggml_tensor * qa_conv = nullptr;
struct ggml_tensor * kv_conv = nullptr; // compresskv_conv
struct ggml_tensor * o_conv = nullptr;
struct ggml_tensor * param_sink_kv = nullptr;
struct ggml_tensor * param_sink_k_pe = nullptr;
struct ggml_tensor * param_sink_blk = nullptr;
struct ggml_tensor * param_sink_lat_t = nullptr;
struct ggml_tensor * block_post_norm = nullptr;
struct ggml_tensor * mhc_attn_phi = nullptr;
struct ggml_tensor * mhc_attn_alpha = nullptr;
struct ggml_tensor * mhc_attn_beta = nullptr;
struct ggml_tensor * mhc_attn_gamma = nullptr;
struct ggml_tensor * mhc_mlp_phi = nullptr;
struct ggml_tensor * mhc_mlp_alpha = nullptr;
struct ggml_tensor * mhc_mlp_beta = nullptr;
struct ggml_tensor * mhc_mlp_gamma = nullptr;
std::unique_ptr<ggml_tensor> computed_wk_b;
std::unique_ptr<ggml_tensor> computed_wk_b_pp;
std::unique_ptr<ggml_tensor> computed_wv_b;
std::unique_ptr<ggml_tensor> computed_wkv_b;
std::unique_ptr<ggml_tensor> computed_param_sink_blk;
std::unique_ptr<ggml_tensor> computed_param_sink_lat_t;
// Per-device replicas of computed wk_b/wv_b (-sm graph). Buffers owned via model.bufs.
std::vector<std::unique_ptr<ggml_tensor>> computed_wk_b_replicas;
@@ -442,6 +462,12 @@ struct llama_model {
struct ggml_tensor * output_norm_enc;
struct ggml_tensor * output_mtp = nullptr;
// openPangu-2.0: global mHC stream-merge module (non-block)
struct ggml_tensor * mhc_merge_phi = nullptr;
struct ggml_tensor * mhc_merge_alpha = nullptr;
struct ggml_tensor * mhc_merge_beta = nullptr;
struct ggml_tensor * mhc_merge_gamma = nullptr;
std::unique_ptr<ggml_tensor> output_mtp_ptr;
llama_split_tensor split_output;
@@ -545,7 +571,7 @@ struct llama_model {
return hadamard_size(hparams.n_embd_head_v(il));
}
size_t cache_size(int il, ggml_type type_k, ggml_type type_v, uint32_t kv_size, int mla_attn, int n_seq_max, bool flash_attn) const;
size_t cache_size(int il, ggml_type type_k, ggml_type type_v, ggml_type idx_type_k, uint32_t kv_size, int mla_attn, int n_seq_max, bool flash_attn) const;
void set_tensor_overrides(const llama_model_params& params);
src/llama-vocab.cpp
+1
View File
@@ -2028,6 +2028,7 @@ void llama_vocab::impl::load(llama_model_loader & ml, const LLM_KV & kv) {
tokenizer_pre == "qwen2" ||
tokenizer_pre == "deepseek-r1-qwen" ||
tokenizer_pre == "kormo" ||
tokenizer_pre == "openpangu" ||
tokenizer_pre == "f2llmv2") {
pre_type = LLAMA_VOCAB_PRE_TYPE_QWEN2;
clean_spaces = false;
src/llama.cpp
+706 -66
View File
File diff suppressed because it is too large Load Diff
tests/test-chat-auto-parser.cpp
+189
View File
@@ -65,6 +65,9 @@ static void test_cohere_analysis(testing & t);
// End-to-end Cohere2MoE (North Code) dedicated PEG parser coverage.
static void test_cohere2moe_parser(testing & t);
// End-to-end MiniMax-M3 dedicated PEG parser coverage.
static void test_minimax_m3_partial_reasoning_parser(testing & t);
// SmolLM3 template analysis tests
static void test_smollm3_analysis(testing & t);
@@ -102,6 +105,7 @@ int main(int argc, char * argv[]) {
t.test("seed_oss_diffs", test_seed_oss_tool_analysis);
t.test("cohere", test_cohere_analysis);
t.test("cohere2moe_parser", test_cohere2moe_parser);
t.test("minimax_m3_partial_reasoning_parser", test_minimax_m3_partial_reasoning_parser);
t.test("nemotron", test_nemotron_analysis);
t.test("smollm3", test_smollm3_analysis);
t.test("standard_json_tools", test_standard_json_tools_formats);
@@ -1971,6 +1975,191 @@ static void test_tagged_args_with_embedded_quotes(testing & t) {
}
}
// End-to-end coverage for MiniMax-M3 partial reasoning streams:
// template apply -> PEG parse -> assert message and streaming diff. The partial
// case mirrors a real leaked <mm:think> stream where </mm:think> had not arrived.
static void test_minimax_m3_partial_reasoning_parser(testing & t) {
const std::string template_str = R"(
{%- set ns_token = ']<]minimax[>[' -%}
{%- set toolcall_begin_token = ns_token ~ '<tool_call>' -%}
{%- set toolcall_end_token = ns_token ~ '</tool_call>' -%}
{%- for message in messages -%}
{{- message.role ~ ': ' ~ message.content ~ '\n' -}}
{%- endfor -%}
{%- if tools -%}
{{- toolcall_begin_token ~ ns_token ~ '<invoke name="example">' ~ ns_token ~ '</invoke>' ~ toolcall_end_token -}}
{%- endif -%}
{%- if add_generation_prompt -%}
{{- '<mm:think>' -}}
{%- endif -%}
)";
common_chat_templates_inputs inputs;
common_chat_msg user;
user.role = "user";
user.content = "Please inspect the extension state.";
inputs.messages = { user };
inputs.add_generation_prompt = true;
inputs.reasoning_format = COMMON_REASONING_FORMAT_DEEPSEEK;
inputs.enable_thinking = true;
inputs.tools = {
common_chat_tool{
/* .name = */ "example",
/* .description = */ "Example tool",
/* .parameters = */ R"({"type":"object","properties":{}})",
},
common_chat_tool{
/* .name = */ "bash",
/* .description = */ "Run shell commands",
/* .parameters = */ R"({"type":"object","properties":{"command":{"type":"string"}},"required":["command"]})",
},
};
common_chat_templates_ptr tmpls(common_chat_templates_init(/* model = */ nullptr, template_str));
auto params = common_chat_templates_apply(tmpls.get(), inputs);
t.assert_equal("MiniMax-M3 parser selected", COMMON_CHAT_FORMAT_PEG_MINIMAX_M3, params.format);
t.assert_true("MiniMax-M3 parser generated", !params.parser.empty());
common_peg_arena arena;
arena.load(params.parser);
common_chat_parser_params parser_params(params);
parser_params.parser = arena;
const std::string partial_reasoning =
"The user is asking why I'm implementing F3 directly. Looking at the context:\n\n"
"1. The user said earlier: \"yes we do F3 manually, by";
auto partial_msg = common_chat_parse(partial_reasoning, /* is_partial = */ true, parser_params);
t.assert_equal("partial reasoning", partial_reasoning, partial_msg.reasoning_content);
t.assert_equal("partial visible content", std::string(), partial_msg.content);
t.assert_true("partial has no tool calls", partial_msg.tool_calls.empty());
auto diffs = common_chat_msg_diff::compute_diffs(common_chat_msg{}, partial_msg);
t.assert_equal("partial emits one diff", 1u, diffs.size());
if (diffs.size() == 1) {
t.assert_equal("partial diff is reasoning", partial_reasoning, diffs[0].reasoning_content_delta);
t.assert_equal("partial diff has no visible content", std::string(), diffs[0].content_delta);
t.assert_equal("partial diff has no tool index", std::string::npos, diffs[0].tool_call_index);
}
const std::string closed_output = partial_reasoning + "</mm:think>You're right to question.";
auto closed_msg = common_chat_parse(closed_output, /* is_partial = */ false, parser_params);
t.assert_equal("closed reasoning", partial_reasoning, closed_msg.reasoning_content);
t.assert_equal("closed visible content", std::string("You're right to question."), closed_msg.content);
t.assert_true("closed has no tool calls", closed_msg.tool_calls.empty());
const std::string command =
"# Deploy to electra and erabus\n"
"SSH_AUTH_SOCK=/run/user/1001/ssh-agent.socket scp index.ts electra:/tmp/index.ts";
const std::string tool_output =
"Let me do that."
"]<]minimax[>[<tool_call>\n"
"]<]minimax[>[<invoke name=\"bash\">"
"]<]minimax[>[<command>" + command + "]<]minimax[>[</command>"
"]<]minimax[>[</invoke>\n"
"]<]minimax[>[</tool_call>";
auto tool_msg = common_chat_parse(tool_output, /* is_partial = */ false, parser_params);
t.assert_equal("tool reasoning excludes markers", std::string("Let me do that."), tool_msg.reasoning_content);
t.assert_equal("tool visible content", std::string(), tool_msg.content);
t.assert_equal("tool call count", 1u, tool_msg.tool_calls.size());
if (tool_msg.tool_calls.size() == 1) {
t.assert_equal("tool call name", std::string("bash"), tool_msg.tool_calls[0].name);
t.assert_equal("tool call args", json({{"command", command}}).dump(), tool_msg.tool_calls[0].arguments);
}
const std::string cmux_command =
"cmux rpc surface.scrollback '{\"surface_id\":\"20CD49E8-A0FA-42E3-B349-7CCCC1075425\","
"\"workspace_id\":\"826FA4BA-01E5-4D84-893A-D70B0DAF9970\"}' 2>&1 | head -3";
const std::string malformed_nested_tool_output =
"]<]minimax[>[<tool_call>\n"
"]<]minimax[>[<invoke name=\"bash\">"
"]<]minimax[>[<command>" + cmux_command +
"]<]minimax[>[<tool_call>\n"
"]<]minimax[>[</invoke>\n"
"]<]minimax[>[</tool_call>老实\n"
"</command>]<]minimax[>[</invoke>\n"
"]<]minimax[>[</tool_call>\n"
"]<]minimax[>[<tool_call>\n"
"bash\n"
"</parameter>]<]minimax[>[</command>"
"]<]minimax[>[</invoke>\n"
"]<]minimax[>[</tool_call>";
auto malformed_nested_tool_msg = common_chat_parse(malformed_nested_tool_output,
/* is_partial = */ false,
parser_params);
t.assert_equal("malformed nested tool call count", 1u, malformed_nested_tool_msg.tool_calls.size());
if (malformed_nested_tool_msg.tool_calls.size() == 1) {
t.assert_equal("malformed nested tool call name", std::string("bash"), malformed_nested_tool_msg.tool_calls[0].name);
t.assert_equal(
"malformed nested tool args truncate namespace leak",
json({{"command", cmux_command}}).dump(),
malformed_nested_tool_msg.tool_calls[0].arguments);
t.assert_true("malformed nested tool args contain no minimax marker",
malformed_nested_tool_msg.tool_calls[0].arguments.find("]<]minimax[>[") == std::string::npos);
}
inputs.enable_thinking = false;
auto params_thinking_off = common_chat_templates_apply(tmpls.get(), inputs);
common_peg_arena thinking_off_arena;
thinking_off_arena.load(params_thinking_off.parser);
common_chat_parser_params thinking_off_parser_params(params_thinking_off);
thinking_off_parser_params.parser = thinking_off_arena;
const std::string leaked_reasoning =
"Right pane is up. Now let me think about the workflow lessons and update "
"kickass + smartass to include them explicitly.\n\n"
"Looking at kickass.md:\n"
"- Rule 4 covers L7 partially\n"
"- Rule 2 covers L15\n\n"
"Let me update kickass and smartass.";
const std::string tool_prelude =
"Let me update kickass and smartass with the workflow lessons explicitly. "
"First check what's there:";
const std::string unopened_closed_tool_output =
leaked_reasoning + "</mm:think>" + tool_prelude +
"]<]minimax[>[<tool_call>\n"
"]<]minimax[>[<invoke name=\"bash\">"
"]<]minimax[>[<command>cat /home/leaf/.pi/agent/agents/kickass.md 2>&1]<]minimax[>[</command>"
"]<]minimax[>[</invoke>\n"
"]<]minimax[>[</tool_call>";
auto unopened_closed_tool_msg = common_chat_parse(unopened_closed_tool_output, /* is_partial = */ false, thinking_off_parser_params);
t.assert_equal("unopened closed tool reasoning", leaked_reasoning + tool_prelude, unopened_closed_tool_msg.reasoning_content);
t.assert_equal("unopened closed tool visible content", std::string(), unopened_closed_tool_msg.content);
t.assert_equal("unopened closed tool call count", 1u, unopened_closed_tool_msg.tool_calls.size());
if (unopened_closed_tool_msg.tool_calls.size() == 1) {
t.assert_equal("unopened closed tool name", std::string("bash"), unopened_closed_tool_msg.tool_calls[0].name);
t.assert_equal(
"unopened closed tool args",
json({{"command", "cat /home/leaf/.pi/agent/agents/kickass.md 2>&1"}}).dump(),
unopened_closed_tool_msg.tool_calls[0].arguments);
}
const std::string migrated_prefix =
"Chain 6 started. JakASS is reading the prompt. Now sleeping 1 hour:";
auto migrated_content_msg = common_chat_parse(migrated_prefix, /* is_partial = */ true, thinking_off_parser_params);
t.assert_equal("migrated prefix parser treats as reasoning", migrated_prefix, migrated_content_msg.reasoning_content);
t.assert_equal("migrated prefix parser has no visible content", std::string(), migrated_content_msg.content);
auto migrated_reasoning_msg = common_chat_parse(migrated_prefix + "</mm:think>",
/* is_partial = */ true,
thinking_off_parser_params);
t.assert_equal("migrated prefix later reasoning", migrated_prefix, migrated_reasoning_msg.reasoning_content);
t.assert_equal("migrated prefix later not visible", std::string(), migrated_reasoning_msg.content);
common_chat_msg streamed_as_content;
streamed_as_content.role = "assistant";
streamed_as_content.content = migrated_prefix;
auto migrated_diffs = common_chat_msg_diff::compute_diffs(streamed_as_content, migrated_reasoning_msg);
t.assert_equal("migrated prefix emits no duplicate diff", 0u, migrated_diffs.size());
}
// End-to-end coverage for the dedicated Cohere2MoE (North Code) parser:
// template apply -> PEG parse -> assert message. Exercises the reasoning-mode
// matrix, including the unopened-thinking-under---reasoning-off case (#1968
tests/test-chat-template.cpp
+1 -1
View File
@@ -294,7 +294,7 @@ static void test_minimax_m3_native_tool_parser(void) {
common_chat_templates_ptr tmpls(common_chat_templates_init(/* model= */ nullptr, template_str));
auto params = common_chat_templates_apply(tmpls.get(), inputs);
assert(params.format == COMMON_CHAT_FORMAT_PEG_NATIVE);
assert(params.format == COMMON_CHAT_FORMAT_PEG_MINIMAX_M3);
assert(!params.parser.empty());
common_peg_arena arena;