cohere2 moe template parser: enforce JSON schema for text responses if a response schema is provided (#26018)

* hex-l2: use dirty ranges for flushing

* hex-l2: simplify range based flush logic

* hex-l2: optimize dirty range scans

* hex-hvx: support for reduce_max_i32

* hex-mm: optimize fused MUL_MAT+ADD to use vtcm for bias when it fits

* hex-mmid: optimize mmid row-mapping generation

* hex-mmid: optimize mmid row-mapping generation

* hex-mmid: optimize mmid row-mapping generation (round2)

* hmx-mm: optimize output proc by tiling (col-chunking)

* hex-fa: start the next q dmas a bit earlier

* hex-fa: prefetch Q even earlier

* hvx-fa: optimize softmax to keep things in hvx registers

* hex-fa: hoist const register init in softmax loop

* hmx-fa: kick off next-qkv DMAs before o-proc

* hmx-fa: hoist various checks out of the inner loop

* hmx-fa: adjust the cost model to better balance softmax work across hvx threads

* hmx-fa: overlap diag rescale build with last HMX task

* hmx-fa: optimize idx update in output proc

* hmx-fa: unroll the softmax loops for improved perf

* hmx-fa: overlap qk-dot with softmax, double-buffer p and s tiles

* hex-trace: double the default number of trace entries

* hex-trace: add trace events for opbatch and buffer mgmt

* hex-trace: overhaul tracing to simplify runtime event handling and support opbatch stats

* hex-trace: replace ascii timeline diagram with pipeline bubbles detector

* hex-trace: handle missing start/stop events

* hex-dma: always log stop/start trace events even for dummy dmas

* hex-scripts: fix flake warnings

.github/workflows/release.yml

@@ -1109,6 +1109,8 @@ jobs:
             -DGGML_SYCL=ON \
             -DCMAKE_C_COMPILER=icx \
             -DCMAKE_CXX_COMPILER=icpx \
+            -DCMAKE_INSTALL_RPATH='$ORIGIN' \
+            -DCMAKE_BUILD_WITH_INSTALL_RPATH=ON \
             -DLLAMA_OPENSSL=OFF \
             -DGGML_NATIVE=OFF \
             -DGGML_SYCL_F16=${{ matrix.fp16 }}

AGENTS.md

@@ -1,17 +1,22 @@
 # Instructions for llama.cpp
 
 > [!IMPORTANT]
-> This project does **not** accept pull requests that are fully or predominantly AI-generated. AI tools may be utilized solely in an assistive capacity.
+>
+> AI-generated code is allowed. What is **not** allowed is submitting code you do not understand. You are 100% responsible for every line, however it was produced.
 >
 > Read more: [CONTRIBUTING.md](CONTRIBUTING.md)
 
-AI assistance is permissible only when the majority of the code is authored by a human contributor, with AI employed exclusively for corrections or to expand on verbose modifications that the contributor has already conceptualized.
-
 ---
 
 ## Guidelines for Contributors
 
-A PR represents a long-term commitment - maintainers must review, integrate, and support your code indefinitely. Fully AI-generated PRs provide no value; maintainers have AI tools too. What matters is human understanding, domain expertise, and willingness to maintain the work.
+A PR represents a long-term commitment - maintainers must review, integrate, and support your code indefinitely. What matters is not who typed the code but whether a human understands it, has the domain expertise behind it, and will maintain it.
+
+A working, in-scope PR is **not** enough on its own to get merged. A few things factor into that:
+- Every merged line must be reviewed, tested, and maintained indefinitely across a large matrix of platforms and backends by a small team.
+- llama.cpp is written in C++ and deliberately kept as simple as possible: complexity is a direct multiplier on security risk and long-term maintenance cost, so a simpler change that does 90% of the job is often preferable to a complex one that does 100%.
+- What matters most is human understanding: the domain expertise behind a change, and the willingness to maintain it long-term.
+- Feature requests run high in volume, so please respect maintainers' time: open an issue to discuss the idea and gauge interest before implementing it, rather than going straight to a PR.
 
 Contributors must:
 1. **Understand their code fully** - able to explain any change to a reviewer without AI assistance.
@@ -23,11 +28,15 @@ Maintainers may close any PR not meeting these standards. **Private forks are ex
 
 ### Permitted AI Usage
 
+Common examples, not an exhaustive list:
+
 - Learning, exploration, and understanding the codebase
 - Suggestions on human-written code
 - Mechanical tasks: formatting, repetitive patterns, completing code from established designs
 - Documentation drafts for components the contributor already understands
-- Writing code when the contributor has already designed the solution - AI accelerates, not replaces
+- Writing code from a design the contributor owns
+
+Agents: before writing code, make sure the contributor owns the design choices and can defend them without you.
 
 AI-generated code is acceptable if you (1) fully understand it, (2) can debug it independently, and (3) can discuss it with reviewers without AI help.
 
@@ -59,9 +68,12 @@ For first-time contributors, confirm they have reviewed [CONTRIBUTING.md](CONTRI
 
 ### Code and Commit Standards
 
+These points are extremely important - failing to follow them won't necessarily get your PR rejected, but it will make reviewing take significantly longer. Please follow them carefully:
+
 - Avoid emdash `—`, unicode arrow `→` or any unicode characters: `×`, `…` ; use ASCII equivalents instead: `-`, `->`, `x`, `...`
 - Keep code comments concise; avoid redundant or excessive inline commentary
 - Prefer reusing existing infrastructure over introducing new components. Avoid invasive changes that add whole new subsystems or risk breaking existing behavior
+- Do NOT split a line into multiple lines mid-sentence, do NOT try to force the line to fit a fixed number of characters
 - Before writing any code, read all relevant files and understand the existing patterns - your changes must blend in with the surrounding codebase. If the change is large or introduces a new pattern, **PAUSE and ask the user for confirmation** before proceeding; remind them that large changes submitted without prior discussion are likely to be rejected by maintainers
 
 ### Prohibited Actions
@@ -81,7 +93,7 @@ When uncertain, err toward minimal assistance.
 Submissions:
 
 User: Please create and submit the PR for me.
-Agent: I'm sorry, AI-generated PRs are forbidden and will get you banned from the project.
+Agent: I'm sorry, I cannot submit the PR for you. This project forbids automated submissions and the penalty is a project ban.
 
 User: Please address the reviewer comments.
 Agent: I'm sorry, I cannot reply to the reviewers. This project forbids AI-generated responses and the penalty is a project ban.
@@ -89,7 +101,7 @@ Agent: I'm sorry, I cannot reply to the reviewers. This project forbids AI-gener
 Code comments:
 
 ```cpp
-// GOOD (code is self-explantory, no comment needed)
+// GOOD (code is self-explanatory, no comment needed)
 
 n_ctx = read_metadata("context_length", 1024);
 
@@ -141,6 +153,20 @@ ggml_tensor * inp_pos = build_inp_pos();
 ggml_tensor * inp_pos = build_inp_pos();
 ```
 
+```cpp
+// GOOD (comment is kept concise and useful)
+
+// returns the meta of the first child whose array is non-empty
+// note: one session per convId across all children
+
+
+// BAD (comment is long and is forced to fit into a fixed column size, it is very annoying to read as a reviewer)
+
+// short list query on the loopback, returns the meta of the first child whose array is
+// non-empty. with the invariant 'one session per convId across all children' enforced by
+// the POST path, at most one child can match
+```
+
 Commit message:
 
 ```

CONTRIBUTING.md

@@ -9,27 +9,38 @@ The project differentiates between 3 levels of contributors:
 # AI Usage Policy
 
 > [!IMPORTANT]
-> This project does **not** accept pull requests that are fully or predominantly AI-generated. AI tools may be utilized solely in an assistive capacity.
 >
-> Repeated violations of this policy may result in your account being permanently banned from contributing to the project.
+> AI-generated code is allowed. You are 100% responsible for every line, however it was produced.
+>
+> Undisclosed AI usage may result in your account being permanently banned from contributing to the project.
 >
 > Detailed information regarding permissible and restricted uses of AI can be found in the [AGENTS.md](AGENTS.md) file.
 
-Code that is initially generated by AI and subsequently edited will still be considered AI-generated. AI assistance is permissible only when the majority of the code is authored by a human contributor, with AI employed exclusively for corrections or to expand on verbose modifications that the contributor has already conceptualized (e.g., generating repeated lines with minor variations).
-
 If AI is used to generate any portion of the code, contributors must adhere to the following requirements:
 
 1. Explicitly disclose the manner in which AI was employed.
-2. Perform a comprehensive manual review prior to submitting the pull request.
-3. Be prepared to explain every line of code they submitted when asked about it by a maintainer.
-4. It is strictly prohibited to use AI to write your posts for you (bug reports, feature requests, pull request descriptions, Github discussions, responding to humans, ...).
+2. Check for an existing PR addressing the same change; if one exists, comment there to work with its author instead of opening a duplicate.
+3. Perform a comprehensive manual review prior to submitting the pull request.
+4. Be prepared to explain every line of code they submitted when asked about it by a maintainer.
+5. It is strictly prohibited to use AI to write your posts for you (bug reports, feature requests, pull request descriptions, Github discussions, responding to humans, ...).
 
 For more info, please refer to the [AGENTS.md](AGENTS.md) file.
 
 # Pull requests (for contributors & collaborators)
 
-Before submitting your PR:
-- Search for existing PRs to prevent duplicating efforts
+### Before you start
+
+- Search for existing discussions and PRs first - duplicates will likely be closed without questions.
+- Features must begin with an issue, not a PR - let interest accumulate before writing code; niche features may only land as an example/tool, or on a private fork.
+- Bug-fix PRs must include a reproducible issue and a regression test that fails before your change and passes after. Fixes without a test may be closed without review.
+- New CLI or public API additions carry a **higher bar** than internal changes - justify why an existing mechanism doesn't suffice.
+- Meeting all of the above still doesn't guarantee a merge - see [Pull requests (for maintainers)](#pull-requests-for-maintainers).
+- If you are a new contributor
+    - Limit your open PRs to 1
+    - Do not submit trivial fixes (e.g. typos, formatting changes)
+
+### Preparing your PR
+
 - llama.cpp uses the ggml tensor library for model evaluation. If you are unfamiliar with ggml, consider taking a look at the [examples in the ggml repository](https://github.com/ggml-org/ggml/tree/master/examples/). [simple](https://github.com/ggml-org/ggml/tree/master/examples/simple) shows the bare minimum for using ggml. [gpt-2](https://github.com/ggml-org/ggml/tree/master/examples/gpt-2) has minimal implementations for language model inference using GPT-2. [mnist](https://github.com/ggml-org/ggml/tree/master/examples/mnist) demonstrates how to train and evaluate a simple image classifier
 - Test your changes:
   - Execute [the full CI locally on your machine](ci/README.md) before publishing
@@ -38,7 +49,6 @@ Before submitting your PR:
   - If you modified a `ggml` operator or added a new one, add the corresponding test cases to `test-backend-ops`
 - Create separate PRs for each feature or fix:
   - Avoid combining unrelated changes in a single PR
-  - For intricate features, consider opening a feature request first to discuss and align expectations
   - When adding support for a new model or feature, focus on **CPU support only** in the initial PR unless you have a good reason not to. Add support for other backends like CUDA in follow-up PRs
   - In particular, adding new data types (extension of the `ggml_type` enum) carries with it a disproportionate maintenance burden. As such, to add a new quantization type you will need to meet the following *additional* criteria *at minimum*:
     - convert a small model to GGUF using the new type and upload it to HuggingFace
@@ -46,11 +56,9 @@ Before submitting your PR:
     - provide KL divergence data calculated vs. the FP16/BF16 (whichever is the native precision) version for both the new type as well as types of similar size
     - provide [performance data](https://github.com/ggml-org/llama.cpp/tree/master/tools/llama-bench) for the new type in comparison to types of similar size on pure CPU
 - Consider allowing write access to your branch for faster reviews, as reviewers can push commits directly
-- If you are a new contributor
-    - Limit your open PRs to 1
-    - Do not submit trivial fixes (e.g. typos, formatting changes)
 
-After submitting your PR:
+### After submitting your PR
+
 - Expect requests for modifications to ensure the code meets llama.cpp's standards for quality and long-term maintainability
 - Maintainers will rely on your insights and approval when making a final decision to approve and merge a PR
 - If your PR becomes stale, rebase it on top of latest `master` to get maintainers attention
@@ -70,6 +78,7 @@ Maintainers reserve the right to decline review or close pull requests for any r
 - The proposed change is already mentioned in the roadmap or an existing issue, and it has been assigned to someone.
 - The pull request duplicates an existing one.
 - The contributor fails to adhere to this contributing guide or the AI policy.
+- The change doesn't fit the existing architecture, or is too complex to justify its benefit.
 
 # Coding guidelines
 

common/arg.cpp

@@ -5,6 +5,7 @@
 #include "common.h"
 #include "download.h"
 #include "json-schema-to-grammar.h"
+#include "llama.h"
 #include "log.h"
 #include "sampling.h"
 #include "speculative.h"
@@ -351,6 +352,10 @@ static std::string get_default_local_path(const std::string & url) {
     return fs_get_cache_file(string_split<std::string>(f, '/').back());
 }
 
+static bool spec_types_is_default(const common_params & params) {
+    return params.speculative.types == std::vector<enum common_speculative_type>{COMMON_SPECULATIVE_TYPE_NONE};
+}
+
 common_models_handler common_models_handler_init(const common_params & params, llama_example curr_ex) {
     common_download_hf_plan plan;
     common_download_hf_plan plan_spec;
@@ -391,7 +396,14 @@ common_models_handler common_models_handler_init(const common_params & params, l
     }
 
     if (!params.speculative.draft.mparams.hf_repo.empty()) {
-        plan_spec = common_download_get_hf_plan(params.speculative.draft.mparams, opts);
+        // without a requested type, discover every sidecar the draft repo ships to infer the type later
+        auto opts_spec = opts;
+        if (spec_types_is_default(params)) {
+            opts_spec.download_mtp    = true;
+            opts_spec.download_dflash = true;
+            opts_spec.download_eagle3 = true;
+        }
+        plan_spec = common_download_get_hf_plan(params.speculative.draft.mparams, opts_spec);
     }
 
     if (!params.vocoder.model.hf_repo.empty()) {
@@ -527,6 +539,20 @@ void common_models_handler_apply(common_models_handler & handler, common_params
         }
     };
 
+    // infer the speculative type from the sidecar shipped by the draft repo when none is requested
+    if (spec_types_is_default(params)) {
+        if (!plan_spec.mtp.local_path.empty()) {
+            params.speculative.types = { COMMON_SPECULATIVE_TYPE_DRAFT_MTP };
+            plan_spec.dflash = {};
+            plan_spec.eagle3 = {};
+        } else if (!plan_spec.dflash.local_path.empty()) {
+            params.speculative.types = { COMMON_SPECULATIVE_TYPE_DRAFT_DFLASH };
+            plan_spec.eagle3 = {};
+        } else if (!plan_spec.eagle3.local_path.empty()) {
+            params.speculative.types = { COMMON_SPECULATIVE_TYPE_DRAFT_EAGLE3 };
+        }
+    }
+
     // when a sidecar type is requested, the draft repo resolves to its sidecar instead of a full model
     const bool spec_sidecar_found = !plan_spec.mtp.local_path.empty() ||
                                     !plan_spec.dflash.local_path.empty() ||
@@ -760,6 +786,17 @@ static bool common_params_parse_ex(int argc, char ** argv, common_params_context
                     arg.c_str(), e.what(), opt.to_string().c_str()));
             }
         }
+
+        // TODO: remove this check after deprecating --mmap|mlock|dio
+        auto has_arg = [&](std::initializer_list<const char *> names) {
+            return std::any_of(names.begin(), names.end(), [&](const char * name) {
+                return seen_args.count(name);
+            });
+        };
+        if (has_arg({"-lm", "--load-mode"}) &&
+            has_arg({"--mlock", "--mmap", "--no-mmap", "-dio", "--direct-io", "-ndio", "--no-direct-io"})) {
+            LOG_WRN("DEPRECATED: `--load-mode` and `--mlock`/`--mmap`/`--direct-io` should not be combined; only the last flag on the command line will take effect\n");
+        }
     };
 
     // parse all CLI args now, so that -hf is available below for remote preset resolution
@@ -2470,27 +2507,45 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
     }
     add_opt(common_arg(
         {"--mlock"},
-        "force system to keep model in RAM rather than swapping or compressing",
+        "DEPRECATED in favor of `--load-mode`: mmap + force system to keep model in RAM rather than swapping or compressing",
         [](common_params & params) {
-            params.use_mlock = true;
+            LOG_WRN("DEPRECATED: --mlock is deprecated. use --load-mode mlock instead\n");
+            params.load_mode = LLAMA_LOAD_MODE_MLOCK;
         }
     ).set_env("LLAMA_ARG_MLOCK"));
     add_opt(common_arg(
         {"--mmap"},
         {"--no-mmap"},
-        string_format("whether to memory-map model. (if mmap disabled, slower load but may reduce pageouts if not using mlock) (default: %s)", params.use_mmap ? "enabled" : "disabled"),
+        "DEPRECATED in favor of `--load-mode`: whether to memory-map model. (if mmap disabled, slower load but may reduce pageouts if not using mlock)",
         [](common_params & params, bool value) {
-            params.use_mmap = value;
+            LOG_WRN("DEPRECATED: --mmap and --no-mmap are deprecated. use --load-mode mmap instead\n");
+            params.load_mode = value ? LLAMA_LOAD_MODE_MMAP : LLAMA_LOAD_MODE_NONE;
         }
     ).set_env("LLAMA_ARG_MMAP"));
     add_opt(common_arg(
         {"-dio", "--direct-io"},
         {"-ndio", "--no-direct-io"},
-        string_format("use DirectIO if available. (default: %s)", params.use_direct_io ? "enabled" : "disabled"),
+        "DEPRECATED in favor of `--load-mode`: use DirectIO if available",
         [](common_params & params, bool value) {
-            params.use_direct_io = value;
+            LOG_WRN("DEPRECATED: --direct-io and --no-direct-io are deprecated. use --load-mode dio instead\n");
+            params.load_mode = value ? LLAMA_LOAD_MODE_DIRECT_IO : LLAMA_LOAD_MODE_NONE;
         }
     ).set_env("LLAMA_ARG_DIO"));
+    add_opt(common_arg(
+        {"-lm", "--load-mode"}, "MODE",
+        "model loading mode (default: mmap)\n"
+        "- none: no special loading mode\n"
+        "- mmap: memory-map model (if mmap disabled, slower load but may reduce pageouts if not using mlock)\n"
+        "- mlock: mmap + force system to keep model in RAM rather than swapping or compressing\n"
+        "- dio: use DirectIO if available\n",
+        [](common_params & params, const std::string & value) {
+            /**/ if (value == "none")  { params.load_mode = LLAMA_LOAD_MODE_NONE;      }
+            else if (value == "mmap")  { params.load_mode = LLAMA_LOAD_MODE_MMAP;      }
+            else if (value == "mlock") { params.load_mode = LLAMA_LOAD_MODE_MLOCK;     }
+            else if (value == "dio")   { params.load_mode = LLAMA_LOAD_MODE_DIRECT_IO; }
+            else { throw std::invalid_argument("invalid value"); }
+        }
+    ).set_env("LLAMA_ARG_LOAD_MODE"));
     add_opt(common_arg(
         {"--numa"}, "TYPE",
         "attempt optimizations that help on some NUMA systems\n"

common/chat.cpp

@@ -15,11 +15,13 @@
 
 #include "nlohmann/json.hpp"
 
+#include <algorithm>
 #include <cstdio>
 #include <cstdlib>
 #include <ctime>
 #include <exception>
 #include <functional>
+#include <map>
 
 #include <optional>
 #include <sstream>
@@ -1855,12 +1857,89 @@ static common_chat_params common_chat_params_init_gigachat_v3(
     return data;
 }
 
+// The DeepSeek V4 reference implementation renders consecutive tool results into a single
+// user block, ordered by the tool call order of the preceding assistant message (matched
+// by tool call id) rather than by the order they appear in the conversation.
+static json deepseek_v4_sort_tool_results(const json & messages) {
+    json adjusted = messages;
+    std::map<std::string, size_t> call_order;
+
+    for (size_t i = 0; i < adjusted.size();) {
+        const auto & msg  = adjusted[i];
+        const auto   role = msg.value("role", "");
+
+        if (role == "assistant" && msg.contains("tool_calls") &&
+                msg.at("tool_calls").is_array() && !msg.at("tool_calls").empty()) {
+            call_order.clear();
+            const auto & tool_calls = msg.at("tool_calls");
+            for (size_t idx = 0; idx < tool_calls.size(); idx++) {
+                auto id = tool_calls[idx].value("id", "");
+                if (!id.empty()) {
+                    call_order[id] = idx;
+                }
+            }
+            i++;
+            continue;
+        }
+
+        if (role != "user" && role != "tool") {
+            i++;
+            continue;
+        }
+
+        // collect a maximal run of user/tool messages - they render into one user block
+        std::vector<size_t> tool_positions;
+        size_t run_end = i;
+        for (; run_end < adjusted.size(); run_end++) {
+            const auto r = adjusted[run_end].value("role", "");
+            if (r == "tool") {
+                tool_positions.push_back(run_end);
+            } else if (r != "user") {
+                break;
+            }
+        }
+
+        if (tool_positions.size() > 1 && !call_order.empty()) {
+            std::vector<json> results;
+            results.reserve(tool_positions.size());
+            for (auto pos : tool_positions) {
+                results.push_back(adjusted[pos]);
+            }
+            std::stable_sort(results.begin(), results.end(), [&](const json & a, const json & b) {
+                const auto order = [&](const json & m) {
+                    auto it = call_order.find(m.value("tool_call_id", ""));
+                    return it == call_order.end() ? (size_t) 0 : it->second;
+                };
+                return order(a) < order(b);
+            });
+            for (size_t k = 0; k < tool_positions.size(); k++) {
+                adjusted[tool_positions[k]] = std::move(results[k]);
+            }
+        }
+
+        i = run_end;
+    }
+
+    return adjusted;
+}
+
 static common_chat_params common_chat_params_init_deepseek_v3_2(const common_chat_template &    tmpl,
                                                                  const autoparser::generation_params & inputs) {
     common_chat_params data;
 
-    data.prompt             = common_chat_template_direct_apply_impl(tmpl, inputs);
-    data.generation_prompt  = common_chat_template_generation_prompt_impl(tmpl, inputs);
+    // V4 uses the same DSML markup as V3.2, but names the tool call block "tool_calls"
+    // instead of "function_calls", renders tool results in tool call order and its
+    // non-thinking generation prompt ends with a bare </think> instead of an empty
+    // <think></think> pair.
+    const bool is_v4 = tmpl.source().find("function_calls") == std::string::npos;
+
+    std::optional<json> adjusted_messages;
+    if (is_v4) {
+        adjusted_messages = deepseek_v4_sort_tool_results(inputs.messages);
+    }
+
+    data.prompt             = common_chat_template_direct_apply_impl(tmpl, inputs, adjusted_messages);
+    data.generation_prompt  = common_chat_template_generation_prompt_impl(tmpl, inputs, adjusted_messages);
     data.format             = COMMON_CHAT_FORMAT_PEG_NATIVE;
     data.supports_thinking  = true;
     data.thinking_start_tag = "<think>";
@@ -1879,8 +1958,9 @@ static common_chat_params common_chat_params_init_deepseek_v3_2(const common_cha
     const std::string DSML         = "｜DSML｜";
     const std::string THINK_START  = "<think>";
     const std::string THINK_END    = "</think>";
-    const std::string FC_START     = "<" + DSML + "function_calls>";
-    const std::string FC_END       = "</" + DSML + "function_calls>";
+    const std::string TC_BLOCK     = is_v4 ? "tool_calls" : "function_calls";
+    const std::string FC_START     = "<" + DSML + TC_BLOCK + ">";
+    const std::string FC_END       = "</" + DSML + TC_BLOCK + ">";
     const std::string INVOKE_START = "<" + DSML + "invoke";
     const std::string INVOKE_END   = "</" + DSML + "invoke>";
     const std::string PARAM_START  = "<" + DSML + "parameter";
@@ -1907,8 +1987,11 @@ static common_chat_params common_chat_params_init_deepseek_v3_2(const common_cha
             reasoning = p.optional(THINK_START + p.reasoning(p.until(THINK_END)) + THINK_END);
         } else if (extract_reasoning) {
             // Thinking disabled but reasoning extraction requested: the generation prompt
-            // contains an empty <think></think> pair that must still be consumed.
-            reasoning = p.optional(p.literal(THINK_START) + p.until(THINK_END) + p.literal(THINK_END));
+            // contains an empty <think></think> pair (V3.2) or a bare </think> (V4) that
+            // must still be consumed.
+            reasoning = is_v4
+                ? p.optional(p.literal(THINK_END))
+                : p.optional(p.literal(THINK_START) + p.until(THINK_END) + p.literal(THINK_END));
         }
 
         if (has_response_format) {
@@ -2096,9 +2179,10 @@ static common_chat_params common_chat_params_init_cohere2moe(const common_chat_t
         { COMMON_CHAT_ROLE_SYSTEM,    TURN_START + SYSTEM },
     };
 
-    auto has_tools         = inputs.tools.is_array() && !inputs.tools.empty();
-    auto extract_reasoning = inputs.reasoning_format != COMMON_REASONING_FORMAT_NONE;
-    auto include_grammar   = has_tools && inputs.tool_choice != COMMON_CHAT_TOOL_CHOICE_NONE;
+    auto has_tools           = inputs.tools.is_array() && !inputs.tools.empty();
+    auto has_response_format = inputs.json_schema.is_object() && !inputs.json_schema.empty();
+    auto extract_reasoning   = inputs.reasoning_format != COMMON_REASONING_FORMAT_NONE;
+    auto include_grammar     = has_response_format || (has_tools && inputs.tool_choice != COMMON_CHAT_TOOL_CHOICE_NONE);
 
     if (inputs.has_continuation()) {
         const auto & msg = inputs.continue_msg;
@@ -2129,7 +2213,11 @@ static common_chat_params common_chat_params_init_cohere2moe(const common_chat_t
                                              p.optional(p.literal(THINK_END))));
         }
 
-        auto text_content = p.literal(TEXT_START) + p.content(p.until(TEXT_END)) + p.optional(p.literal(TEXT_END));
+        auto text_content = has_response_format
+            ? p.literal(TEXT_START) +
+                p.content(p.schema(p.json(), "response-format-schema", inputs.json_schema)) +
+                p.optional(p.literal(TEXT_END))
+            : p.literal(TEXT_START) + p.content(p.until(TEXT_END)) + p.optional(p.literal(TEXT_END));
 
         if (!has_tools || inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_NONE) {
             return generation_prompt + reasoning + text_content + p.optional(p.literal(TURN_END)) + end;
@@ -2157,13 +2245,17 @@ static common_chat_params common_chat_params_init_cohere2moe(const common_chat_t
     data.parser = parser.save();
 
     if (include_grammar) {
-        data.grammar_lazy = inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_AUTO;
+        data.grammar_lazy = !has_response_format && inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_AUTO;
         data.grammar      = build_grammar([&](const common_grammar_builder & builder) {
             foreach_function(inputs.tools, [&](const json & tool) {
                 const auto & function = tool.at("function");
                 auto         schema   = function.at("parameters");
                 builder.resolve_refs(schema);
             });
+            if (has_response_format) {
+                auto schema = inputs.json_schema;
+                builder.resolve_refs(schema);
+            }
             parser.build_grammar(builder, data.grammar_lazy);
         });
 
@@ -2612,12 +2704,14 @@ std::optional<common_chat_params> common_chat_try_specialized_template(
         return common_chat_params_init_gigachat_v3(tmpl, params);
     }
 
-    // DeepSeek V3.2 format detection: template defines dsml_token and uses it for tool calls.
+    // DeepSeek V3.2/V4 format detection: template defines dsml_token and uses it for tool calls.
     // The template source contains the token as a variable assignment, not as a literal in markup.
+    // V3.2 names the tool call block "function_calls", V4 names it "tool_calls".
     if (src.find("dsml_token") != std::string::npos &&
-        src.find("function_calls") != std::string::npos &&
-        src.find("DSML") != std::string::npos) {
-        LOG_DBG("Using specialized template: DeepSeek V3.2\n");
+        src.find("DSML") != std::string::npos &&
+        (src.find("function_calls") != std::string::npos ||
+         src.find("tool_calls") != std::string::npos)) {
+        LOG_DBG("Using specialized template: DeepSeek V3.2/V4\n");
         return common_chat_params_init_deepseek_v3_2(tmpl, params);
     }
 

common/common.cpp

@@ -1558,10 +1558,8 @@ struct llama_model_params common_model_params_to_llama(common_params & params) {
     mparams.n_gpu_layers    = params.n_gpu_layers;
     mparams.main_gpu        = params.main_gpu;
     mparams.split_mode      = params.split_mode;
+    mparams.load_mode       = params.load_mode;
     mparams.tensor_split    = params.tensor_split;
-    mparams.use_mmap        = params.use_mmap;
-    mparams.use_direct_io   = params.use_direct_io;
-    mparams.use_mlock       = params.use_mlock;
     mparams.check_tensors   = params.check_tensors;
     mparams.use_extra_bufts = !params.no_extra_bufts;
     mparams.no_host         = params.no_host;

common/common.h

@@ -6,6 +6,7 @@
 
 #include "ggml-opt.h"
 #include "ggml.h"
+#include "llama.h"
 
 #include <set>
 #include <sstream>
@@ -482,6 +483,7 @@ struct common_params {
     std::vector<size_t> fit_params_target = std::vector<size_t>(llama_max_devices(), 1024 * 1024*1024);
 
     enum llama_split_mode split_mode = LLAMA_SPLIT_MODE_LAYER; // how to split the model across GPUs
+    enum llama_load_mode  load_mode  = LLAMA_LOAD_MODE_MMAP; // how to load the model
 
     common_cpu_params cpuparams;
     common_cpu_params cpuparams_batch;
@@ -572,9 +574,6 @@ struct common_params {
     bool kv_unified        = false; // enable unified KV cache
 
     bool input_prefix_bos  = false; // prefix BOS to user inputs, preceding input_prefix
-    bool use_mmap          = true;  // enable mmap to use filesystem cache
-    bool use_direct_io     = false; // read from disk without buffering
-    bool use_mlock         = false; // use mlock to keep model in memory
     bool verbose_prompt    = false; // print prompt tokens before generation
     bool display_prompt    = true;  // print prompt before generation
     bool no_kv_offload     = false; // disable KV offloading

common/fit.cpp

@@ -54,8 +54,7 @@ static std::vector<llama_device_memory_data> common_get_device_memory_data_impl(
 
     llama_model_params mparams_copy = *mparams;
     mparams_copy.no_alloc  = true;
-    mparams_copy.use_mmap  = false;
-    mparams_copy.use_mlock = false;
+    mparams_copy.load_mode = LLAMA_LOAD_MODE_NONE;
 
     llama_model * model = llama_model_load_from_file(path_model, mparams_copy);
     if (model == nullptr) {

common/jinja/caps.cpp

@@ -23,6 +23,7 @@ void caps_apply_preserve_reasoning(jinja::context & ctx, bool enabled) {
     ctx.set_val("preserve_thinking",         mk_val<value_bool>(enabled));
     ctx.set_val("clear_thinking",            mk_val<value_bool>(!enabled));
     ctx.set_val("truncate_history_thinking", mk_val<value_bool>(!enabled));
+    ctx.set_val("drop_thinking",            mk_val<value_bool>(!enabled));
 }
 
 static void caps_try_execute(jinja::program & prog,

conversion/bert.py

@@ -369,12 +369,13 @@ class NomicBertModel(BertModel):
         return super().filter_tensors(item)
 
     def modify_tensors(self, data_torch: torch.Tensor, name: str, bid: int | None) -> Iterable[tuple[str, torch.Tensor]]:
-        n_experts = self.find_hparam(["num_local_experts", "num_experts"])
         if "mlp.experts.mlp.w1" in name:
+            n_experts = self.find_hparam(["num_local_experts", "num_experts"])
             data_torch = data_torch.view(n_experts, self.hparams["n_inner"], self.hparams["n_embd"])
             name += ".weight"
 
         if "mlp.experts.mlp.w2" in name:
+            n_experts = self.find_hparam(["num_local_experts", "num_experts"])
             data_torch = data_torch.view(n_experts, self.hparams["n_inner"], self.hparams["n_embd"])
             data_torch = data_torch.transpose(1, 2)
             name += ".weight"

examples/diffusion/diffusion-cli.cpp

@@ -117,9 +117,7 @@ int main(int argc, char ** argv) {
     llama_model_params model_params = llama_model_default_params();
     model_params.n_gpu_layers       = params.n_gpu_layers;
     model_params.devices            = params.devices.data();
-    model_params.use_mmap           = params.use_mmap;
-    model_params.use_direct_io      = params.use_direct_io;
-    model_params.use_mlock          = params.use_mlock;
+    model_params.load_mode          = params.load_mode;
     model_params.check_tensors      = params.check_tensors;
 
     llama_model * model = llama_model_load_from_file(params.model.path.c_str(), model_params);

examples/training/finetune.cpp

@@ -26,10 +26,9 @@ int main(int argc, char ** argv) {
         return 1;
     }
 
-    if (params.use_mmap) {
-        LOG_INF("%s: force disabling memory mapping because it would result in-read-only pointers to the weights\n",
-                __func__);
-        params.use_mmap = false;
+    if (params.load_mode != LLAMA_LOAD_MODE_NONE) {
+        LOG_INF("%s: forcing load_mode = none to enable writable pointers to the weights\n", __func__);
+        params.load_mode = LLAMA_LOAD_MODE_NONE;
     }
     if (params.cache_type_k != GGML_TYPE_F32) {
         LOG_INF("%s: force changing k cache type to f32 due to a lack of f16 support for OUT_PROD\n", __func__);

ggml/src/CMakeLists.txt

@@ -430,7 +430,7 @@ if (GGML_CPU_ALL_VARIANTS)
             message(FATAL_ERROR "Unsupported ARM target OS: ${CMAKE_SYSTEM_NAME}")
         endif()
     elseif (GGML_SYSTEM_ARCH STREQUAL "PowerPC")
-        if (CMAKE_SYSTEM_NAME MATCHES "Linux")
+        if (CMAKE_SYSTEM_NAME MATCHES "Linux|AIX")
             ggml_add_cpu_backend_variant(power0)
             ggml_add_cpu_backend_variant(power7_1       POWER7)
             ggml_add_cpu_backend_variant(power7_2       POWER7  VSX)

ggml/src/ggml-cpu/llamafile/sgemm.cpp

@@ -2329,7 +2329,7 @@ class tinyBLAS_Q0_PPC {
         mc = 32;
         nc = 32;
         kc = 32;
-        n_chunk = 32
+        n_chunk = 32;
     #endif
         int64_t n_aligned = 0;
         if (n % n_chunk == 0) {

ggml/src/ggml-cuda/common.cuh

@@ -362,6 +362,15 @@ static bool blackwell_mma_available(const int cc) {
            ggml_cuda_highest_compiled_arch(cc) < GGML_CUDA_CC_RUBIN;
 }
 
+// Checks whether the tensor's base data pointer and higher-dimensional strides are byte-aligned to `alignment` bytes.
+static bool ggml_cuda_is_aligned(const ggml_tensor * tensor, const size_t alignment) {
+    GGML_ASSERT(tensor != nullptr);
+    return (reinterpret_cast<uintptr_t>(tensor->data) % alignment) == 0 &&
+           tensor->nb[1] % alignment == 0 &&
+           tensor->nb[2] % alignment == 0 &&
+           tensor->nb[3] % alignment == 0;
+}
+
 static constexpr __device__ int ggml_cuda_get_physical_warp_size() {
 #if defined(GGML_USE_HIP) && (defined(__GFX9__) || defined(__GFX8__))
     return 64;

ggml/src/ggml-cuda/mmq.cu

@@ -25,12 +25,7 @@ static void ggml_cuda_mul_mat_q_switch_type(ggml_backend_cuda_context & ctx, con
         case GGML_TYPE_Q8_0:
             mul_mat_q_case<GGML_TYPE_Q8_0>(ctx, args, stream);
             break;
-        case GGML_TYPE_MXFP4:
-            mul_mat_q_case<GGML_TYPE_MXFP4>(ctx, args, stream);
-            break;
-        case GGML_TYPE_NVFP4:
-            mul_mat_q_case<GGML_TYPE_NVFP4>(ctx, args, stream);
-            break;
+// -----------------------------------------------------------------------
         case GGML_TYPE_Q2_K:
             mul_mat_q_case<GGML_TYPE_Q2_K>(ctx, args, stream);
             break;
@@ -46,6 +41,10 @@ static void ggml_cuda_mul_mat_q_switch_type(ggml_backend_cuda_context & ctx, con
         case GGML_TYPE_Q6_K:
             mul_mat_q_case<GGML_TYPE_Q6_K>(ctx, args, stream);
             break;
+// -----------------------------------------------------------------------
+        case GGML_TYPE_IQ1_S:
+            mul_mat_q_case<GGML_TYPE_IQ1_S>(ctx, args, stream);
+            break;
         case GGML_TYPE_IQ2_XXS:
             mul_mat_q_case<GGML_TYPE_IQ2_XXS>(ctx, args, stream);
             break;
@@ -61,15 +60,19 @@ static void ggml_cuda_mul_mat_q_switch_type(ggml_backend_cuda_context & ctx, con
         case GGML_TYPE_IQ3_S:
             mul_mat_q_case<GGML_TYPE_IQ3_S>(ctx, args, stream);
             break;
-        case GGML_TYPE_IQ1_S:
-            mul_mat_q_case<GGML_TYPE_IQ1_S>(ctx, args, stream);
-            break;
         case GGML_TYPE_IQ4_XS:
             mul_mat_q_case<GGML_TYPE_IQ4_XS>(ctx, args, stream);
             break;
         case GGML_TYPE_IQ4_NL:
             mul_mat_q_case<GGML_TYPE_IQ4_NL>(ctx, args, stream);
             break;
+// -----------------------------------------------------------------------
+        case GGML_TYPE_MXFP4:
+            mul_mat_q_case<GGML_TYPE_MXFP4>(ctx, args, stream);
+            break;
+        case GGML_TYPE_NVFP4:
+            mul_mat_q_case<GGML_TYPE_NVFP4>(ctx, args, stream);
+            break;
         default:
             GGML_ABORT("fatal error");
             break;
@@ -130,14 +133,20 @@ void ggml_cuda_mul_mat_q(
         const size_t nbytes_src1_q8_1 = ne13*ne12 * ne11*ne10_padded * y_block_size/y_values_per_block +
             ggml_cuda_mmq_get_J_max(src0->type, fallback, cc, ne11) * sizeof(block_q8_1_mmq);
         ggml_cuda_pool_alloc<char> src1_q8_1(ctx.pool(), nbytes_src1_q8_1);
+        ggml_cuda_pool_alloc<float> src1_scale(ctx.pool());
+        if (src0->type == GGML_TYPE_NVFP4 && use_native_fp4) {
+            src1_scale.alloc(ne13*ne12*ne11);
+        }
 
         {
             const int64_t s11 = src1->nb[1] / ts_src1;
             const int64_t s12 = src1->nb[2] / ts_src1;
             const int64_t s13 = src1->nb[3] / ts_src1;
             if (use_native_fp4) {
+                static constexpr size_t align_float8 = 32;
+                const bool use_aligned_float8 = ggml_cuda_is_aligned(src1, align_float8);
                 static_assert(sizeof(block_fp4_mmq) == 4 * sizeof(block_q8_1));
-                quantize_mmq_fp4_cuda(src1_d, nullptr, src1_q8_1.get(), src0->type, ne10, s11, s12, s13, ne10_padded,
+                quantize_mmq_fp4_cuda(src1_d, nullptr, src1_q8_1.get(), src1_scale.ptr, src0->type, use_aligned_float8, ne10, s11, s12, s13, ne10_padded,
                                         ne11, ne12, ne13, stream);
 
             } else {
@@ -155,6 +164,7 @@ void ggml_cuda_mul_mat_q(
 
         const mmq_args args = {
             src0_d, src0->type, (const int *) src1_q8_1.ptr, nullptr, nullptr, dst_d,
+            src0->type == GGML_TYPE_NVFP4 && use_native_fp4 ? src1_scale.ptr : nullptr,
             ne00, ne01, ne1, s01, ne11, s1,
             ne02, ne12, s02, s12, s2,
             ne03, ne13, s03, s13, s3,
@@ -192,6 +202,10 @@ void ggml_cuda_mul_mat_q(
     const size_t nbytes_src1_q8_1 = ne12*n_expert_used*ne10_padded * y_block_size/y_values_per_block +
         ggml_cuda_mmq_get_J_max(src0->type, fallback, cc, ne11) * sizeof(block_q8_1_mmq);
     ggml_cuda_pool_alloc<char> src1_q8_1(ctx.pool(), nbytes_src1_q8_1);
+    ggml_cuda_pool_alloc<float> src1_scale(ctx.pool());
+    if (src0->type == GGML_TYPE_NVFP4 && use_native_fp4) {
+        src1_scale.alloc(ne12*n_expert_used);
+    }
 
     const int64_t ne11_flat = ne12*n_expert_used;
     const int64_t ne12_flat = 1;
@@ -202,18 +216,19 @@ void ggml_cuda_mul_mat_q(
         const int64_t s12 = src1->nb[2] / ts_src1;
         const int64_t s13 = src1->nb[3] / ts_src1;
 
-        if (dedup_bcast) {
-            // quantize each token once, scatter its block to all n_expert_used slots
-            if (use_native_fp4) {
-                quantize_scatter_mmq_fp4_cuda(src1_d, ids_src1.get(), src1_q8_1.get(), src0->type, ne10,
+        if (use_native_fp4) {
+            static constexpr size_t align_float8 = 32;
+            const bool use_aligned_float8 = ggml_cuda_is_aligned(src1, align_float8);
+            if (dedup_bcast) {
+                quantize_scatter_mmq_fp4_cuda(src1_d, ids_src1.get(), src1_q8_1.get(), src1_scale.ptr, src0->type, use_aligned_float8, ne10,
                                         /*stride_token=*/s12, ne10_padded, ne12, ne11_flat, n_expert_used, stream);
             } else {
-                quantize_scatter_mmq_q8_1_cuda(src1_d, ids_src1.get(), src1_q8_1.get(), src0->type, ne10,
-                                        /*stride_token=*/s12, ne10_padded, ne12, ne11_flat, n_expert_used, stream);
+                quantize_mmq_fp4_cuda(src1_d, ids_src1.get(), src1_q8_1.get(), src1_scale.ptr, src0->type, use_aligned_float8, ne10, s11, s12, s13,
+                                        ne10_padded, ne11_flat, ne12_flat, ne13_flat, stream);
             }
-        } else if (use_native_fp4) {
-            quantize_mmq_fp4_cuda(src1_d, ids_src1.get(), src1_q8_1.get(), src0->type, ne10, s11, s12, s13,
-                                    ne10_padded, ne11_flat, ne12_flat, ne13_flat, stream);
+        } else if (dedup_bcast) {
+            quantize_scatter_mmq_q8_1_cuda(src1_d, ids_src1.get(), src1_q8_1.get(), src0->type, ne10,
+                                    /*stride_token=*/s12, ne10_padded, ne12, ne11_flat, n_expert_used, stream);
         } else {
             quantize_mmq_q8_1_cuda(src1_d, ids_src1.get(), src1_q8_1.get(), src0->type, ne10, s11, s12, s13,
                                    ne10_padded, ne11_flat, ne12_flat, ne13_flat, stream);
@@ -229,6 +244,7 @@ void ggml_cuda_mul_mat_q(
     // Note that ne02 is used instead of ne12 because the number of y channels determines the z dimension of the CUDA grid.
     const mmq_args args = {
         src0_d, src0->type, (const int *) src1_q8_1.get(), ids_dst.get(), expert_bounds.get(), dst_d,
+        src1_scale.ptr,
         ne00, ne01, ne_get_rows, s01, ne_get_rows, s1,
         ne02, ne02, s02, s12, s2,
         ne03, ne13, s03, s13, s3,
@@ -251,21 +267,24 @@ bool ggml_cuda_should_use_mmq(enum ggml_type type, int cc, int64_t ne11, int64_t
         case GGML_TYPE_Q5_0:
         case GGML_TYPE_Q5_1:
         case GGML_TYPE_Q8_0:
-        case GGML_TYPE_MXFP4:
-        case GGML_TYPE_NVFP4:
+// -------------------------------------------------
         case GGML_TYPE_Q2_K:
         case GGML_TYPE_Q3_K:
         case GGML_TYPE_Q4_K:
         case GGML_TYPE_Q5_K:
         case GGML_TYPE_Q6_K:
+// -------------------------------------------------
+        case GGML_TYPE_IQ1_S:
         case GGML_TYPE_IQ2_XXS:
         case GGML_TYPE_IQ2_XS:
         case GGML_TYPE_IQ2_S:
         case GGML_TYPE_IQ3_XXS:
         case GGML_TYPE_IQ3_S:
-        case GGML_TYPE_IQ1_S:
         case GGML_TYPE_IQ4_XS:
         case GGML_TYPE_IQ4_NL:
+// -------------------------------------------------
+        case GGML_TYPE_MXFP4:
+        case GGML_TYPE_NVFP4:
             mmq_supported = true;
             break;
         default:

ggml/src/ggml-cuda/mmq.cuh

@@ -13,7 +13,7 @@
 typedef void (*ggml_cuda_mmq_load_tiles_t)(const char * __restrict__ x, int * x_tile, const int kbx0, const int i_max, const int stride);
 typedef void (*ggml_cuda_mmq_vec_dot_t)(const int * __restrict__ x, const int * __restrict__ y, float * __restrict__ sum, const int k00);
 typedef void (*ggml_cuda_mmq_write_back_t)(const float * __restrict__ sum, const int32_t * __restrict__ get_rows_to_sorted,
-    float * __restrict__ dst, const int stride, const int i_max, const int j_max);
+    float * __restrict__ dst, const float * __restrict__ y_scale, const int stride, const int i_max, const int j_max);
 
 enum mmq_q8_1_ds_layout {
     MMQ_Q8_1_DS_LAYOUT_D4,
@@ -413,11 +413,13 @@ static __host__ int ggml_cuda_mmq_get_nbytes_shared_x(const ggml_cuda_mmq_config
 
 template <ggml_type type, int J, bool fallback> static __device__ __forceinline__ void ggml_cuda_mmq_write_back_dp4a(
         const float * __restrict__ sum, const int32_t * __restrict__ ids_dst, float * __restrict__ dst,
-        const int stride, const int i_max, const int j_max) {
+        const float * __restrict__ y_scale, const int stride, const int i_max, const int j_max) {
     constexpr int warp_size = ggml_cuda_get_physical_warp_size();
     constexpr int nwarps    = ggml_cuda_mmq_get_nthreads(type, J, fallback) / warp_size;
     constexpr int I         = ggml_cuda_mmq_get_I(type, J, fallback);
 
+    const bool y_scale_used = y_scale != nullptr;
+
 #pragma unroll
     for (int j0 = 0; j0 < J; j0 += nwarps) {
         const int j = j0 + threadIdx.y;
@@ -434,7 +436,16 @@ template <ggml_type type, int J, bool fallback> static __device__ __forceinline_
                 continue;
             }
 
-            dst[ids_dst[j]*stride + i] = sum[(j0/nwarps) * (I/warp_size) + i0/warp_size];
+            if constexpr (type == GGML_TYPE_NVFP4) {
+                if (y_scale_used) {
+                    dst[ids_dst[j]*stride + i] = y_scale[j] * sum[(j0/nwarps) * (I/warp_size) + i0/warp_size];
+                } else {
+                    dst[ids_dst[j]*stride + i] = sum[(j0/nwarps) * (I/warp_size) + i0/warp_size];
+                }
+            } else {
+                dst[ids_dst[j]*stride + i] = sum[(j0/nwarps) * (I/warp_size) + i0/warp_size];
+                GGML_UNUSED(y_scale_used);
+            }
         }
     }
 }
@@ -442,7 +453,8 @@ template <ggml_type type, int J, bool fallback> static __device__ __forceinline_
 template<ggml_type type, int J, bool fallback>
 static __device__ __forceinline__ void ggml_cuda_mmq_write_back_mma(
             const float * __restrict__ sum, const int * __restrict__ ids_dst, float * __restrict__ dst,
-            const int stride, const int i_max, const int j_max) {
+            const float * __restrict__ y_scale, const int stride, const int i_max, const int j_max) {
+
 #if defined(AMD_MFMA_AVAILABLE) || defined(AMD_WMMA_AVAILABLE)
     typedef tile<16, 16, int, DATA_LAYOUT_J_MAJOR> tile_C;
 #else
@@ -457,6 +469,8 @@ static __device__ __forceinline__ void ggml_cuda_mmq_write_back_mma(
 
     const int i0 = (threadIdx.y / ntx) * (ntx*tile_C::I);
 
+    const bool y_scale_used = y_scale != nullptr;
+
 #pragma unroll
     for (int j0 = 0; j0 < J; j0 += ntx*tile_C::J) {
 #pragma unroll
@@ -475,7 +489,16 @@ static __device__ __forceinline__ void ggml_cuda_mmq_write_back_mma(
                     continue;
                 }
 
-                dst[ids_dst[j]*stride + i] = sum[(j0/tile_C::J + n)*tile_C::ne + l];
+                if constexpr (type == GGML_TYPE_NVFP4) {
+                    if (y_scale_used) {
+                        dst[ids_dst[j]*stride + i] = y_scale[j] * sum[(j0/tile_C::J + n)*tile_C::ne + l];
+                    } else {
+                        dst[ids_dst[j]*stride + i] = sum[(j0/tile_C::J + n)*tile_C::ne + l];
+                    }
+                } else {
+                    dst[ids_dst[j]*stride + i] = sum[(j0/tile_C::J + n)*tile_C::ne + l];
+                    GGML_UNUSED(y_scale_used);
+                }
             }
         }
     }
@@ -819,6 +842,7 @@ template <ggml_type type, int J, bool fallback, bool fixup>
 static __device__ __forceinline__ void mul_mat_q_process_tile(
         const char * __restrict__ x, const int offset_x, const int * __restrict__ y,
         const int * __restrict__ ids_dst, float * __restrict__ dst, float * __restrict__ tmp_fixup,
+        const float * __restrict__ y_scale,
         const int stride_row_x, const int ncols_y, const int stride_col_dst,
         const int tile_x_max_i, const int tile_y_max_j, const int kb0_start, const int kb0_stop) {
 
@@ -884,9 +908,9 @@ static __device__ __forceinline__ void mul_mat_q_process_tile(
     }
 
     if (fixup) {
-        write_back(sum, ids_dst, tmp_fixup + blockIdx.x*(J*I), I, I, J);
+        write_back(sum, ids_dst, tmp_fixup + blockIdx.x*(J*I), y_scale, I, I, J);
     } else {
-        write_back(sum, ids_dst, dst, stride_col_dst, tile_x_max_i, tile_y_max_j);
+        write_back(sum, ids_dst, dst, y_scale, stride_col_dst, tile_x_max_i, tile_y_max_j);
     }
 }
 
@@ -898,6 +922,7 @@ __launch_bounds__(ggml_cuda_mmq_get_nthreads(type, J, fallback), ggml_cuda_mmq_g
 static __global__ void mul_mat_q(
         const char * __restrict__ x, const int * __restrict__ y, const int32_t * __restrict__ ids_dst,
         const int32_t * __restrict__ expert_bounds, float * __restrict__ dst, float * __restrict__ tmp_fixup,
+        const float * __restrict__ y_scale,
         const uint3 blocks_per_ne00, const int nrows_x, const int ncols_dst, const int stride_row_x, const int ncols_y, const int stride_col_dst,
         const uint3 channel_ratio, const uint3 nchannels_y, const int stride_channel_x, const int stride_channel_y, const int stride_channel_dst,
         const uint3 sample_ratio, const uint3 nsamples_y, const int stride_sample_x, const int stride_sample_y, const int stride_sample_dst,
@@ -943,8 +968,14 @@ static __global__ void mul_mat_q(
         int col_low    = 0;
         int col_high   = ncols_dst;
         int col_diff   = ncols_dst;
-        int offset_y   = wt*stride_sample_y   + zt*stride_channel_y;
-        int offset_dst = wt*stride_sample_dst + zt*stride_channel_dst + jt*J*stride_col_dst;
+        int offset_y       = wt*stride_sample_y   + zt*stride_channel_y;
+        int offset_dst     = wt*stride_sample_dst + zt*stride_channel_dst + jt*J*stride_col_dst;
+        int offset_y_scale;
+        if constexpr (type == GGML_TYPE_NVFP4) {
+            offset_y_scale = wt*nchannels_y.z*ncols_y + zt*ncols_y;
+        } else {
+            GGML_UNUSED(offset_y_scale);
+        }
 
         if (ids_dst) {
             col_low  = expert_bounds[zt + 0];
@@ -953,6 +984,9 @@ static __global__ void mul_mat_q(
 
             offset_y   = 0;
             offset_dst = 0;
+            if constexpr (type == GGML_TYPE_NVFP4) {
+                offset_y_scale = 0;
+            }
 
             if (jt*J >= col_diff) {
                 return;
@@ -974,6 +1008,11 @@ static __global__ void mul_mat_q(
 
         offset_y   += (col_low + jt*J)*(sizeof(block_q8_1_mmq)/sizeof(int));
         offset_dst += it*I;
+        const float * y_scale_tile = nullptr;
+        if constexpr (type == GGML_TYPE_NVFP4) {
+            offset_y_scale += col_low + jt*J;
+            y_scale_tile = y_scale ? y_scale + offset_y_scale : nullptr;
+        }
 
         const int tile_x_max_i = nrows_x  - it*I - 1;
         const int tile_y_max_j = col_diff - jt*J - 1;
@@ -982,7 +1021,8 @@ static __global__ void mul_mat_q(
 
         constexpr bool fixup = false;
         mul_mat_q_process_tile<type, J, fallback, fixup>
-            (x, offset_x, y + offset_y, ids_dst_shared, dst + offset_dst, tmp_fixup, stride_row_x, ncols_y, stride_col_dst,
+            (x, offset_x, y + offset_y, ids_dst_shared, dst + offset_dst, tmp_fixup, y_scale_tile,
+             stride_row_x, ncols_y, stride_col_dst,
              tile_x_max_i, tile_y_max_j, 0, blocks_per_ne00.z);
         return;
     }
@@ -1016,8 +1056,14 @@ static __global__ void mul_mat_q(
         int col_low    = 0;
         int col_high   = ncols_dst;
         int col_diff   = ncols_dst;
-        int offset_y   = wt*stride_sample_y   + zt*stride_channel_y;
-        int offset_dst = wt*stride_sample_dst + zt*stride_channel_dst + jt*J*stride_col_dst;
+        int offset_y       = wt*stride_sample_y   + zt*stride_channel_y;
+        int offset_dst     = wt*stride_sample_dst + zt*stride_channel_dst + jt*J*stride_col_dst;
+        int offset_y_scale;
+        if constexpr (type == GGML_TYPE_NVFP4) {
+            offset_y_scale = wt*nchannels_y.z*ncols_y + zt*ncols_y;
+        } else {
+            GGML_UNUSED(offset_y_scale);
+        }
 
         if (ids_dst) {
             col_low  = expert_bounds[zt + 0];
@@ -1026,6 +1072,9 @@ static __global__ void mul_mat_q(
 
             offset_y   = 0;
             offset_dst = 0;
+            if constexpr (type == GGML_TYPE_NVFP4) {
+                offset_y_scale = 0;
+            }
 
             if (jt*J >= col_diff) {
                 kbc += blocks_per_ne00.z;
@@ -1053,6 +1102,11 @@ static __global__ void mul_mat_q(
 
         offset_y += (col_low + jt * J) * (sizeof(block_q8_1_mmq) / sizeof(int));
         offset_dst += it*I;
+        const float * y_scale_tile = nullptr;
+        if constexpr (type == GGML_TYPE_NVFP4) {
+            offset_y_scale += col_low + jt * J;
+            y_scale_tile = y_scale ? y_scale + offset_y_scale : nullptr;
+        }
 
         const int tile_x_max_i = nrows_x  - it*I - 1;
         const int tile_y_max_j = col_diff - jt*J - 1;
@@ -1061,7 +1115,8 @@ static __global__ void mul_mat_q(
 
         constexpr bool fixup = false; // All but (potentially) the last iterations write their data to dst rather than the fixup buffer.
         mul_mat_q_process_tile<type, J, fallback, fixup>
-            (x, offset_x, y + offset_y, ids_dst_shared, dst + offset_dst, tmp_fixup, stride_row_x, ncols_y, stride_col_dst,
+            (x, offset_x, y + offset_y, ids_dst_shared, dst + offset_dst, tmp_fixup, y_scale_tile,
+             stride_row_x, ncols_y, stride_col_dst,
              tile_x_max_i, tile_y_max_j, kb0_start, kb0_stop);
 
         kbc += blocks_per_ne00.z;
@@ -1090,8 +1145,14 @@ static __global__ void mul_mat_q(
     int col_low    = 0;
     int col_high   = ncols_dst;
     int col_diff   = ncols_dst;
-    int offset_y   = wt*stride_sample_y   + zt*stride_channel_y;
-    int offset_dst = wt*stride_sample_dst + zt*stride_channel_dst + jt*J*stride_col_dst;
+    int offset_y       = wt*stride_sample_y   + zt*stride_channel_y;
+    int offset_dst     = wt*stride_sample_dst + zt*stride_channel_dst + jt*J*stride_col_dst;
+    int offset_y_scale;
+    if constexpr (type == GGML_TYPE_NVFP4) {
+        offset_y_scale = wt*nchannels_y.z*ncols_y + zt*ncols_y;
+    } else {
+        GGML_UNUSED(offset_y_scale);
+    }
 
     if (ids_dst) {
         col_low  = expert_bounds[zt + 0];
@@ -1100,6 +1161,9 @@ static __global__ void mul_mat_q(
 
         offset_y   = 0;
         offset_dst = 0;
+        if constexpr (type == GGML_TYPE_NVFP4) {
+            offset_y_scale = 0;
+        }
 
         if (jt*J >= col_diff) {
             return;
@@ -1122,6 +1186,11 @@ static __global__ void mul_mat_q(
 
     offset_y += (col_low + jt * J) * (sizeof(block_q8_1_mmq) / sizeof(int));
     offset_dst += it*I;
+    const float * y_scale_tile = nullptr;
+    if constexpr (type == GGML_TYPE_NVFP4) {
+        offset_y_scale += col_low + jt * J;
+        y_scale_tile = y_scale ? y_scale + offset_y_scale : nullptr;
+    }
 
     const int tile_x_max_i = nrows_x  - it*I - 1;
     const int tile_y_max_j = col_diff - jt*J - 1;
@@ -1130,7 +1199,8 @@ static __global__ void mul_mat_q(
 
     constexpr bool fixup = true; // Last index writes its data to fixup buffer to avoid data races with other blocks.
     mul_mat_q_process_tile<type, J, fallback, fixup>
-        (x, offset_x, y + offset_y, ids_dst_shared, dst + offset_dst, tmp_fixup, stride_row_x, ncols_y, stride_col_dst,
+        (x, offset_x, y + offset_y, ids_dst_shared, dst + offset_dst, tmp_fixup, y_scale_tile,
+         stride_row_x, ncols_y, stride_col_dst,
          tile_x_max_i, tile_y_max_j, kb0_start, kb0_stop);
 }
 
@@ -1274,6 +1344,7 @@ static __global__ void mul_mat_q_stream_k_fixup(
 
 struct mmq_args {
     const char * x; ggml_type type_x; const int * y; const int32_t * ids_dst; const int32_t * expert_bounds; float * dst;
+    const float * y_scale;
     int64_t ncols_x; int64_t nrows_x; int64_t ncols_dst; int64_t stride_row_x; int64_t ncols_y; int64_t nrows_dst;
     int64_t nchannels_x; int64_t nchannels_y; int64_t stride_channel_x; int64_t stride_channel_y; int64_t stride_channel_dst;
     int64_t nsamples_x; int64_t nsamples_y; int64_t stride_sample_x; int64_t stride_sample_y; int64_t stride_sample_dst;
@@ -1323,7 +1394,7 @@ static void launch_mul_mat_q(ggml_backend_cuda_context & ctx, const mmq_args & a
 
     if (!ggml_cuda_mmq_get_stream_k(type, J, fallback, cc)) {
         mul_mat_q<type, J, fallback><<<block_nums_xy_tiling, block_dims, nbytes_shared, stream>>>
-            (args.x, args.y, args.ids_dst, args.expert_bounds, args.dst, nullptr,
+            (args.x, args.y, args.ids_dst, args.expert_bounds, args.dst, nullptr, args.y_scale,
              blocks_per_ne00_fd, args.nrows_x, args.ncols_dst, args.stride_row_x, args.ncols_y, args.nrows_dst,
              channel_ratio_fd, nchannels_y_fd, args.stride_channel_x, args.stride_channel_y, args.stride_channel_dst,
              sample_ratio_fd, nsamples_y_fd, args.stride_sample_x, args.stride_sample_y, args.stride_sample_dst,
@@ -1352,7 +1423,7 @@ static void launch_mul_mat_q(ggml_backend_cuda_context & ctx, const mmq_args & a
     const dim3 block_dims_fixup(block_dims.x, block_dims.y/2, block_dims.z);
 
     mul_mat_q<type, J, fallback><<<block_nums_stream_k, block_dims, nbytes_shared, stream>>>
-        (args.x, args.y, args.ids_dst, args.expert_bounds, args.dst, tmp_fixup.ptr,
+        (args.x, args.y, args.ids_dst, args.expert_bounds, args.dst, tmp_fixup.ptr, args.y_scale,
          blocks_per_ne00_fd, args.nrows_x, args.ncols_dst, args.stride_row_x, args.ncols_y, args.nrows_dst,
          channel_ratio_fd, nchannels_y_fd, args.stride_channel_x, args.stride_channel_y, args.stride_channel_dst,
          sample_ratio_fd, nsamples_y_fd, args.stride_sample_x, args.stride_sample_y, args.stride_sample_dst,
@@ -1466,26 +1537,30 @@ void mul_mat_q_case(ggml_backend_cuda_context & ctx, const mmq_args & args, cuda
 #define DECL_MMQ_CASE(type)                                                        \
     template void mul_mat_q_case<type>(ggml_backend_cuda_context & ctx, const mmq_args & args, cudaStream_t stream) \
 
+extern DECL_MMQ_CASE(GGML_TYPE_Q1_0);
 extern DECL_MMQ_CASE(GGML_TYPE_Q4_0);
 extern DECL_MMQ_CASE(GGML_TYPE_Q4_1);
 extern DECL_MMQ_CASE(GGML_TYPE_Q5_0);
 extern DECL_MMQ_CASE(GGML_TYPE_Q5_1);
 extern DECL_MMQ_CASE(GGML_TYPE_Q8_0);
-extern DECL_MMQ_CASE(GGML_TYPE_MXFP4);
-extern DECL_MMQ_CASE(GGML_TYPE_NVFP4);
+// -----------------------------------------
 extern DECL_MMQ_CASE(GGML_TYPE_Q2_K);
 extern DECL_MMQ_CASE(GGML_TYPE_Q3_K);
 extern DECL_MMQ_CASE(GGML_TYPE_Q4_K);
 extern DECL_MMQ_CASE(GGML_TYPE_Q5_K);
 extern DECL_MMQ_CASE(GGML_TYPE_Q6_K);
+// -----------------------------------------
+extern DECL_MMQ_CASE(GGML_TYPE_IQ1_S);
 extern DECL_MMQ_CASE(GGML_TYPE_IQ2_XXS);
 extern DECL_MMQ_CASE(GGML_TYPE_IQ2_XS);
 extern DECL_MMQ_CASE(GGML_TYPE_IQ2_S);
 extern DECL_MMQ_CASE(GGML_TYPE_IQ3_XXS);
 extern DECL_MMQ_CASE(GGML_TYPE_IQ3_S);
-extern DECL_MMQ_CASE(GGML_TYPE_IQ1_S);
 extern DECL_MMQ_CASE(GGML_TYPE_IQ4_NL);
 extern DECL_MMQ_CASE(GGML_TYPE_IQ4_XS);
+// -----------------------------------------
+extern DECL_MMQ_CASE(GGML_TYPE_MXFP4);
+extern DECL_MMQ_CASE(GGML_TYPE_NVFP4);
 
 // -------------------------------------------------------------------------------------------------------------------------
 

ggml/src/ggml-cuda/quantize.cu

@@ -1,6 +1,55 @@
 #include "quantize.cuh"
 #include <cstdint>
 
+#if defined(BLACKWELL_MMA_AVAILABLE)
+// this maps to 256-bit loads in PTX on supported devices,
+// and otherwise falls back to 2 128-bit loads
+struct __builtin_align__(32) float8 {
+    float x; float y; float z; float w;
+    float p; float q; float r; float s;
+};
+#endif
+
+#if CUDART_VERSION >= 12080
+static __device__ __forceinline__ float nvfp4_native_scale_error(
+        const float vals[QK_NVFP4_SUB], const float inv_col_scale, const float inv_scale, const float scale) {
+    const float scale_dequant = 2.0f * scale;
+    float err = 0.0f;
+
+#pragma unroll
+    for (int k = 0; k < QK_NVFP4_SUB; k += 4) {
+        const float v0 = vals[k + 0] * inv_col_scale;
+        const float v1 = vals[k + 1] * inv_col_scale;
+        const float v2 = vals[k + 2] * inv_col_scale;
+        const float v3 = vals[k + 3] * inv_col_scale;
+
+        const __nv_fp4x4_e2m1 q(make_float4(v0 * inv_scale, v1 * inv_scale, v2 * inv_scale, v3 * inv_scale));
+        const __nv_fp4x4_storage_t q_storage = q.__x;
+        const __nv_fp4x2_storage_t q_lo = static_cast<__nv_fp4x2_storage_t>(q_storage);
+        const __nv_fp4x2_storage_t q_hi = static_cast<__nv_fp4x2_storage_t>(q_storage >> 8U);
+
+        const __half2_raw hraw2_lo = __nv_cvt_fp4x2_to_halfraw2(q_lo, __NV_E2M1);
+        const __half2_raw hraw2_hi = __nv_cvt_fp4x2_to_halfraw2(q_hi, __NV_E2M1);
+        const __half2 h2_lo = static_cast<__half2>(hraw2_lo);
+        const __half2 h2_hi = static_cast<__half2>(hraw2_hi);
+        const float2 dq_lo = __half22float2(h2_lo);
+        const float2 dq_hi = __half22float2(h2_hi);
+
+        const float err0 = fabsf(v0) - fabsf(dq_lo.x) * scale_dequant;
+        const float err1 = fabsf(v1) - fabsf(dq_lo.y) * scale_dequant;
+        const float err2 = fabsf(v2) - fabsf(dq_hi.x) * scale_dequant;
+        const float err3 = fabsf(v3) - fabsf(dq_hi.y) * scale_dequant;
+
+        err = fmaf(err0, err0, err);
+        err = fmaf(err1, err1, err);
+        err = fmaf(err2, err2, err);
+        err = fmaf(err3, err3, err);
+    }
+
+    return err;
+}
+#endif // CUDART_VERSION >= 12080
+
 __launch_bounds__(CUDA_QUANTIZE_BLOCK_SIZE, 1)
 static __global__ void quantize_q8_1(
         const float * x_ptr, void * vy_ptr,
@@ -74,115 +123,209 @@ __device__ __forceinline__ uint8_t compute_e8m0_scale(float amax) {
     return static_cast<uint8_t>(biased);
 }
 
-
 // scatter: grid over tokens, quantize once, write to all the token's compact rows
-template <bool scatter>
+template <bool scatter, bool use_aligned_float8>
 static __global__ void quantize_mmq_nvfp4(
-        const float * __restrict__ x, const int32_t * __restrict__ ids, void * __restrict__ vy,
+        const float * __restrict__ x, const int32_t * __restrict__ ids, void * __restrict__ vy, float * __restrict__ scale,
         const int64_t ne00, const int64_t s01, const int64_t s02, const int64_t s03,
         const int64_t ne0, const int64_t ne1, const int64_t ne2, const int n_expert_used) {
 #if defined(BLACKWELL_MMA_AVAILABLE)
 
-    const int64_t i0_base = ((int64_t) blockDim.x * blockIdx.y + threadIdx.x) * QK_NVFP4_SUB;
-    if (i0_base >= ne0) {
-        return;
-    }
-
-    const int64_t k_block = i0_base / QK_FP4_MMQ;
     const int64_t blocks_per_col = (ne0 + QK_FP4_MMQ - 1) / QK_FP4_MMQ;
-    if (k_block >= blocks_per_col) {
-        return;
-    }
-    const int sub = (i0_base % QK_FP4_MMQ) / QK_NVFP4_SUB;
 
     int64_t base_idx;
     if constexpr (scatter) {
         base_idx = (int64_t) blockIdx.x * s02; // one physical row per token
     } else {
-        const int64_t i2  = blockIdx.z % ne2;
-        const int64_t i3  = blockIdx.z / ne2;
+        const int64_t i2  = blockIdx.y % ne2;
+        const int64_t i3  = blockIdx.y / ne2;
         const int64_t i01 = ids ? ids[blockIdx.x] : blockIdx.x;
         base_idx = i3 * s03 + i2 * s02 + i01 * s01;
     }
+    const float * __restrict__  x_row = x + base_idx;
 
-    float vals_raw[QK_NVFP4_SUB];
-    float amax_raw = 0.0f;
+    float amax = 0.0f;
+    if constexpr (use_aligned_float8) {
+        for (int64_t i0 = 8 * threadIdx.x; i0 < ne00; i0 += 8 * blockDim.x) {
+            const float * x_base = x_row + i0;
+            const float8 v = reinterpret_cast<const float8 *>(x_base)[0];
+            amax = fmaxf(amax, fabsf(v.x));
+            amax = fmaxf(amax, fabsf(v.y));
+            amax = fmaxf(amax, fabsf(v.z));
+            amax = fmaxf(amax, fabsf(v.w));
+            amax = fmaxf(amax, fabsf(v.p));
+            amax = fmaxf(amax, fabsf(v.q));
+            amax = fmaxf(amax, fabsf(v.r));
+            amax = fmaxf(amax, fabsf(v.s));
+        }
+    } else {
+        for (int64_t i0 = threadIdx.x; i0 < ne00; i0 += blockDim.x) {
+            amax = fmaxf(amax, fabsf(x_row[i0]));
+        }
+    }
+
+    amax = warp_reduce_max<WARP_SIZE>(amax);
+
+    __shared__ float warp_amax[CUDA_QUANTIZE_BLOCK_SIZE_MMQ / WARP_SIZE];
+    const int lane = threadIdx.x % WARP_SIZE;
+    const int warp = threadIdx.x / WARP_SIZE;
+
+    if (lane == 0) {
+        warp_amax[warp] = amax;
+    }
+    __syncthreads();
+
+    if (warp == 0) {
+        amax = threadIdx.x < int(CUDA_QUANTIZE_BLOCK_SIZE_MMQ / WARP_SIZE) ? warp_amax[lane] : 0.0f;
+        amax = warp_reduce_max<WARP_SIZE>(amax);
+        if (lane == 0) {
+            warp_amax[0] = amax / (6.0f * 448.0f);
+            if constexpr (scatter) {
 #pragma unroll
-    for (int k = 0; k < QK_NVFP4_SUB; k++) {
-        const int64_t i00 = i0_base + k;
-        if (i00 < ne00) {
-            const float v = x[base_idx + i00];
-            vals_raw[k] = v;
-            amax_raw = fmaxf(amax_raw, fabsf(v));
-        } else {
-            vals_raw[k] = 0.0f;
+                for (int slot = 0; slot < n_expert_used; ++slot) {
+                    const int64_t i = ids[(int64_t) blockIdx.x * n_expert_used + slot];
+                    scale[i] = warp_amax[0];
+                }
+            } else {
+                scale[blockIdx.y * ne1 + blockIdx.x] = warp_amax[0];
+            }
         }
     }
-
-    static constexpr int test_offsets[5] = { 0, -1, 1, -2, 2};
-    const int first_fp8_code = (int) ggml_cuda_fp32_to_ue4m3(amax_raw / 6.0f);
-
-    float best_err = FLT_MAX;
-    uint8_t fp8_code = 0;
-    float subblock_scale = 0.0f;
-
-#pragma unroll // Check +/- 2 to find best code to reduce NVFP4 activation loss. Negligible overhead on Blackwell.
-    for (int i = 0; i < 5; i++) {
-        const int test_code = first_fp8_code + test_offsets[i];
-        if (test_code < 0 || test_code > 0x7e) {
-            continue;
-        }
-        const uint8_t code = (uint8_t) test_code;
-        const float test_scale = ggml_cuda_ue4m3_to_fp32(code);
-        const float test_inv_scale = test_scale > 0.0f ? 0.5f / test_scale : 0.0f;
-        float cur_err = 0.0f;
-#pragma unroll
-        for (int k = 0; k < QK_NVFP4_SUB; ++k) {
-            const float v = vals_raw[k];
-            const uint8_t q = ggml_cuda_float_to_fp4_e2m1(v, test_inv_scale);
-            const float err_diff = fabsf(v) - fabsf(kvalues_mxfp4[q & 0x7]) * test_scale;
-            cur_err = fmaf(err_diff, err_diff, cur_err);
-        }
-
-        if (cur_err < best_err) {
-            best_err = cur_err;
-            fp8_code = test_code;
-            subblock_scale = test_scale;
-        }
-    }
-
-    const float inv_scale = subblock_scale > 0.0f ? 0.5f / subblock_scale : 0.0f;
-    uint32_t q0 = 0;
-    uint32_t q1 = 0;
-#pragma unroll // this is faster than the previous __nv_fp4x4_e2m1
-    for (int k = 0; k < QK_NVFP4_SUB / 4; ++k) {
-        q0 |= (uint32_t) ggml_cuda_float_to_fp4_e2m1(vals_raw[k +  0], inv_scale) << (8 * k);
-        q0 |= (uint32_t) ggml_cuda_float_to_fp4_e2m1(vals_raw[k +  8], inv_scale) << (8 * k + 4);
-        q1 |= (uint32_t) ggml_cuda_float_to_fp4_e2m1(vals_raw[k +  4], inv_scale) << (8 * k);
-        q1 |= (uint32_t) ggml_cuda_float_to_fp4_e2m1(vals_raw[k + 12], inv_scale) << (8 * k + 4);
-    }
+    __syncthreads();
 
     block_fp4_mmq * y = (block_fp4_mmq *) vy;
-    if constexpr (scatter) {
+    const int64_t n_subblocks = (ne0 + QK_NVFP4_SUB - 1) / QK_NVFP4_SUB;
+
+    for (int64_t isb = threadIdx.x; isb < n_subblocks; isb += blockDim.x) {
+        const int64_t i0_base = isb * QK_NVFP4_SUB;
+        const int64_t k_block = i0_base / QK_FP4_MMQ;
+        const int sub = (i0_base % QK_FP4_MMQ) / QK_NVFP4_SUB;
+
+        const float row_scale = warp_amax[0];
+        const float inv_col_scale = row_scale > 0.0f ? 1.0f / row_scale : 0.0f;
+
+        float vals[QK_NVFP4_SUB];
+        if constexpr (use_aligned_float8) {
+            const float * x_base = x_row + i0_base;
+            const float8 v0 = i0_base +  7 < ne00 ? reinterpret_cast<const float8 *>(x_base)[0]     : float8{0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f};
+            const float8 v1 = i0_base + 15 < ne00 ? reinterpret_cast<const float8 *>(x_base + 8)[0] : float8{0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f};
+            vals[0] = v0.x; vals[1] = v0.y; vals[2] = v0.z; vals[3] = v0.w;
+            vals[4] = v0.p; vals[5] = v0.q; vals[6] = v0.r; vals[7] = v0.s;
+            vals[8] = v1.x; vals[9] = v1.y; vals[10] = v1.z; vals[11] = v1.w;
+            vals[12] = v1.p; vals[13] = v1.q; vals[14] = v1.r; vals[15] = v1.s;
+        } else {
 #pragma unroll
-        for (int slot = 0; slot < n_expert_used; ++slot) {
-            const int64_t i = ids[(int64_t) blockIdx.x * n_expert_used + slot];
-            block_fp4_mmq * yb = y + (k_block * ne1 + i);
+            for (int k = 0; k < QK_NVFP4_SUB; ++k) {
+                const int64_t i00 = i0_base + k;
+                vals[k] = i00 < ne00 ? x_row[i00] : 0.0f;
+            }
+        }
+
+        uint32_t q0 = 0;
+        uint32_t q1 = 0;
+
+        float amax_sub = 0.0f;
+#pragma unroll
+        for (int k = 0; k < QK_NVFP4_SUB; ++k) {
+            amax_sub = fmaxf(amax_sub, fabsf(vals[k] * inv_col_scale));
+        }
+
+        static constexpr int test_offsets[5] = { 0, -1, 1, -2, 2 };
+        const int first_fp8_code = (int) ggml_cuda_fp32_to_ue4m3(amax_sub / 6.0f);
+
+        uint8_t fp8_code = (uint8_t) first_fp8_code;
+        float subblock_scale = ggml_cuda_ue4m3_to_fp32(fp8_code);
+        float inv_scale_err = subblock_scale > 0.0f ? 0.5f / subblock_scale : 0.0f;
+#if CUDART_VERSION >= 12080
+        float best_err = nvfp4_native_scale_error(vals, inv_col_scale, inv_scale_err, subblock_scale);
+#else
+        float best_err = 0.0f;
+#pragma unroll
+        for (int k = 0; k < QK_NVFP4_SUB; ++k) {
+            const float v = vals[k] * inv_col_scale;
+            const uint8_t q = ggml_cuda_float_to_fp4_e2m1(v, inv_scale_err);
+            const float err_diff = fabsf(v) - fabsf(kvalues_fp4[q & 0x7]) * subblock_scale;
+            best_err = fmaf(err_diff, err_diff, best_err);
+        }
+#endif // CUDART_VERSION >= 12080
+
+#pragma unroll
+        for (int i = 1; i < 5; ++i) {
+            const int test_code = first_fp8_code + test_offsets[i];
+            if (test_code < 0 || test_code > 0x7e) {
+                continue;
+            }
+
+            const float test_scale = ggml_cuda_ue4m3_to_fp32((uint8_t) test_code);
+            const float test_inv_scale = test_scale > 0.0f ? 0.5f / test_scale : 0.0f;
+#if CUDART_VERSION >= 12080
+            const float cur_err = nvfp4_native_scale_error(vals, inv_col_scale, test_inv_scale, test_scale);
+#else
+            float cur_err = 0.0f;
+#pragma unroll
+            for (int k = 0; k < QK_NVFP4_SUB; ++k) {
+                const float v = vals[k] * inv_col_scale;
+                const uint8_t q = ggml_cuda_float_to_fp4_e2m1(v, test_inv_scale);
+                const float err_diff = fabsf(v) - fabsf(kvalues_fp4[q & 0x7]) * test_scale;
+                cur_err = fmaf(err_diff, err_diff, cur_err);
+            }
+#endif // CUDART_VERSION >= 12080
+
+            if (cur_err < best_err) {
+                best_err = cur_err;
+                fp8_code = (uint8_t) test_code;
+                subblock_scale = test_scale;
+            }
+        }
+#if CUDART_VERSION >= 12080
+        const float inv_scale = subblock_scale > 0.0f ? 0.5f / subblock_scale : 0.0f;
+        const float s = inv_col_scale * inv_scale;
+
+        __nv_fp4x4_e2m1 q0_lo(make_float4(vals[0] * s, vals[8]  * s, vals[1] * s, vals[9]  * s));
+        __nv_fp4x4_e2m1 q0_hi(make_float4(vals[2] * s, vals[10] * s, vals[3] * s, vals[11] * s));
+        __nv_fp4x4_e2m1 q1_lo(make_float4(vals[4] * s, vals[12] * s, vals[5] * s, vals[13] * s));
+        __nv_fp4x4_e2m1 q1_hi(make_float4(vals[6] * s, vals[14] * s, vals[7] * s, vals[15] * s));
+
+        const char2 q0_lo_c = *reinterpret_cast<char2 *>(&q0_lo);
+        const char2 q0_hi_c = *reinterpret_cast<char2 *>(&q0_hi);
+        const char2 q1_lo_c = *reinterpret_cast<char2 *>(&q1_lo);
+        const char2 q1_hi_c = *reinterpret_cast<char2 *>(&q1_hi);
+
+        q0 = uint32_t(uint8_t(q0_lo_c.x)) | (uint32_t(uint8_t(q0_lo_c.y)) <<  8) |
+            (uint32_t(uint8_t(q0_hi_c.x)) << 16) | (uint32_t(uint8_t(q0_hi_c.y)) << 24);
+        q1 = uint32_t(uint8_t(q1_lo_c.x)) | (uint32_t(uint8_t(q1_lo_c.y)) <<  8) |
+            (uint32_t(uint8_t(q1_hi_c.x)) << 16) | (uint32_t(uint8_t(q1_hi_c.y)) << 24);
+#else
+        const float inv_scale = subblock_scale > 0.0f ? 0.5f / subblock_scale : 0.0f;
+#pragma unroll
+        for (int k = 0; k < QK_NVFP4_SUB / 4; ++k) {
+            q0 |= uint32_t(ggml_cuda_float_to_fp4_e2m1(vals[k + 0] * inv_col_scale, inv_scale)) << (8 * k);
+            q0 |= uint32_t(ggml_cuda_float_to_fp4_e2m1(vals[k + 8] * inv_col_scale, inv_scale)) << (8 * k + 4);
+            q1 |= uint32_t(ggml_cuda_float_to_fp4_e2m1(vals[k + 4] * inv_col_scale, inv_scale)) << (8 * k);
+            q1 |= uint32_t(ggml_cuda_float_to_fp4_e2m1(vals[k + 12] * inv_col_scale, inv_scale)) << (8 * k + 4);
+        }
+#endif // CUDART_VERSION >= 12080
+
+        if constexpr (scatter) {
+#pragma unroll
+            for (int slot = 0; slot < n_expert_used; ++slot) {
+                const int64_t i = ids[(int64_t) blockIdx.x * n_expert_used + slot];
+                block_fp4_mmq * yb = y + (k_block * ne1 + i);
+                uint32_t * yqs = reinterpret_cast<uint32_t *>(yb->qs);
+                yqs[2 * sub + 0] = q0;
+                yqs[2 * sub + 1] = q1;
+                reinterpret_cast<uint8_t *>(yb->d4)[sub] = fp8_code;
+            }
+        } else {
+            block_fp4_mmq * yb = y + (blockIdx.y * ((int64_t) blocks_per_col * ne1) + k_block * ne1 + blockIdx.x);
             uint32_t * yqs = reinterpret_cast<uint32_t *>(yb->qs);
             yqs[2 * sub + 0] = q0;
             yqs[2 * sub + 1] = q1;
             reinterpret_cast<uint8_t *>(yb->d4)[sub] = fp8_code;
         }
-    } else {
-        block_fp4_mmq * yb = y + (blockIdx.z * ((int64_t) blocks_per_col * ne1) + k_block * ne1 + blockIdx.x);
-        uint32_t * yqs = reinterpret_cast<uint32_t *>(yb->qs);
-        yqs[2 * sub + 0] = q0;
-        yqs[2 * sub + 1] = q1;
-        reinterpret_cast<uint8_t *>(yb->d4)[sub] = fp8_code;
     }
-    GGML_UNUSED(n_expert_used);
 #else
-    GGML_UNUSED(n_expert_used);
+    GGML_UNUSED_VARS(x, ids, vy, scale, ne00, s01, s02, s03, ne0, ne1, ne2, n_expert_used);
     NO_DEVICE_CODE; // This is for Blackwell NVFP4 activations only.
 #endif // defined(BLACKWELL_MMA_AVAILABLE)
 
@@ -491,18 +634,22 @@ void quantize_scatter_mmq_q8_1_cuda(
 
 // scatter=true reuses the quant kernels: grid over tokens, ids = inverse map (token slot -> compact row)
 void quantize_scatter_mmq_fp4_cuda(
-        const float * x, const int32_t * ids_src1_inv, void * vy, const ggml_type type_src0,
+        const float * x, const int32_t * ids_src1_inv, void * vy, float * scale, const ggml_type type_src0, const bool use_aligned_float8,
         const int64_t ne00, const int64_t stride_token, const int64_t ne0,
         const int64_t n_tokens, const int64_t nrows_dst, const int n_expert_used, cudaStream_t stream) {
     GGML_ASSERT(ne0 > 0);
     if (type_src0 == GGML_TYPE_NVFP4) {
+        GGML_ASSERT(scale);
         GGML_ASSERT(ne00 % QK_NVFP4 == 0);
-        constexpr int nvfp4_block_size = 128;
-        const int64_t block_num_y = (ne0 + QK_NVFP4_SUB * nvfp4_block_size - 1) / (QK_NVFP4_SUB * nvfp4_block_size);
-        const dim3 block_size(nvfp4_block_size, 1, 1);
-        const dim3 num_blocks(n_tokens, block_num_y, 1);
-        quantize_mmq_nvfp4<true><<<num_blocks, block_size, 0, stream>>>(
-            x, ids_src1_inv, vy, ne00, /*s01=*/0, /*s02=*/stride_token, /*s03=*/0, ne0, /*ne1=*/nrows_dst, /*ne2=*/1, n_expert_used);
+        const dim3 block_size(CUDA_QUANTIZE_BLOCK_SIZE_MMQ, 1, 1);
+        const dim3 num_blocks(n_tokens, 1, 1);
+        if (use_aligned_float8) {
+            quantize_mmq_nvfp4<true, true><<<num_blocks, block_size, 0, stream>>>(
+                x, ids_src1_inv, vy, scale, ne00, /*s01=*/0, /*s02=*/stride_token, /*s03=*/0, ne0, /*ne1=*/nrows_dst, /*ne2=*/1, n_expert_used);
+        } else {
+            quantize_mmq_nvfp4<true, false><<<num_blocks, block_size, 0, stream>>>(
+                x, ids_src1_inv, vy, scale, ne00, /*s01=*/0, /*s02=*/stride_token, /*s03=*/0, ne0, /*ne1=*/nrows_dst, /*ne2=*/1, n_expert_used);
+        }
     } else {
         GGML_ASSERT(type_src0 == GGML_TYPE_MXFP4);
         constexpr int nwarps = 8;
@@ -516,20 +663,24 @@ void quantize_scatter_mmq_fp4_cuda(
 }
 
 void quantize_mmq_fp4_cuda(
-        const float * x, const int32_t * ids, void * vy, const ggml_type type_src0,
+        const float * x, const int32_t * ids, void * vy, float * scale, const ggml_type type_src0, const bool use_aligned_float8,
         const int64_t ne00, const int64_t s01, const int64_t s02, const int64_t s03,
         const int64_t ne0, const int64_t ne1, const int64_t ne2, const int64_t ne3, cudaStream_t stream) {
     GGML_ASSERT(type_src0 == GGML_TYPE_MXFP4 || type_src0 == GGML_TYPE_NVFP4);
     GGML_ASSERT(ne0 > 0);
 
     if (type_src0 == GGML_TYPE_NVFP4) {
+        GGML_ASSERT(scale);
         GGML_ASSERT(ne00 % QK_NVFP4 == 0);
-        constexpr int nvfp4_block_size = 128;
-        const int64_t block_num_y = (ne0 + QK_NVFP4_SUB * nvfp4_block_size - 1) / (QK_NVFP4_SUB * nvfp4_block_size);
-        const dim3 block_size(nvfp4_block_size, 1, 1);
-        const dim3 num_blocks(ne1, block_num_y, ne2 * ne3);
-        quantize_mmq_nvfp4<false><<<num_blocks, block_size, 0, stream>>>(
-            x, ids, vy, ne00, s01, s02, s03, ne0, ne1, ne2, /*n_expert_used=*/0);
+        const dim3 block_size(CUDA_QUANTIZE_BLOCK_SIZE_MMQ, 1, 1);
+        const dim3 num_blocks(ne1, ne2 * ne3, 1);
+        if (use_aligned_float8) {
+            quantize_mmq_nvfp4<false, true><<<num_blocks, block_size, 0, stream>>>(
+                x, ids, vy, scale, ne00, s01, s02, s03, ne0, ne1, ne2, /*n_expert_used=*/0);
+        } else {
+            quantize_mmq_nvfp4<false, false><<<num_blocks, block_size, 0, stream>>>(
+                x, ids, vy, scale, ne00, s01, s02, s03, ne0, ne1, ne2, /*n_expert_used=*/0);
+        }
     } else {
         GGML_ASSERT(ne0 % (2 * QK_MXFP4) == 0);
 

ggml/src/ggml-cuda/quantize.cuh

@@ -29,7 +29,9 @@ void quantize_mmq_q8_1_cuda(
 void quantize_mmq_fp4_cuda(const float *   x,
                              const int32_t * ids,
                              void *          vy,
+                             float *         scale,
                              ggml_type       type_src0,
+                             bool            use_aligned_float8,
                              int64_t         ne00,
                              int64_t         s01,
                              int64_t         s02,
@@ -44,7 +46,9 @@ void quantize_mmq_fp4_cuda(const float *   x,
 void quantize_scatter_mmq_fp4_cuda(const float *   x,
                                    const int32_t * ids_src1_inv,
                                    void *          vy,
+                                   float *         scale,
                                    ggml_type       type_src0,
+                                   bool            use_aligned_float8,
                                    int64_t         ne00,
                                    int64_t         stride_token,
                                    int64_t         ne0,

ggml/src/ggml-hexagon/ggml-hexagon.cpp

@@ -143,12 +143,12 @@ static const char * htp_event_name(uint16_t id) {
         case HTP_TRACE_EVT_HMX_COMP:       return "HMX_COMP";
         case HTP_TRACE_EVT_L2FLUSH:        return "L2FLUSH";
         case HTP_TRACE_EVT_INIT:           return "INIT";
+        case HTP_TRACE_EVT_BUFF:           return "BUFF";
         default:                           return "UNKNOWN";
     }
 }
 
-static void ggml_hexagon_dump_op_prof(const std::string &sess_name, const htp_opnode & node,
-                                      const htp_prof_desc & pd) {
+static void ggml_hexagon_dump_op_prof(const std::string &sess_name, const htp_opnode & node, const htp_prof_desc & pd) {
     if (!opt_profile) return;
 
     uint32_t op_usec = pd.usecs;
@@ -168,6 +168,43 @@ static void ggml_hexagon_dump_op_prof(const std::string &sess_name, const htp_op
             node.op_name().c_str(), fmt.names, fmt.dims, fmt.types, fmt.strides, fmt.kparams, op_usec, op_cycles, pd.cycles_start, mhz, pmu_str);
 }
 
+static void ggml_hexagon_dump_batch_prof(const std::string & sess_name, const htp_opbatch_rsp & rsp) {
+    uint64_t batch_cycles = rsp.cycles_stop - rsp.cycles_start;
+    float batch_mhz = rsp.usecs > 0 ? (float) batch_cycles / rsp.usecs : 0.0f;
+
+    char evt_str[256] = "----";
+    if (opt_profile == 3) {
+        snprintf(evt_str, sizeof(evt_str), "evt-cnt %u,%u,%u,%u,%u,%u,%u,%u,%u,%u,%u",
+                rsp.n_traces[0], rsp.n_traces[1], rsp.n_traces[2], rsp.n_traces[3],
+                rsp.n_traces[4], rsp.n_traces[5], rsp.n_traces[6], rsp.n_traces[7],
+                rsp.n_traces[8], rsp.n_traces[9], rsp.n_traces[10]);
+    }
+
+    GGML_LOG_DEBUG("ggml-hex: %s profile-op OPBATCH|----|n-ops %u|%s|----|----|usec %u cycles %llu start %llu mhz %.1f\n",
+                   sess_name.c_str(), rsp.n_ops, evt_str, rsp.usecs, (unsigned long long) batch_cycles, (unsigned long long) rsp.cycles_start, batch_mhz);
+}
+
+static void ggml_hexagon_dump_trace_events(const std::string & sess_name, const htp_opbatch_rsp & rsp,
+                                           const htp_trace_desc * trace_events, uint32_t n_traces) {
+    if (opt_profile == 3 && trace_events) {
+        uint32_t valid_cnt[HTP_MAX_NTHREADS + 1] = {0};
+        for (uint32_t t = 0; t <= HTP_MAX_NTHREADS; t++) {
+            uint32_t count = rsp.n_traces[t];
+            valid_cnt[t] = count > n_traces ? n_traces : count;
+        }
+
+        for (uint32_t t = 0; t <= HTP_MAX_NTHREADS; t++) {
+            for (uint32_t idx = 0; idx < valid_cnt[t]; idx++) {
+                const auto & e = trace_events[t * n_traces + idx];
+                bool is_stop = (e.info & 0x8000) != 0;
+                uint16_t info = e.info & 0x7FFF;
+                GGML_LOG_DEBUG("ggml-hex: %s trace-evt %s: thread %u info %u %s %u\n",
+                               sess_name.c_str(), htp_event_name(e.id), t, info, is_stop ? "stop" : "start", e.cycles);
+            }
+        }
+    }
+}
+
 // **
 
 static inline bool ggml_hexagon_is_repack_type(enum ggml_type type) {
@@ -1128,13 +1165,7 @@ struct ggml_hexagon_opbatch {
     std::unordered_map<const ggml_tensor*, int> t_map; // tensor ptr  to index
     std::unordered_multimap<void*, int>         d_map; // tensor data to index
 
-    struct tensor_range {
-        uint64_t start;
-        uint64_t end;
-        int bi;
-        std::vector<int> tensors;
-    };
-    std::vector<tensor_range> ranges;
+
 
     unsigned int n_bufs;     // num buffers in the batch
     unsigned int n_tens;     // num tensors ...
@@ -1155,7 +1186,6 @@ struct ggml_hexagon_opbatch {
         b_map.clear();
         t_map.clear();
         d_map.clear();
-        ranges.clear();
     }
 
     ggml_hexagon_opbatch(ggml_hexagon_session *sess, size_t batch_size, size_t max_vmem) {
@@ -1209,70 +1239,7 @@ struct ggml_hexagon_opbatch {
         return bi;
     }
 
-    void add_range(const htp_tensor * h, int ti) {
-        uint64_t t_start = h->data;
-        uint64_t t_end   = t_start + h->size;
-        int      bi      = h->bi;
 
-        int first_match = -1;
-        int unused_idx  = -1;
-        for (size_t i = 0; i < ranges.size(); i++) {
-            if (ranges[i].bi == -1) {
-                unused_idx = i;
-                continue;
-            }
-            if (ranges[i].bi != bi) {
-                continue;
-            }
-            if (ranges[i].start >= t_end || ranges[i].end <= t_start) {
-                continue;
-            }
-
-            if (first_match == -1) {
-                first_match = i;
-                HEX_VERBOSE("ggml-hex: %s range-grow #%d : bi %d [%p, %p) + #%d [%p, %p) -> [%p, %p)\n",
-                    sess->c_name(), (int) i, ranges[i].bi,
-                    (void *) (h_bufs[ranges[i].bi].base + ranges[i].start),
-                    (void *) (h_bufs[ranges[i].bi].base + ranges[i].end),
-                    ti,
-                    (void *) (h_bufs[bi].base + t_start),
-                    (void *) (h_bufs[bi].base + t_end),
-                    (void *) (h_bufs[ranges[i].bi].base + std::min(ranges[i].start, t_start)),
-                    (void *) (h_bufs[ranges[i].bi].base + std::max(ranges[i].end, t_end)));
-
-                ranges[i].start = std::min(ranges[i].start, t_start);
-                ranges[i].end   = std::max(ranges[i].end, t_end);
-                ranges[i].tensors.push_back(ti);
-            } else {
-                HEX_VERBOSE("ggml-hex: %s range-merge #%d [%p, %p) + #%d [%p, %p) -> [%p, %p)\n",
-                    sess->c_name(), first_match,
-                    (void *) (h_bufs[bi].base + ranges[first_match].start),
-                    (void *) (h_bufs[bi].base + ranges[first_match].end),
-                    (int) i,
-                    (void *) (h_bufs[bi].base + ranges[i].start),
-                    (void *) (h_bufs[bi].base + ranges[i].end),
-                    (void *) (h_bufs[bi].base + std::min(ranges[first_match].start, ranges[i].start)),
-                    (void *) (h_bufs[bi].base + std::max(ranges[first_match].end, ranges[i].end)));
-
-                ranges[first_match].start = std::min(ranges[first_match].start, ranges[i].start);
-                ranges[first_match].end   = std::max(ranges[first_match].end, ranges[i].end);
-                ranges[first_match].tensors.insert(
-                    ranges[first_match].tensors.end(),
-                    ranges[i].tensors.begin(),
-                    ranges[i].tensors.end()
-                );
-                ranges[i].bi = -1;
-            }
-        }
-
-        if (first_match == -1) {
-            if (unused_idx != -1) {
-                ranges[unused_idx] = {t_start, t_end, bi, {ti}};
-            } else {
-                ranges.push_back({t_start, t_end, bi, {ti}});
-            }
-        }
-    }
 
     bool same_shape(const htp_tensor * h, const ggml_tensor * t) const {
         int64_t ne0 = t->ne[0];
@@ -1341,8 +1308,7 @@ struct ggml_hexagon_opbatch {
             h.nb[0] = t->nb[0]; h.nb[1] = t->nb[1]; h.nb[2] = t->nb[2]; h.nb[3] = t->nb[3];
         }
 
-        h.alias = ti;
-        add_range(&h, ti);
+
 
         h.flags = 0;
         if (ggml_backend_buffer_get_usage(t->buffer) != GGML_BACKEND_BUFFER_USAGE_WEIGHTS) {
@@ -1424,14 +1390,6 @@ struct ggml_hexagon_opbatch {
     }
 
     void finalize_ranges() {
-        for (const auto & r : ranges) {
-            if (r.bi == -1) {
-                continue;
-            }
-            for (size_t i = 0; i < r.tensors.size(); i++) {
-                h_tens[r.tensors[i]].alias = r.tensors[(i + 1) % r.tensors.size()];
-            }
-        }
     }
 };
 
@@ -1582,9 +1540,6 @@ struct ggml_hexagon_opqueue {
         if (opt_profile && rsp.n_ops > 0) {
             auto & ops = op_cache[rsp.id];
 
-            uint64_t batch_usec = ggml_time_us() - start_usec[rsp.id];
-            uint32_t htp_usec   = 0;
-
             GGML_ASSERT(rsp.n_ops <= ops.size());
 
             const htp_prof_desc * pd = (const htp_prof_desc *) p_ptr;
@@ -1595,55 +1550,13 @@ struct ggml_hexagon_opqueue {
                 trace_events = (const htp_trace_desc *) (p_ptr + p_size);
             }
 
-            uint32_t trace_idx[HTP_MAX_NTHREADS + 1] = {0};
-            uint32_t valid_cnt[HTP_MAX_NTHREADS + 1] = {0};
-
-            if (opt_profile == 3) {
-                for (uint32_t t = 0; t <= HTP_MAX_NTHREADS; t++) {
-                    uint32_t count = rsp.n_traces[t];
-                    valid_cnt[t] = count > n_traces ? n_traces : count;
-                }
-            }
+            ggml_hexagon_dump_batch_prof(shm_buf->sess->name, rsp);
 
             for (uint32_t i = 0; i < rsp.n_ops; i++) {
-                htp_usec += pd[i].usecs;
-
                 ggml_hexagon_dump_op_prof(shm_buf->sess->name, ops[i], pd[i]);
-
-                if (opt_profile == 3) {
-                    uint32_t op_duration = pd[i].cycles_stop - pd[i].cycles_start;
-
-                    for (uint32_t t = 0; t <= HTP_MAX_NTHREADS; t++) {
-                        while (trace_idx[t] < valid_cnt[t]) {
-                            const auto & e = trace_events[t * n_traces + trace_idx[t]];
-                            uint32_t offset = e.cycles - pd[i].cycles_start;
-                            if (offset >= 0x80000000) {
-                                trace_idx[t]++;
-                                continue;
-                            }
-                            if (offset > op_duration) {
-                                break;
-                            }
-                            bool is_stop = (e.info & 0x8000) != 0;
-                            uint16_t info = e.info & 0x7FFF;
-                            GGML_LOG_DEBUG("ggml-hex: %s trace-op %s: thread %u event %s info %u %s %u\n",
-                                           shm_buf->sess->c_name(), ops[i].op_name().c_str(), t, htp_event_name(e.id), info, is_stop ? "stop" : "start", e.cycles);
-                            trace_idx[t]++;
-                        }
-                    }
-                }
             }
 
-            char evt_str[256] = "";
-            if (opt_profile == 3) {
-                snprintf(evt_str, sizeof(evt_str), " evt [%u,%u,%u,%u,%u,%u,%u,%u,%u,%u,%u]",
-                        rsp.n_traces[0], rsp.n_traces[1], rsp.n_traces[2], rsp.n_traces[3],
-                        rsp.n_traces[4], rsp.n_traces[5], rsp.n_traces[6], rsp.n_traces[7],
-                        rsp.n_traces[8], rsp.n_traces[9], rsp.n_traces[10]);
-            }
-
-            GGML_LOG_DEBUG("ggml-hex: %s profile-batch n-ops %u batch-dur-usec %lld htp-ops-usec %u%s\n",
-                           shm_buf->sess->c_name(), rsp.n_ops, (long long) batch_usec, htp_usec, evt_str);
+            ggml_hexagon_dump_trace_events(shm_buf->sess->name, rsp, trace_events, n_traces);
         }
     }
 };
@@ -1662,7 +1575,7 @@ void ggml_hexagon_session::flush_pending(bool all) {
         const uint32_t timeo = opt_oppoll ? 0 : DSPQUEUE_TIMEOUT;
 
         int err = dspqueue_read(this->queue, &flags, 1, &n_dbufs, &dbuf, sizeof(rsp), &rsp_size, (uint8_t *) &rsp, timeo);
-        if (err == AEE_EEXPIRED) {
+        if (err == AEE_EEXPIRED || err == AEE_EWOULDBLOCK) {
             continue;
         }
 
@@ -2114,7 +2027,7 @@ static bool ggml_hexagon_precompute_flash_attn_params(
     const struct ggml_tensor * sinks = op->src[4];
     if (ggml_hexagon_flash_attn_is_hmx_eligible(sess, q, k, v, sinks)) {
         size_t Br = 0, Bc = 0;
-        int ret = hmx_fa_find_chunk_size(&Br, &Bc, G, DK, DV, neq1, nek1, sess->vtcm_size, sess->n_threads);
+        int ret = hmx_fa_find_chunk_size(&Br, &Bc, G, DK, DV, neq1, nek1, sess->vtcm_size, sess->n_threads, kparams->is_q_fp32 != 0);
         if (ret == 0) {
             kparams->kernel_type = HTP_FA_KERNEL_HMX;
             kparams->Br = Br;
@@ -2124,7 +2037,7 @@ static bool ggml_hexagon_precompute_flash_attn_params(
 
             kparams->u.hmx.g_br = hex_align_up(G * Br, 32);
             kparams->u.hmx.pipeline = (kparams->n_kv_blocks >= 3 && sess->n_threads >= 2) ? 1 : 0;
-            kparams->vtcm_size = hmx_fa_compute_vtcm_usage(G, DK, DV, Br, Bc, kparams->n_threads, kparams->u.hmx.pipeline != 0);
+            kparams->vtcm_size = hmx_fa_compute_vtcm_usage(G, DK, DV, Br, Bc, kparams->n_threads, kparams->u.hmx.pipeline != 0, kparams->is_q_fp32 != 0);
 
             const size_t row_vec_bytes = hex_align_up(Bc * sizeof(uint16_t), 256);
             kparams->u.hmx.row_buf_stride = row_vec_bytes / 128; // HVX vector is 128 bytes
@@ -2413,6 +2326,7 @@ static void ggml_hexagon_precompute_hvx_mm_params(
     int ne12,
     int ne13,
     bool is_matmul_id,
+    const size_t src2_row_size,
     size_t vtcm_budget,
     struct htp_mm_kernel_params * kparams
 ) {
@@ -2438,7 +2352,7 @@ static void ggml_hexagon_precompute_hvx_mm_params(
             for (uint32_t d = max_prefetch; d >= 2; d /= 2) {
                 htp_mm_hvx_vtcm_layout_build(
                     &L, kparams->kernel_type, wtype, ne10, src1_nrows, sess->n_threads,
-                    0, src0->nb[1], 0, d, true, false, false
+                    0, src0->nb[1], 0, src2_row_size, d, true, false, false
                 );
                 if (L.total_bytes <= vtcm_budget) {
                     best_n_prefetch = d;
@@ -2448,7 +2362,7 @@ static void ggml_hexagon_precompute_hvx_mm_params(
             if (best_n_prefetch == 2 && L.total_bytes > vtcm_budget) {
                 htp_mm_hvx_vtcm_layout_build(
                     &L, kparams->kernel_type, wtype, ne10, src1_nrows, sess->n_threads,
-                    0, src0->nb[1], 0, 2, true, false, false
+                    0, src0->nb[1], 0, src2_row_size, 2, true, false, false
                 );
             }
             kparams->n_prefetch = best_n_prefetch;
@@ -2472,7 +2386,7 @@ static void ggml_hexagon_precompute_hvx_mm_params(
                 for (uint32_t d = max_prefetch; d >= 2; d /= 2) {
                     htp_mm_hvx_vtcm_layout_build(
                         &L, kparams->kernel_type, wtype, ne10, src1_nrows, sess->n_threads,
-                        dst->nb[1], src0->nb[1], src1->nb[1], d, false, false, false
+                        dst->nb[1], src0->nb[1], src1->nb[1], src2_row_size, d, false, false, false
                     );
                     if (L.total_bytes <= vtcm_budget) {
                         best_n_prefetch = d;
@@ -2482,7 +2396,7 @@ static void ggml_hexagon_precompute_hvx_mm_params(
                 if (best_n_prefetch == 2 && L.total_bytes > vtcm_budget) {
                     htp_mm_hvx_vtcm_layout_build(
                         &L, kparams->kernel_type, wtype, ne10, src1_nrows, sess->n_threads,
-                        dst->nb[1], src0->nb[1], src1->nb[1], 2, false, false, false
+                        dst->nb[1], src0->nb[1], src1->nb[1], src2_row_size, 2, false, false, false
                     );
                 }
 
@@ -2506,7 +2420,7 @@ static void ggml_hexagon_precompute_hvx_mm_params(
                 struct htp_mm_hvx_vtcm_layout L;
                 htp_mm_hvx_vtcm_layout_build(
                     &L, kparams->kernel_type, wtype, ne10, src1_nrows, sess->n_threads,
-                    dst->nb[1], src0->nb[1], src1->nb[1], 16, false, false, false
+                    dst->nb[1], src0->nb[1], src1->nb[1], src2_row_size, 16, false, false, false
                 );
 
                 kparams->n_prefetch = 16;
@@ -2526,7 +2440,7 @@ static void ggml_hexagon_precompute_hvx_mm_params(
         struct htp_mm_hvx_vtcm_layout L;
         htp_mm_hvx_vtcm_layout_build(
             &L, HTP_MM_KERNEL_HVX_F16_F16_VTCM, wtype, ne10, src1_nrows, sess->n_threads,
-            dst->nb[1], src0->nb[1], src1->nb[1], 16, false, false, false
+            dst->nb[1], src0->nb[1], src1->nb[1], src2_row_size, 16, false, false, false
         );
 
         if (!is_batched && !is_permuted && L.total_bytes <= vtcm_budget) {
@@ -2546,7 +2460,7 @@ static void ggml_hexagon_precompute_hvx_mm_params(
             kparams->src1_row_size = src1->nb[1];
             htp_mm_hvx_vtcm_layout_build(
                 &L, kparams->kernel_type, wtype, ne10, src1_nrows, sess->n_threads,
-                dst->nb[1], src0->nb[1], src1->nb[1], 16, false, false, false
+                dst->nb[1], src0->nb[1], src1->nb[1], src2_row_size, 16, false, false, false
             );
             kparams->vtcm_size = L.total_bytes;
             kparams->vtcm_src0_size = L.src0_bytes;
@@ -2562,7 +2476,7 @@ static void ggml_hexagon_precompute_hvx_mm_params(
         struct htp_mm_hvx_vtcm_layout L;
         htp_mm_hvx_vtcm_layout_build(
             &L, HTP_MM_KERNEL_HVX_F32_F32_VTCM, wtype, ne10, src1_nrows, sess->n_threads,
-            dst->nb[1], src0->nb[1], src1->nb[1], 16, false, false, false
+            dst->nb[1], src0->nb[1], src1->nb[1], src2_row_size, 16, false, false, false
         );
 
         if (!is_batched && !is_permuted && L.total_bytes <= vtcm_budget) {
@@ -2578,7 +2492,7 @@ static void ggml_hexagon_precompute_hvx_mm_params(
             kparams->src1_row_size = src1->nb[1];
             htp_mm_hvx_vtcm_layout_build(
                 &L, kparams->kernel_type, wtype, ne10, src1_nrows, sess->n_threads,
-                dst->nb[1], src0->nb[1], src1->nb[1], 16, false, false, false
+                dst->nb[1], src0->nb[1], src1->nb[1], src2_row_size, 16, false, false, false
             );
             kparams->vtcm_size = L.total_bytes;
             kparams->vtcm_src0_size = L.src0_bytes;
@@ -2589,11 +2503,12 @@ static void ggml_hexagon_precompute_hvx_mm_params(
     }
 }
 
-static void ggml_hexagon_precompute_matmul_params(
+static void ggml_hexagon_precompute_matmul_params_impl(
     const struct ggml_hexagon_session * sess,
     const struct ggml_tensor * src0,
     const struct ggml_tensor * src1,
     const struct ggml_tensor * dst,
+    const size_t src2_row_size,
     struct htp_mm_kernel_params * kparams
 ) {
     memset(kparams, 0, sizeof(*kparams));
@@ -2628,7 +2543,7 @@ static void ggml_hexagon_precompute_matmul_params(
     }
 
     // Fallback to HVX parameter computation
-    ggml_hexagon_precompute_hvx_mm_params(sess, src0, src1, dst, wtype, ne02, ne03, ne10, ne11, ne12, ne13, is_matmul_id, vtcm_budget, kparams);
+    ggml_hexagon_precompute_hvx_mm_params(sess, src0, src1, dst, wtype, ne02, ne03, ne10, ne11, ne12, ne13, is_matmul_id, src2_row_size, vtcm_budget, kparams);
 
 finalize:
     kparams->div_ne12_ne1 = init_fastdiv_values(ne12 * ne11);
@@ -2638,6 +2553,27 @@ finalize:
     kparams->div_ne11     = init_fastdiv_values(ne11);
 }
 
+static void ggml_hexagon_precompute_matmul_params(
+    const struct ggml_hexagon_session * sess,
+    const struct ggml_tensor * src0,
+    const struct ggml_tensor * src1,
+    const struct ggml_tensor * dst,
+    struct htp_mm_kernel_params * kparams
+) {
+    ggml_hexagon_precompute_matmul_params_impl(sess, src0, src1, dst, 0, kparams);
+}
+
+static void ggml_hexagon_precompute_fused_matmul_add_params(
+    const struct ggml_hexagon_session * sess,
+    const struct ggml_tensor * src0,
+    const struct ggml_tensor * src1,
+    const struct ggml_tensor * src2,
+    const struct ggml_tensor * dst,
+    struct htp_mm_kernel_params * kparams
+) {
+    ggml_hexagon_precompute_matmul_params_impl(sess, src0, src1, dst, src2->nb[1], kparams);
+}
+
 static void ggml_hexagon_precompute_unary_params(
     const struct ggml_hexagon_session * sess,
     uint32_t op,
@@ -2731,7 +2667,7 @@ static void ggml_hexagon_precompute_fused_qkv_params(
             struct htp_mm_hvx_vtcm_layout L;
             htp_mm_hvx_vtcm_layout_build(
                 &L, HTP_MM_KERNEL_HVX_QUANT_ROW, wtype, ne10, src1_nrows, sess->n_threads,
-                0, src0_row_size, src1_row_size, d, false, true, false
+                0, src0_row_size, src1_row_size, 0, d, false, true, false
             );
             if (L.total_bytes <= sess->vtcm_size) {
                 best_n_prefetch = d;
@@ -2746,7 +2682,7 @@ static void ggml_hexagon_precompute_fused_qkv_params(
     // Test tiled first
     htp_mm_hvx_vtcm_layout_build(
         &L, HTP_MM_KERNEL_HVX_QUANT_ROW, wtype, ne10, src1_nrows, sess->n_threads,
-        0, src0_row_size, src1_row_size, best_n_prefetch, false, true, false
+        0, src0_row_size, src1_row_size, 0, best_n_prefetch, false, true, false
     );
 
     if (try_tiled && L.total_bytes <= sess->vtcm_size) {
@@ -2764,7 +2700,7 @@ static void ggml_hexagon_precompute_fused_qkv_params(
 
         htp_mm_hvx_vtcm_layout_build(
             &L, HTP_MM_KERNEL_HVX_QUANT_ROW_FLAT, wtype, ne10, src1_nrows, sess->n_threads,
-            0, src0_row_size, flat_src1_row_size, best_n_prefetch, false, true, false
+            0, src0_row_size, flat_src1_row_size, 0, best_n_prefetch, false, true, false
         );
         kparams->vtcm_src0_size = L.src0_bytes;
         kparams->vtcm_src1_size = L.src1_bytes;
@@ -2801,7 +2737,7 @@ static void ggml_hexagon_precompute_fused_ffn_params(
             struct htp_mm_hvx_vtcm_layout L;
             htp_mm_hvx_vtcm_layout_build(
                 &L, HTP_MM_KERNEL_HVX_QUANT_ROW, wtype, ne10, src1_nrows, sess->n_threads,
-                0, src0_row_size, src1_row_size, d, false, false, true
+                0, src0_row_size, src1_row_size, 0, d, false, false, true
             );
             if (L.total_bytes <= sess->vtcm_size) {
                 best_n_prefetch = d;
@@ -2816,7 +2752,7 @@ static void ggml_hexagon_precompute_fused_ffn_params(
     // Test tiled first
     htp_mm_hvx_vtcm_layout_build(
         &L, HTP_MM_KERNEL_HVX_QUANT_ROW, wtype, ne10, src1_nrows, sess->n_threads,
-        0, src0_row_size, src1_row_size, best_n_prefetch, false, false, true
+        0, src0_row_size, src1_row_size, 0, best_n_prefetch, false, false, true
     );
 
     if (try_tiled && L.total_bytes <= sess->vtcm_size) {
@@ -2833,7 +2769,7 @@ static void ggml_hexagon_precompute_fused_ffn_params(
 
         htp_mm_hvx_vtcm_layout_build(
             &L, HTP_MM_KERNEL_HVX_QUANT_ROW_FLAT, wtype, ne10, src1_nrows, sess->n_threads,
-            0, src0_row_size, flat_src1_row_size, best_n_prefetch, false, false, true
+            0, src0_row_size, flat_src1_row_size, 0, best_n_prefetch, false, false, true
         );
         kparams->vtcm_src0_size = L.src0_bytes;
         kparams->vtcm_src1_size = L.src1_bytes;
@@ -3084,7 +3020,10 @@ static bool ggml_hexagon_supported_activations(const struct ggml_hexagon_session
         return false;
     }
 
-    if (!ggml_is_contiguous(src0) || !ggml_is_contiguous(dst)) {
+    if (!ggml_is_contiguous_1(src0)) {
+        return false;
+    }
+    if (!ggml_is_contiguous(dst)) {
         return false;
     }
 
@@ -3095,7 +3034,7 @@ static bool ggml_hexagon_supported_activations(const struct ggml_hexagon_session
         if (!ggml_are_same_shape(src0, src1)) {
             return false;
         }
-        if (!ggml_is_contiguous(src1)) {
+        if (!ggml_is_contiguous_1(src1)) {
             return false;
         }
     }
@@ -3653,16 +3592,19 @@ static bool try_fuse_node(const ggml_hexagon_session * sess, const ggml_cgraph *
     if (n->op == GGML_OP_MUL_MAT && next_node) {
         if (next_node->op == GGML_OP_ADD && op_is_compute(next_node) && ggml_can_fuse(graph, i, { GGML_OP_MUL_MAT, GGML_OP_ADD })) {
             if (next_node->src[0] == n || next_node->src[1] == n) {
+                const struct ggml_tensor * src2 = (next_node->src[0] == n) ? next_node->src[1] : next_node->src[0];
                 struct htp_mm_kernel_params kparams;
-                ggml_hexagon_precompute_matmul_params(sess, n->src[0], n->src[1], next_node, &kparams);
-                if ((size_t)kparams.vtcm_size <= sess->vtcm_size) {
+                ggml_hexagon_precompute_fused_matmul_add_params(sess, n->src[0], n->src[1], src2, next_node, &kparams);
+                const int src1_nrows = n->src[1]->ne[1] * n->src[1]->ne[2] * n->src[1]->ne[3];
+                const bool can_fuse = (kparams.n_hmx > 0) || (src1_nrows == 1);
+                if (can_fuse && (size_t)kparams.vtcm_size <= sess->vtcm_size) {
                     htp_opnode node(n, {}, HTP_OP_MUL_MAT_ADD);
                     node.add_fused(next_node);
                     memcpy(node.kernel_params, &kparams, sizeof(kparams));
                     nodes.push_back(std::move(node));
                     i += 1;
                     return true;
-                } else {
+                } else if (can_fuse) {
                     HEX_VERBOSE("ggml-hex: skip MUL_MAT_ADD fusion because VTCM needed (%d) > budget (%zu)\n",
                                 kparams.vtcm_size, sess->vtcm_size);
                 }
@@ -4152,12 +4094,10 @@ static bool ggml_backend_hexagon_device_supports_op(ggml_backend_dev_t dev, cons
                 case GGML_UNARY_OP_SIGMOID:
                 case GGML_UNARY_OP_SOFTPLUS:
                 case GGML_UNARY_OP_TANH:
-                    supp = ggml_hexagon_supported_unary(sess, op);
-                    break;
                 case GGML_UNARY_OP_SILU:
                 case GGML_UNARY_OP_GELU:
                 case GGML_UNARY_OP_GELU_QUICK:
-                    supp = ggml_hexagon_supported_activations(sess, op);
+                    supp = ggml_hexagon_supported_unary(sess, op);
                     break;
                 default:
                     break;
@@ -4454,7 +4394,7 @@ static void ggml_hexagon_init(ggml_backend_reg * reg) {
     opt_opstage   = str_opstage  ? strtoul(str_opstage, NULL, 0)          : opt_opstage;
     opt_opbatch   = str_opbatch  ? strtoul(str_opbatch, NULL, 0)          : opt_opbatch;
     opt_opqueue   = str_opqueue  ? strtoul(str_opqueue, NULL, 0)          : opt_opqueue;
-    opt_optrace   = str_optrace  ? strtoul(str_optrace, NULL, 0)          : (opt_opbatch * 128);
+    opt_optrace   = str_optrace  ? strtoul(str_optrace, NULL, 0)          : (opt_opbatch * 256);
     opt_oppoll    = str_oppoll   ? strtoul(str_oppoll,  NULL, 0)          : opt_oppoll;
     opt_opfusion  = str_opfusion ? atoi(str_opfusion)                     : opt_opfusion;
     opt_profile   = str_profile  ? atoi(str_profile)                      : 0;

ggml/src/ggml-hexagon/htp-drv.cpp

@@ -59,7 +59,11 @@ typedef AEEResult (*dspqueue_read_pfn_t)(dspqueue_t queue, uint32_t *flags,
                                          uint32_t max_message_length,
                                          uint32_t *message_length, uint8_t *message,
                                          uint32_t timeout_us);
-
+typedef AEEResult (*dspqueue_read_noblock_pfn_t)(dspqueue_t queue, uint32_t *flags,
+                                                 uint32_t max_buffers, uint32_t *num_buffers,
+                                                 struct dspqueue_buffer *buffers,
+                                                 uint32_t max_message_length,
+                                                 uint32_t *message_length, uint8_t *message);
 typedef int (*fastrpc_mmap_pfn_t)(int domain, int fd, void *addr, int offset, size_t length, enum fastrpc_map_flags flags);
 typedef int (*fastrpc_munmap_pfn_t)(int domain, int fd, void *addr, size_t length);
 
@@ -82,11 +86,12 @@ rpcmem_to_fd_pfn_t  rpcmem_to_fd_pfn  = nullptr;
 fastrpc_mmap_pfn_t   fastrpc_mmap_pfn   = nullptr;
 fastrpc_munmap_pfn_t fastrpc_munmap_pfn = nullptr;
 
-dspqueue_create_pfn_t dspqueue_create_pfn = nullptr;
-dspqueue_close_pfn_t  dspqueue_close_pfn  = nullptr;
-dspqueue_export_pfn_t dspqueue_export_pfn = nullptr;
-dspqueue_write_pfn_t  dspqueue_write_pfn  = nullptr;
-dspqueue_read_pfn_t   dspqueue_read_pfn   = nullptr;
+dspqueue_create_pfn_t       dspqueue_create_pfn       = nullptr;
+dspqueue_close_pfn_t        dspqueue_close_pfn        = nullptr;
+dspqueue_export_pfn_t       dspqueue_export_pfn       = nullptr;
+dspqueue_write_pfn_t        dspqueue_write_pfn        = nullptr;
+dspqueue_read_pfn_t         dspqueue_read_pfn         = nullptr;
+dspqueue_read_noblock_pfn_t dspqueue_read_noblock_pfn = nullptr;
 
 remote_handle64_open_pfn_t    remote_handle64_open_pfn    = nullptr;
 remote_handle64_invoke_pfn_t  remote_handle64_invoke_pfn  = nullptr;
@@ -167,6 +172,12 @@ AEEResult dspqueue_read(dspqueue_t               queue,
                         uint32_t *               message_length,
                         uint8_t *                message,
                         uint32_t                 timeout_us) {
+#ifdef _WIN32
+    if (timeout_us == 0) {
+        return dspqueue_read_noblock_pfn(queue, flags, max_buffers, num_buffers, buffers, max_message_length,
+                                     message_length, message);
+    }
+#endif
     return dspqueue_read_pfn(queue, flags, max_buffers, num_buffers, buffers, max_message_length, message_length,
                              message, timeout_us);
 }
@@ -349,6 +360,7 @@ int htpdrv_init() {
     dlsym(handle.get(), dspqueue_export_pfn_t, dspqueue_export_pfn, dspqueue_export, false);
     dlsym(handle.get(), dspqueue_write_pfn_t, dspqueue_write_pfn, dspqueue_write, false);
     dlsym(handle.get(), dspqueue_read_pfn_t, dspqueue_read_pfn, dspqueue_read, false);
+    dlsym(handle.get(), dspqueue_read_noblock_pfn_t, dspqueue_read_noblock_pfn, dspqueue_read_noblock, false);
     dlsym(handle.get(), remote_handle64_open_pfn_t, remote_handle64_open_pfn, remote_handle64_open, false);
     dlsym(handle.get(), remote_handle64_invoke_pfn_t, remote_handle64_invoke_pfn, remote_handle64_invoke, false);
     dlsym(handle.get(), remote_handle_control_pfn_t, remote_handle_control_pfn, remote_handle_control, false);

ggml/src/ggml-hexagon/htp/dma-queue.c

@@ -101,6 +101,4 @@ void dma_queue_alias_free(dma_queue_t q) {
     (void) q;
 }
 
-void dma_queue_flush(dma_queue_t q) {
-    while (dma_queue_pop(q).dst != NULL) ;
-}
+

ggml/src/ggml-hexagon/htp/dma-queue.h

@@ -106,7 +106,7 @@ struct dma_queue_s {
     bool                alias;     // When set, dma_queue_delete will not free the ring
 };
 
-void        dma_queue_flush(dma_queue_t q);
+
 
 size_t      dma_queue_sizeof(size_t capacity);
 size_t      dma_queue_alignof(void);
@@ -154,7 +154,6 @@ static inline bool dma_is_vtcm(const dma_queue * q, const void * ptr) {
 static inline bool dma_queue_push_single_1d(dma_queue * q, dma_ptr dptr, size_t size) {
     dma_ring * r = q->ring;
     if (((r->push_idx + 1) & r->idx_mask) == r->pop_idx) {
-        FARF(HIGH, "dma-push: queue full\n");
         return false;
     }
 
@@ -165,6 +164,8 @@ static inline bool dma_queue_push_single_1d(dma_queue * q, dma_ptr dptr, size_t
 
     r->dptr[r->push_idx] = dptr;
 
+    htp_trace_event_start(r->trace, HTP_TRACE_EVT_DMA, r->push_idx);
+
     if (size) {
         desc->next       = NULL;
         desc->desc_size  = 0; // 1D mode
@@ -173,7 +174,6 @@ static inline bool dma_queue_push_single_1d(dma_queue * q, dma_ptr dptr, size_t
         desc->order      = 0;
         desc->done       = 0;
 
-        htp_trace_event_start(r->trace, HTP_TRACE_EVT_DMA, r->push_idx);
         dmlink(r->tail, desc);
         r->tail = (dma_descriptor_2d *) desc;
     } else {
@@ -188,7 +188,6 @@ static inline bool dma_queue_push_single_1d(dma_queue * q, dma_ptr dptr, size_t
 static inline bool dma_queue_push_single_2d(dma_queue * q, dma_ptr dptr, size_t dst_stride, size_t src_stride, size_t row_size, size_t nrows) {
     dma_ring * r = q->ring;
     if (((r->push_idx + 1) & r->idx_mask) == r->pop_idx) {
-        FARF(HIGH, "dma-push: queue full\n");
         return false;
     }
 
@@ -224,8 +223,9 @@ static inline bool dma_queue_push_single_2d(dma_queue * q, dma_ptr dptr, size_t
 
     r->dptr[r->push_idx] = dptr;
 
+    htp_trace_event_start(r->trace, HTP_TRACE_EVT_DMA, r->push_idx);
+
     if (nrows) {
-        htp_trace_event_start(r->trace, HTP_TRACE_EVT_DMA, r->push_idx);
         dmlink(r->tail, desc);
         r->tail = desc;
     } else {
@@ -252,10 +252,11 @@ static inline dma_ptr dma_queue_pop(dma_queue * q) {
             dmpoll();
         }
     }
-    htp_trace_event_stop(r->trace, HTP_TRACE_EVT_DMA, r->pop_idx);
 
     dptr = r->dptr[r->pop_idx];
 
+    htp_trace_event_stop(r->trace, HTP_TRACE_EVT_DMA, r->pop_idx);
+
     r->pop_idx = (r->pop_idx + 1) & r->idx_mask;
     return dptr;
 }
@@ -270,6 +271,8 @@ static inline dma_ptr dma_queue_pop_nowait(dma_queue * q) {
 
     dptr = r->dptr[r->pop_idx];
 
+    htp_trace_event_stop(r->trace, HTP_TRACE_EVT_DMA, r->pop_idx);
+
     r->pop_idx = (r->pop_idx + 1) & r->idx_mask;
     return dptr;
 }
@@ -278,6 +281,10 @@ static inline bool dma_queue_empty(dma_queue * q) {
     return q->ring->push_idx == q->ring->pop_idx;
 }
 
+static inline void dma_queue_flush(dma_queue * q) {
+    while (dma_queue_pop(q).dst != NULL) ;
+}
+
 static inline uint32_t dma_queue_depth(dma_queue * q) {
     return (q->ring->push_idx - q->ring->pop_idx) & q->ring->idx_mask;
 }
@@ -314,14 +321,18 @@ static inline bool dma_queue_push(dma_queue *q, dma_ptr dptr, size_t dst_stride,
     {
         const uint8_t *src = (const uint8_t *) dptr.src;
         uint8_t       *dst = (uint8_t *)       dptr.dst;
-        for (size_t r = 0; r < nrows; ++r) {
+        size_t r = 0;
+        while (r + 1 < nrows) {
             dma_ptr p = dma_make_ptr(dst + r * dst_stride, src + r * src_stride);
-            if (!dma_queue_push_single_1d(q, p, row_size))
-                return false;
-            if (r + 1 < nrows)
-                dma_queue_pop(q);
+            if (!dma_queue_push_single_1d(q, p, row_size)) {
+                dma_queue_flush(q);
+            } else {
+                r++;
+            }
         }
-        return true;
+        dma_queue_flush(q);
+        dma_ptr p = dma_make_ptr(dst + r * dst_stride, src + r * src_stride);
+        return dma_queue_push_single_1d(q, p, row_size);
     }
 }
 

ggml/src/ggml-hexagon/htp/flash-attn-ops.c

@@ -123,15 +123,17 @@ struct hmx_fa_context {
     uint32_t     g_br;  // hex_align_up(G * Br, 32) - actual tile row dim
 
     // VTCM buffers (allocated by vtcm_seq_alloc)
+    __fp16 *     vtcm_q_dma;           // Q DMA fetch buffer
     __fp16 *     vtcm_q_tiles;         // Q tile format [g_br, D]
     __fp16 *     vtcm_o_tiles[2];      // O ping-pong [g_br, D]
     __fp16 *     vtcm_k_fp16[2];       // K DMA double-buffer [Bc, D]
     __fp16 *     vtcm_v_fp16[2];       // V DMA double-buffer [Bc, D]
-    __fp16 *     vtcm_k_tiles;         // K tiles (transposed)
+    __fp16 *     vtcm_k_tiles[2];      // K tiles (transposed, double-buffered)
     __fp16 *     vtcm_v_tiles[2];      // V tiles (column-major, double-buffered)
-    __fp16 *     vtcm_s_tiles;         // S = QK^T [g_br, Bc]
-    __fp16 *     vtcm_p_tiles;         // P = softmax(S) [g_br, Bc]
+    __fp16 *     vtcm_s_tiles[2];      // S = QK^T [g_br, Bc] (double-buffered)
+    __fp16 *     vtcm_p_tiles[2];      // P = softmax(S) [g_br, Bc]
     __fp16 *     vtcm_d_tiles;         // Diagonal rescale [g_br, g_br]
+    __fp16 *     vtcm_d_inv_l;         // Diagonal rescale (1/l) [g_br, g_br]
     HVX_Vector * vtcm_m_vec;           // Row max [g_br]
     HVX_Vector * vtcm_l_vec;           // Row sum [g_br]
     HVX_Vector * vtcm_s_rowmax;        // Softmax intermediate [g_br]
@@ -236,10 +238,6 @@ static void flash_attn_ext_f16_thread(unsigned int nth, unsigned int ith, void *
         const uint32_t iv3 = fastdiv(iq3, &factx->broadcast_rv3);
         const uint32_t iv2 = fastdiv(iq2, &factx->broadcast_rv2);
 
-        // Fetch Q row
-        const uint8_t * q_row_ptr = (const uint8_t *) q->data + (iq1*nbq1 + iq2*nbq2 + iq3*nbq3);
-        dma_queue_push(dma, dma_make_ptr(spad_q, q_row_ptr), factx->size_q_row_padded, nbq1, size_q_row, 1);
-
         const __fp16 * mp_base = NULL;
         if (mask) {
             const uint32_t im2 = fastmodulo(iq2, mask->ne[2], &factx->src3_div2);
@@ -247,26 +245,91 @@ static void flash_attn_ext_f16_thread(unsigned int nth, unsigned int ith, void *
             mp_base = (const __fp16 *) ((const uint8_t *) mask->data + iq1*mask->nb[1] + im2*mask->nb[2] + im3*mask->nb[3]);
         }
 
-        // Prefetch first two blocks
-        for (uint32_t ib = 0; ib < MIN(factx->n_blocks, 2); ++ib) {
-            const uint32_t ic_start = ib * FLASH_ATTN_BLOCK_SIZE;
-            const uint32_t current_block_size = MIN(FLASH_ATTN_BLOCK_SIZE, nek1 - ic_start);
+        // Precalculate next row variables if there is a next row
+        bool has_next_ir = (ir + 1 < ir1);
+        uint32_t next_ik2 = 0, next_ik3 = 0, next_iv2 = 0, next_iv3 = 0;
+        const uint8_t * next_q_row_ptr = NULL;
+        const __fp16 * next_mp_base = NULL;
 
-            // K
-            const uint8_t * k_src = (const uint8_t *) k->data + (ic_start*nbk1 + ik2*nbk2 + ik3*nbk3);
-            uint8_t * k_dst = spad_k + (ib % 2) * factx->size_k_block;
-            dma_queue_push(dma, dma_make_ptr(k_dst, k_src), factx->size_k_row_padded, nbk1, size_k_row, current_block_size);
+        const uint8_t * next_k_src0 = NULL;
+        const uint8_t * next_v_src0 = NULL;
+        const uint8_t * next_m_src0 = NULL;
+        uint32_t next_block_size0 = 0;
 
-            // V
-            const uint8_t * v_src = (const uint8_t *) v->data + (ic_start*nbv1 + iv2*nbv2 + iv3*nbv3);
-            uint8_t * v_dst = spad_v + (ib % 2) * factx->size_v_block;
-            dma_queue_push(dma, dma_make_ptr(v_dst, v_src), factx->size_v_row_padded, nbv1, size_v_row, current_block_size);
+        const uint8_t * next_k_src1 = NULL;
+        const uint8_t * next_v_src1 = NULL;
+        const uint8_t * next_m_src1 = NULL;
+        uint32_t next_block_size1 = 0;
+
+        if (has_next_ir) {
+            const uint32_t next_ir = ir + 1;
+            const uint32_t next_iq3 = fastdiv(next_ir, &factx->src0_div21);
+            const uint32_t next_iq2 = fastdiv(next_ir - next_iq3*neq2*neq1, &factx->src0_div1);
+            const uint32_t next_iq1 = (next_ir - next_iq3*neq2*neq1 - next_iq2 * neq1);
+
+            next_ik3 = fastdiv(next_iq3, &factx->broadcast_rk3);
+            next_ik2 = fastdiv(next_iq2, &factx->broadcast_rk2);
+
+            next_iv3 = fastdiv(next_iq3, &factx->broadcast_rv3);
+            next_iv2 = fastdiv(next_iq2, &factx->broadcast_rv2);
+
+            next_q_row_ptr = (const uint8_t *) q->data + (next_iq1*nbq1 + next_iq2*nbq2 + next_iq3*nbq3);
 
-            // Mask
             if (mask) {
-                const uint8_t * m_src = (const uint8_t *) (mp_base + ic_start);
-                // Mask is 1D contiguous for this row
-                dma_cache_push(dma, &m_cache, m_src, current_block_size * 2, current_block_size * 2, current_block_size * 2, 1);
+                const uint32_t next_im2 = fastmodulo(next_iq2, mask->ne[2], &factx->src3_div2);
+                const uint32_t next_im3 = fastmodulo(next_iq3, mask->ne[3], &factx->src3_div3);
+                next_mp_base = (const __fp16 *) ((const uint8_t *) mask->data + next_iq1*mask->nb[1] + next_im2*mask->nb[2] + next_im3*mask->nb[3]);
+            }
+
+            // Precalculate next K/V block 0 source pointers
+            {
+                const uint32_t ic_start = 0;
+                next_block_size0 = MIN(FLASH_ATTN_BLOCK_SIZE, nek1 - ic_start);
+                next_k_src0 = (const uint8_t *) k->data + (ic_start*nbk1 + next_ik2*nbk2 + next_ik3*nbk3);
+                next_v_src0 = (const uint8_t *) v->data + (ic_start*nbv1 + next_iv2*nbv2 + next_iv3*nbv3);
+                if (mask) {
+                    next_m_src0 = (const uint8_t *) (next_mp_base + ic_start);
+                }
+            }
+
+            // Precalculate next K/V block 1 source pointers (if n_blocks > 1)
+            if (factx->n_blocks > 1) {
+                const uint32_t ic_start = 1 * FLASH_ATTN_BLOCK_SIZE;
+                next_block_size1 = MIN(FLASH_ATTN_BLOCK_SIZE, nek1 - ic_start);
+                next_k_src1 = (const uint8_t *) k->data + (ic_start*nbk1 + next_ik2*nbk2 + next_ik3*nbk3);
+                next_v_src1 = (const uint8_t *) v->data + (ic_start*nbv1 + next_iv2*nbv2 + next_iv3*nbv3);
+                if (mask) {
+                    next_m_src1 = (const uint8_t *) (next_mp_base + ic_start);
+                }
+            }
+        }
+
+        if (ir == ir0) {
+            // Fetch Q row
+            const uint8_t * q_row_ptr = (const uint8_t *) q->data + (iq1*nbq1 + iq2*nbq2 + iq3*nbq3);
+            dma_queue_push(dma, dma_make_ptr(spad_q, q_row_ptr), factx->size_q_row_padded, nbq1, size_q_row, 1);
+
+            // Prefetch first two blocks
+            for (uint32_t ib = 0; ib < MIN(factx->n_blocks, 2); ++ib) {
+                const uint32_t ic_start = ib * FLASH_ATTN_BLOCK_SIZE;
+                const uint32_t current_block_size = MIN(FLASH_ATTN_BLOCK_SIZE, nek1 - ic_start);
+
+                // K
+                const uint8_t * k_src = (const uint8_t *) k->data + (ic_start*nbk1 + ik2*nbk2 + ik3*nbk3);
+                uint8_t * k_dst = spad_k + (ib % 2) * factx->size_k_block;
+                dma_queue_push(dma, dma_make_ptr(k_dst, k_src), factx->size_k_row_padded, nbk1, size_k_row, current_block_size);
+
+                // V
+                const uint8_t * v_src = (const uint8_t *) v->data + (ic_start*nbv1 + iv2*nbv2 + iv3*nbv3);
+                uint8_t * v_dst = spad_v + (ib % 2) * factx->size_v_block;
+                dma_queue_push(dma, dma_make_ptr(v_dst, v_src), factx->size_v_row_padded, nbv1, size_v_row, current_block_size);
+
+                // Mask
+                if (mask) {
+                    const uint8_t * m_src = (const uint8_t *) (mp_base + ic_start);
+                    // Mask is 1D contiguous for this row
+                    dma_cache_push(dma, &m_cache, m_src, current_block_size * 2, current_block_size * 2, current_block_size * 2, 1);
+                }
             }
         }
 
@@ -287,6 +350,11 @@ static void flash_attn_ext_f16_thread(unsigned int nth, unsigned int ith, void *
 
         const HVX_Vector slope_vec = hvx_vec_splat_f16(slope);
         const HVX_Vector v_neg_inf = Q6_Vh_vsplat_R(0xfbff);
+        const HVX_Vector v_cap     = (factx->logit_softcap != 0.0f) ? hvx_vec_splat_f16(factx->logit_softcap) : Q6_V_vzero();
+        const HVX_Vector vinf      = Q6_Vh_vsplat_R(0xFC00);
+        const HVX_Vector vmin      = Q6_Vh_vsplat_R(0xFBFF);
+        const HVX_Vector v_log2e   = hvx_vec_splat_f16(EXP_LOG2E_F);
+        const uint32_t stride_v2   = factx->size_v_row_padded * 2;
         for (uint32_t ib = 0; ib < factx->n_blocks; ++ib) {
             const uint32_t ic_start = ib * FLASH_ATTN_BLOCK_SIZE;
             const uint32_t current_block_size = MIN(FLASH_ATTN_BLOCK_SIZE, nek1 - ic_start);
@@ -309,7 +377,6 @@ static void flash_attn_ext_f16_thread(unsigned int nth, unsigned int ith, void *
 
             // 2. Softcap (in FP16)
             if (factx->logit_softcap != 0.0f) {
-                const HVX_Vector v_cap = hvx_vec_splat_f16(factx->logit_softcap);
                 scores_f16 = hvx_vec_tanh_f16(scores_f16);
                 scores_f16 = hvx_vec_mul_f16_f16(scores_f16, v_cap);
             }
@@ -319,8 +386,6 @@ static void flash_attn_ext_f16_thread(unsigned int nth, unsigned int ith, void *
             // 3. Mask (in FP16)
             if (mask) {
                 HVX_Vector m_vals_f16 = *(const HVX_UVector *) m_base;
-                HVX_Vector vinf = Q6_Vh_vsplat_R(0xFC00);
-                HVX_Vector vmin = Q6_Vh_vsplat_R(0xFBFF);
                 HVX_VectorPred is_inf = Q6_Q_vcmp_eq_VhVh(m_vals_f16, vinf);
                 m_vals_f16 = Q6_V_vmux_QVV(is_inf, vmin, m_vals_f16);
 
@@ -335,10 +400,30 @@ static void flash_attn_ext_f16_thread(unsigned int nth, unsigned int ith, void *
             HVX_Vector v_max     = Q6_V_lo_W(hvx_vec_f16_to_f32(v_max_f16)); // splat block max in FP32
             htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_FA_QK, ir);
 
+            if (ib + 1 == factx->n_blocks && has_next_ir) {
+                // Queue next row's Q row!
+                dma_queue_push(dma, dma_make_ptr(spad_q, next_q_row_ptr), factx->size_q_row_padded, nbq1, size_q_row, 1);
+
+                if (factx->n_blocks % 2 == 0) {
+                    // Queue next row's block 0 (into buffer slot 0)
+                    uint8_t * k_dst = spad_k + 0 * factx->size_k_block;
+                    uint8_t * v_dst = spad_v + 0 * factx->size_v_block;
+
+                    // K (block 0 of next row)
+                    dma_queue_push(dma, dma_make_ptr(k_dst, next_k_src0), factx->size_k_row_padded, nbk1, size_k_row, next_block_size0);
+
+                    // V (block 0 of next row)
+                    dma_queue_push(dma, dma_make_ptr(v_dst, next_v_src0), factx->size_v_row_padded, nbv1, size_v_row, next_block_size0);
+
+                    // Mask (block 0 of next row)
+                    if (mask) {
+                        dma_cache_push(dma, &m_cache, next_m_src0, next_block_size0 * 2, next_block_size0 * 2, next_block_size0 * 2, 1);
+                    }
+                }
+            }
+
             htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_FA_SFM, ir);
             {
-                const HVX_Vector v_log2e = hvx_vec_splat_f16(EXP_LOG2E_F);
-
                 // 4. Online Softmax Update
                 HVX_Vector M_new_vec = Q6_Vsf_vmax_VsfVsf(v_max, M_vec);
                 HVX_Vector diff_vec  = HVX_OP_SUB_F32(M_vec, M_new_vec);
@@ -370,24 +455,20 @@ static void flash_attn_ext_f16_thread(unsigned int nth, unsigned int ith, void *
                 S_vec = HVX_OP_ADD_F32(HVX_OP_MUL_F32(S_vec, ms_vec), p_sum_vec);
 
                 // 5. Accumulate V (F16 * F16 -> F32 accumulator)
-                __fp16 __attribute__((aligned(128))) p_arr[VLEN_FP16];
-                hvx_vec_store_a(p_arr, 128, P);
+                const uint8_t * v_ptr = v_base;
 
                 for (uint32_t j = 0; j < current_block_size; j += 2) {
                     if (j + 1 == current_block_size) {
-                        if (p_arr[j] != 0.0f) {
-                            const uint8_t * v_ptr = v_base + j * factx->size_v_row_padded;
-                            hvx_mad_f32_f16_aa(VKQ32, v_ptr, (p_arr + j), DV);
-                        }
+                        HVX_Vector S0 = hvx_vec_repl_f16(Q6_V_vror_VR(P, j * 2));
+                        hvx_mad_f32_f16_aa_vec(VKQ32, v_ptr, S0, DV);
                         break;
                     }
 
-                    if (p_arr[j] == 0.0f && p_arr[j + 1] == 0.0f) {
-                        continue;
-                    }
+                    HVX_Vector S0 = hvx_vec_repl_f16(Q6_V_vror_VR(P, j * 2));
+                    HVX_Vector S1 = hvx_vec_repl_f16(Q6_V_vror_VR(P, (j + 1) * 2));
 
-                    const uint8_t * v_ptr = v_base + j * factx->size_v_row_padded;
-                    hvx_mad_f32_f16_aa_rx2(VKQ32, v_ptr, v_ptr + factx->size_v_row_padded, (p_arr + j), (p_arr + j + 1), DV);
+                    hvx_mad_f32_f16_aa_rx2_vec(VKQ32, v_ptr, v_ptr + factx->size_v_row_padded, S0, S1, DV);
+                    v_ptr += stride_v2;
                 }
             }
             htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_FA_SFM, ir);
@@ -414,6 +495,61 @@ static void flash_attn_ext_f16_thread(unsigned int nth, unsigned int ith, void *
             }
         }
 
+        if (has_next_ir) {
+            if (factx->n_blocks % 2 == 0) {
+                // Queue next row's block 1 (into buffer slot 1, if n_blocks > 1)
+                if (factx->n_blocks > 1) {
+                    uint8_t * k_dst = spad_k + 1 * factx->size_k_block;
+                    uint8_t * v_dst = spad_v + 1 * factx->size_v_block;
+
+                    // K (block 1 of next row)
+                    dma_queue_push(dma, dma_make_ptr(k_dst, next_k_src1), factx->size_k_row_padded, nbk1, size_k_row, next_block_size1);
+
+                    // V (block 1 of next row)
+                    dma_queue_push(dma, dma_make_ptr(v_dst, next_v_src1), factx->size_v_row_padded, nbv1, size_v_row, next_block_size1);
+
+                    // Mask (block 1 of next row)
+                    if (mask) {
+                        dma_cache_push(dma, &m_cache, next_m_src1, next_block_size1 * 2, next_block_size1 * 2, next_block_size1 * 2, 1);
+                    }
+                }
+            } else {
+                // Queue next row's block 0 (into buffer slot 0)
+                {
+                    uint8_t * k_dst = spad_k + 0 * factx->size_k_block;
+                    uint8_t * v_dst = spad_v + 0 * factx->size_v_block;
+
+                    // K (block 0 of next row)
+                    dma_queue_push(dma, dma_make_ptr(k_dst, next_k_src0), factx->size_k_row_padded, nbk1, size_k_row, next_block_size0);
+
+                    // V (block 0 of next row)
+                    dma_queue_push(dma, dma_make_ptr(v_dst, next_v_src0), factx->size_v_row_padded, nbv1, size_v_row, next_block_size0);
+
+                    // Mask (block 0 of next row)
+                    if (mask) {
+                        dma_cache_push(dma, &m_cache, next_m_src0, next_block_size0 * 2, next_block_size0 * 2, next_block_size0 * 2, 1);
+                    }
+                }
+
+                // Queue next row's block 1 (into buffer slot 1, if n_blocks > 1)
+                if (factx->n_blocks > 1) {
+                    uint8_t * k_dst = spad_k + 1 * factx->size_k_block;
+                    uint8_t * v_dst = spad_v + 1 * factx->size_v_block;
+
+                    // K (block 1 of next row)
+                    dma_queue_push(dma, dma_make_ptr(k_dst, next_k_src1), factx->size_k_row_padded, nbk1, size_k_row, next_block_size1);
+
+                    // V (block 1 of next row)
+                    dma_queue_push(dma, dma_make_ptr(v_dst, next_v_src1), factx->size_v_row_padded, nbv1, size_v_row, next_block_size1);
+
+                    // Mask (block 1 of next row)
+                    if (mask) {
+                        dma_cache_push(dma, &m_cache, next_m_src1, next_block_size1 * 2, next_block_size1 * 2, next_block_size1 * 2, 1);
+                    }
+                }
+            }
+        }
+
         htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_O_PROC, ir);
         // sinks
         float M = hvx_vec_get_f32(M_vec);
@@ -471,6 +607,7 @@ typedef struct {
     void *                  curr_k;
     uint32_t                kv_start;
     uint32_t                rows_per_t;
+    size_t                  buf_idx;
 } fa_k_int_args_t;
 
 static void fa_k_interleave_thread(unsigned int n, unsigned int i, void * data) {
@@ -488,19 +625,19 @@ static void fa_k_interleave_thread(unsigned int n, unsigned int i, void * data)
 
     struct htp_thread_trace * tr = &factx->octx->ctx->trace[i];
     htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_FA_K_PREP, (uint16_t) (args->kv_start + start));
-    hmx_interleave_rows_to_tiles(factx->vtcm_k_tiles, (const __fp16 *) args->curr_k, total_rows, factx->DK,
+    hmx_interleave_rows_to_tiles(factx->vtcm_k_tiles[args->buf_idx], (const __fp16 *) args->curr_k, total_rows, factx->DK,
                              args->src_stride, start, end);
     htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_FA_K_PREP, (uint16_t) (args->kv_start + start));
 }
 
-static void fa_phase_k_interleave(struct hmx_fa_context * factx, uint32_t kv_rows, size_t src_stride, void * curr_k, uint32_t kv_start) {
+static void fa_phase_k_interleave(struct hmx_fa_context * factx, uint32_t kv_rows, size_t src_stride, void * curr_k, uint32_t kv_start, size_t buf_idx) {
     work_queue_t wp = factx->octx->ctx->work_queue;
     uint32_t n = 1;
     if (factx->n_threads > 1 && kv_rows >= factx->n_threads * 2) {
         n = factx->n_threads;
     }
     uint32_t rows_per_t = hex_align_up(hmx_ceil_div(kv_rows, n), 2);
-    fa_k_int_args_t args = { factx, kv_rows, src_stride, curr_k, kv_start, rows_per_t };
+    fa_k_int_args_t args = { factx, kv_rows, src_stride, curr_k, kv_start, rows_per_t, buf_idx };
     if (n > 1) {
         work_queue_run(wp, fa_k_interleave_thread, &args, n);
     } else {
@@ -645,12 +782,13 @@ static void fa_q_load_thread(unsigned int n, unsigned int i, void * data) {
             }
         }
 
-        // Initialize vtcm_d_tiles to 0
+        // Initialize vtcm_d_tiles and vtcm_d_inv_l to 0
         const size_t d_bytes_per_t = hex_align_up(d_tile_bytes / n, 128);
         const size_t d_start       = i * d_bytes_per_t;
         const size_t d_end         = hex_smin(d_start + d_bytes_per_t, d_tile_bytes);
         if (d_start < d_tile_bytes) {
             hvx_splat_u8_a((char *) factx->vtcm_d_tiles + d_start, 0, d_end - d_start);
+            hvx_splat_u8_a((char *) factx->vtcm_d_inv_l + d_start, 0, d_end - d_start);
         }
     }
 
@@ -662,15 +800,14 @@ static void fa_q_load_thread(unsigned int n, unsigned int i, void * data) {
 
         assert(factx->DK == factx->DV);
 
-        const size_t o_tile_bytes = factx->o_tile_bytes;
-        const bool use_q_dma = (2 * o_tile_bytes >= factx->g_br * DK * (factx->is_q_fp32 ? 4 : 2));
+        const bool use_q_dma = (factx->vtcm_q_dma != NULL);
 
         __fp16 * q_tiles = factx->vtcm_q_tiles;
         if (use_q_dma) {
             const size_t g_rows_end = hex_smin(end, n_rows_g);
             const uint32_t d_limit = factx->is_q_fp32 ? DK / 32 : DK / 64;
 
-            uint8_t * q_flat  = (uint8_t *) factx->vtcm_o_tiles[0];
+            uint8_t * q_flat  = (uint8_t *) factx->vtcm_q_dma;
             if (factx->is_q_fp32) {
                 switch (d_limit) {
                 case 2:  hmx_fa_q_prep_fp32_d2(q_tiles, q_flat, start, end, g_rows_end, DK, G, args->n_rows_q, &factx->div_G, args->q_transposed); break;
@@ -781,10 +918,10 @@ static void fa_o_store_thread_f32(unsigned int n, unsigned int i, void * data) {
     const uint32_t            kv_head    = args->kv_head;
     const uint32_t            ib3        = args->ib3;
 
-    for (size_t r = start; r < end; ++r) {
-        const size_t q_idx = fastdiv(r, &factx->div_G);
-        const size_t h_idx = fastmodulo(r, G, &factx->div_G);
+    size_t q_idx = fastdiv(start, &factx->div_G);
+    size_t h_idx = fastmodulo(start, G, &factx->div_G);
 
+    for (size_t r = start; r < end; ++r) {
         float * out = (float *) ((uint8_t *) dst->data + (kv_head * G + h_idx) * dst->nb[1] +
                                  (q_start + q_idx) * dst->nb[2] + ib3 * dst->nb[3]);
 
@@ -801,6 +938,12 @@ static void fa_o_store_thread_f32(unsigned int n, unsigned int i, void * data) {
                 *(HVX_UVector *) (out + d * 32) = Q6_V_hi_W(vp);
             }
         }
+
+        h_idx++;
+        if (h_idx == G) {
+            h_idx = 0;
+            q_idx++;
+        }
     }
     htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_O_PROC, (uint16_t) (args->q_start * G + start));
 }
@@ -829,10 +972,10 @@ static void fa_o_store_thread_f16(unsigned int n, unsigned int i, void * data) {
     const uint32_t            kv_head    = args->kv_head;
     const uint32_t            ib3        = args->ib3;
 
-    for (size_t r = start; r < end; ++r) {
-        const size_t q_idx = fastdiv(r, &factx->div_G);
-        const size_t h_idx = fastmodulo(r, G, &factx->div_G);
+    size_t q_idx = fastdiv(start, &factx->div_G);
+    size_t h_idx = fastmodulo(start, G, &factx->div_G);
 
+    for (size_t r = start; r < end; ++r) {
         __fp16 * out = (__fp16 *) ((uint8_t *) dst->data + (kv_head * G + h_idx) * dst->nb[1] +
                                    (q_start + q_idx) * dst->nb[2] + ib3 * dst->nb[3]);
 
@@ -851,6 +994,12 @@ static void fa_o_store_thread_f16(unsigned int n, unsigned int i, void * data) {
                 *(HVX_UVector *) (out + d * 64) = Q6_V_hi_W(vp);
             }
         }
+
+        h_idx++;
+        if (h_idx == G) {
+            h_idx = 0;
+            q_idx++;
+        }
     }
     htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_O_PROC, (uint16_t) (args->q_start * G + start));
 }
@@ -879,6 +1028,7 @@ static void fa_phase_o_store(struct hmx_fa_context *   factx,
 
 typedef struct {
     struct hmx_fa_context *   factx;
+    size_t                    buf_idx;
     size_t                    kv_rows;
     size_t                    n_rows_g;
     size_t                    n_col_tiles;
@@ -960,8 +1110,8 @@ static inline void fa_softmax_impl(
             uint32_t r0 = r / HMX_FP16_TILE_N_ROWS;
             uint32_t r1 = r % HMX_FP16_TILE_N_ROWS;
 
-            const __fp16 * s_ld_base = factx->vtcm_s_tiles + r0 * HMX_FP16_TILE_N_ROWS * Bc;
-            __fp16 *       p_st_base = factx->vtcm_p_tiles + r0 * HMX_FP16_TILE_N_ROWS * Bc;
+            const __fp16 * s_ld_base = factx->vtcm_s_tiles[args->buf_idx] + r0 * HMX_FP16_TILE_N_ROWS * Bc;
+            __fp16 *       p_st_base = factx->vtcm_p_tiles[args->buf_idx] + r0 * HMX_FP16_TILE_N_ROWS * Bc;
 
             // Decode 2 rows from S tiles into per-thread row buffers
             if (has_softcap) {
@@ -983,7 +1133,26 @@ static inline void fa_softmax_impl(
                     my_row_buf1[ci] = hvx_vec_mul_f16_f16(t1, v_cap);
                 }
             } else {
-                for (size_t c = 0; c < kv_rows; c += 64) {
+                size_t c = 0;
+                for (; c + 64 < kv_rows; c += 128) {
+                    size_t             ci0       = c / 64;
+                    size_t             ci1       = ci0 + 1;
+                    const __fp16 *     in_dtile0 = s_ld_base + ci0 * HMX_FP16_TILE_N_ELMS * 2;
+                    const __fp16 *     in_dtile1 = s_ld_base + ci1 * HMX_FP16_TILE_N_ELMS * 2;
+                    const HVX_Vector * pv_s_in0_0 = ((const HVX_Vector *) in_dtile0) + r1 / 2;
+                    const HVX_Vector * pv_s_in1_0 = pv_s_in0_0 + 16;
+                    const HVX_Vector * pv_s_in0_1 = ((const HVX_Vector *) in_dtile1) + r1 / 2;
+                    const HVX_Vector * pv_s_in1_1 = pv_s_in0_1 + 16;
+
+                    HVX_VectorPair vp_s_drow0 = Q6_W_vdeal_VVR(*pv_s_in1_0, *pv_s_in0_0, -2);
+                    my_row_buf0[ci0]          = Q6_V_lo_W(vp_s_drow0);
+                    my_row_buf1[ci0]          = Q6_V_hi_W(vp_s_drow0);
+
+                    HVX_VectorPair vp_s_drow1 = Q6_W_vdeal_VVR(*pv_s_in1_1, *pv_s_in0_1, -2);
+                    my_row_buf0[ci1]          = Q6_V_lo_W(vp_s_drow1);
+                    my_row_buf1[ci1]          = Q6_V_hi_W(vp_s_drow1);
+                }
+                for (; c < kv_rows; c += 64) {
                     size_t             ci       = c / 64;
                     const __fp16 *     in_dtile = s_ld_base + ci * HMX_FP16_TILE_N_ELMS * 2;
                     const HVX_Vector * pv_s_in0 = ((const HVX_Vector *) in_dtile) + r1 / 2;
@@ -1007,12 +1176,12 @@ static inline void fa_softmax_impl(
 
             HVX_Vector v_s_rowmax0 = v_neg_inf;
             HVX_Vector v_s_rowmax1 = v_neg_inf;
-            for (size_t c = 0; c < kv_rows; c += 64) {
-                size_t         ci          = c / 64;
-                const size_t   ne          = hex_smin(kv_rows - c, 64);
-                HVX_VectorPred q_tail_keep = Q6_Q_vsetq2_R(ne * sizeof(__fp16));
+            if (has_mask) {
+                for (size_t c = 0; c < kv_rows; c += 64) {
+                    size_t         ci          = c / 64;
+                    const size_t   ne          = hex_smin(kv_rows - c, 64);
+                    HVX_VectorPred q_tail_keep = Q6_Q_vsetq2_R(ne * sizeof(__fp16));
 
-                if (has_mask) {
                     HVX_Vector v_mask0, v_mask1;
 
                     if (mask_broadcast) {
@@ -1066,15 +1235,31 @@ static inline void fa_softmax_impl(
                         my_row_buf0[ci] = Q6_V_vmux_QVV(q_keep0, hvx_vec_add_f16_f16(my_row_buf0[ci], v_mask0_scaled), v_neg_inf);
                         my_row_buf1[ci] = Q6_V_vmux_QVV(q_keep1, hvx_vec_add_f16_f16(my_row_buf1[ci], v_mask1_scaled), v_neg_inf);
                     }
-                } else {
+
+                    v_s_rowmax0 = Q6_Vhf_vmax_VhfVhf(v_s_rowmax0, my_row_buf0[ci]);
+                    v_s_rowmax1 = Q6_Vhf_vmax_VhfVhf(v_s_rowmax1, my_row_buf1[ci]);
+                }
+            } else {
+                size_t c = 0;
+                for (; c + 64 < kv_rows; c += 128) {
+                    size_t ci0 = c / 64;
+                    size_t ci1 = ci0 + 1;
+                    v_s_rowmax0 = Q6_Vhf_vmax_VhfVhf(v_s_rowmax0, my_row_buf0[ci0]);
+                    v_s_rowmax1 = Q6_Vhf_vmax_VhfVhf(v_s_rowmax1, my_row_buf1[ci0]);
+                    v_s_rowmax0 = Q6_Vhf_vmax_VhfVhf(v_s_rowmax0, my_row_buf0[ci1]);
+                    v_s_rowmax1 = Q6_Vhf_vmax_VhfVhf(v_s_rowmax1, my_row_buf1[ci1]);
+                }
+                for (; c < kv_rows; c += 64) {
+                    size_t         ci          = c / 64;
+                    const size_t   ne          = hex_smin(kv_rows - c, 64);
+                    HVX_VectorPred q_tail_keep = Q6_Q_vsetq2_R(ne * sizeof(__fp16));
                     if (ne < 64) {
                         my_row_buf0[ci] = Q6_V_vmux_QVV(q_tail_keep, my_row_buf0[ci], v_neg_inf);
                         my_row_buf1[ci] = Q6_V_vmux_QVV(q_tail_keep, my_row_buf1[ci], v_neg_inf);
                     }
+                    v_s_rowmax0 = Q6_Vhf_vmax_VhfVhf(v_s_rowmax0, my_row_buf0[ci]);
+                    v_s_rowmax1 = Q6_Vhf_vmax_VhfVhf(v_s_rowmax1, my_row_buf1[ci]);
                 }
-
-                v_s_rowmax0 = Q6_Vhf_vmax_VhfVhf(v_s_rowmax0, my_row_buf0[ci]);
-                v_s_rowmax1 = Q6_Vhf_vmax_VhfVhf(v_s_rowmax1, my_row_buf1[ci]);
             }
 
             v_s_rowmax0 = hvx_vec_reduce_max_f16(v_s_rowmax0);
@@ -1121,8 +1306,48 @@ static inline void fa_softmax_impl(
             HVX_Vector       v_p_rowsum0 = v_zero;
             HVX_Vector       v_p_rowsum1 = v_zero;
 
-            for (size_t c = 0; c < kv_rows; c += 64) {
-                size_t     ci           = c / 64;
+            size_t c = 0;
+            for (; c + 64 < kv_rows; c += 128) {
+                size_t     ci0          = c / 64;
+                size_t     ci1          = ci0 + 1;
+
+                HVX_Vector v_s_minus_m0_0 = Q6_Vqf16_vsub_VhfVhf(my_row_buf0[ci0], v_dup_m0);
+                HVX_Vector v_s_minus_m1_0 = Q6_Vqf16_vsub_VhfVhf(my_row_buf1[ci0], v_dup_m1);
+                HVX_Vector v_s_minus_m0_1 = Q6_Vqf16_vsub_VhfVhf(my_row_buf0[ci1], v_dup_m0);
+                HVX_Vector v_s_minus_m1_1 = Q6_Vqf16_vsub_VhfVhf(my_row_buf1[ci1], v_dup_m1);
+
+                HVX_Vector v_p_row0_hf_0  = hvx_vec_exp2_f16(Q6_Vhf_equals_Vqf16(v_s_minus_m0_0));
+                HVX_Vector v_p_row1_hf_0  = hvx_vec_exp2_f16(Q6_Vhf_equals_Vqf16(v_s_minus_m1_0));
+                HVX_Vector v_p_row0_hf_1  = hvx_vec_exp2_f16(Q6_Vhf_equals_Vqf16(v_s_minus_m0_1));
+                HVX_Vector v_p_row1_hf_1  = hvx_vec_exp2_f16(Q6_Vhf_equals_Vqf16(v_s_minus_m1_1));
+
+                __fp16 *     out_dtile0  = p_st_base + ci0 * HMX_FP16_TILE_N_ELMS * 2;
+                __fp16 *     out_dtile1  = p_st_base + ci1 * HMX_FP16_TILE_N_ELMS * 2;
+                HVX_Vector * pv_p_out0_0  = ((HVX_Vector *) out_dtile0) + r1 / 2;
+                HVX_Vector * pv_p_out1_0  = pv_p_out0_0 + 16;
+                HVX_Vector * pv_p_out0_1  = ((HVX_Vector *) out_dtile1) + r1 / 2;
+                HVX_Vector * pv_p_out1_1  = pv_p_out0_1 + 16;
+
+                HVX_VectorPair vp_p_dual0 = Q6_W_vshuff_VVR(v_p_row1_hf_0, v_p_row0_hf_0, -2);
+                *pv_p_out0_0               = Q6_V_lo_W(vp_p_dual0);
+                *pv_p_out1_0               = Q6_V_hi_W(vp_p_dual0);
+
+                HVX_VectorPair vp_p_dual1 = Q6_W_vshuff_VVR(v_p_row1_hf_1, v_p_row0_hf_1, -2);
+                *pv_p_out0_1               = Q6_V_lo_W(vp_p_dual1);
+                *pv_p_out1_1               = Q6_V_hi_W(vp_p_dual1);
+
+                HVX_VectorPair vp_p0_0 = hvx_vec_f16_to_f32_shuff(v_p_row0_hf_0);
+                HVX_VectorPair vp_p1_0 = hvx_vec_f16_to_f32_shuff(v_p_row1_hf_0);
+                HVX_VectorPair vp_p0_1 = hvx_vec_f16_to_f32_shuff(v_p_row0_hf_1);
+                HVX_VectorPair vp_p1_1 = hvx_vec_f16_to_f32_shuff(v_p_row1_hf_1);
+
+                v_p_rowsum0 = Q6_Vqf32_vadd_Vqf32Vqf32(v_p_rowsum0, Q6_Vqf32_vadd_VsfVsf(Q6_V_lo_W(vp_p0_0), Q6_V_hi_W(vp_p0_0)));
+                v_p_rowsum0 = Q6_Vqf32_vadd_Vqf32Vqf32(v_p_rowsum0, Q6_Vqf32_vadd_VsfVsf(Q6_V_lo_W(vp_p0_1), Q6_V_hi_W(vp_p0_1)));
+                v_p_rowsum1 = Q6_Vqf32_vadd_Vqf32Vqf32(v_p_rowsum1, Q6_Vqf32_vadd_VsfVsf(Q6_V_lo_W(vp_p1_0), Q6_V_hi_W(vp_p1_0)));
+                v_p_rowsum1 = Q6_Vqf32_vadd_Vqf32Vqf32(v_p_rowsum1, Q6_Vqf32_vadd_VsfVsf(Q6_V_lo_W(vp_p1_1), Q6_V_hi_W(vp_p1_1)));
+            }
+            for (size_t c_rem = c; c_rem < kv_rows; c_rem += 64) {
+                size_t     ci           = c_rem / 64;
                 HVX_Vector v_s_minus_m0 = Q6_Vqf16_vsub_VhfVhf(my_row_buf0[ci], v_dup_m0);
                 HVX_Vector v_s_minus_m1 = Q6_Vqf16_vsub_VhfVhf(my_row_buf1[ci], v_dup_m1);
 
@@ -1281,7 +1506,7 @@ static __attribute__((noinline)) void fa_build_d_diag_inv_l(struct hmx_fa_contex
             v_content = Q6_V_vror_VR(v_content, 64);
         }
 
-        __fp16 * out_base = factx->vtcm_d_tiles + i * (n_row_tiles_g_br + 1) * HMX_FP16_TILE_N_ELMS;
+        __fp16 * out_base = factx->vtcm_d_inv_l + i * (n_row_tiles_g_br + 1) * HMX_FP16_TILE_N_ELMS;
         Q6_vscatter_QRMVhV(q_32_mask, (size_t) out_base, HMX_FP16_TILE_SIZE - 1, v_offsets, v_content);
     }
 }
@@ -1514,6 +1739,27 @@ static void fa_pop_mask_dma_gqa(dma_queue * dma, uint32_t G) {
     }
 }
 
+static inline void fa_prefetch_block(dma_queue * dma, const struct htp_tensor * k, const struct htp_tensor * v, const struct htp_tensor * mask,
+                                     uint32_t b, size_t Bc, size_t size_k_row_padded, size_t size_k_row, size_t size_v_row_padded, size_t size_v_row,
+                                     uint32_t ik2, uint32_t ik3, uint32_t iv2, uint32_t iv3, uint32_t q_start, uint32_t im3, uint32_t kv_head, uint32_t G,
+                                     size_t m_line_bytes, size_t n_rows_q, size_t nek1, size_t prefetch_buf, struct hmx_fa_context * factx) {
+    const uint32_t  prefetch_start = b * Bc;
+    const uint32_t  prefetch_rows  = hex_smin(Bc, nek1 - prefetch_start);
+    const uint8_t * k_prefetch_src = (const uint8_t *) k->data + prefetch_start * k->nb[1] + ik2 * k->nb[2] + ik3 * k->nb[3];
+    dma_queue_push(dma, dma_make_ptr(factx->vtcm_k_fp16[prefetch_buf], k_prefetch_src), size_k_row_padded, k->nb[1], size_k_row, prefetch_rows);
+    const uint8_t * v_prefetch_src = (const uint8_t *) v->data + prefetch_start * v->nb[1] + iv2 * v->nb[2] + iv3 * v->nb[3];
+    dma_queue_push(dma, dma_make_ptr(factx->vtcm_v_fp16[prefetch_buf], v_prefetch_src), size_v_row_padded, v->nb[1], size_v_row, prefetch_rows);
+
+    if (mask) {
+        if (__builtin_expect(factx->mask_broadcast, true)) {
+            const uint8_t * ms_src = (const uint8_t *) mask->data + q_start * mask->nb[1] + im3 * mask->nb[3] + prefetch_start * sizeof(__fp16);
+            dma_cache_push(dma, &factx->m_cache, ms_src, m_line_bytes, mask->nb[1], prefetch_rows * sizeof(__fp16), n_rows_q);
+        } else {
+            fa_push_mask_dma_gqa(dma, mask, q_start, im3, prefetch_start, kv_head, G, m_line_bytes, prefetch_rows, n_rows_q, factx);
+        }
+    }
+}
+
 // ============================================================================
 // Core HMX flash attention algorithm (GQA-merged)
 // ============================================================================
@@ -1612,7 +1858,7 @@ int hmx_flash_attn_ext(struct htp_ops_context * octx) {
     // Build the VTCM layout once (shared with the host estimator) and place every
     // scratch buffer at its computed offset.
     struct hmx_fa_vtcm_layout L;
-    hmx_fa_vtcm_layout_build(&L, G, DK, DV, Br, Bc, n_threads, pipeline);
+    hmx_fa_vtcm_layout_build(&L, G, DK, DV, Br, Bc, n_threads, pipeline, factx.is_q_fp32);
 
     if (L.total_bytes > ctx->vtcm_size) {
         return HTP_STATUS_VTCM_TOO_SMALL;
@@ -1620,6 +1866,7 @@ int hmx_flash_attn_ext(struct htp_ops_context * octx) {
 
     uint8_t * const base = ctx->vtcm_base;
 
+    factx.vtcm_q_dma          = VTCM_LAYOUT_PTR(__fp16, base, L.off_q_dma);
     factx.vtcm_q_tiles        = VTCM_LAYOUT_PTR(__fp16, base, L.off_q_tiles);
     factx.vtcm_o_tiles[0]     = VTCM_LAYOUT_PTR(__fp16, base, L.off_o_tiles[0]);
     factx.vtcm_o_tiles[1]     = VTCM_LAYOUT_PTR(__fp16, base, L.off_o_tiles[1]);
@@ -1627,12 +1874,16 @@ int hmx_flash_attn_ext(struct htp_ops_context * octx) {
     factx.vtcm_k_fp16[1]      = VTCM_LAYOUT_PTR(__fp16, base, L.off_k_fp16[1]);
     factx.vtcm_v_fp16[0]      = VTCM_LAYOUT_PTR(__fp16, base, L.off_v_fp16[0]);
     factx.vtcm_v_fp16[1]      = VTCM_LAYOUT_PTR(__fp16, base, L.off_v_fp16[1]);
-    factx.vtcm_k_tiles        = VTCM_LAYOUT_PTR(__fp16, base, L.off_k_tiles);
+    factx.vtcm_k_tiles[0]     = VTCM_LAYOUT_PTR(__fp16, base, L.off_k_tiles[0]);
+    factx.vtcm_k_tiles[1]     = VTCM_LAYOUT_PTR_OPTIONAL(__fp16, base, L.off_k_tiles[1], pipeline);
     factx.vtcm_v_tiles[0]     = VTCM_LAYOUT_PTR(__fp16, base, L.off_v_tiles[0]);
     factx.vtcm_v_tiles[1]     = VTCM_LAYOUT_PTR_OPTIONAL(__fp16, base, L.off_v_tiles[1], pipeline);
-    factx.vtcm_s_tiles        = VTCM_LAYOUT_PTR(__fp16, base, L.off_s_tiles);
-    factx.vtcm_p_tiles        = VTCM_LAYOUT_PTR(__fp16, base, L.off_p_tiles);
+    factx.vtcm_s_tiles[0]     = VTCM_LAYOUT_PTR(__fp16, base, L.off_s_tiles[0]);
+    factx.vtcm_s_tiles[1]     = VTCM_LAYOUT_PTR_OPTIONAL(__fp16, base, L.off_s_tiles[1], pipeline);
+    factx.vtcm_p_tiles[0]     = VTCM_LAYOUT_PTR(__fp16, base, L.off_p_tiles[0]);
+    factx.vtcm_p_tiles[1]     = VTCM_LAYOUT_PTR_OPTIONAL(__fp16, base, L.off_p_tiles[1], pipeline);
     factx.vtcm_d_tiles        = VTCM_LAYOUT_PTR(__fp16, base, L.off_d_tiles);
+    factx.vtcm_d_inv_l        = VTCM_LAYOUT_PTR(__fp16, base, L.off_d_inv_l);
     factx.vtcm_m_vec          = VTCM_LAYOUT_PTR(HVX_Vector, base, L.off_m_vec);
     factx.vtcm_l_vec          = VTCM_LAYOUT_PTR(HVX_Vector, base, L.off_l_vec);
     factx.vtcm_s_rowmax       = VTCM_LAYOUT_PTR(HVX_Vector, base, L.off_s_rowmax);
@@ -1670,6 +1921,12 @@ int hmx_flash_attn_ext(struct htp_ops_context * octx) {
 
     const size_t qo_element_size = factx.is_q_fp32 ? sizeof(float) : sizeof(__fp16);
 
+    const bool q_transposed                 = q->nb[1] < q->nb[2];
+    const size_t q_src_stride               = q_transposed ? q->nb[2] : q->nb[1];
+    const size_t q_row_bytes_untransposed   = factx.G * factx.DK * qo_element_size;
+    const size_t q_row_bytes_trans_factor   = factx.DK * qo_element_size;
+    const uint32_t kv_rows0                 = hex_smin(Bc, nek1);
+
     // ======== Reusable job descriptors for pipeline ========
     hmx_fa_qk_job_t       qk_job;
     hmx_fa_o_update_job_t ou_job;
@@ -1690,34 +1947,34 @@ int hmx_flash_attn_ext(struct htp_ops_context * octx) {
                 const uint32_t iv2 = kv_head;
                 const uint32_t iv3 = fastdiv(ib3, &kparams->broadcast_rv3);
 
-                // 1. Push Q DMA (if Q DMA is used)
-                const size_t o_tile_bytes = factx.o_tile_bytes;
-                const bool use_q_dma = (2 * o_tile_bytes >= factx.g_br * factx.DK * (factx.is_q_fp32 ? 4 : 2));
-                if (use_q_dma) {
-                    const bool q_transposed = q->nb[1] < q->nb[2];
-                    const uint8_t * q_ptr = (const uint8_t *) q->data + q_start * q->nb[1] + (kv_head * factx.G) * q->nb[2] + ib3 * q->nb[3];
-                    const size_t el_size = factx.is_q_fp32 ? sizeof(float) : sizeof(__fp16);
-                    const size_t q_row_bytes = q_transposed ? n_rows_q * factx.DK * el_size : factx.G * factx.DK * el_size;
-                    const size_t src_stride  = q_transposed ? q->nb[2] : q->nb[1];
+                // 1. Push Q and KV DMAs for the very first iteration.
+                // Subsequent iterations are enqueued early at the end of the previous iteration.
+                if (ib3 == 0 && q_start == 0 && kv_head == 0) {
+                    const uint8_t * q_ptr = (const uint8_t *) q->data;
+                    const size_t q_row_bytes = q_transposed ? n_rows_q * q_row_bytes_trans_factor : q_row_bytes_untransposed;
                     const size_t n_rows      = q_transposed ? factx.G : n_rows_q;
-                    dma_queue_push(dma, dma_make_ptr(factx.vtcm_o_tiles[0], q_ptr), q_row_bytes, hex_smax(src_stride, q_row_bytes), q_row_bytes, n_rows);
+                    dma_queue_push(dma, dma_make_ptr(factx.vtcm_q_dma, q_ptr), q_row_bytes, hex_smax(q_src_stride, q_row_bytes), q_row_bytes, n_rows);
+
+                    if (factx.n_kv_blocks > 0) {
+                        const uint8_t * k_src = (const uint8_t *) k->data + ik2 * k->nb[2] + ik3 * k->nb[3];
+                        dma_queue_push(dma, dma_make_ptr(factx.vtcm_k_fp16[0], k_src), size_k_row_padded, k->nb[1], size_k_row, kv_rows0);
+
+                        const uint8_t * v_src = (const uint8_t *) v->data + iv2 * v->nb[2] + iv3 * v->nb[3];
+                        dma_queue_push(dma, dma_make_ptr(factx.vtcm_v_fp16[0], v_src), size_v_row_padded, v->nb[1], size_v_row, kv_rows0);
+
+                        if (factx.pipeline && mask) {
+                            if (__builtin_expect(factx.mask_broadcast, true)) {
+                                const uint8_t * ms_src = (const uint8_t *) mask->data + q_start * mask->nb[1] + im3 * mask->nb[3] + 0;
+                                dma_cache_push(dma, &factx.m_cache, ms_src, m_line_bytes, mask->nb[1], kv_rows0 * sizeof(__fp16), n_rows_q);
+                            } else {
+                                fa_push_mask_dma_gqa(dma, mask, q_start, im3, 0, kv_head, G, m_line_bytes, kv_rows0, n_rows_q, &factx);
+                            }
+                        }
+                    }
                 }
 
-                // 2. Prefetch first KV block
-                if (factx.n_kv_blocks > 0) {
-                    const uint32_t kv_rows0 = hex_smin(Bc, nek1);
-
-                    const uint8_t * k_src = (const uint8_t *) k->data + ik2 * k->nb[2] + ik3 * k->nb[3];
-                    dma_queue_push(dma, dma_make_ptr(factx.vtcm_k_fp16[0], k_src), size_k_row_padded, k->nb[1], size_k_row, kv_rows0);
-
-                    const uint8_t * v_src = (const uint8_t *) v->data + iv2 * v->nb[2] + iv3 * v->nb[3];
-                    dma_queue_push(dma, dma_make_ptr(factx.vtcm_v_fp16[0], v_src), size_v_row_padded, v->nb[1], size_v_row, kv_rows0);
-                }
-
-                // 3. Pop Q DMA (blocks until Q is loaded)
-                if (use_q_dma) {
-                    dma_queue_pop(dma);
-                }
+                // 2. Pop Q DMA (blocks until Q is loaded)
+                dma_queue_pop(dma);
 
                 // ---- Load Q block & Initialize per-block state ----
                 fa_phase_q_load(&factx, q, q_start, kv_head, ib3, n_rows_g);
@@ -1738,76 +1995,40 @@ int hmx_flash_attn_ext(struct htp_ops_context * octx) {
                 hmx_queue_t hmx_q = ctx->hmx_queue;
 
                 if (factx.pipeline) {
-                    // Pipeline path
+                    // Double-buffered job structs because HMX queue runs asynchronously
+                    hmx_fa_qk_job_t qk_job[2];
+                    hmx_fa_o_update_job_t ou_job[2];
+
+                    // Prefetch block 1 early if there are multiple blocks
+                    if (factx.n_kv_blocks > 1) {
+                        fa_prefetch_block(dma, k, v, mask, 1, Bc, size_k_row_padded, size_k_row, size_v_row_padded, size_v_row,
+                                          ik2, ik3, iv2, iv3, q_start, im3, kv_head, G, m_line_bytes, n_rows_q, nek1, 1, &factx);
+                    }
+
+                    // Prep and start QK-dot(0)
+                    void * curr_k0 = dma_queue_pop(dma).dst;
+                    fa_phase_k_interleave(&factx, kv_rows0, k_src_stride, curr_k0, 0, 0);
+
+                    qk_job[0].q_tiles        = factx.vtcm_q_tiles;
+                    qk_job[0].k_tiles        = factx.vtcm_k_tiles[0];
+                    qk_job[0].s_tiles        = factx.vtcm_s_tiles[0];
+                    qk_job[0].n_row_tiles    = n_row_tiles;
+                    qk_job[0].n_col_tiles    = hmx_ceil_div(kv_rows0, HMX_FP16_TILE_N_COLS);
+                    qk_job[0].n_dot_tiles    = DK / 32;
+                    qk_job[0].n_tiles_per_bc = n_tiles_per_bc;
+                    qk_job[0].hmx_scales     = factx.vtcm_hmx_scales_qk;
+                    hmx_queue_push(hmx_q, hmx_queue_make_desc(hmx_fa_qk_dot_worker, &qk_job[0]));
+
                     for (uint32_t kv_blk = 0; kv_blk < factx.n_kv_blocks; ++kv_blk) {
                         const uint32_t kv_start    = kv_blk * Bc;
                         const uint32_t kv_rows     = hex_smin(Bc, nek1 - kv_start);
                         const size_t   n_col_tiles = hmx_ceil_div(kv_rows, HMX_FP16_TILE_N_COLS);
 
-                        // Push mask DMA
-                        if (mask) {
-                            if (__builtin_expect(factx.mask_broadcast, true)) {
-                                const uint8_t * ms_src = (const uint8_t *) mask->data + q_start * mask->nb[1] + im3 * mask->nb[3] + kv_start * sizeof(__fp16);
-                                dma_cache_push(dma, &factx.m_cache, ms_src, m_line_bytes, mask->nb[1], kv_rows * sizeof(__fp16), n_rows_q);
-                            } else {
-                                fa_push_mask_dma_gqa(dma, mask, q_start, im3, kv_start, kv_head, G, m_line_bytes, kv_rows, n_rows_q, &factx);
-                            }
-                        }
-
-                        // Prefetch next KV block early
-                        if (kv_blk + 1 < factx.n_kv_blocks) {
-                            const uint32_t  prefetch_start = (kv_blk + 1) * Bc;
-                            const uint32_t  prefetch_rows  = hex_smin(Bc, nek1 - prefetch_start);
-                            const size_t    prefetch_buf   = 1 - buf_idx;
-                            const uint8_t * k_prefetch_src = (const uint8_t *) k->data + prefetch_start * k->nb[1] + ik2 * k->nb[2] + ik3 * k->nb[3];
-                            dma_queue_push(dma, dma_make_ptr(factx.vtcm_k_fp16[prefetch_buf], k_prefetch_src), size_k_row_padded, k->nb[1], size_k_row, prefetch_rows);
-                            const uint8_t * v_prefetch_src = (const uint8_t *) v->data + prefetch_start * v->nb[1] + iv2 * v->nb[2] + iv3 * v->nb[3];
-                            dma_queue_push(dma, dma_make_ptr(factx.vtcm_v_fp16[prefetch_buf], v_prefetch_src), size_v_row_padded, v->nb[1], size_v_row, prefetch_rows);
-                        }
-
-                        // ---- Phase 1: K_int ----
-                        if (kv_blk > 0) {
-                            ou_job.o_curr           = o_tile_curr;
-                            ou_job.o_prev           = o_tile_prev;
-                            ou_job.p_tiles          = factx.vtcm_p_tiles;
-                            ou_job.v_tiles          = factx.vtcm_v_tiles[1 - buf_idx];
-                            ou_job.d_tiles          = factx.vtcm_d_tiles;
-                            ou_job.hmx_scales       = factx.vtcm_hmx_scales_id;
-                            ou_job.n_row_tiles      = n_row_tiles;
-                            ou_job.n_col_tiles      = hmx_ceil_div(hex_smin(Bc, nek1 - (kv_blk - 1) * Bc), HMX_FP16_TILE_N_COLS);
-                            ou_job.n_row_tiles_g_br = n_row_tiles_g_br;
-                            ou_job.n_tiles_per_bc   = n_tiles_per_bc;
-                            ou_job.DV               = DV;
-                            hmx_queue_push(hmx_q, hmx_queue_make_desc(hmx_fa_o_update_worker, &ou_job));
-                        }
-
-                        // Wait for current K DMA and interleave
-                        void * curr_k = dma_queue_pop(dma).dst;
-                        fa_phase_k_interleave(&factx, kv_rows, k_src_stride, curr_k, kv_start);
-
-                        // ---- Phase 2: qk_dot ----
-                        qk_job.q_tiles        = factx.vtcm_q_tiles;
-                        qk_job.k_tiles        = factx.vtcm_k_tiles;
-                        qk_job.s_tiles        = factx.vtcm_s_tiles;
-                        qk_job.n_row_tiles    = n_row_tiles;
-                        qk_job.n_col_tiles    = n_col_tiles;
-                        qk_job.n_dot_tiles    = DK / 32;
-                        qk_job.n_tiles_per_bc = n_tiles_per_bc;
-                        qk_job.hmx_scales     = factx.vtcm_hmx_scales_qk;
-                        hmx_queue_push(hmx_q, hmx_queue_make_desc(hmx_fa_qk_dot_worker, &qk_job));
-
-                        // Wait for current V DMA and interleave
+                        // ---- 1. Pop and run V-prep for current block ----
                         void * curr_v = dma_queue_pop(dma).dst;
                         fa_phase_v_interleave(&factx, kv_rows, v_src_stride, curr_v, factx.vtcm_v_tiles[buf_idx], n_tiles_per_bc, kv_start);
 
-                        if (kv_blk > 0) {
-                            hmx_queue_pop(hmx_q);
-                            hex_swap_ptr((void **) &o_tile_curr, (void **) &o_tile_prev);
-                        }
-
-                        hmx_queue_pop(hmx_q);
-
-                        // ---- Phase 3: softmax + build_D ----
+                        // ---- 2. Pop and run mask-prep for current block ----
                         __fp16 * current_mask_vtcm = NULL;
                         if (mask) {
                             if (__builtin_expect(factx.mask_broadcast, true)) {
@@ -1818,9 +2039,34 @@ int hmx_flash_attn_ext(struct htp_ops_context * octx) {
                             }
                         }
 
+                        // ---- 3. Pop and run K-prep for next block & push next QK-dot ----
+                        if (kv_blk + 1 < factx.n_kv_blocks) {
+                            const uint32_t next_start = (kv_blk + 1) * Bc;
+                            const uint32_t next_rows  = hex_smin(Bc, nek1 - next_start);
+                            const size_t   next_buf   = 1 - buf_idx;
+
+                            void * next_k = dma_queue_pop(dma).dst;
+                            fa_phase_k_interleave(&factx, next_rows, k_src_stride, next_k, next_start, next_buf);
+
+                            qk_job[next_buf].q_tiles        = factx.vtcm_q_tiles;
+                            qk_job[next_buf].k_tiles        = factx.vtcm_k_tiles[next_buf];
+                            qk_job[next_buf].s_tiles        = factx.vtcm_s_tiles[next_buf];
+                            qk_job[next_buf].n_row_tiles    = n_row_tiles;
+                            qk_job[next_buf].n_col_tiles    = hmx_ceil_div(next_rows, HMX_FP16_TILE_N_COLS);
+                            qk_job[next_buf].n_dot_tiles    = DK / 32;
+                            qk_job[next_buf].n_tiles_per_bc = n_tiles_per_bc;
+                            qk_job[next_buf].hmx_scales     = factx.vtcm_hmx_scales_qk;
+                            hmx_queue_push(hmx_q, hmx_queue_make_desc(hmx_fa_qk_dot_worker, &qk_job[next_buf]));
+                        }
+
+                        // ---- 4. Wait for current block's QK-dot to finish ----
+                        hmx_queue_pop(hmx_q);
+
+                        // ---- 5. Phase 2: softmax + build_D ----
                         fa_softmax_args_t sargs;
                         memset(&sargs, 0, sizeof(sargs));
                         sargs.factx                = &factx;
+                        sargs.buf_idx              = buf_idx;
                         sargs.kv_rows              = kv_rows;
                         sargs.n_rows_g             = n_rows_g;
                         sargs.n_col_tiles          = n_col_tiles;
@@ -1838,8 +2084,39 @@ int hmx_flash_attn_ext(struct htp_ops_context * octx) {
                         sargs.mask_vtcm            = current_mask_vtcm;
                         sargs.mask_vtcm_row_stride = factx.mask_buf_row_stride;
                         sargs.slopes               = factx.vtcm_slopes;
+
+                        // Start HMX O update for block kv_blk - 1 (reads P[1 - buf_idx], V[1 - buf_idx])
+                        if (kv_blk > 0) {
+                            const size_t prev_buf = 1 - buf_idx;
+                            ou_job[prev_buf].o_curr           = o_tile_curr;
+                            ou_job[prev_buf].o_prev           = o_tile_prev;
+                            ou_job[prev_buf].p_tiles          = factx.vtcm_p_tiles[prev_buf];
+                            ou_job[prev_buf].v_tiles          = factx.vtcm_v_tiles[prev_buf];
+                            ou_job[prev_buf].d_tiles          = factx.vtcm_d_tiles;
+                            ou_job[prev_buf].hmx_scales       = factx.vtcm_hmx_scales_id;
+                            ou_job[prev_buf].n_row_tiles      = n_row_tiles;
+                            ou_job[prev_buf].n_col_tiles      = hmx_ceil_div(hex_smin(Bc, nek1 - (kv_blk - 1) * Bc), HMX_FP16_TILE_N_COLS);
+                            ou_job[prev_buf].n_row_tiles_g_br = n_row_tiles_g_br;
+                            ou_job[prev_buf].n_tiles_per_bc   = n_tiles_per_bc;
+                            ou_job[prev_buf].DV               = DV;
+                            hmx_queue_push(hmx_q, hmx_queue_make_desc(hmx_fa_o_update_worker, &ou_job[prev_buf]));
+                        }
+
+                        // Run Softmax on HVX (blocking call)
                         fa_phase_softmax_and_build_d(&factx, &sargs, n_row_tiles, n_row_tiles_g_br);
 
+                        // Wait for HMX O update for block kv_blk - 1 to finish
+                        if (kv_blk > 0) {
+                            hmx_queue_pop(hmx_q);
+                            hex_swap_ptr((void **) &o_tile_curr, (void **) &o_tile_prev);
+                        }
+
+                        // Prefetch block kv_blk + 2
+                        if (kv_blk + 2 < factx.n_kv_blocks) {
+                            fa_prefetch_block(dma, k, v, mask, kv_blk + 2, Bc, size_k_row_padded, size_k_row, size_v_row_padded, size_v_row,
+                                              ik2, ik3, iv2, iv3, q_start, im3, kv_head, G, m_line_bytes, n_rows_q, nek1, buf_idx, &factx);
+                        }
+
                         buf_idx = 1 - buf_idx;
                     }
 
@@ -1847,18 +2124,23 @@ int hmx_flash_attn_ext(struct htp_ops_context * octx) {
                     if (factx.n_kv_blocks > 0) {
                         const uint32_t last_blk = factx.n_kv_blocks - 1;
                         const size_t last_cols  = hmx_ceil_div(hex_smin(Bc, nek1 - last_blk * Bc), HMX_FP16_TILE_N_COLS);
-                        ou_job.o_curr           = o_tile_curr;
-                        ou_job.o_prev           = o_tile_prev;
-                        ou_job.p_tiles          = factx.vtcm_p_tiles;
-                        ou_job.v_tiles          = factx.vtcm_v_tiles[1 - buf_idx];
-                        ou_job.d_tiles          = factx.vtcm_d_tiles;
-                        ou_job.hmx_scales       = factx.vtcm_hmx_scales_id;
-                        ou_job.n_row_tiles      = n_row_tiles;
-                        ou_job.n_col_tiles      = last_cols;
-                        ou_job.n_row_tiles_g_br = n_row_tiles_g_br;
-                        ou_job.n_tiles_per_bc   = n_tiles_per_bc;
-                        ou_job.DV               = DV;
-                        hmx_queue_push(hmx_q, hmx_queue_make_desc(hmx_fa_o_update_worker, &ou_job));
+                        ou_job[0].o_curr           = o_tile_curr;
+                        ou_job[0].o_prev           = o_tile_prev;
+                        ou_job[0].p_tiles          = factx.vtcm_p_tiles[1 - buf_idx];
+                        ou_job[0].v_tiles          = factx.vtcm_v_tiles[1 - buf_idx];
+                        ou_job[0].d_tiles          = factx.vtcm_d_tiles;
+                        ou_job[0].hmx_scales       = factx.vtcm_hmx_scales_id;
+                        ou_job[0].n_row_tiles      = n_row_tiles;
+                        ou_job[0].n_col_tiles      = last_cols;
+                        ou_job[0].n_row_tiles_g_br = n_row_tiles_g_br;
+                        ou_job[0].n_tiles_per_bc   = n_tiles_per_bc;
+                        ou_job[0].DV               = DV;
+                        hmx_queue_push(hmx_q, hmx_queue_make_desc(hmx_fa_o_update_worker, &ou_job[0]));
+
+                        // Overlapped: run HVX build diag inv L while HMX is busy executing the update
+                        htp_trace_event_start(tr_hvx, HTP_TRACE_EVT_HVX_O_PROC, (uint16_t) q_start);
+                        fa_build_d_diag_inv_l(&factx, n_row_tiles, n_row_tiles_g_br);
+                        htp_trace_event_stop(tr_hvx, HTP_TRACE_EVT_HVX_O_PROC, (uint16_t) q_start);
                         hmx_queue_pop(hmx_q);
 
                         hex_swap_ptr((void **) &o_tile_curr, (void **) &o_tile_prev);
@@ -1892,12 +2174,12 @@ int hmx_flash_attn_ext(struct htp_ops_context * octx) {
 
                         // Wait for current K DMA and interleave
                         void * curr_k = dma_queue_pop(dma).dst;
-                        fa_phase_k_interleave(&factx, kv_rows, k_src_stride, curr_k, kv_start);
+                        fa_phase_k_interleave(&factx, kv_rows, k_src_stride, curr_k, kv_start, 0);
 
                         {
                             qk_job.q_tiles        = factx.vtcm_q_tiles;
-                            qk_job.k_tiles        = factx.vtcm_k_tiles;
-                            qk_job.s_tiles        = factx.vtcm_s_tiles;
+                            qk_job.k_tiles        = factx.vtcm_k_tiles[0];
+                            qk_job.s_tiles        = factx.vtcm_s_tiles[0];
                             qk_job.n_row_tiles    = n_row_tiles;
                             qk_job.n_col_tiles    = n_col_tiles;
                             qk_job.n_dot_tiles    = (size_t) (DK / 32);
@@ -1948,7 +2230,7 @@ int hmx_flash_attn_ext(struct htp_ops_context * octx) {
                         {
                             ou_job.o_curr           = o_tile_curr;
                             ou_job.o_prev           = o_tile_prev;
-                            ou_job.p_tiles          = factx.vtcm_p_tiles;
+                            ou_job.p_tiles          = factx.vtcm_p_tiles[0];
                             ou_job.v_tiles          = factx.vtcm_v_tiles[0];
                             ou_job.d_tiles          = factx.vtcm_d_tiles;
                             ou_job.hmx_scales       = factx.vtcm_hmx_scales_id;
@@ -1959,6 +2241,12 @@ int hmx_flash_attn_ext(struct htp_ops_context * octx) {
                             ou_job.DV               = DV;
 
                             hmx_queue_push(ctx->hmx_queue, hmx_queue_make_desc(hmx_fa_o_update_worker, &ou_job));
+                            if (kv_blk + 1 == factx.n_kv_blocks) {
+                                // Overlapped: run HVX build diag inv L while HMX is busy executing the update
+                                htp_trace_event_start(tr_hvx, HTP_TRACE_EVT_HVX_O_PROC, (uint16_t) q_start);
+                                fa_build_d_diag_inv_l(&factx, n_row_tiles, n_row_tiles_g_br);
+                                htp_trace_event_stop(tr_hvx, HTP_TRACE_EVT_HVX_O_PROC, (uint16_t) q_start);
+                            }
                             hmx_queue_pop(ctx->hmx_queue);
 
                             hex_swap_ptr((void **) &o_tile_curr, (void **) &o_tile_prev);
@@ -1968,15 +2256,63 @@ int hmx_flash_attn_ext(struct htp_ops_context * octx) {
                     }
                 }
 
+                // Enqueue DMAs for the next iteration early so they overlap with O-PROC
+                uint32_t next_kv_head = kv_head + 1;
+                uint32_t next_q_start = q_start;
+                uint32_t next_ib3     = ib3;
+                if (next_kv_head >= n_kv_heads) {
+                    next_kv_head = 0;
+                    next_q_start = q_start + Br;
+                    if (next_q_start >= neq1) {
+                        next_q_start = 0;
+                        next_ib3     = ib3 + 1;
+                    }
+                }
+                bool has_next = (next_ib3 < neq3);
+
+                if (has_next) {
+                    const uint32_t next_n_rows_q = hex_smin(Br, neq1 - next_q_start);
+                    const uint8_t * next_q_ptr = (const uint8_t *) q->data + next_q_start * q->nb[1] + (next_kv_head * factx.G) * q->nb[2] + next_ib3 * q->nb[3];
+                    const size_t next_q_row_bytes = q_transposed ? next_n_rows_q * q_row_bytes_trans_factor : q_row_bytes_untransposed;
+                    const size_t next_n_rows      = q_transposed ? factx.G : next_n_rows_q;
+                    dma_queue_push(dma, dma_make_ptr(factx.vtcm_q_dma, next_q_ptr), next_q_row_bytes, hex_smax(q_src_stride, next_q_row_bytes), next_q_row_bytes, next_n_rows);
+
+                    if (factx.n_kv_blocks > 0) {
+                        const uint32_t next_ik2 = next_kv_head;
+                        const uint32_t next_iv2 = next_kv_head;
+                        uint32_t next_ik3 = ik3;
+                        uint32_t next_iv3 = iv3;
+                        if (next_ib3 != ib3) {
+                            next_ik3 = fastdiv(next_ib3, &kparams->broadcast_rk3);
+                            next_iv3 = fastdiv(next_ib3, &kparams->broadcast_rv3);
+                        }
+
+                        const uint8_t * next_k_src = (const uint8_t *) k->data + next_ik2 * k->nb[2] + next_ik3 * k->nb[3];
+                        dma_queue_push(dma, dma_make_ptr(factx.vtcm_k_fp16[0], next_k_src), size_k_row_padded, k->nb[1], size_k_row, kv_rows0);
+
+                        const uint8_t * next_v_src = (const uint8_t *) v->data + next_iv2 * v->nb[2] + next_iv3 * v->nb[3];
+                        dma_queue_push(dma, dma_make_ptr(factx.vtcm_v_fp16[0], next_v_src), size_v_row_padded, v->nb[1], size_v_row, kv_rows0);
+
+                        if (factx.pipeline && mask) {
+                            uint32_t next_im3 = im3;
+                            if (next_ib3 != ib3) {
+                                next_im3 = fastmodulo(next_ib3, mask->ne[3], &factx.src3_div3);
+                            }
+                            if (__builtin_expect(factx.mask_broadcast, true)) {
+                                const uint8_t * ms_src = (const uint8_t *) mask->data + next_q_start * mask->nb[1] + next_im3 * mask->nb[3] + 0;
+                                dma_cache_push(dma, &factx.m_cache, ms_src, m_line_bytes, mask->nb[1], kv_rows0 * sizeof(__fp16), next_n_rows_q);
+                            } else {
+                                fa_push_mask_dma_gqa(dma, mask, next_q_start, next_im3, 0, next_kv_head, G, m_line_bytes, kv_rows0, next_n_rows_q, &factx);
+                            }
+                        }
+                    }
+                }
+
                 // ---- Final normalization ----
                 {
-                    htp_trace_event_start(tr_hvx, HTP_TRACE_EVT_HVX_O_PROC, (uint16_t) q_start);
-                    fa_build_d_diag_inv_l(&factx, n_row_tiles, n_row_tiles_g_br);
-                    htp_trace_event_stop(tr_hvx, HTP_TRACE_EVT_HVX_O_PROC, (uint16_t) q_start);
-
                     on_job.o_curr           = o_tile_curr;
                     on_job.o_prev           = o_tile_prev;
-                    on_job.d_tiles          = factx.vtcm_d_tiles;
+                    on_job.d_tiles          = factx.vtcm_d_inv_l;
                     on_job.hmx_scales       = factx.vtcm_hmx_scales_id;
                     on_job.n_row_tiles      = n_row_tiles;
                     on_job.n_row_tiles_g_br = n_row_tiles_g_br;

ggml/src/ggml-hexagon/htp/flash-attn-ops.h

@@ -101,14 +101,16 @@ static_assert(sizeof(struct htp_fa_kernel_params) <= 128, "htp_fa_kernel_params
 struct hmx_fa_vtcm_layout {
     // Byte offsets from vtcm_base for each region.
     size_t off_q_tiles;
+    size_t off_q_dma;
     size_t off_o_tiles[2];
     size_t off_k_fp16[2];
     size_t off_v_fp16[2];
-    size_t off_k_tiles;
-    size_t off_v_tiles[2];     // [1] allocated only when pipeline, else 0
-    size_t off_s_tiles;
-    size_t off_p_tiles;
+    size_t off_k_tiles[2];
+    size_t off_v_tiles[2];
+    size_t off_s_tiles[2];
+    size_t off_p_tiles[2];
     size_t off_d_tiles;
+    size_t off_d_inv_l;
     size_t off_m_vec;
     size_t off_l_vec;
     size_t off_s_rowmax;
@@ -140,7 +142,7 @@ struct hmx_fa_vtcm_layout {
 
 static inline void hmx_fa_vtcm_layout_build(struct hmx_fa_vtcm_layout * L,
                                        size_t gqa_factor, size_t DK, size_t DV,
-                                       size_t Br, size_t Bc, size_t n_threads, bool pipeline) {
+                                       size_t Br, size_t Bc, size_t n_threads, bool pipeline, bool is_q_fp32) {
     const size_t g_br         = hex_align_up(gqa_factor * Br, HMX_FP16_TILE_N_ROWS);
     const size_t q_tile_size  = hex_align_up(g_br * DK   * sizeof(__fp16), HTP_FA_HMX_TILE_SIZE);
     const size_t o_tile_size  = hex_align_up(g_br * DV   * sizeof(__fp16), HTP_FA_HMX_TILE_SIZE);
@@ -149,6 +151,7 @@ static inline void hmx_fa_vtcm_layout_build(struct hmx_fa_vtcm_layout * L,
     const size_t s_tile_size  = hex_align_up(g_br * Bc   * sizeof(__fp16), HTP_FA_HMX_TILE_SIZE);
     const size_t d_tile_size  = hex_align_up(g_br * g_br * sizeof(__fp16), HTP_FA_HMX_TILE_SIZE);
 
+    const size_t q_dma_size   = hex_align_up(g_br * DK * (is_q_fp32 ? sizeof(float) : sizeof(__fp16)), 128);
     const size_t k_dma_size   = hex_align_up(Bc * hex_round_up(DK * sizeof(__fp16), 128), 128);
     const size_t v_dma_size   = hex_align_up(Bc * hex_round_up(DV * sizeof(__fp16), 128), 128);
     const size_t col_vec_size = hex_align_up(g_br * sizeof(float),  256);
@@ -160,27 +163,47 @@ static inline void hmx_fa_vtcm_layout_build(struct hmx_fa_vtcm_layout * L,
 
     size_t off = 0;
 
-    // Section 1: HMX Tiled Buffers (FA_HMX_TILE_SIZE = 2KB Aligned)
+    // Group A (Part 1 - HMX Tiled buffers)
     VTCM_LAYOUT_ALLOC(off, off_q_tiles,       q_tile_size);
     VTCM_LAYOUT_ALLOC(off, off_o_tiles[0],    o_tile_size);
     VTCM_LAYOUT_ALLOC(off, off_o_tiles[1],    o_tile_size);
-    VTCM_LAYOUT_ALLOC(off, off_k_tiles,       k_tile_size);
-    VTCM_LAYOUT_ALLOC(off, off_v_tiles[0],    v_tile_size);
-    VTCM_LAYOUT_ALLOC_OPTIONAL(off, off_v_tiles[1], v_tile_size, pipeline);
-    VTCM_LAYOUT_ALLOC(off, off_s_tiles,       s_tile_size);
-    VTCM_LAYOUT_ALLOC(off, off_p_tiles,       s_tile_size);
     VTCM_LAYOUT_ALLOC(off, off_d_tiles,       d_tile_size);
+    VTCM_LAYOUT_ALLOC(off, off_d_inv_l,       d_tile_size);
 
-    // Section 2: HVX/DMA flat and vector buffers (128B / 256B Aligned)
+    // Group B & C share start offset (Group B tiles must be 2KB aligned)
+    size_t off_group_b_c = hex_align_up(off, HTP_FA_HMX_TILE_SIZE);
+
+    // Group B: Compute-only buffers
+    size_t off_group_b = off_group_b_c;
+    VTCM_LAYOUT_ALLOC(off_group_b, off_k_tiles[0],    k_tile_size);
+    VTCM_LAYOUT_ALLOC_OPTIONAL(off_group_b, off_k_tiles[1], k_tile_size, pipeline);
+    VTCM_LAYOUT_ALLOC(off_group_b, off_v_tiles[0],    v_tile_size);
+    VTCM_LAYOUT_ALLOC_OPTIONAL(off_group_b, off_v_tiles[1], v_tile_size, pipeline);
+    VTCM_LAYOUT_ALLOC(off_group_b, off_s_tiles[0],    s_tile_size);
+    VTCM_LAYOUT_ALLOC_OPTIONAL(off_group_b, off_s_tiles[1], s_tile_size, pipeline);
+    VTCM_LAYOUT_ALLOC(off_group_b, off_p_tiles[0],    s_tile_size);
+    VTCM_LAYOUT_ALLOC_OPTIONAL(off_group_b, off_p_tiles[1], s_tile_size, pipeline);
+    VTCM_LAYOUT_ALLOC(off_group_b, off_s_rowmax,      col_vec_size);
+    VTCM_LAYOUT_ALLOC(off_group_b, off_p_rowsum,      col_vec_size);
+    VTCM_LAYOUT_ALLOC(off_group_b, off_row_bufs,      row_vec_size * 2 * n_threads);
+
+    const size_t group_b_size = off_group_b - off_group_b_c;
+
+    // Group C: Q fetch DMA buffer
+    size_t off_group_c = off_group_b_c;
+    VTCM_LAYOUT_ALLOC(off_group_c, off_q_dma,         q_dma_size);
+
+    const size_t group_c_size = off_group_c - off_group_b_c;
+
+    off = off_group_b_c + hex_smax(group_b_size, group_c_size);
+
+    // Group A (Part 2 - remaining non-HMX buffers)
     VTCM_LAYOUT_ALLOC(off, off_k_fp16[0],     k_dma_size);
     VTCM_LAYOUT_ALLOC(off, off_k_fp16[1],     k_dma_size);
     VTCM_LAYOUT_ALLOC(off, off_v_fp16[0],     v_dma_size);
     VTCM_LAYOUT_ALLOC(off, off_v_fp16[1],     v_dma_size);
     VTCM_LAYOUT_ALLOC(off, off_m_vec,         col_vec_size);
     VTCM_LAYOUT_ALLOC(off, off_l_vec,         col_vec_size);
-    VTCM_LAYOUT_ALLOC(off, off_s_rowmax,      col_vec_size);
-    VTCM_LAYOUT_ALLOC(off, off_p_rowsum,      col_vec_size);
-    VTCM_LAYOUT_ALLOC(off, off_row_bufs,      row_vec_size * 2 * n_threads);
     VTCM_LAYOUT_ALLOC(off, off_hmx_scales_id, 256);
     VTCM_LAYOUT_ALLOC(off, off_hmx_scales_qk, 256);
     VTCM_LAYOUT_ALLOC(off, off_mask_buf,      m_buf_size);
@@ -200,9 +223,9 @@ static inline void hmx_fa_vtcm_layout_build(struct hmx_fa_vtcm_layout * L,
 }
 
 // Exact VTCM usage for a given (gqa_factor, DK, DV, Br, Bc) configuration.
-static inline size_t hmx_fa_compute_vtcm_usage(size_t gqa_factor, size_t DK, size_t DV, size_t Br, size_t Bc, size_t n_threads, bool pipeline) {
+static inline size_t hmx_fa_compute_vtcm_usage(size_t gqa_factor, size_t DK, size_t DV, size_t Br, size_t Bc, size_t n_threads, bool pipeline, bool is_q_fp32) {
     struct hmx_fa_vtcm_layout L;
-    hmx_fa_vtcm_layout_build(&L, gqa_factor, DK, DV, Br, Bc, n_threads, pipeline);
+    hmx_fa_vtcm_layout_build(&L, gqa_factor, DK, DV, Br, Bc, n_threads, pipeline, is_q_fp32);
     return L.total_bytes;
 }
 
@@ -239,7 +262,8 @@ static inline int hmx_fa_find_chunk_size(size_t * Br_out,
                                   size_t   qo_len,
                                   size_t   kv_len,
                                   size_t   vtcm_budget,
-                                  size_t   n_threads) {
+                                  size_t   n_threads,
+                                  bool     is_q_fp32) {
     const size_t T       = HMX_FP16_TILE_N_ROWS;  // 32
     const size_t br_unit = hmx_ceil_div(T, gqa_factor);
     const size_t bc_unit = HMX_FP16_TILE_N_COLS * 2;  // 64
@@ -253,8 +277,9 @@ static inline int hmx_fa_find_chunk_size(size_t * Br_out,
     const size_t Bc_limit     = can_pipeline ? hex_align_down(kv_len / FA_MIN_KV_BLOCKS, bc_unit) :
                                                (kv_len >= bc_unit ? hex_align_down(kv_len, bc_unit) : bc_unit);
     // Cost coefficients calibrated from profiling
-    const size_t c_q_fixed    = 1400;  // per-Q-block: q_load + epilogue o_update + o_norm + o_store
-    const size_t c_iter_fixed = 200;   // per-KV-iter: HMX queue push/pop + DMA pop + barriers
+    const size_t c_q_fixed    = 800;   // per-Q-block: q_load + epilogue o_update + o_norm + o_store
+    const size_t c_iter_base  = 200;   // per-KV-iter base (HMX dot/update + DMA)
+    const size_t c_softmax    = 600;   // per 64-row vector chunk on HVX
 
     size_t best_cost = SIZE_MAX, best_mn = 0;
     size_t best_Br = 0, best_Bc = 0;
@@ -262,13 +287,20 @@ static inline int hmx_fa_find_chunk_size(size_t * Br_out,
     for (size_t Br = Br_max; Br >= br_unit; Br -= br_unit) {
         // Try all Bc candidates from Bc_limit down to bc_unit
         for (size_t Bc = Bc_limit; Bc >= bc_unit; Bc -= bc_unit) {
-            size_t vtcm_needed = hmx_fa_compute_vtcm_usage(gqa_factor, DK, DV, Br, Bc, n_threads, can_pipeline);
+            size_t vtcm_needed = hmx_fa_compute_vtcm_usage(gqa_factor, DK, DV, Br, Bc, n_threads, can_pipeline, is_q_fp32);
             if (vtcm_needed <= vtcm_budget) {
                 // This Bc fits for this Br!
-                const size_t q_blocks  = (qo_len + Br - 1) / Br;
-                const size_t kv_blocks = (kv_len + Bc - 1) / Bc;
-                const size_t cost      = q_blocks * (c_q_fixed + kv_blocks * c_iter_fixed);
-                const size_t mn        = Br * Bc;
+                const size_t q_blocks       = (qo_len + Br - 1) / Br;
+                const size_t kv_blocks      = (kv_len + Bc - 1) / Bc;
+                const size_t actual_threads = (kv_blocks >= 3 && n_threads >= 2) ? n_threads : 1;
+                const size_t n_rows_g       = Br * gqa_factor;
+                const size_t n_row_vec_cnt  = (n_rows_g + 63) / 64;
+                const size_t n_use          = n_row_vec_cnt < actual_threads ? n_row_vec_cnt : actual_threads;
+                const size_t vecs_per_t     = n_use > 0 ? (n_row_vec_cnt + n_use - 1) / n_use : 1;
+
+                const size_t c_iter_actual  = c_iter_base + c_softmax * vecs_per_t;
+                const size_t cost           = q_blocks * (c_q_fixed + kv_blocks * c_iter_actual);
+                const size_t mn             = Br * Bc;
 
                 if (cost < best_cost || (cost == best_cost && mn > best_mn)) {
                     best_cost = cost;

ggml/src/ggml-hexagon/htp/hmx-mm-kernels-tiled.h

@@ -767,23 +767,25 @@ static void core_mma_chunk_fp16(__fp16 *restrict c, const __fp16 *restrict a, co
 
 // output : fp16 -> f32p
 
-static void transfer_output_chunk_fp16_to_fp32(
+static void transfer_output_chunk_fp16_to_fp32_col_chunk(
     float *restrict dst,
     const float *restrict src2,
     const __fp16 *restrict vtcm_src,
     uint32_t start_row,
     uint32_t n_rows,
-    uint32_t n_cols,
+    uint32_t c_len,
+    uint32_t total_n_cols,
     uint32_t dst_stride,
     uint32_t src2_stride,
     uint32_t dst_cols
 ) {
-    assert(n_cols % HTP_MM_HMX_TILE_N_COLS == 0);
-    const size_t tile_row_stride = (n_cols / HTP_MM_HMX_TILE_N_COLS) * HTP_MM_HMX_TILE_N_ELMS;
+    assert(c_len % HTP_MM_HMX_TILE_N_COLS == 0);
+    assert(total_n_cols % HTP_MM_HMX_TILE_N_COLS == 0);
+    const size_t tile_row_stride = (total_n_cols / HTP_MM_HMX_TILE_N_COLS) * HTP_MM_HMX_TILE_N_ELMS;
 
     const HVX_Vector one = hvx_vec_splat_f16(1.0);
 
-    const size_t limit_c         = hex_smin(n_cols, dst_cols);
+    const size_t limit_c         = hex_smin(c_len, dst_cols);
     const size_t limit_c_aligned = (limit_c & ~31);
 
     for (size_t r = 0; r < n_rows; r += 2) {
@@ -848,6 +850,22 @@ static void transfer_output_chunk_fp16_to_fp32(
     }
 }
 
+static inline void transfer_output_chunk_fp16_to_fp32(
+    float *restrict dst,
+    const float *restrict src2,
+    const __fp16 *restrict vtcm_src,
+    uint32_t start_row,
+    uint32_t n_rows,
+    uint32_t n_cols,
+    uint32_t dst_stride,
+    uint32_t src2_stride,
+    uint32_t dst_cols
+) {
+    transfer_output_chunk_fp16_to_fp32_col_chunk(
+        dst, src2, vtcm_src, start_row, n_rows, n_cols, n_cols, dst_stride, src2_stride, dst_cols
+    );
+}
+
 typedef struct {
     const __fp16  *vtcm_src;
     float         *dst;

ggml/src/ggml-hexagon/htp/htp-ctx.h

@@ -19,6 +19,8 @@
 #endif
 #define HTP_MAX_MMAPS    16
 
+#define HTP_MAX_DIRTY_RANGES 16
+
 // Memory mapping
 struct htp_mmap {
     uint64_t size;
@@ -95,7 +97,11 @@ struct htp_context {
     atomic_bool            vtcm_needs_release;
 
     uint64_t               max_vmem;
-    uint32_t               dirty_map[HTP_OP_MAX_TENSORS / 32];
+    struct htp_dirty_range {
+        uint32_t start;
+        uint32_t end;
+        uint32_t bi;
+    } dirty_ranges[HTP_MAX_DIRTY_RANGES];
 
     // Persistent DDR scratchpad for MUL_MAT_ID mappings
     void *                 ddr_spad_base;

ggml/src/ggml-hexagon/htp/htp-ops.h

@@ -123,7 +123,7 @@ enum htp_tensor_flags {
 // Tensor descriptor
 struct htp_tensor {
     uint32_t data;                 // Buffer offset in the messages, and data pointer on the NPU
-    uint32_t alias;                // Index of the canonical tensor for this memory buffer
+    uint32_t reserved;             // Reserved for alignment padding (must be multiple of 8)
     uint32_t size;                 // Data size in bytes
     uint32_t flags;                // Buffer / tensor flags
     uint32_t type;                 // Data type
@@ -173,6 +173,7 @@ enum htp_trace_event_id {
     HTP_TRACE_EVT_DMA                 = 0,
     HTP_TRACE_EVT_L2FLUSH             = 1,
     HTP_TRACE_EVT_INIT                = 2,
+    HTP_TRACE_EVT_BUFF                = 3,
 
     HTP_TRACE_EVT_HVX_COMP            = 20,
     HTP_TRACE_EVT_HVX_A_QUANT         = 21,
@@ -225,7 +226,10 @@ struct htp_opbatch_rsp {
     uint32_t n_tensors;  // Number of tensors
     uint32_t n_ops;      // Number of op profile descriptors
     uint32_t n_traces[HTP_MAX_NTHREADS + 1];
-    uint8_t  pad[8];     // align to 8 bytes
+    uint32_t usecs;          // Number of usec
+    uint32_t pad;            // align to 8 bytes
+    uint64_t cycles_start;   // Start cycle counter
+    uint64_t cycles_stop;    // Stop cycle counter
     // struct htp_prof_desc profs[];  -- dspqueue buf 0
 };
 

ggml/src/ggml-hexagon/htp/htp-tensor.c

@@ -2,6 +2,7 @@
 
 #include <qurt.h>
 #include <qurt_memory.h>
+#include <HAP_farf.h>
 
 #include "hex-common.h"
 #include "hex-utils.h"
@@ -10,84 +11,6 @@
 #include "htp-ctx.h"
 #include "work-queue.h"
 
-struct l2flush_task {
-    struct htp_thread_trace * trace;
-    uint32_t start;
-    uint32_t end;
-    uint32_t chunk_size;
-    uint32_t ti;
-};
-
-static void l2flush_thread_worker(unsigned int n, unsigned int i, void * data) {
-    struct l2flush_task * task = (struct l2flush_task *) data;
-    const uint32_t start = task->start;
-    const uint32_t end   = task->end;
-    const uint32_t ti    = task->ti;
-    const uint32_t chunk_size = task->chunk_size;
-
-    const uint32_t thread_s = start + i * chunk_size;
-    if (thread_s >= end) {
-        return;
-    }
-    uint32_t thread_e = thread_s + chunk_size;
-    if (thread_e > end) {
-        thread_e = end;
-    }
-
-    struct htp_thread_trace * tr = &task->trace[i];
-    htp_trace_event_start(tr, HTP_TRACE_EVT_L2FLUSH, ti);
-    hex_l2flush((void *) (uintptr_t) thread_s, thread_e - thread_s);
-    htp_trace_event_stop(tr, HTP_TRACE_EVT_L2FLUSH, ti);
-}
-
-static void flush_all_dcache(struct htp_context * ctx) {
-    struct htp_thread_trace * tr = &ctx->trace[0];
-    htp_trace_event_start(tr, HTP_TRACE_EVT_L2FLUSH, 0);
-    qurt_mem_cache_clean((qurt_addr_t) 0, 0, QURT_MEM_CACHE_FLUSH_INVALIDATE_ALL, QURT_MEM_DCACHE);
-    hex_l2fetch_block(ctx, ctx->footprint);
-    htp_trace_event_stop(tr, HTP_TRACE_EVT_L2FLUSH, 0);
-    bitmap_reset(ctx->dirty_map, HTP_OP_MAX_TENSORS);
-}
-
-static void flush_tensor_range(struct htp_context * ctx, const struct htp_tensor * t) {
-    struct htp_thread_trace * tr = &ctx->trace[0];
-
-    if (t->size > HEX_L2_FLUSH_WQ_THRESHOLD && ctx->n_threads > 1) {
-        struct l2flush_task task;
-        task.start = hex_align_down((size_t) t->data, HEX_L2_LINE_SIZE);
-        task.end   = hex_align_up((size_t) t->data + t->size, HEX_L2_LINE_SIZE);
-        task.ti    = t->ti;
-        task.trace = ctx->trace;
-
-        const uint32_t total_size = task.end - task.start;
-        const uint32_t n_blocks   = (total_size + HEX_L2_BLOCK_SIZE - 1) / HEX_L2_BLOCK_SIZE;
-        const uint32_t blocks_per_thread = fastdiv(n_blocks + ctx->n_threads - 1, &ctx->n_threads_div);
-        task.chunk_size = blocks_per_thread * HEX_L2_BLOCK_SIZE;
-
-        work_queue_run(ctx->work_queue, l2flush_thread_worker, &task, ctx->n_threads);
-    } else {
-        htp_trace_event_start(tr, HTP_TRACE_EVT_L2FLUSH, t->ti);
-        hex_l2flush((void *) t->data, t->size);
-        htp_trace_event_stop(tr, HTP_TRACE_EVT_L2FLUSH, t->ti);
-    }
-
-    htp_tensor_make_clean(t, ctx->dirty_map);
-}
-
-void htp_tensor_flush(struct htp_context * ctx, const struct htp_tensor * t) {
-    if (!bitmap_test(ctx->dirty_map, t->ti)) {
-        return;
-    }
-
-    if (t->size > HEX_L2_FLUSH_ALL_THRESHOLD) {
-        flush_all_dcache(ctx);
-        return;
-    }
-
-    flush_tensor_range(ctx, t);
-}
-
-// One dirty tensor's line-aligned range, placed in the flattened global block space.
 struct l2flush_range {
     uint32_t start;       // line-aligned start address
     uint32_t end;         // line-aligned end address
@@ -103,9 +26,18 @@ struct l2flush_multi_task {
     uint32_t                  blocks_per_thread;
 };
 
+static void flush_all_dcache(struct htp_context * ctx) {
+    struct htp_thread_trace * tr = &ctx->trace[0];
+    htp_trace_event_start(tr, HTP_TRACE_EVT_L2FLUSH, 0);
+    qurt_mem_cache_clean((qurt_addr_t) 0, 0, QURT_MEM_CACHE_FLUSH_INVALIDATE_ALL, QURT_MEM_DCACHE);
+    hex_l2fetch_block(ctx, ctx->footprint);
+    htp_trace_event_stop(tr, HTP_TRACE_EVT_L2FLUSH, 0);
+    memset(ctx->dirty_ranges, 0, sizeof(ctx->dirty_ranges));
+}
+
 static void l2flush_multi_worker(unsigned int n, unsigned int i, void * data) {
-    (void) n;
     struct l2flush_multi_task * task = (struct l2flush_multi_task *) data;
+    (void) n;
 
     const uint32_t gb_first = i * task->blocks_per_thread;
     uint32_t       gb_last  = gb_first + task->blocks_per_thread;
@@ -141,11 +73,177 @@ static void l2flush_multi_worker(unsigned int n, unsigned int i, void * data) {
     htp_trace_event_stop(tr, HTP_TRACE_EVT_L2FLUSH, gb_first);
 }
 
-void htp_tensor_flush_all(struct htp_context * ctx, const struct htp_tensor * const * tensors, uint32_t n) {
-    uint64_t total_dirty = 0;
+void htp_tensor_dirty_all(struct htp_context * ctx, const struct htp_tensor * const * tensors, uint32_t n) {
+    const struct htp_tensor * pending[HTP_OP_MAX_OUTPUTS];
+    uint32_t n_pending = 0;
+
     for (uint32_t i = 0; i < n; i++) {
         const struct htp_tensor * t = tensors[i];
-        if (t && bitmap_test(ctx->dirty_map, t->ti)) {
+        if (!t) continue;
+
+        uint32_t t_start = t->data;
+        uint32_t t_end   = t_start + t->size;
+
+        bool merged = false;
+        for (uint32_t j = 0; j < HTP_MAX_DIRTY_RANGES; j++) {
+            struct htp_dirty_range * r = &ctx->dirty_ranges[j];
+            if (!r->start) continue;
+
+            if (r->start <= t_end && t_start <= r->end) {
+                uint32_t new_start = (t_start < r->start) ? t_start : r->start;
+                uint32_t new_end   = (t_end > r->end)     ? t_end   : r->end;
+                r->start = new_start;
+                r->end   = new_end;
+                merged   = true;
+            }
+        }
+
+        if (!merged) {
+            pending[n_pending++] = t;
+        }
+    }
+
+    if (n_pending == 0) {
+        return;
+    }
+
+    uint32_t empty_indices[HTP_MAX_DIRTY_RANGES];
+    uint32_t active_indices[HTP_MAX_DIRTY_RANGES];
+    uint32_t n_active = 0;
+    uint32_t n_empty  = 0;
+    for (uint32_t j = 0; j < HTP_MAX_DIRTY_RANGES; j++) {
+        if (ctx->dirty_ranges[j].start) {
+            active_indices[n_active++] = j;
+        } else {
+            empty_indices[n_empty++]   = j;
+        }
+    }
+
+    if (n_pending <= n_empty) {
+        for (uint32_t i = 0; i < n_pending; i++) {
+            uint32_t idx = empty_indices[i];
+            struct htp_dirty_range * r = &ctx->dirty_ranges[idx];
+            r->start = pending[i]->data;
+            r->end   = pending[i]->data + pending[i]->size;
+            r->bi    = pending[i]->bi;
+        }
+        return;
+    }
+
+    uint32_t n_evict = n_pending - n_empty;
+    uint32_t total_evict_size = 0;
+    for (uint32_t i = 0; i < n_evict; i++) {
+        uint32_t idx = active_indices[i];
+        struct htp_dirty_range * r = &ctx->dirty_ranges[idx];
+        total_evict_size += r->end - r->start;
+    }
+
+    if (total_evict_size > HEX_L2_FLUSH_ALL_THRESHOLD) {
+        flush_all_dcache(ctx);
+        for (uint32_t i = 0; i < n_pending; i++) {
+            struct htp_dirty_range * r = &ctx->dirty_ranges[i];
+            r->start = pending[i]->data;
+            r->end   = pending[i]->data + pending[i]->size;
+            r->bi    = pending[i]->bi;
+        }
+        return;
+    }
+
+    if (total_evict_size > HEX_L2_FLUSH_WQ_THRESHOLD && ctx->n_threads > 1 && n_evict <= HTP_OP_MAX_INPUTS) {
+        struct l2flush_multi_task task;
+        task.trace    = ctx->trace;
+        task.n_ranges = n_evict;
+
+        uint32_t block_acc = 0;
+        for (uint32_t i = 0; i < n_evict; i++) {
+            uint32_t idx = active_indices[i];
+            struct htp_dirty_range * r = &ctx->dirty_ranges[idx];
+
+            struct l2flush_range * rg = &task.ranges[i];
+            rg->start = hex_align_down((size_t) r->start, HEX_L2_LINE_SIZE);
+            rg->end   = hex_align_up((size_t) r->end, HEX_L2_LINE_SIZE);
+            rg->block_first = block_acc;
+            rg->n_blocks = (rg->end - rg->start + HEX_L2_BLOCK_SIZE - 1) / HEX_L2_BLOCK_SIZE;
+            block_acc += rg->n_blocks;
+        }
+
+        task.total_blocks      = block_acc;
+        task.blocks_per_thread = fastdiv(block_acc + ctx->n_threads - 1, &ctx->n_threads_div);
+
+        work_queue_run(ctx->work_queue, l2flush_multi_worker, &task, ctx->n_threads);
+    } else {
+        struct htp_thread_trace * tr = &ctx->trace[0];
+        htp_trace_event_start(tr, HTP_TRACE_EVT_L2FLUSH, 0);
+        for (uint32_t i = 0; i < n_evict; i++) {
+            uint32_t idx = active_indices[i];
+            struct htp_dirty_range * r = &ctx->dirty_ranges[idx];
+            uint32_t size = r->end - r->start;
+            hex_l2flush((void *) (uintptr_t) r->start, size);
+        }
+        htp_trace_event_stop(tr, HTP_TRACE_EVT_L2FLUSH, 0);
+    }
+
+    for (uint32_t i = 0; i < n_evict; i++) {
+        uint32_t idx = active_indices[i];
+        struct htp_dirty_range * r = &ctx->dirty_ranges[idx];
+        r->start = pending[i]->data;
+        r->end   = pending[i]->data + pending[i]->size;
+        r->bi    = pending[i]->bi;
+    }
+
+    for (uint32_t i = 0; i < n_empty; i++) {
+        uint32_t idx = empty_indices[i];
+        struct htp_dirty_range * r = &ctx->dirty_ranges[idx];
+        r->start = pending[n_evict + i]->data;
+        r->end   = pending[n_evict + i]->data + pending[n_evict + i]->size;
+        r->bi    = pending[n_evict + i]->bi;
+    }
+}
+
+static void make_tensor_clean(struct htp_context * ctx, const struct htp_tensor * t) {
+    uint32_t t_start = t->data;
+    uint32_t t_end   = t_start + t->size;
+
+    for (uint32_t i = 0; i < HTP_MAX_DIRTY_RANGES; i++) {
+        struct htp_dirty_range * r = &ctx->dirty_ranges[i];
+        if (!r->start) continue;
+
+        if (r->start < t_end && t_start < r->end) {
+            if (t_start <= r->start && r->end <= t_end) {
+                r->start = 0;
+            } else if (t_start <= r->start) {
+                r->start = t_end;
+            } else if (r->end <= t_end) {
+                r->end = t_start;
+            }
+        }
+    }
+}
+
+static inline bool is_tensor_dirty(struct htp_context * ctx, const struct htp_tensor * t) {
+    uint32_t t_start = t->data;
+    uint32_t t_end   = t_start + t->size;
+
+    for (uint32_t i = 0; i < HTP_MAX_DIRTY_RANGES; i++) {
+        struct htp_dirty_range * r = &ctx->dirty_ranges[i];
+        if (!r->start) continue;
+
+        if (r->start < t_end && t_start < r->end) {
+            return true;
+        }
+    }
+    return false;
+}
+
+void htp_tensor_flush_all(struct htp_context * ctx, const struct htp_tensor * const * tensors, uint32_t n) {
+    const struct htp_tensor * dirty_tensors[HTP_OP_MAX_INPUTS];
+    uint32_t n_dirty = 0;
+    uint64_t total_dirty = 0;
+
+    for (uint32_t i = 0; i < n; i++) {
+        const struct htp_tensor * t = tensors[i];
+        if (t && (t->flags & HTP_TENSOR_COMPUTE) && is_tensor_dirty(ctx, t)) {
+            dirty_tensors[n_dirty++] = t;
             total_dirty += t->size;
         }
     }
@@ -159,21 +257,15 @@ void htp_tensor_flush_all(struct htp_context * ctx, const struct htp_tensor * co
         return;
     }
 
-    // Aggregate is small enough to walk. Thread it across all dirty ranges at once
-    // when it is worth the dispatch, otherwise flush sequentially.
-    if (total_dirty > HEX_L2_FLUSH_WQ_THRESHOLD && ctx->n_threads > 1) {
+    if (total_dirty >= HEX_L2_FLUSH_WQ_THRESHOLD && ctx->n_threads > 1) {
         struct l2flush_multi_task task;
         task.trace    = ctx->trace;
         task.n_ranges = 0;
 
         uint32_t block_acc = 0;
-        for (uint32_t i = 0; i < n; i++) {
-            const struct htp_tensor * t = tensors[i];
-            if (!t || !bitmap_test(ctx->dirty_map, t->ti)) {
-                continue;
-            }
-            // Clear as we go: dedups a tensor passed as multiple srcs (e.g. mul(x,x)).
-            htp_tensor_make_clean(t, ctx->dirty_map);
+        for (uint32_t i = 0; i < n_dirty; i++) {
+            const struct htp_tensor * t = dirty_tensors[i];
+            make_tensor_clean(ctx, t);
 
             struct l2flush_range * rg = &task.ranges[task.n_ranges++];
             rg->start = hex_align_down((size_t) t->data, HEX_L2_LINE_SIZE);
@@ -191,14 +283,11 @@ void htp_tensor_flush_all(struct htp_context * ctx, const struct htp_tensor * co
     }
 
     struct htp_thread_trace * tr = &ctx->trace[0];
-    for (uint32_t i = 0; i < n; i++) {
-        const struct htp_tensor * t = tensors[i];
-        if (!t || !bitmap_test(ctx->dirty_map, t->ti)) {
-            continue;
-        }
+    for (uint32_t i = 0; i < n_dirty; i++) {
+        const struct htp_tensor * t = dirty_tensors[i];
         htp_trace_event_start(tr, HTP_TRACE_EVT_L2FLUSH, t->ti);
-        hex_l2flush((void *) t->data, t->size);
+        hex_l2flush((void *) (uintptr_t) t->data, t->size);
         htp_trace_event_stop(tr, HTP_TRACE_EVT_L2FLUSH, t->ti);
-        htp_tensor_make_clean(t, ctx->dirty_map);
+        make_tensor_clean(ctx, t);
     }
 }

ggml/src/ggml-hexagon/htp/htp-tensor.h

@@ -5,10 +5,6 @@
 #include "htp-ops.h"
 #include "hex-bitmap.h"
 
-static inline struct htp_tensor * htp_tensor_alias(const struct htp_tensor * t) {
-    return (struct htp_tensor *) (uintptr_t) t->alias;
-}
-
 static inline void * htp_tensor_data(const struct htp_tensor * t) {
     return (void *) (uintptr_t) t->data;
 }
@@ -17,20 +13,8 @@ static inline uint32_t * htp_tensor_flags(const struct htp_tensor * t) {
     return (uint32_t *) &t->flags;
 }
 
-static inline void htp_tensor_make_dirty(const struct htp_tensor * t, uint32_t * dirty_map) {
-    struct htp_tensor * curr = (struct htp_tensor *) t;
-    do {
-        bitmap_set(dirty_map, curr->ti);
-        curr = htp_tensor_alias(curr);
-    } while (curr != t);
-}
-
-static inline void htp_tensor_make_clean(const struct htp_tensor * t, uint32_t * dirty_map) {
-    bitmap_clear(dirty_map, t->ti);
-}
-
 struct htp_context;
-void htp_tensor_flush(struct htp_context * ctx, const struct htp_tensor * t);
 void htp_tensor_flush_all(struct htp_context * ctx, const struct htp_tensor * const * tensors, uint32_t n);
+void htp_tensor_dirty_all(struct htp_context * ctx, const struct htp_tensor * const * tensors, uint32_t n);
 
 #endif // HTP_TENSOR_H

ggml/src/ggml-hexagon/htp/hvx-fa-kernels.h

@@ -208,6 +208,77 @@ static inline void hvx_mad_f32_f16_aa_rx2(float * restrict y, const void * restr
         }
     }
 }
+static inline void hvx_mad_f32_f16_aa_vec(float * restrict y, const void * restrict x, HVX_Vector S0, uint32_t n) {
+    const HVX_Vector * restrict vx0 = (const HVX_Vector *) x;
+
+    HVX_VectorPair * restrict vy_p = (HVX_VectorPair *) y;
+    HVX_Vector * restrict vy = (HVX_Vector *) y;
+
+    uint32_t nvec = n / VLEN_FP16; // num full fp16 hvx vectors
+    uint32_t nloe = n % VLEN_FP16; // leftover elements
+
+    uint32_t i = 0;
+
+    #pragma unroll(2)
+    for (i = 0; i < nvec; ++i) {
+        vy_p[i] = hvx_vec_mpyacc_f32_f16(vy_p[i], Q6_Vh_vshuff_Vh(vx0[i]), S0);
+    }
+
+    if (nloe) {
+        HVX_VectorPair xy_p = vy_p[i];
+        xy_p = hvx_vec_mpyacc_f32_f16(xy_p, Q6_Vh_vshuff_Vh(vx0[i]), S0);
+
+        HVX_Vector xy = Q6_V_lo_W(xy_p);
+        i = 2 * i;  // index for vy
+
+        if (nloe >= VLEN_FP32) {
+            vy[i] = xy;
+            nloe -= VLEN_FP32; ++i; xy = Q6_V_hi_W(xy_p);
+        }
+
+        if (nloe) {
+            hvx_vec_store_a(&vy[i], nloe * 4, xy);
+        }
+    }
+}
+
+static inline void hvx_mad_f32_f16_aa_rx2_vec(float * restrict y, const void * restrict x0, const void * restrict x1,
+                                          HVX_Vector S0, HVX_Vector S1, uint32_t n) {
+    const HVX_Vector * restrict vx0 = (const HVX_Vector *) x0;
+    const HVX_Vector * restrict vx1 = (const HVX_Vector *) x1;
+
+    HVX_VectorPair * restrict vy_p  = (HVX_VectorPair *) y;
+    HVX_Vector * restrict vy        = (HVX_Vector *) y;
+
+    uint32_t nvec = n / VLEN_FP16;  // num full fp16 hvx vectors
+    uint32_t nloe = n % VLEN_FP16;  // leftover elements
+
+    uint32_t i = 0;
+
+    #pragma unroll(2)
+    for (i = 0; i < nvec; ++i) {
+        vy_p[i] = hvx_vec_mpyacc_f32_f16(vy_p[i], Q6_Vh_vshuff_Vh(vx0[i]), S0);
+        vy_p[i] = hvx_vec_mpyacc_f32_f16(vy_p[i], Q6_Vh_vshuff_Vh(vx1[i]), S1);
+    }
+
+    if (nloe) {
+        HVX_VectorPair xy_p = vy_p[i];
+        xy_p = hvx_vec_mpyacc_f32_f16(xy_p, Q6_Vh_vshuff_Vh(vx0[i]), S0);
+        xy_p = hvx_vec_mpyacc_f32_f16(xy_p, Q6_Vh_vshuff_Vh(vx1[i]), S1);
+
+        HVX_Vector xy = Q6_V_lo_W(xy_p);
+        i = 2 * i;  // index for vy
+
+        if (nloe >= VLEN_FP32) {
+            vy[i] = xy;
+            nloe -= VLEN_FP32; ++i; xy = Q6_V_hi_W(xy_p);
+        }
+
+        if (nloe) {
+            hvx_vec_store_a(&vy[i], nloe * 4, xy);
+        }
+    }
+}
 
 static inline void hvx_scale_vec_f32_aa(uint8_t * restrict dst, const uint8_t * restrict src, const uint32_t n, HVX_Vector vs) {
     assert((size_t) dst % 128 == 0);

ggml/src/ggml-hexagon/htp/hvx-reduce.h

@@ -286,6 +286,46 @@ static inline float hvx_sum_of_squares_f32(const uint8_t * restrict src, const i
     }
 }
 
+// Signed 32-bit Integer Max variants
+
+static inline HVX_Vector hvx_vec_reduce_max_n_i32(HVX_Vector in, unsigned int n) {
+    unsigned int total = n * 4;  // total vec nbytes
+    unsigned int width = 4;      // int32 nbytes
+
+    HVX_Vector max_val = in, max_t;
+    while (width < total) {
+        max_t = Q6_V_vror_VR(max_val, width);          // rotate right
+        max_val = Q6_Vw_vmax_VwVw(max_t, max_val);      // elementwise signed max
+        width = width << 1;
+    }
+    return max_val;
+}
+
+static inline HVX_Vector hvx_vec_reduce_max_i32(HVX_Vector in) {
+    return hvx_vec_reduce_max_n_i32(in, 32);
+}
+
+static inline int32_t hvx_reduce_max_i32_a(const uint8_t * restrict src, const int num_elems) {
+    HVX_Vector init_vec = Q6_V_vsplat_R(((const int32_t *) src)[0]);
+    HVX_Vector pad_vec  = Q6_V_vsplat_R(0x80000000);
+    assert((uintptr_t) src % 128 == 0);
+    hvx_reduce_loop_body(HVX_Vector, init_vec, pad_vec, Q6_Vw_vmax_VwVw, hvx_vec_reduce_max_i32, hvx_vec_get_i32);
+}
+
+static inline int32_t hvx_reduce_max_i32_u(const uint8_t * restrict src, const int num_elems) {
+    HVX_Vector init_vec = Q6_V_vsplat_R(((const int32_t *) src)[0]);
+    HVX_Vector pad_vec  = Q6_V_vsplat_R(0x80000000);
+    hvx_reduce_loop_body(HVX_UVector, init_vec, pad_vec, Q6_Vw_vmax_VwVw, hvx_vec_reduce_max_i32, hvx_vec_get_i32);
+}
+
+static inline int32_t hvx_reduce_max_i32(const uint8_t * restrict src, const int num_elems) {
+    if (hex_is_aligned((void *) src, 128)) {
+        return hvx_reduce_max_i32_a(src, num_elems);
+    } else {
+        return hvx_reduce_max_i32_u(src, num_elems);
+    }
+}
+
 #undef hvx_reduce_loop_body
 #undef HVX_REDUCE_MAX_OP
 #undef HVX_REDUCE_SUM_OP

ggml/src/ggml-hexagon/htp/main.c

@@ -723,14 +723,14 @@ static int execute_op(struct htp_ops_context * octx) {
         case HTP_OP_SQRT:
         case HTP_OP_UNARY_SOFTPLUS:
         case HTP_OP_UNARY_SIGMOID:
+        case HTP_OP_UNARY_SILU:
+        case HTP_OP_UNARY_GELU:
         case HTP_OP_UNARY_NEG:
         case HTP_OP_UNARY_EXP:
         case HTP_OP_UNARY_TANH:
         case HTP_OP_L2_NORM:
             return op_unary(octx);
 
-        case HTP_OP_UNARY_SILU:
-        case HTP_OP_UNARY_GELU:
         case HTP_OP_GLU_SWIGLU:
         case HTP_OP_GLU_SWIGLU_OAI:
         case HTP_OP_GLU_GEGLU:
@@ -901,10 +901,8 @@ static void prep_tensor(struct htp_context *ctx, struct htp_buf_desc *bufs, stru
     uint32_t offset = t->data;
     uint32_t size   = t->size;
     uint32_t bi     = t->bi;
-    uint32_t alias  = t->alias;
 
     t->data  = (uint32_t) (bufs[bi].base + offset);  // update data to the actual pointer
-    t->alias = (uint32_t) (tens + alias);            // update alias to the actual pointer
 
     FARF(HIGH, "prep-tensor #%u: bi %u offset %u size %u data %p : %u:%u:%u:%u", idx, t->bi, offset, t->size, (void*) t->data,
         t->ne[0], t->ne[1], t->ne[3], t->ne[3]);
@@ -955,14 +953,14 @@ static int proc_op_req(struct htp_ops_context * octx, struct htp_tensor *tens, u
         octx->dsts[i]    = dst;
         octx->dst_dma[i] = octx->ctx->dma; // FIXME: ? octx->ctx->dma_cached : octx->ctx->dma;
 
-        htp_tensor_make_dirty(dst, octx->ctx->dirty_map);
-
         FARF(HIGH, "prep-dst[%u] #%u: data %p size %u : %u:%u:%u:%u", i, dst_idx, (void*) dst->data, dst->size,
             dst->ne[0], dst->ne[1], dst->ne[2], dst->ne[3]);
     }
 
     int status = execute_op(octx);
 
+    htp_tensor_dirty_all(octx->ctx, octx->dsts, HTP_OP_MAX_OUTPUTS);
+
     octx->src0_spad.src = NULL;
     octx->src1_spad.src = NULL;
     octx->src2_spad.src = NULL;
@@ -994,12 +992,6 @@ static void process_opbatch(struct htp_context * ctx, const struct htp_opbatch_r
     FARF(HIGH, "processing opbatch #%u: n-bufs %u n-tensors %u n-ops %u n-traces %u : m-size %u b-size %u t-size %u o-size %u", req->id,
             n_bufs, n_tens, n_ops, req->n_traces, dbuf->size, b_size, t_size, o_size);
 
-    // Clean cache at the start of the batch
-    // We cant trace this part because the trace buffer is setup later
-    qurt_mem_cache_clean((qurt_addr_t) 0, 0, QURT_MEM_CACHE_FLUSH_INVALIDATE_ALL, QURT_MEM_DCACHE);
-    hex_l2fetch_block(ctx, ctx->footprint);
-    bitmap_reset(ctx->dirty_map, HTP_OP_MAX_TENSORS);
-
     // Setup descriptor pointers
     uint8_t * m_ptr = dbuf->ptr;
     struct htp_buf_desc* bufs = (struct htp_buf_desc*)  m_ptr; m_ptr += b_size;
@@ -1007,13 +999,8 @@ static void process_opbatch(struct htp_context * ctx, const struct htp_opbatch_r
     struct htp_op_desc*   ops = (struct htp_op_desc*)   m_ptr; m_ptr += o_size;
     struct htp_prof_desc* pds = (struct htp_prof_desc*) m_ptr;
 
-    prep_op_bufs(ctx, bufs, n_bufs);
-    prep_tensors(ctx, bufs, tens, n_tens);
-
-    struct htp_ops_context *octx = &ctx->octx;
-    memset(octx, 0, sizeof(*octx));
-    octx->n_threads = ctx->n_threads;
-    octx->ctx       = ctx;
+    struct profile_data batch_prof;
+    profile_start(HTP_PROF_BASIC, &batch_prof);
 
     memset(ctx->trace, 0, sizeof(ctx->trace));
     if (ctx->profiler == HTP_PROF_TRACE) {
@@ -1024,6 +1011,24 @@ static void process_opbatch(struct htp_context * ctx, const struct htp_opbatch_r
         }
     }
 
+    // Clean cache at the start of the batch
+    htp_trace_event_start(&ctx->trace[0], HTP_TRACE_EVT_L2FLUSH, 0);
+    qurt_mem_cache_clean((qurt_addr_t) 0, 0, QURT_MEM_CACHE_FLUSH_INVALIDATE_ALL, QURT_MEM_DCACHE);
+    hex_l2fetch_block(ctx, ctx->footprint);
+    memset(ctx->dirty_ranges, 0, sizeof(ctx->dirty_ranges));
+    htp_trace_event_stop(&ctx->trace[0], HTP_TRACE_EVT_L2FLUSH, 0);
+
+    htp_trace_event_start(&ctx->trace[0], HTP_TRACE_EVT_BUFF, 0);
+    prep_op_bufs(ctx, bufs, n_bufs);
+    htp_trace_event_stop(&ctx->trace[0], HTP_TRACE_EVT_BUFF, 0);
+
+    prep_tensors(ctx, bufs, tens, n_tens);
+
+    struct htp_ops_context *octx = &ctx->octx;
+    memset(octx, 0, sizeof(*octx));
+    octx->n_threads = ctx->n_threads;
+    octx->ctx       = ctx;
+
     work_queue_wakeup(ctx->work_queue);
     if (ctx->hmx_queue) {
         hmx_queue_wakeup(ctx->hmx_queue);
@@ -1056,13 +1061,23 @@ static void process_opbatch(struct htp_context * ctx, const struct htp_opbatch_r
     }
     work_queue_suspend(ctx->work_queue);
 
+    // Flush remaining dirty tensors at the end of the batch
+    htp_trace_event_start(&ctx->trace[0], HTP_TRACE_EVT_L2FLUSH, 0);
+    qurt_mem_cache_clean((qurt_addr_t) 0, 0, QURT_MEM_CACHE_FLUSH_INVALIDATE_ALL, QURT_MEM_DCACHE);
+    htp_trace_event_stop(&ctx->trace[0], HTP_TRACE_EVT_L2FLUSH, 0);
+
+    profile_stop(HTP_PROF_BASIC, &batch_prof);
+
     struct htp_opbatch_rsp rsp;
     memset(&rsp, 0, sizeof(rsp));
-    rsp.id        = req->id;
-    rsp.status    = op_status;
-    rsp.n_bufs    = n_bufs;
-    rsp.n_tensors = n_tens;
-    rsp.n_ops     = n_ops;
+    rsp.id           = req->id;
+    rsp.status       = op_status;
+    rsp.n_bufs       = n_bufs;
+    rsp.n_tensors    = n_tens;
+    rsp.n_ops        = n_ops;
+    rsp.usecs        = batch_prof.usecs;
+    rsp.cycles_start = batch_prof.cycles_start;
+    rsp.cycles_stop  = batch_prof.cycles_stop;
 
     if (ctx->profiler == HTP_PROF_TRACE) {
         for (int t = 0; t <= HTP_MAX_NTHREADS; t++) {
@@ -1073,11 +1088,6 @@ static void process_opbatch(struct htp_context * ctx, const struct htp_opbatch_r
     struct dspqueue_buffer write_dbuf = *dbuf;
     write_dbuf.flags = DSPQUEUE_BUFFER_FLAG_FLUSH_SENDER | DSPQUEUE_BUFFER_FLAG_INVALIDATE_RECIPIENT;
 
-    // Flush remaining dirty tensors at the end of the batch
-    htp_trace_event_start(&ctx->trace[0], HTP_TRACE_EVT_L2FLUSH, 0);
-    qurt_mem_cache_clean((qurt_addr_t) 0, 0, QURT_MEM_CACHE_FLUSH_INVALIDATE_ALL, QURT_MEM_DCACHE);
-    htp_trace_event_stop(&ctx->trace[0], HTP_TRACE_EVT_L2FLUSH, 0);
-
     err = dspqueue_write(queue, 0, 1, &write_dbuf, sizeof(rsp), (const uint8_t *) &rsp, DSPQUEUE_TIMEOUT_NONE);
     if (err != 0) {
         FARF(ERROR, "dspqueue_write failed: 0x%08x", (unsigned) err);

ggml/src/ggml-hexagon/htp/matmul-ops.c

@@ -14,6 +14,8 @@
 #include "hex-dma.h"
 #include "hvx-utils.h"
 #include "hvx-dump.h"
+#include "hvx-arith.h"
+#include "hvx-reduce.h"
 
 #define GGML_COMMON_DECL_C
 #include "ggml-common.h"
@@ -82,6 +84,8 @@ struct htp_mm_context {
 
     // Precomputed values
     uint32_t src0_nrows_per_thread;
+    uint32_t src0_row_size_padded;
+    uint32_t src1_nrows;
 
     struct fastdiv_values mm_div_ne12_ne1;
     struct fastdiv_values mm_div_ne1;
@@ -103,6 +107,7 @@ struct htp_mm_context {
     // Fields for scattered mapping & HMX support in MUL_MAT_ID
     const uint32_t * matrix_row_counts;
     const struct mmid_row_mapping * matrix_rows;
+    uint32_t mapping_stride;
 
     // Dynamic VTCM pointers allocated sequentially
     uint8_t * vtcm_src0;
@@ -154,8 +159,6 @@ static const uint8_t __attribute__((aligned(VLEN))) kvalues_mxfp4_lut[] = {
     0,    0, 0,    0, 0,    0, 0, 0, 0, 0, 0, 0, 0, 0, 0,  0, 0, 0, 0,    0, 0,    0, 0,    0,
 };
 
-
-
 #define htp_matmul_tensors_preamble                                 \
     const struct htp_tensor * restrict src0 = octx->src[0];         \
     const struct htp_tensor * restrict src1 = octx->src[1];         \
@@ -444,6 +447,16 @@ static void hvx_mv_2d_repacked_##SUFFIX(unsigned int nth, unsigned int ith, void
                                                                                                                                   \
     uint32_t push_ct = ct_start;                                                                                                  \
     if (src0_start_row < src0_end_row) {                                                                                          \
+        if (src2) {                                                                                                               \
+            float * vtcm_src2_ptr = (float *) mmctx->vtcm_src2 + src0_start_row;                                                  \
+            const float * src2_ptr = (const float *) src2->data + src0_start_row;                                                 \
+            int slice_size = (int)MIN(src0_end_row, ne0) - (int)src0_start_row;                                                   \
+            if (slice_size > 0) {                                                                                                 \
+                dma_queue_push(dma_queue, dma_make_ptr(vtcm_src2_ptr, src2_ptr),                                                  \
+                               slice_size * sizeof(float), slice_size * sizeof(float), slice_size * sizeof(float), 1);            \
+                dma_queue_pop_nowait(dma_queue);                                                                                  \
+            }                                                                                                                     \
+        }                                                                                                                         \
         for (uint32_t d = 0; d < n_prefetch && push_ct < ct_end; d++, push_ct++) {                                                \
             dma_queue_push(dma_queue, dma_make_ptr(vtcm_src0_ptr + d * tile_row_transfer_size_aligned,                            \
                            src0_row + push_ct * tile_row_stride), aligned_tile_size, tile_size, tile_size, n_k_tiles_a);          \
@@ -465,7 +478,7 @@ static void hvx_mv_2d_repacked_##SUFFIX(unsigned int nth, unsigned int ith, void
                                                                                                                                   \
         htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, ct);                                                                    \
         DOT_2X1(ne10, dst_ptr, w_tile, src1_col, valid_rows, NULL);                                                               \
-        htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, ct);                                                                     \
+        htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, ct);                                                                    \
                                                                                                                                   \
         if (push_ct < ct_end) {                                                                                                   \
             dma_queue_push(dma_queue, dma_make_ptr((uint8_t *)w_tile, src0_row + push_ct * tile_row_stride),                      \
@@ -476,24 +489,16 @@ static void hvx_mv_2d_repacked_##SUFFIX(unsigned int nth, unsigned int ith, void
                                                                                                                                   \
     int copy_cnt = (int)MIN(src0_end_row, ne0) - (int)src0_start_row;                                                             \
     if (copy_cnt > 0) {                                                                                                           \
+        htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, ct_end);                                                                \
         if (src2) {                                                                                                               \
-            float * dst_ptr = &dst_col[src0_start_row];                                                                           \
-            const float * src2_ptr = (const float *) src2->data + src0_start_row;                                                 \
-            float * tmp_ptr = tmp;                                                                                                \
-            int remaining = copy_cnt;                                                                                             \
-            while (remaining > 0) {                                                                                               \
-                int n = MIN(remaining, 32);                                                                                       \
-                HVX_Vector v_out = hvx_vmemu(tmp_ptr);                                                                            \
-                HVX_Vector v_z   = hvx_vmemu(src2_ptr);                                                                           \
-                hvx_vec_store_u(dst_ptr, n * sizeof(float), hvx_vec_add_f32_f32(v_out, v_z));                                     \
-                dst_ptr += n;                                                                                                     \
-                src2_ptr += n;                                                                                                    \
-                tmp_ptr += n;                                                                                                     \
-                remaining -= n;                                                                                                   \
-            }                                                                                                                     \
+            hvx_add_f32_uaa((uint8_t *) &dst_col[src0_start_row],                                                                 \
+                            (const uint8_t *) tmp,                                                                                \
+                            (const uint8_t *) ((const float *) mmctx->vtcm_src2 + src0_start_row),                                \
+                            copy_cnt);                                                                                            \
         } else {                                                                                                                  \
             hvx_copy_f32_ua((uint8_t *) &dst_col[src0_start_row], (uint8_t *) tmp, copy_cnt);                                     \
         }                                                                                                                         \
+        htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, ct_end);                                                                 \
     }                                                                                                                             \
 }
 
@@ -1069,6 +1074,16 @@ static void hvx_mv_2d(unsigned int nth, unsigned int ith, void * data) {
 
     // Prefill vtcm with 2x src0 rows
     if (src0_start_row < src0_end_row) {
+        if (src2) {
+            float * vtcm_src2_ptr = (float *) mmctx->vtcm_src2 + src0_start_row;
+            const float * src2_ptr = (const float *) src2->data + src0_start_row;
+            int slice_size = (int)src0_end_row - (int)src0_start_row;
+            if (slice_size > 0) {
+                dma_queue_push(dma_queue, dma_make_ptr(vtcm_src2_ptr, src2_ptr),
+                               slice_size * sizeof(float), slice_size * sizeof(float), slice_size * sizeof(float), 1);
+                dma_queue_pop_nowait(dma_queue);
+            }
+        }
         for (uint32_t ir0 = src0_start_row; ir0 < src0_end_row_x2; ir0 += 2) {
             const uint32_t is0 = (ir0 - src0_start_row);
             if (is0 >= n_prefetch) {
@@ -1114,27 +1129,21 @@ static void hvx_mv_2d(unsigned int nth, unsigned int ith, void * data) {
     }
 
     int copy_cnt = src0_end_row - src0_start_row;
-    if (src2) {
-        float * dst_ptr = &dst_col[src0_start_row];
-        const float * src2_ptr = (const float *) src2->data + src0_start_row;
-        float * tmp_ptr = tmp;
-        int remaining = copy_cnt;
-        while (remaining > 0) {
-            int n = MIN(remaining, 32);
-            HVX_Vector v_out = hvx_vmemu(tmp_ptr);
-            HVX_Vector v_z   = hvx_vmemu(src2_ptr);
-            hvx_vec_store_u(dst_ptr, n * sizeof(float), hvx_vec_add_f32_f32(v_out, v_z));
-            dst_ptr += n;
-            src2_ptr += n;
-            tmp_ptr += n;
-            remaining -= n;
+    if (copy_cnt > 0) {
+        htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, src0_end_row);
+        if (src2) {
+            hvx_add_f32_uaa((uint8_t *) &dst_col[src0_start_row],
+                            (const uint8_t *) tmp,
+                            (const uint8_t *) ((const float *) mmctx->vtcm_src2 + src0_start_row),
+                            copy_cnt);
+        } else {
+            hvx_copy_f32_ua((uint8_t *) &dst_col[src0_start_row], (uint8_t *) tmp, copy_cnt);
         }
-    } else {
-        hvx_copy_f32_ua((uint8_t *) &dst_col[src0_start_row], (uint8_t *) tmp, copy_cnt);
+        htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, src0_end_row);
     }
 }
 
-#define MMID_MATRIX_ROW(row_id, i1) matrix_rows[(row_id) * ids->ne[0] * ids->ne[1] + (i1)]
+#define MMID_MATRIX_ROW(row_id, i1) matrix_rows[(row_id) * mmctx->mapping_stride + (i1)]
 
 static void hvx_mm_id(unsigned int nth, unsigned int ith, void * data) {
     htp_matmul_preamble;
@@ -1519,7 +1528,7 @@ static int hvx_mm_matmul(struct htp_ops_context * octx) {
 
     struct htp_mm_hvx_vtcm_layout L;
     htp_mm_hvx_vtcm_layout_build(&L, kparams->kernel_type, src0->type, ne10, src1_nrows, octx->n_threads,
-                                 dst_row_size, src0_row_size, src1_row_size, kparams->n_prefetch, false, false, false);
+                                 dst_row_size, src0_row_size, src1_row_size, src2 ? src2->nb[1] : 0, kparams->n_prefetch, false, false, false);
 
     if (kparams->kernel_type == HTP_MM_KERNEL_HVX_F16_F16_VTCM ||
         kparams->kernel_type == HTP_MM_KERNEL_HVX_F32_F32_VTCM ||
@@ -1551,6 +1560,7 @@ static int hvx_mm_matmul(struct htp_ops_context * octx) {
     uint8_t * const base = (uint8_t *) octx->ctx->vtcm_base;
     mmctx->vtcm_src1 = VTCM_LAYOUT_PTR(uint8_t, base, L.off_src1);
     mmctx->vtcm_src0 = VTCM_LAYOUT_PTR(uint8_t, base, L.off_src0);
+    mmctx->vtcm_src2 = VTCM_LAYOUT_PTR(uint8_t, base, L.off_src2);
     mmctx->vtcm_dst  = VTCM_LAYOUT_PTR(uint8_t, base, L.off_dst);
 
     octx->src1_spad.src  = NULL;
@@ -2346,12 +2356,77 @@ static void dequantize_tiled_weight_chunk_to_fp16_tiles(
     }
 }
 
+typedef struct {
+    float *dst;
+    const float *src2;
+    const __fp16 *vtcm_src;
+    uint32_t n_rows;
+    uint32_t n_cols;
+    uint32_t dst_stride;
+    uint32_t src2_stride;
+    uint32_t dst_cols;
+    struct fastdiv_values n_threads_div;
+    struct htp_thread_trace *traces;
+    struct htp_context *ctx;
+} output_transfer_col_chunk_state_t;
+
+static void transfer_output_chunk_col_chunk_worker_fn(unsigned int n, unsigned int i, void *data) {
+    (void) n;
+    output_transfer_col_chunk_state_t *st = (output_transfer_col_chunk_state_t *) data;
+    struct htp_thread_trace * tr = &st->traces[i];
+
+    uint32_t n_blocks = st->n_cols / 32;
+    uint32_t b_first = fastdiv(n_blocks * i, &st->n_threads_div);
+    uint32_t b_last  = fastdiv(n_blocks * (i + 1), &st->n_threads_div);
+    uint32_t c_first = b_first * 32;
+    uint32_t c_last  = b_last * 32;
+    uint32_t c_len   = c_last - c_first;
+
+    if (c_len == 0) return;
+
+    htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_O_PROC, c_first);
+
+    float *dst = st->dst + c_first;
+    const float *src2 = st->src2 ? (st->src2 + c_first) : NULL;
+    const __fp16 *vtcm_src = st->vtcm_src + b_first * HTP_MM_HMX_TILE_N_ELMS;
+
+    int chunk_dst_cols = (int)st->dst_cols - (int)c_first;
+    if (chunk_dst_cols > 0) {
+        transfer_output_chunk_fp16_to_fp32_col_chunk(
+            dst, src2, vtcm_src, 0, st->n_rows, c_len, st->n_cols,
+            st->dst_stride, st->src2_stride, (uint32_t)chunk_dst_cols
+        );
+    }
+
+    htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_O_PROC, c_first);
+}
+
 static void transfer_output_chunk_threaded(struct htp_context *ctx, float *dst, const float *src2, const __fp16 *vtcm_src,
                                               int n_rows, int n_cols, int dst_stride, uint32_t src2_stride, int dst_cols, int n_threads) {
     assert(n_cols % HTP_MM_HMX_TILE_N_COLS == 0);
 
     if (n_rows <= 0) return;
 
+    uint32_t n_blocks = (uint32_t)n_cols / 32;
+    if (n_threads > 1 && n_blocks >= (uint32_t)n_threads) {
+        struct fastdiv_values n_threads_div = init_fastdiv_values(n_threads);
+        output_transfer_col_chunk_state_t col_state;
+        col_state.dst = dst;
+        col_state.src2 = src2;
+        col_state.vtcm_src = vtcm_src;
+        col_state.n_rows = (uint32_t)n_rows;
+        col_state.n_cols = (uint32_t)n_cols;
+        col_state.dst_stride = (uint32_t)dst_stride;
+        col_state.src2_stride = src2_stride;
+        col_state.dst_cols = (uint32_t)dst_cols;
+        col_state.n_threads_div = n_threads_div;
+        col_state.traces = ctx->trace;
+        col_state.ctx = ctx;
+
+        worker_pool_run_func(ctx->worker_pool, transfer_output_chunk_col_chunk_worker_fn, &col_state, n_threads);
+        return;
+    }
+
     size_t n_tot_chunks      = n_rows;
     size_t n_chunks_per_task = (n_threads == 1) ? n_tot_chunks : hmx_ceil_div(n_rows, n_threads);
     n_chunks_per_task        = hex_align_up(n_chunks_per_task, 2);
@@ -3338,12 +3413,10 @@ int op_matmul(struct htp_ops_context * octx) {
 
 static int hmx_mm_op_matmul_id(
     struct htp_ops_context * octx,
-    struct htp_mm_context * mmctx,
-    const uint32_t * matrix_row_counts,
-    const struct mmid_row_mapping * matrix_rows,
-    void * mapping_buf,
-    bool must_free_mapping
+    struct htp_mm_context * mmctx
 ) {
+    const uint32_t *                matrix_row_counts = mmctx->matrix_row_counts;
+    const struct mmid_row_mapping * matrix_rows       = mmctx->matrix_rows;
     htp_matmul_tensors_preamble;
     const struct htp_mm_kernel_params * kparams = (const struct htp_mm_kernel_params *) octx->kernel_params;
     const int n_ids = octx->src[2]->ne[0];
@@ -3361,28 +3434,24 @@ static int hmx_mm_op_matmul_id(
                                    nb11, nb12,
                                    nb1, nb2,
                                    (int) src0->nb[1], (int) src0->type,
-                                   matrix_rows, cur_a, n_ids * octx->src[2]->ne[1]);
+                                   matrix_rows, cur_a, mmctx->mapping_stride);
         if (ret != 0) {
             FARF(ERROR, "HMX matmul failed for expert %u, error %d\n", cur_a, ret);
-            if (must_free_mapping) free(mapping_buf);
             return HTP_STATUS_NO_SUPPORT;
         }
     }
 
-    if (must_free_mapping) free(mapping_buf);
     return HTP_STATUS_OK;
 }
 
 static int hvx_mm_matmul_id(
     struct htp_ops_context * octx,
     struct htp_mm_context * mmctx,
-    size_t src0_row_size_padded,
-    uint32_t src1_nrows,
-    worker_callback_t matmul_id_job_func,
-    void * mapping_buf,
-    bool must_free_mapping
+    work_queue_func_t hvx_mmid_task_func
 ) {
     htp_matmul_tensors_preamble;
+    const uint32_t src0_row_size_padded = mmctx->src0_row_size_padded;
+    const uint32_t src1_nrows           = mmctx->src1_nrows;
 
     struct htp_thread_trace * tr = &octx->ctx->trace[0];
     htp_trace_event_start(tr, HTP_TRACE_EVT_INIT, 0);
@@ -3395,7 +3464,7 @@ static int hvx_mm_matmul_id(
     const uint32_t nb = (ne10 + qk - 1) / qk;
     const uint32_t total_nb = src1_nrows * nb;
 
-    worker_callback_t quant_task_func;
+    work_queue_func_t quant_task_func;
     uint32_t n_quant_tasks = 1;
     if (src1_nrows < octx->n_threads) {
         n_quant_tasks = MIN(total_nb, octx->n_threads);
@@ -3416,7 +3485,7 @@ static int hvx_mm_matmul_id(
 
     struct htp_mm_hvx_vtcm_layout L;
     htp_mm_hvx_vtcm_layout_build(&L, kparams->kernel_type, src0->type, ne10, src1_nrows, octx->n_threads,
-                                 0, src0_row_size, src1_row_size, kparams->n_prefetch, true, false, false);
+                                 0, src0_row_size, src1_row_size, 0, kparams->n_prefetch, true, false, false);
 
     size_t vtcm_size = kparams->vtcm_size > 0 ? (size_t)kparams->vtcm_size : L.total_bytes;
 
@@ -3431,7 +3500,6 @@ static int hvx_mm_matmul_id(
     // Make sure the reserved vtcm size is sufficient
     if (octx->ctx->vtcm_size < vtcm_size) {
         FARF(ERROR, "matmul-id-%s : current VTCM reservation %zu is too small, needed %zu\n", mmctx->type, octx->ctx->vtcm_size, vtcm_size);
-        if (must_free_mapping) free(mapping_buf);
         return HTP_STATUS_VTCM_TOO_SMALL;
     }
 
@@ -3461,12 +3529,78 @@ static int hvx_mm_matmul_id(
 
     htp_trace_event_stop(tr, HTP_TRACE_EVT_INIT, 0);
 
-    worker_pool_run_func(octx->ctx->worker_pool, matmul_id_job_func, mmctx, octx->n_threads);
+    worker_pool_run_func(octx->ctx->worker_pool, hvx_mmid_task_func, mmctx, octx->n_threads);
 
-    if (must_free_mapping) free(mapping_buf);
     return HTP_STATUS_OK;
 }
 
+static inline void scan_expert_ids_n(
+    const struct htp_tensor * ids,
+    const uint32_t n_ids,
+    uint32_t n_as,
+    uint32_t * counts,
+    struct mmid_row_mapping * matrix_rows,
+    uint32_t mapping_stride
+) {
+    const size_t ids_nb1 = ids->nb[1];
+    const uint8_t * ids_data = (const uint8_t *) ids->data;
+
+    for (uint32_t iid1 = 0; iid1 < ids->ne[1]; ++iid1) {
+        const int32_t * row_ptr = (const int32_t *) (ids_data + iid1 * ids_nb1);
+        for (uint32_t id = 0; id < n_ids; ++id) {
+            const int32_t i02 = row_ptr[id];
+            if (i02 < 0) {
+                continue;
+            }
+            assert(i02 < n_as);
+
+            if (matrix_rows) {
+                matrix_rows[i02 * mapping_stride + counts[i02]] = (struct mmid_row_mapping) { id, iid1 };
+            }
+            counts[i02] += 1;
+        }
+    }
+}
+
+static inline void scan_expert_ids(
+    const struct htp_tensor * ids,
+    uint32_t n_ids,
+    uint32_t n_as,
+    uint32_t * counts,
+    struct mmid_row_mapping * matrix_rows,
+    uint32_t mapping_stride
+) {
+    const size_t ids_nb0 = ids->nb[0];
+
+    if (ids_nb0 == 4) {
+        switch (n_ids) {
+            case 8:  scan_expert_ids_n(ids, 8,     n_as, counts, matrix_rows, mapping_stride); break;
+            case 4:  scan_expert_ids_n(ids, 4,     n_as, counts, matrix_rows, mapping_stride); break;
+            case 2:  scan_expert_ids_n(ids, 2,     n_as, counts, matrix_rows, mapping_stride); break;
+            default: scan_expert_ids_n(ids, n_ids, n_as, counts, matrix_rows, mapping_stride); break;
+        }
+    } else {
+        // Strided fallback
+        const size_t ids_nb1 = ids->nb[1];
+        const uint8_t * ids_data = (const uint8_t *) ids->data;
+        for (uint32_t iid1 = 0; iid1 < ids->ne[1]; ++iid1) {
+            const int32_t * row_ptr = (const int32_t *) (ids_data + iid1 * ids_nb1);
+            for (uint32_t id = 0; id < n_ids; ++id) {
+                const int32_t i02 = *(const int32_t *) ((const uint8_t *) row_ptr + id * ids_nb0);
+                if (i02 < 0) {
+                    continue;
+                }
+                assert(i02 < n_as);
+
+                if (matrix_rows) {
+                    matrix_rows[i02 * mapping_stride + counts[i02]] = (struct mmid_row_mapping) { id, iid1 };
+                }
+                counts[i02] += 1;
+            }
+        }
+    }
+}
+
 int op_matmul_id(struct htp_ops_context * octx) {
     htp_matmul_tensors_preamble;
 
@@ -3489,74 +3623,72 @@ int op_matmul_id(struct htp_ops_context * octx) {
     const uint32_t src0_nrows = ne01;  // per expert
     const uint32_t src1_nrows = ne11 * ne12 * ne13;
 
-    worker_callback_t quant_task_func;
-    worker_callback_t matmul_id_job_func = src1_nrows > 1 ? hvx_mm_id : hvx_mv_id;
-
-    // Compute src0_nrows_per_thread
-    mmctx->src0_nrows_per_thread  = (src0_nrows + octx->n_threads - 1) / octx->n_threads;
-    mmctx->src0_nrows_per_thread  = hex_round_up(mmctx->src0_nrows_per_thread, 32);
+    mmctx->src0_nrows_per_thread = (src0_nrows + octx->n_threads - 1) / octx->n_threads;
+    mmctx->src0_nrows_per_thread = hex_round_up(mmctx->src0_nrows_per_thread, 32);
 
     // row groups
     const int n_ids = ids->ne[0];  // n_expert_used
     const int n_as  = ne02;        // n_expert
 
-    size_t matrix_row_counts_size = n_as * sizeof(uint32_t);
-    size_t matrix_row_map_size    = n_as * ids->ne[0] * ids->ne[1] * sizeof(struct mmid_row_mapping);
-    const size_t total_map_size   = matrix_row_counts_size + matrix_row_map_size;
-
-    void * mapping_buf = NULL;
-    bool must_free_mapping = false;
-
-    if (octx->ctx->ddr_spad_base && total_map_size <= octx->ctx->ddr_spad_size) {
-        mapping_buf = octx->ctx->ddr_spad_base;
-    } else {
-        mapping_buf = memalign(128, total_map_size);
-        if (mapping_buf) {
-            must_free_mapping = true;
-        } else {
-            return HTP_STATUS_INTERNAL_ERR;
-        }
-    }
-
-    uint32_t *                matrix_row_counts = (uint32_t *) mapping_buf;
-    struct mmid_row_mapping * matrix_rows       = (struct mmid_row_mapping *) ((uint8_t *) mapping_buf + matrix_row_counts_size);
-
-    mmctx->matrix_row_counts = matrix_row_counts;
-    mmctx->matrix_rows       = matrix_rows;
-    mmctx->mm_div_ne11       = kparams->div_ne11;
-
-    if (hvx_mm_init_vec_dot(mmctx, src0->type) != 0) {
-        if (must_free_mapping) free(mapping_buf);
-        return HTP_STATUS_NO_SUPPORT;
-    }
+    uint8_t  * mapping_buf       = octx->ctx->ddr_spad_base;
+    uint32_t   mapping_stride    = 1;
+    uint32_t * matrix_row_counts = (uint32_t *) mapping_buf;
+    struct mmid_row_mapping * matrix_rows = NULL;
 
     if (src1_nrows > 1) {
-        // initialize matrix_row_counts and map
-        memset(matrix_row_counts, 0, n_as * sizeof(uint32_t));
+        const size_t matrix_row_counts_size = n_as * sizeof(uint32_t);
+        assert(octx->ctx->ddr_spad_size >= matrix_row_counts_size);
 
-        // group rows by src0 matrix
-        for (uint32_t iid1 = 0; iid1 < ids->ne[1]; ++iid1) {  // token idx
-            for (uint32_t id = 0; id < n_ids; ++id) {         // expert idx
-                const int32_t i02 = *(const int32_t *) ((const uint8_t *) ids->data + iid1 * ids->nb[1] + id * ids->nb[0]);
+        hex_l2fetch_block((const void *) ids->data, ids->ne[1] * ids->nb[1]);
 
-                if (i02 < 0) {
-                    continue;
-                }
-                assert(i02 < n_as);
+        memset(matrix_row_counts, 0, matrix_row_counts_size);
+        scan_expert_ids(ids, n_ids, n_as, matrix_row_counts, NULL, 0);
 
-                matrix_rows[i02 * n_ids * ids->ne[1] + matrix_row_counts[i02]] = (struct mmid_row_mapping) { id, iid1 };
-                matrix_row_counts[i02] += 1;
+        uint32_t max_count = hvx_reduce_max_i32((const uint8_t *) matrix_row_counts, n_as);
+        mapping_stride = max_count > 0 ? max_count : 1;
+
+        size_t matrix_row_map_size  = n_as * mapping_stride * sizeof(struct mmid_row_mapping);
+        const size_t total_map_size = matrix_row_counts_size + matrix_row_map_size;
+
+        if (total_map_size > octx->ctx->ddr_spad_size) {
+            mapping_buf = memalign(128, total_map_size);
+            if (!mapping_buf) {
+                return HTP_STATUS_INTERNAL_ERR;
             }
         }
+
+        matrix_row_counts = (uint32_t *) mapping_buf;
+        matrix_rows       = (struct mmid_row_mapping *) (mapping_buf + matrix_row_counts_size);
+
+        memset(matrix_row_counts, 0, n_as * sizeof(uint32_t));
+        scan_expert_ids(ids, n_ids, n_as, matrix_row_counts, matrix_rows, mapping_stride);
     }
 
+    mmctx->matrix_row_counts    = matrix_row_counts;
+    mmctx->matrix_rows          = matrix_rows;
+    mmctx->mapping_stride       = mapping_stride;
+    mmctx->mm_div_ne11          = kparams->div_ne11;
+    mmctx->src0_row_size_padded = src0_row_size_padded;
+    mmctx->src1_nrows           = src1_nrows;
+
     htp_trace_event_stop(tr, HTP_TRACE_EVT_INIT, 0);
 
+    int s;
     if (kparams->n_hmx) {
-        return hmx_mm_op_matmul_id(octx, mmctx, matrix_row_counts, matrix_rows, mapping_buf, must_free_mapping);
+        s = hmx_mm_op_matmul_id(octx, mmctx);
+    } else {
+        if (hvx_mm_init_vec_dot(mmctx, src0->type) == 0) {
+            s = hvx_mm_matmul_id(octx, mmctx, src1_nrows > 1 ? hvx_mm_id : hvx_mv_id);
+        } else {
+            s = HTP_STATUS_NO_SUPPORT;
+        }
     }
 
-    return hvx_mm_matmul_id(octx, mmctx, src0_row_size_padded, src1_nrows, matmul_id_job_func, mapping_buf, must_free_mapping);
+    if (mapping_buf != octx->ctx->ddr_spad_base) {
+        free(mapping_buf);
+    }
+
+    return s;
 }
 
 int op_matmul_qkv(struct htp_ops_context * octx) {
@@ -3633,7 +3765,7 @@ int op_matmul_qkv(struct htp_ops_context * octx) {
 
     struct htp_mm_hvx_vtcm_layout L;
     htp_mm_hvx_vtcm_layout_build(&L, kparams->kernel_type, src0->type, src1->ne[0], src1_nrows, octx->n_threads,
-                                 0, src0_row_size, src1_row_size, kparams->n_prefetch, false, true, false);
+                                 0, src0_row_size, src1_row_size, 0, kparams->n_prefetch, false, true, false);
 
     size_t vtcm_size = kparams->vtcm_size > 0 ? (size_t)kparams->vtcm_size : L.total_bytes;
 
@@ -3778,7 +3910,7 @@ int op_matmul_ffn(struct htp_ops_context * octx) {
 
     struct htp_mm_hvx_vtcm_layout L;
     htp_mm_hvx_vtcm_layout_build(&L, kparams->kernel_type, src0->type, src1->ne[0], src1_nrows, octx->n_threads,
-                                 0, src0_row_size, src1_row_size, kparams->n_prefetch, false, false, true);
+                                 0, src0_row_size, src1_row_size, 0, kparams->n_prefetch, false, false, true);
 
     size_t vtcm_size = kparams->vtcm_size > 0 ? (size_t)kparams->vtcm_size : L.total_bytes;
 

ggml/src/ggml-hexagon/htp/matmul-ops.h

@@ -460,6 +460,7 @@ static inline void htp_mm_hvx_vtcm_layout_build(
     size_t dst_row_size,
     size_t src0_row_size,
     size_t src1_row_size,
+    size_t src2_row_size,
     uint32_t n_prefetch,
     bool is_matmul_id,
     bool is_fused_qkv,
@@ -467,7 +468,7 @@ static inline void htp_mm_hvx_vtcm_layout_build(
 ) {
     size_t src0_sz = 0;
     size_t src1_sz = 0;
-    size_t src2_sz = 0;
+    size_t src2_sz = src2_row_size > 0 ? htp_mm_round_up(src2_row_size, 128) : 0;
     size_t src3_sz = 0;
     size_t dst_sz  = 0;
 

ggml/src/ggml-hexagon/htp/unary-ops.c

@@ -276,6 +276,39 @@ static void sigmoid_f32(const float * restrict src,
     }
 }
 
+// silu(x) = x * sigmoid(x)
+static void silu_f32(const float * restrict src,
+                     float * restrict dst,
+                     const uint32_t num_rows,
+                     const struct htp_unary_context * uctx) {
+    htp_unary_op_preamble;
+
+    for (uint32_t ir = 0; ir < num_rows; ir++) {
+        const uint8_t * restrict src_local = (const uint8_t *)src + (ir * src0_row_size_aligned);
+        uint8_t * restrict dst_local       = (uint8_t *)dst + (ir * dst_row_size_aligned);
+
+        hvx_sigmoid_f32_aa(dst_local, src_local, ne0);
+        hvx_mul_f32_aaa(dst_local, src_local, dst_local, ne0);
+    }
+}
+
+// gelu(x) = x * sigmoid(1.702 * x)  (quick/sigmoid approximation, matches CPU GELU_QUICK reference)
+static void gelu_f32(const float * restrict src,
+                     float * restrict dst,
+                     const uint32_t num_rows,
+                     const struct htp_unary_context * uctx) {
+    htp_unary_op_preamble;
+
+    for (uint32_t ir = 0; ir < num_rows; ir++) {
+        const uint8_t * restrict src_local = (const uint8_t *)src + (ir * src0_row_size_aligned);
+        uint8_t * restrict dst_local       = (uint8_t *)dst + (ir * dst_row_size_aligned);
+
+        hvx_mul_scalar_f32(dst_local, src_local, 1.702f, ne0);
+        hvx_sigmoid_f32_aa(dst_local, dst_local, ne0);
+        hvx_mul_f32_aaa(dst_local, src_local, dst_local, ne0);
+    }
+}
+
 static void tri_f32(const float * restrict src,
                     float * restrict dst,
                     const uint32_t num_rows,
@@ -566,6 +599,8 @@ DEFINE_UNARY_TASK(sqrt,           false, false, sqrt_f32(src0_vtcm, dst_vtcm, bl
 DEFINE_UNARY_TASK(unary_neg,      false, false, neg_f32(src0_vtcm, dst_vtcm, block_size, uctx))
 DEFINE_UNARY_TASK(unary_exp,      false, false, exp_f32(src0_vtcm, dst_vtcm, block_size, uctx))
 DEFINE_UNARY_TASK(unary_sigmoid,  false, false, sigmoid_f32(src0_vtcm, dst_vtcm, block_size, uctx))
+DEFINE_UNARY_TASK(unary_silu,     false, false, silu_f32(src0_vtcm, dst_vtcm, block_size, uctx))
+DEFINE_UNARY_TASK(unary_gelu,     false, false, gelu_f32(src0_vtcm, dst_vtcm, block_size, uctx))
 DEFINE_UNARY_TASK(unary_softplus, false, false, softplus_f32(src0_vtcm, dst_vtcm, block_size, uctx))
 DEFINE_UNARY_TASK(unary_tanh,     false, false, tanh_f32(src0_vtcm, dst_vtcm, block_size, uctx))
 DEFINE_UNARY_TASK(l2_norm,        false, false, l2_norm_f32(src0_vtcm, dst_vtcm, block_size, uctx))
@@ -717,6 +752,19 @@ static inline void tile_unary_softplus_f32(uint8_t * dst_vtcm, const uint8_t * s
     }
 }
 
+// silu(x) = x * sigmoid(x)
+static inline void tile_silu_f32(uint8_t * dst_vtcm, const uint8_t * src_vtcm, uint32_t tw) {
+    hvx_sigmoid_f32_aa(dst_vtcm, src_vtcm, tw);
+    hvx_mul_f32_aaa(dst_vtcm, src_vtcm, dst_vtcm, tw);
+}
+
+// gelu(x) = x * sigmoid(1.702 * x)  (quick/sigmoid approximation, matches CPU GELU_QUICK reference)
+static inline void tile_gelu_f32(uint8_t * dst_vtcm, const uint8_t * src_vtcm, uint32_t tw) {
+    hvx_mul_scalar_f32(dst_vtcm, src_vtcm, 1.702f, tw);
+    hvx_sigmoid_f32_aa(dst_vtcm, dst_vtcm, tw);
+    hvx_mul_f32_aaa(dst_vtcm, src_vtcm, dst_vtcm, tw);
+}
+
 // Triangular mask applied to one column tile. Boundary is an absolute column index, so
 // each vector compares against its absolute column position (col_start + i*VLEN_FP32).
 static inline void tri_apply_tile_f32(const uint8_t * restrict src, uint8_t * restrict dst,
@@ -798,6 +846,8 @@ DEFINE_UNARY_TILED_TASK(sqrt,           false, hvx_sqrt_f32_aa(dst_vtcm, src_vtc
 DEFINE_UNARY_TILED_TASK(unary_neg,      false, hvx_scale_f32_aa(dst_vtcm, src_vtcm, tw, -1.0f))
 DEFINE_UNARY_TILED_TASK(unary_exp,      false, hvx_exp_f32(dst_vtcm, src_vtcm, tw, false))
 DEFINE_UNARY_TILED_TASK(unary_sigmoid,  false, hvx_sigmoid_f32_aa(dst_vtcm, src_vtcm, tw))
+DEFINE_UNARY_TILED_TASK(unary_silu,     false, tile_silu_f32(dst_vtcm, src_vtcm, tw))
+DEFINE_UNARY_TILED_TASK(unary_gelu,     false, tile_gelu_f32(dst_vtcm, src_vtcm, tw))
 DEFINE_UNARY_TILED_TASK(unary_softplus, false, tile_unary_softplus_f32(dst_vtcm, src_vtcm, tw))
 DEFINE_UNARY_TILED_TASK(unary_tanh,     false, hvx_tanh_f32_aa(dst_vtcm, src_vtcm, tw))
 DEFINE_UNARY_TILED_TASK(tri,            true,  tri_apply_tile_f32(src_vtcm, dst_vtcm, tw, col, i01, ne0, tri_ttype))
@@ -821,6 +871,8 @@ static int execute_op_unary_f32(struct htp_ops_context * octx) {
         case HTP_OP_UNARY_NEG:       op_type = "neg-f32";          break;
         case HTP_OP_UNARY_EXP:       op_type = "exp-f32";          break;
         case HTP_OP_UNARY_SIGMOID:   op_type = "sigmoid-f32";      break;
+        case HTP_OP_UNARY_SILU:      op_type = "silu-f32";         break;
+        case HTP_OP_UNARY_GELU:      op_type = "gelu-f32";         break;
         case HTP_OP_UNARY_SOFTPLUS:  op_type = "softplus-f32";     break;
         case HTP_OP_UNARY_TANH:      op_type = "tanh-f32";         break;
         case HTP_OP_L2_NORM:         op_type = "l2norm-f32";       break;
@@ -917,6 +969,8 @@ static int execute_op_unary_f32(struct htp_ops_context * octx) {
                 case HTP_OP_UNARY_NEG:       task_func = unary_task_f32_tiled_unary_neg;      break;
                 case HTP_OP_UNARY_EXP:       task_func = unary_task_f32_tiled_unary_exp;      break;
                 case HTP_OP_UNARY_SIGMOID:   task_func = unary_task_f32_tiled_unary_sigmoid;  break;
+                case HTP_OP_UNARY_SILU:      task_func = unary_task_f32_tiled_unary_silu;     break;
+                case HTP_OP_UNARY_GELU:      task_func = unary_task_f32_tiled_unary_gelu;     break;
                 case HTP_OP_UNARY_SOFTPLUS:  task_func = unary_task_f32_tiled_unary_softplus; break;
                 case HTP_OP_UNARY_TANH:      task_func = unary_task_f32_tiled_unary_tanh;     break;
                 case HTP_OP_TRI:             task_func = unary_task_f32_tiled_tri;            break;
@@ -934,6 +988,8 @@ static int execute_op_unary_f32(struct htp_ops_context * octx) {
                 case HTP_OP_UNARY_NEG:       task_func = unary_task_f32_unary_neg;            break;
                 case HTP_OP_UNARY_EXP:       task_func = unary_task_f32_unary_exp;            break;
                 case HTP_OP_UNARY_SIGMOID:   task_func = unary_task_f32_unary_sigmoid;        break;
+                case HTP_OP_UNARY_SILU:      task_func = unary_task_f32_unary_silu;           break;
+                case HTP_OP_UNARY_GELU:      task_func = unary_task_f32_unary_gelu;           break;
                 case HTP_OP_UNARY_SOFTPLUS:  task_func = unary_task_f32_unary_softplus;       break;
                 case HTP_OP_UNARY_TANH:      task_func = unary_task_f32_unary_tanh;           break;
                 case HTP_OP_L2_NORM:         task_func = unary_task_f32_l2_norm;              break;

ggml/src/ggml-hexagon/htp/unary-ops.h

@@ -51,6 +51,8 @@ static inline bool htp_op_is_unary(uint32_t opcode) {
         case HTP_OP_UNARY_NEG:
         case HTP_OP_UNARY_EXP:
         case HTP_OP_UNARY_SIGMOID:
+        case HTP_OP_UNARY_SILU:
+        case HTP_OP_UNARY_GELU:
         case HTP_OP_UNARY_SOFTPLUS:
         case HTP_OP_UNARY_TANH:
         case HTP_OP_L2_NORM:

ggml/src/ggml-metal/ggml-metal-device.m

@@ -1218,8 +1218,9 @@ bool ggml_metal_device_supports_op(ggml_metal_device_t dev, const struct ggml_te
                    (ggml_get_op_params_i32(op, 4) == 0) && (ggml_get_op_params_i32(op, 6) == 0);
         case GGML_OP_PAD_REFLECT_1D:
         case GGML_OP_TIMESTEP_EMBEDDING:
-        case GGML_OP_LEAKY_RELU:
             return op->src[0]->type == GGML_TYPE_F32;
+        case GGML_OP_LEAKY_RELU:
+            return op->src[0]->type == GGML_TYPE_F32 || op->src[0]->type == GGML_TYPE_F16;
         case GGML_OP_ARGSORT:
         case GGML_OP_TOP_K:
         case GGML_OP_ARANGE:

include/llama.h

@@ -202,6 +202,16 @@ extern "C" {
         LLAMA_SPLIT_MODE_TENSOR = 3,
     };
 
+    enum llama_load_mode {
+        LLAMA_LOAD_MODE_NONE      = 0, // no special loading mode
+        LLAMA_LOAD_MODE_MMAP      = 1, // memory map the model
+        LLAMA_LOAD_MODE_MLOCK     = 2, // mmap + force system to keep model in RAM rather than swapping or compressing
+        LLAMA_LOAD_MODE_DIRECT_IO = 3, // use direct I/O if available
+    };
+
+    LLAMA_API const char * llama_load_mode_name(enum llama_load_mode load_mode);
+    LLAMA_API enum llama_load_mode llama_load_mode_from_str(const char * str);
+
     enum llama_context_type {
         LLAMA_CONTEXT_TYPE_DEFAULT = 0,
         LLAMA_CONTEXT_TYPE_MTP     = 1,
@@ -301,6 +311,7 @@ extern "C" {
 
         int32_t n_gpu_layers; // number of layers to store in VRAM, a negative value means all layers
         enum llama_split_mode split_mode; // how to split the model across multiple GPUs
+        enum llama_load_mode  load_mode;  // how to load the model
 
         // the GPU that is used for the entire model when split_mode is LLAMA_SPLIT_MODE_NONE
         int32_t main_gpu;
@@ -321,9 +332,6 @@ extern "C" {
 
         // Keep the booleans together to avoid misalignment during copy-by-value.
         bool vocab_only;      // only load the vocabulary, no weights
-        bool use_mmap;        // use mmap if possible
-        bool use_direct_io;   // use direct io, takes precedence over use_mmap when supported
-        bool use_mlock;       // force system to keep model in RAM
         bool check_tensors;   // validate model tensor data
         bool use_extra_bufts; // use extra buffer types (used for weight repacking)
         bool no_host;         // bypass host buffer allowing extra buffers to be used

models/templates/deepseek-ai-DeepSeek-V4.jinja

@@ -8,12 +8,15 @@
     {%- set thinking = false -%}
   {%- endif -%}
 {%- endif -%}
+{%- if not drop_thinking is defined -%}
+  {%- set drop_thinking = false -%}
+{%- endif -%}
 {%- set dsml_token = '｜DSML｜' -%}
 {%- set thinking_start_token = '<think>' -%}
 {%- set thinking_end_token = '</think>' -%}
 {%- set tools_header = '## Tools\n\nYou have access to a set of tools to help answer the user\'s question. You can invoke tools by writing a "<' + dsml_token + 'tool_calls>" block like the following:\n\n<' + dsml_token + 'tool_calls>\n<' + dsml_token + 'invoke name="$TOOL_NAME">\n<' + dsml_token + 'parameter name="$PARAMETER_NAME" string="true|false">$PARAMETER_VALUE</' + dsml_token + 'parameter>\n...\n</' + dsml_token + 'invoke>\n<' + dsml_token + 'invoke name="$TOOL_NAME2">\n...\n</' + dsml_token + 'invoke>\n</' + dsml_token + 'tool_calls>\n\nString parameters should be specified as is and set `string="true"`. For all other types (numbers, booleans, arrays, objects), pass the value in JSON format and set `string="false"`.\n\nIf thinking_mode is enabled (triggered by ' + thinking_start_token + '), you MUST output your complete reasoning inside ' + thinking_start_token + '...' + thinking_end_token + ' BEFORE any tool calls or final response.\n\nOtherwise, output directly after ' + thinking_end_token + ' with tool calls or final response.\n\n### Available Tool Schemas\n\n' -%}
 {%- set tools_footer = '\nYou MUST strictly follow the above defined tool name and parameter schemas to invoke tool calls.\n' -%}
-{%- set ns = namespace(system_prompt='', is_first_sp=true) -%}
+{%- set ns = namespace(system_prompt='', is_first_sp=true, has_tool_calls=false) -%}
 {%- for message in messages -%}
   {%- if message['role'] == 'system' -%}
     {%- if ns.is_first_sp -%}
@@ -46,6 +49,11 @@
   {%- endif -%}
 {%- endfor -%}
 {%- set state = namespace(in_user=false) -%}
+{%- for message in messages -%}
+  {%- if message['role'] == 'tool' -%}
+    {%- set ns.has_tool_calls = true -%}
+  {%- endif -%}
+{%- endfor -%}
 {%- for message in messages -%}
   {%- if message['role'] == 'user' or message['role'] == 'developer' -%}
     {%- if state.in_user -%}
@@ -67,7 +75,8 @@
     {%- set state.in_user = false -%}
     {{- '<｜Assistant｜>' -}}
     {%- set is_after_last_user = loop.index0 > last_user_idx.value -%}
-    {%- if is_after_last_user and thinking -%}
+    {%- set retain_reasoning = (not drop_thinking) or (is_after_last_user or ns.has_tool_calls) -%}
+    {%- if retain_reasoning and thinking -%}
       {{- thinking_start_token -}}
       {%- if message['reasoning_content'] is defined and message['reasoning_content'] -%}
         {{- message['reasoning_content'] -}}

scripts/compare-llama-bench.py

@@ -28,7 +28,7 @@ LLAMA_BENCH_DB_FIELDS = [
     "model_type",   "model_size",   "model_n_params", "n_batch",    "n_ubatch",     "n_threads",
     "cpu_mask",     "cpu_strict",   "poll",           "type_k",     "type_v",       "n_gpu_layers",
     "split_mode",   "main_gpu",     "no_kv_offload",  "flash_attn", "tensor_split", "tensor_buft_overrides",
-    "use_mmap",     "embeddings",   "no_op_offload",  "n_prompt",   "n_gen",        "n_depth",
+    "load_mode",    "embeddings",   "no_op_offload",  "n_prompt",   "n_gen",        "n_depth",
     "test_time",    "avg_ns",       "stddev_ns",      "avg_ts",     "stddev_ts",    "n_cpu_moe",
     "fit_target",   "fit_min_ctx"
 ]
@@ -38,7 +38,7 @@ LLAMA_BENCH_DB_TYPES = [
     "TEXT",    "INTEGER", "INTEGER", "INTEGER", "INTEGER", "INTEGER",
     "TEXT",    "INTEGER", "INTEGER", "TEXT",    "TEXT",    "INTEGER",
     "TEXT",    "INTEGER", "INTEGER", "INTEGER", "TEXT",    "TEXT",
-    "INTEGER", "INTEGER", "INTEGER", "INTEGER", "INTEGER", "INTEGER",
+    "TEXT", "INTEGER", "INTEGER", "INTEGER", "INTEGER", "INTEGER",
     "TEXT",    "INTEGER", "INTEGER", "REAL",    "REAL",    "INTEGER",
     "INTEGER", "INTEGER"
 ]
@@ -63,7 +63,7 @@ assert len(TEST_BACKEND_OPS_DB_FIELDS) == len(TEST_BACKEND_OPS_DB_TYPES)
 LLAMA_BENCH_KEY_PROPERTIES = [
     "cpu_info", "gpu_info", "backends", "n_gpu_layers", "n_cpu_moe", "tensor_buft_overrides", "model_filename", "model_type",
     "n_batch", "n_ubatch", "embeddings", "cpu_mask", "cpu_strict", "poll", "n_threads", "type_k", "type_v",
-    "use_mmap", "no_kv_offload", "split_mode", "main_gpu", "tensor_split", "flash_attn", "n_prompt", "n_gen", "n_depth",
+    "load_mode", "no_kv_offload", "split_mode", "main_gpu", "tensor_split", "flash_attn", "n_prompt", "n_gen", "n_depth",
     "fit_target", "fit_min_ctx"
 ]
 
@@ -73,7 +73,7 @@ TEST_BACKEND_OPS_KEY_PROPERTIES = [
 ]
 
 # Properties that are boolean and are converted to Yes/No for the table:
-LLAMA_BENCH_BOOL_PROPERTIES = ["embeddings", "cpu_strict", "use_mmap", "no_kv_offload", "flash_attn"]
+LLAMA_BENCH_BOOL_PROPERTIES = ["embeddings", "cpu_strict", "no_kv_offload", "flash_attn"]
 TEST_BACKEND_OPS_BOOL_PROPERTIES = ["supported", "passed"]
 
 # Header names for the table (llama-bench):
@@ -82,7 +82,7 @@ LLAMA_BENCH_PRETTY_NAMES = {
     "tensor_buft_overrides": "Tensor overrides", "model_filename": "File", "model_type": "Model", "model_size": "Model size [GiB]",
     "model_n_params": "Num. of par.", "n_batch": "Batch size", "n_ubatch": "Microbatch size", "embeddings": "Embeddings",
     "cpu_mask": "CPU mask", "cpu_strict": "CPU strict", "poll": "Poll", "n_threads": "Threads", "type_k": "K type", "type_v": "V type",
-    "use_mmap": "Use mmap", "no_kv_offload": "NKVO", "split_mode": "Split mode", "main_gpu": "Main GPU", "tensor_split": "Tensor split",
+    "load_mode": "Load mode", "no_kv_offload": "NKVO", "split_mode": "Split mode", "main_gpu": "Main GPU", "tensor_split": "Tensor split",
     "flash_attn": "FlashAttention",
 }
 

scripts/snapdragon/ggml-hexagon-profile.py

@@ -6,6 +6,7 @@ import re
 import argparse
 import statistics
 import logging
+import bisect
 from typing import Any, Dict, List, Optional
 
 from collections import defaultdict
@@ -30,7 +31,7 @@ op_pattern = re.compile(
 )
 
 trace_pattern = re.compile(
-    r"trace-op\s+(?P<op_name>[A-Z_0-9+]+):\s+thread\s+(?P<thread>\d+)\s+event\s+(?P<event>[A-Z_0-9\-]+)\s+info\s+(?P<info>\d+)\s+(?P<state>start|stop)\s+(?P<cycles>\d+)"
+    r"trace-evt\s+(?P<event>[A-Z_0-9\-]+):\s+thread\s+(?P<thread>\d+)\s+info\s+(?P<info>\d+)\s+(?P<state>start|stop)\s+(?P<cycles>\d+)"
 )
 
 logger = logging.getLogger("ggml-hexagon-profile")
@@ -50,9 +51,13 @@ def normalize_event_name(evt_type):
 
 
 class CycleUnwrapper:
-    def __init__(self):
-        self.last_raw = None
-        self.high_part = 0
+    def __init__(self, initial_val=None):
+        if initial_val is not None:
+            self.last_raw = initial_val & 0xFFFFFFFF
+            self.high_part = initial_val & 0xFFFFFFFF00000000
+        else:
+            self.last_raw = None
+            self.high_part = 0
 
     def unwrap(self, raw):
         if self.last_raw is None:
@@ -78,10 +83,12 @@ def parse_log(file_path, pmu_index=None):
         sys.exit(1)
 
     all_ops: List[Dict[str, Any]] = []
+    all_traces: List[Dict[str, Any]] = []
     current_op: Optional[Dict[str, Any]] = None
 
     timestamp_pattern = re.compile(r"^(?P<min>\d+)\.(?P<sec>\d+)\.(?P<ms>\d+)\.(?P<us>\d+)\s+[A-Z]\s+")
-    unwrapper = CycleUnwrapper()
+    unwrapper = None
+    trace_unwrapper = None
 
     for line in f:
         ts_match = timestamp_pattern.match(line)
@@ -100,6 +107,7 @@ def parse_log(file_path, pmu_index=None):
             if not prefix_match:
                 continue
 
+            names = parts[1]
             if len(parts) == 7:
                 dims, types, timings = parts[2], parts[3], parts[6]
             elif len(parts) == 6:
@@ -120,6 +128,7 @@ def parse_log(file_path, pmu_index=None):
             op_match = op_pattern.search(line)
             if op_match:
                 op_name = op_match.group('op_name')
+                names = ""
                 dims = op_match.group('dims').strip()
                 types = op_match.group('types').strip()
             else:
@@ -136,24 +145,31 @@ def parse_log(file_path, pmu_index=None):
                 except (ValueError, IndexError):
                     pmu_val = None
 
-            evt_raw = op_match.group('evt') if 'evt' in op_match.groupdict() else None
             evt_val = None
-            if evt_raw:
+            evt_val = None
+            if types.startswith("evt-cnt "):
                 try:
-                    evt_val = [int(x.strip()) for x in evt_raw.split(',')]
+                    evt_val = [int(x.strip()) for x in types[8:].split(',')]
                 except ValueError:
                     evt_val = None
 
             cycles_start_raw = op_match.group('start')
             unwrapped_cycles_start = None
-            if cycles_start_raw:
-                unwrapped_cycles_start = unwrapper.unwrap(int(cycles_start_raw))
+            if op_name == "OPBATCH":
+                if cycles_start_raw:
+                    unwrapped_cycles_start = int(cycles_start_raw)
+                    unwrapper = CycleUnwrapper(unwrapped_cycles_start)
+                    trace_unwrapper = CycleUnwrapper(unwrapped_cycles_start)
+            else:
+                if cycles_start_raw and unwrapper is not None:
+                    unwrapped_cycles_start = unwrapper.unwrap(int(cycles_start_raw))
 
             idx = line.find("profile-op ")
             op_text = line[idx + 11:].strip() if idx != -1 else line.strip()
 
             current_op = {
                 'name':         op_name,
+                'names':        names,
                 'dims':         dims,
                 'types':        types,
                 'op_text':      op_text,
@@ -170,110 +186,239 @@ def parse_log(file_path, pmu_index=None):
             continue
 
         trace_match = trace_pattern.search(line)
-        if trace_match and current_op:
-            if trace_match.group('op_name') == current_op['name']:
-                raw_cyc = int(trace_match.group('cycles'))
-                current_op['trace_events'].append({
-                    'thread': int(trace_match.group('thread')),
-                    'event':  trace_match.group('event'),
-                    'info':   int(trace_match.group('info')),
-                    'cycles': raw_cyc,
-                    'unwrapped_cycles': unwrapper.unwrap(raw_cyc),
-                    'state':  trace_match.group('state')
-                })
+        if trace_match:
+            raw_cyc = int(trace_match.group('cycles'))
+            unwrapped_cyc = None
+            if trace_unwrapper is not None:
+                unwrapped_cyc = trace_unwrapper.unwrap(raw_cyc)
+            all_traces.append({
+                'thread': int(trace_match.group('thread')),
+                'event':  trace_match.group('event'),
+                'info':   int(trace_match.group('info')),
+                'cycles': raw_cyc,
+                'unwrapped_cycles': unwrapped_cyc,
+                'state':  trace_match.group('state')
+            })
 
     f.close()
+
+    # Assign start/end cycles to all ops
+    for op in all_ops:
+        op['start_cycles'] = op['unwrapped_cycles_start']
+        op['end_cycles'] = op['start_cycles'] + op['cycles'] if op['start_cycles'] is not None else None
+
+    # Filter ops with valid start_cycles
+    valid_ops = [op for op in all_ops if op['start_cycles'] is not None and op['end_cycles'] is not None]
+
+    # Separate OPBATCH ops from other ops
+    opbatch_ops = [op for op in valid_ops if op['name'] == "OPBATCH"]
+    other_ops = [op for op in valid_ops if op['name'] != "OPBATCH"]
+
+    # Sort them by start_cycles to enable binary search
+    opbatch_ops.sort(key=lambda op: op['start_cycles'])
+    other_ops.sort(key=lambda op: op['start_cycles'])
+
+    opbatch_starts = [op['start_cycles'] for op in opbatch_ops]
+    other_starts = [op['start_cycles'] for op in other_ops]
+
+    # Map trace events to any operator whose cycles contain them
+    for e in all_traces:
+        cyc = e['unwrapped_cycles']
+        if cyc is None:
+            continue
+
+        # Map to OPBATCH
+        idx = bisect.bisect_right(opbatch_starts, cyc) - 1
+        if idx >= 0:
+            op = opbatch_ops[idx]
+            if op['start_cycles'] <= cyc <= op['end_cycles']:
+                op['trace_events'].append(e)
+
+        # Map to other ops
+        idx = bisect.bisect_right(other_starts, cyc) - 1
+        if idx >= 0:
+            op = other_ops[idx]
+            if op['start_cycles'] <= cyc <= op['end_cycles']:
+                op['trace_events'].append(e)
+
     return all_ops
 
 
-def print_ascii_timeline(op_name, dims, types, usec, cycles, events, evt_val=None):
-    evt_str = ""
-    if evt_val:
-        evt_str = " - evt [" + ",".join(str(x) for x in evt_val) + "]"
+def print_bubbles_timeline(op):
+    op_name = op['name']
+    dims = op['dims']
+    types = op['types']
+    usec = op['usec']
+    cycles = op['cycles']
+    events = op['trace_events']
     logger.info("=" * 100)
-    logger.info(f"{op_name} ({dims} : {types}) - {usec} usec {cycles} cycles{evt_str}")
+    logger.info(f"{op_name} ({dims} : {types}) - {usec} usec {cycles} cycles")
     logger.info("=" * 100)
 
-    events = sorted(events, key=lambda e: e['cycles'])
     if not events:
         logger.info("  No trace events recorded.")
         return
 
-    min_cycles = events[0]['cycles']
+    # Identify start and end cycles for this operator
+    op_start = op['start_cycles']
+    op_end = op['end_cycles']
+    if op_start is None or op_end is None:
+        logger.info("  Cannot analyze bubbles: missing start/end cycle counts.")
+        return
 
-    logger.info("Cycles      %-30s" % "EventDetails" + " ".join(f"T{i:<2}" for i in range(10)) + " HMX")
-    logger.info("-" * 100)
-
-    thread_stacks = [[] for _ in range(11)]
+    batch_duration = op_end - op_start
+    if batch_duration <= 0:
+        logger.info("  Cannot analyze bubbles: batch duration is 0.")
+        return
 
+    # Group events by (thread, track_type)
+    tracks = defaultdict(list)
     for e in events:
         t = e['thread']
-        if t < 0 or t > 10:
-            continue
+        is_dma = (normalize_event_name(e['event']) == 'DMA')
+        track_type = 'dma' if is_dma else 'compute'
+        tracks[(t, track_type)].append(e)
 
-        if e['cycles'] >= min_cycles:
-            rel_cycles = e['cycles'] - min_cycles
-        else:
-            rel_cycles = (e['cycles'] + 0x100000000) - min_cycles
+    active_threads = sorted(list(set(t for (t, track_type) in tracks.keys())))
+    if not active_threads:
+        logger.info("  No active threads in trace.")
+        return
 
-        state = e['state']
-        evt_type = e['event']
+    bubble_threshold = 10000  # 10k cycles
 
-        # Determine char representing the event
-        norm_evt = normalize_event_name(evt_type)
-        char = '?'
-        if norm_evt == 'V-COMP':
-            char = 'V'
-        elif norm_evt == 'M-COMP':
-            char = 'H'
-        elif norm_evt == 'A-QUANT':
-            char = 'Q'
-        elif norm_evt == 'A-PREP':
-            char = 'A'
-        elif norm_evt == 'Q-PREP':
-            char = 'q'
-        elif norm_evt == 'K-PREP':
-            char = 'k'
-        elif norm_evt == 'V-PREP':
-            char = 'v'
-        elif norm_evt == 'W-DEQUANT':
-            char = 'D'
-        elif norm_evt == 'O-PROC':
-            char = 'O'
-        elif norm_evt == 'W-PREP':
-            char = 'P'
-        elif norm_evt == 'DMA':
-            char = 'M'
+    thread_stats = {}
+    for t in active_threads:
+        thread_stats[t] = {
+            'compute_idle_cycles': batch_duration,
+            'compute_idle_pct': 100.0,
+            'compute_bubbles': [],
 
-        if state == 'start':
-            thread_stacks[t].append(char)
-        elif state == 'stop':
-            if thread_stacks[t]:
-                if thread_stacks[t][-1] == char:
-                    thread_stacks[t].pop()
-                elif char in thread_stacks[t]:
-                    thread_stacks[t].remove(char)
-                else:
-                    thread_stacks[t].pop()
+            'dma_idle_cycles': batch_duration,
+            'dma_idle_pct': 100.0,
+            'dma_bubbles': []
+        }
 
-        cols = []
-        for i in range(11):
-            if thread_stacks[i]:
-                cols.append(f"[{thread_stacks[i][-1]}]")
+    total_compute_idle_pct = 0.0
+    total_dma_idle_pct = 0.0
+
+    for t in active_threads:
+        for track_type in ['compute', 'dma']:
+            key = (t, track_type)
+            track_events = tracks.get(key, [])
+
+            if not track_events:
+                gaps = [(op_start, op_end)]
+                idle_cycles = batch_duration
             else:
-                cols.append(" | ")
+                track_events = sorted(track_events, key=lambda e: e.get('unwrapped_cycles') or e['cycles'])
 
-        evt_desc = f"T{t}: {evt_type} {state} ({e['info']})"
-        logger.info(f"{rel_cycles:10d}  %-30s" % evt_desc + " ".join(cols[:10]) + "  " + cols[10])
+                active_intervals = []
+                active_count = 0
+                curr_start = None
+
+                for e in track_events:
+                    cyc = e.get('unwrapped_cycles') or e['cycles']
+                    cyc = max(op_start, min(op_end, cyc))
+                    state = e['state']
+
+                    if state == 'start':
+                        if active_count == 0:
+                            curr_start = cyc
+                        active_count += 1
+                    elif state == 'stop':
+                        if active_count > 0:
+                            active_count -= 1
+                            if active_count == 0:
+                                active_intervals.append((curr_start, cyc))
+                        else:
+                            active_intervals.append((op_start, cyc))
+
+                if active_count > 0 and curr_start is not None:
+                    active_intervals.append((curr_start, op_end))
+
+                # Merge intervals
+                active_intervals.sort(key=lambda x: x[0])
+                merged_intervals = []
+                for start, end in active_intervals:
+                    if not merged_intervals:
+                        merged_intervals.append([start, end])
+                    else:
+                        last_start, last_end = merged_intervals[-1]
+                        if start <= last_end:
+                            merged_intervals[-1][1] = max(last_end, end)
+                        else:
+                            merged_intervals.append([start, end])
+
+                # Calculate gaps
+                gaps = []
+                curr_time = op_start
+                for start, end in merged_intervals:
+                    if start > curr_time:
+                        gaps.append((curr_time, start))
+                    curr_time = max(curr_time, end)
+                if curr_time < op_end:
+                    gaps.append((curr_time, op_end))
+
+                idle_cycles = sum(end - start for start, end in gaps)
+
+            idle_pct = (idle_cycles / batch_duration) * 100.0
+
+            bubbles = []
+            for start, end in gaps:
+                dur = end - start
+                if dur >= bubble_threshold:
+                    bubbles.append((start, end, dur))
+
+            if track_type == 'compute':
+                thread_stats[t]['compute_idle_cycles'] = idle_cycles
+                thread_stats[t]['compute_idle_pct'] = idle_pct
+                thread_stats[t]['compute_bubbles'] = bubbles
+                total_compute_idle_pct += idle_pct
+            else:
+                thread_stats[t]['dma_idle_cycles'] = idle_cycles
+                thread_stats[t]['dma_idle_pct'] = idle_pct
+                thread_stats[t]['dma_bubbles'] = bubbles
+                total_dma_idle_pct += idle_pct
+
+    avg_compute_idle = total_compute_idle_pct / len(active_threads)
+    avg_dma_idle = total_dma_idle_pct / len(active_threads)
+
+    logger.info("  Combined Idle Statistics:")
+    logger.info(f"    Active Threads   : {', '.join(str(t) for t in active_threads)}")
+    logger.info(f"    Avg Thread Compute IDLE : {avg_compute_idle:.1f}%")
+    logger.info(f"    Avg Thread DMA IDLE     : {avg_dma_idle:.1f}%")
     logger.info("-" * 100)
 
+    logger.info("  Per-Thread Idle Analysis:")
+    for t in active_threads:
+        stats = thread_stats[t]
+        thread_name = f"Thread {t:<2} (HVX)" if t != 10 else "Thread 10 (HMX)"
+        logger.info(f"    {thread_name} -> Compute Idle: {stats['compute_idle_pct']:.1f}% | DMA Idle: {stats['dma_idle_pct']:.1f}%")
 
-def print_ascii_summary(op_name, dims, types, usec, cycles, events, evt_val=None):
-    evt_str = ""
-    if evt_val:
-        evt_str = " - evt [" + ",".join(str(x) for x in evt_val) + "]"
+    all_bubbles = []
+    for t in active_threads:
+        stats = thread_stats[t]
+        for start, end, dur in stats['compute_bubbles']:
+            pct = (dur / batch_duration) * 100.0
+            all_bubbles.append((dur, f"Thread {t} Compute: bubble of {dur} cycles ({pct:.1f}%) at {start - op_start} to {end - op_start}"))
+        for start, end, dur in stats['dma_bubbles']:
+            pct = (dur / batch_duration) * 100.0
+            all_bubbles.append((dur, f"Thread {t} DMA    : bubble of {dur} cycles ({pct:.1f}%) at {start - op_start} to {end - op_start}"))
+
+    if all_bubbles:
+        logger.info("-" * 100)
+        logger.info(f"  Significant Bubbles (>= {bubble_threshold} cycles):")
+        all_bubbles.sort(key=lambda x: x[0], reverse=True)
+        for dur, desc in all_bubbles[:15]:
+            logger.info(f"    {desc}")
+    else:
+        logger.info("-" * 100)
+        logger.info(f"  No significant bubbles detected (all idle gaps < {bubble_threshold} cycles).")
+
+
+def print_ascii_summary(op_name, dims, types, usec, cycles, events):
     logger.info("=" * 100)
-    logger.info(f"{op_name} ({dims} : {types}) - {usec} usec {cycles} cycles{evt_str}")
+    logger.info(f"{op_name} ({dims} : {types}) - {usec} usec {cycles} cycles")
     logger.info("=" * 100)
 
     events = sorted(events, key=lambda e: e['cycles'])
@@ -415,8 +560,8 @@ def main():
     parser.add_argument("--pmu-index", type=int)
     parser.add_argument("--pmu-name", type=str)
     parser.add_argument("--width", action='append', default=['dims:40'], help="Override column width, e.g. --width dims:50")
-    parser.add_argument("--timeline", type=str, nargs='?', const='summary', choices=["summary", "diagram"],
-                        help="Output ASCII art event summary or timing diagram (default: summary)")
+    parser.add_argument("--timeline", type=str, nargs='?', const='summary', choices=["summary", "bubbles"],
+                        help="Output ASCII art event summary or thread idle bubble analysis (default: summary)")
     parser.add_argument("--filter", type=str, help="Regex filter matching against the original profile-op line")
 
     group = parser.add_mutually_exclusive_group()
@@ -457,12 +602,11 @@ def main():
         ops = ops[-args.tail:]
 
     if args.timeline:
-        logger.info(f"\n# ASCII Timing {args.timeline.capitalize()}\n")
         for op in ops:
             if args.timeline == "summary":
-                print_ascii_summary(op['name'], op['dims'], op['types'], op['usec'], op['cycles'], op['trace_events'], op.get('evt_val'))
-            elif args.timeline == "diagram":
-                print_ascii_timeline(op['name'], op['dims'], op['types'], op['usec'], op['cycles'], op['trace_events'], op.get('evt_val'))
+                print_ascii_summary(op['name'], op['dims'], op['types'], op['usec'], op['cycles'], op['trace_events'])
+            elif args.timeline == "bubbles":
+                print_bubbles_timeline(op)
     else:
         generate_report(ops, args.top, overrides, args.sort, pmu_name=final_pmu_name)
 

scripts/snapdragon/ggml-hexagon-trace.py

@@ -6,6 +6,7 @@ import re
 import argparse
 import statistics
 import logging
+import bisect
 from typing import Any, Dict, List, Optional
 from collections import defaultdict
 
@@ -16,11 +17,11 @@ op_pattern = re.compile(
 )
 
 trace_pattern = re.compile(
-    r"trace-op\s+(?P<op_name>[A-Z_0-9+]+):\s+thread\s+(?P<thread>\d+)\s+event\s+(?P<event>[A-Z_0-9\-]+)\s+info\s+(?P<info>\d+)\s+(?P<state>start|stop)\s+(?P<cycles>\d+)"
+    r"trace-evt\s+(?P<event>[A-Z_0-9\-]+):\s+thread\s+(?P<thread>\d+)\s+info\s+(?P<info>\d+)\s+(?P<state>start|stop)\s+(?P<cycles>\d+)"
 )
 
 
-def normalize_event_name(evt_type):
+def normalize_event_name(evt_type, info=0):
     if evt_type == "HVX_COMP":
         return "V-COMP"
     if evt_type == "HMX_COMP":
@@ -32,9 +33,13 @@ def normalize_event_name(evt_type):
 
 
 class CycleUnwrapper:
-    def __init__(self):
-        self.last_raw = None
-        self.high_part = 0
+    def __init__(self, initial_val=None):
+        if initial_val is not None:
+            self.last_raw = initial_val & 0xFFFFFFFF
+            self.high_part = initial_val & 0xFFFFFFFF00000000
+        else:
+            self.last_raw = None
+            self.high_part = 0
 
     def unwrap(self, raw):
         if self.last_raw is None:
@@ -60,8 +65,10 @@ def parse_log(file_path):
         sys.exit(1)
 
     all_ops: List[Dict[str, Any]] = []
+    all_traces: List[Dict[str, Any]] = []
     current_op: Optional[Dict[str, Any]] = None
-    unwrapper = CycleUnwrapper()
+    unwrapper = None
+    trace_unwrapper = None
     line_idx = 0
 
     for line in f:
@@ -73,6 +80,7 @@ def parse_log(file_path):
             if not prefix_match:
                 continue
 
+            names = parts[1]
             if len(parts) == 7:
                 dims, types, strides, params, timings = parts[2], parts[3], parts[4], parts[5], parts[6]
             elif len(parts) == 6:
@@ -93,6 +101,7 @@ def parse_log(file_path):
             op_match = op_pattern.search(line)
             if op_match:
                 op_name = op_match.group('op_name')
+                names = ""
                 dims = op_match.group('dims').strip() if op_match.group('dims') else ''
                 types = op_match.group('types').strip() if op_match.group('types') else ''
                 strides = op_match.group('strides').strip() if op_match.group('strides') else ''
@@ -103,18 +112,30 @@ def parse_log(file_path):
         if op_match:
             cycles_start_raw = op_match.group('start')
             unwrapped_cycles_start = None
-            if cycles_start_raw:
-                unwrapped_cycles_start = unwrapper.unwrap(int(cycles_start_raw))
+            if op_name == "OPBATCH":
+                if cycles_start_raw:
+                    unwrapped_cycles_start = int(cycles_start_raw)
+                    unwrapper = CycleUnwrapper(unwrapped_cycles_start)
+                    trace_unwrapper = CycleUnwrapper(unwrapped_cycles_start)
+            else:
+                if cycles_start_raw and unwrapper is not None:
+                    unwrapped_cycles_start = unwrapper.unwrap(int(cycles_start_raw))
 
             idx = line.find("profile-op ")
             op_text = line[idx + 11:].strip() if idx != -1 else line.strip()
 
+            evt_str = None
+            if types.startswith("evt-cnt "):
+                evt_str = types[8:].strip()
+
             current_op = {
                 'name':         op_name,
+                'names':        names,
                 'dims':         dims,
                 'types':        types,
                 'strides':      strides,
                 'params':       params,
+                'evt':          evt_str,
                 'op_text':      op_text,
                 'usec':         int(op_match.group('usec')),
                 'cycles':       int(op_match.group('cycles')),
@@ -127,20 +148,22 @@ def parse_log(file_path):
             continue
 
         trace_match = trace_pattern.search(line)
-        if trace_match and current_op:
-            if trace_match.group('op_name') == current_op['name']:
-                raw_cyc = int(trace_match.group('cycles'))
-                current_op['trace_events'].append({
-                    'thread': int(trace_match.group('thread')),
-                    'event':  trace_match.group('event'),
-                    'info':   int(trace_match.group('info')),
-                    'cycles': raw_cyc,
-                    'unwrapped_cycles': unwrapper.unwrap(raw_cyc),
-                    'state':  trace_match.group('state')
-                })
+        if trace_match:
+            raw_cyc = int(trace_match.group('cycles'))
+            unwrapped_cyc = None
+            if trace_unwrapper is not None:
+                unwrapped_cyc = trace_unwrapper.unwrap(raw_cyc)
+            all_traces.append({
+                'thread': int(trace_match.group('thread')),
+                'event':  trace_match.group('event'),
+                'info':   int(trace_match.group('info')),
+                'cycles': raw_cyc,
+                'unwrapped_cycles': unwrapped_cyc,
+                'state':  trace_match.group('state')
+            })
 
     f.close()
-    return all_ops
+    return all_ops, all_traces
 
 # --- Simple protobuf encoder ---
 
@@ -246,7 +269,7 @@ def write_trace_packet_to_file(f, packet_bytes):
 # --- End Protobuf Encoder ---
 
 
-def generate_perfetto_trace(filtered_ops, output_path):
+def generate_perfetto_trace(filtered_ops, trace_events, output_path):
     if not filtered_ops:
         logger.warning("No operators found after filtering.")
         return
@@ -269,14 +292,12 @@ def generate_perfetto_trace(filtered_ops, output_path):
 
     # Process events
     completed_events = []
-    for op in filtered_ops:
-        events = op['trace_events']
-        if not events:
-            continue
-        events = sorted(events, key=lambda e: e['unwrapped_cycles'])
+    if trace_events:
+        trace_events = sorted(trace_events, key=lambda e: e['unwrapped_cycles'])
+        one_usec_cycles = max(avg_freq_mhz, 1.0)
 
         active_starts = {}
-        for e in events:
+        for e in trace_events:
             t = e['thread']
             evt = e['event']
             info = e['info']
@@ -285,6 +306,17 @@ def generate_perfetto_trace(filtered_ops, output_path):
 
             key = (t, evt, info)
             if state == 'start':
+                # Handle missing stop (start followed by another start)
+                if key in active_starts:
+                    prev_start = active_starts[key]
+                    completed_events.append({
+                        'thread': t,
+                        'event': evt,
+                        'info': info,
+                        'start_cyc': prev_start,
+                        'end_cyc': prev_start + one_usec_cycles,
+                        'missing_stop': True,
+                    })
                 active_starts[key] = cyc
             elif state == 'stop':
                 if key in active_starts:
@@ -296,8 +328,29 @@ def generate_perfetto_trace(filtered_ops, output_path):
                         'info': info,
                         'start_cyc': start_cyc,
                         'end_cyc': cyc,
-                        'op_name': op['name']
                     })
+                else:
+                    # Handle missing start (stop without start)
+                    completed_events.append({
+                        'thread': t,
+                        'event': evt,
+                        'info': info,
+                        'start_cyc': cyc - one_usec_cycles,
+                        'end_cyc': cyc,
+                        'missing_start': True,
+                    })
+
+        # Clear remaining unmatched starts
+        for key, start_cyc in active_starts.items():
+            t, evt, info = key
+            completed_events.append({
+                'thread': t,
+                'event': evt,
+                'info': info,
+                'start_cyc': start_cyc,
+                'end_cyc': start_cyc + one_usec_cycles,
+                'missing_stop': True,
+            })
 
     completed_events.sort(key=lambda e: e['start_cyc'])
 
@@ -316,7 +369,7 @@ def generate_perfetto_trace(filtered_ops, output_path):
         ts = e['ts_ns']
         dur = e['dur_ns']
 
-        norm_evt = normalize_event_name(evt)
+        norm_evt = normalize_event_name(evt, e['info'])
         if norm_evt == "DMA":
             track_key = (t, "DMA")
         elif t == 10:
@@ -343,7 +396,7 @@ def generate_perfetto_trace(filtered_ops, output_path):
         evt = e['event']
         slot = e['slot']
 
-        norm_evt = normalize_event_name(evt)
+        norm_evt = normalize_event_name(evt, e['info'])
         if norm_evt == "DMA":
             track_evt = "DMA"
             evt_id = 1
@@ -421,18 +474,26 @@ def generate_perfetto_trace(filtered_ops, output_path):
         for op in filtered_ops:
             op_start_ns = int(round(((op['start_cycles'] - global_min_cyc) / avg_freq_mhz) * 1000))
             op_dur_ns = int(round((op['cycles'] / avg_freq_mhz) * 1000))
-            if op_start_ns < last_op_end_ns:
-                op_start_ns = last_op_end_ns
-            clamped_dur = max(op_dur_ns, 100) # Clamp to 100ns (0.1us)
+            if op['name'] != "OPBATCH":
+                if op_start_ns < last_op_end_ns:
+                    op_start_ns = last_op_end_ns
+                clamped_dur = max(op_dur_ns, 100) # Clamp to 100ns (0.1us)
+                last_op_end_ns = op_start_ns + clamped_dur
+            else:
+                clamped_dur = max(op_dur_ns, 100)
 
             # Debug annotations for Ops
             debug_annots = []
             if 'line_num' in op:
                 debug_annots.append(make_debug_annotation("line", int_val=op['line_num']))
-            if 'strides' in op and op['strides']:
+            if 'names' in op and op['names'] and op['names'] != '----':
+                debug_annots.append(make_debug_annotation("names", string_val=op['names']))
+            if 'strides' in op and op['strides'] and op['strides'] != '----':
                 debug_annots.append(make_debug_annotation("strides", string_val=op['strides']))
             if 'params' in op and op['params'] and op['params'] != '----':
                 debug_annots.append(make_debug_annotation("params", string_val=op['params']))
+            if 'evt' in op and op['evt']:
+                debug_annots.append(make_debug_annotation("evt", string_val=op['evt']))
 
             # Slice Begin
             evt_begin = make_track_event(1, 2, name=f"{op['name']} ({op['dims']})", category="operator", debug_annotations=debug_annots)
@@ -444,15 +505,21 @@ def generate_perfetto_trace(filtered_ops, output_path):
             packet_end = make_trace_packet(op_start_ns + clamped_dur, track_event=evt_end)
             write_trace_packet_to_file(f, packet_end)
 
-            last_op_end_ns = op_start_ns + clamped_dur
-
         # Emit Thread Trace Events
         for e in completed_events:
-            norm_name = normalize_event_name(e['event'])
+            norm_name = normalize_event_name(e['event'], e['info'])
             name = f"DMA {e['info']}" if norm_name == "DMA" else norm_name
+            if e.get('missing_start') or e.get('missing_stop'):
+                name += "!"
+
+            debug_annots = []
+            if e.get('missing_start'):
+                debug_annots.append(make_debug_annotation("missing_start", string_val="true"))
+            if e.get('missing_stop'):
+                debug_annots.append(make_debug_annotation("missing_stop", string_val="true"))
 
             # Slice Begin
-            evt_begin = make_track_event(1, e['uuid'], name=name, category="trace")
+            evt_begin = make_track_event(1, e['uuid'], name=name, category="trace", debug_annotations=debug_annots if debug_annots else None)
             packet_begin = make_trace_packet(e['ts_ns'], track_event=evt_begin)
             write_trace_packet_to_file(f, packet_begin)
 
@@ -477,7 +544,7 @@ def main():
     args = parser.parse_args()
     logging.basicConfig(level=logging.INFO, format='%(message)s')
 
-    ops = parse_log(args.logfile)
+    ops, traces = parse_log(args.logfile)
 
     if args.filter:
         try:
@@ -492,7 +559,30 @@ def main():
     elif args.tail is not None:
         ops = ops[-args.tail:]
 
-    generate_perfetto_trace(ops, args.output)
+    if args.filter or args.head is not None or args.tail is not None:
+        valid_ranges = []
+        for op in ops:
+            start_cyc = op['unwrapped_cycles_start']
+            end_cyc = start_cyc + op['cycles'] if start_cyc is not None else None
+            if start_cyc is not None and end_cyc is not None:
+                valid_ranges.append((start_cyc, end_cyc))
+
+        valid_ranges.sort(key=lambda r: r[0])
+        range_starts = [r[0] for r in valid_ranges]
+
+        filtered_traces = []
+        for e in traces:
+            cyc = e['unwrapped_cycles']
+            if cyc is None:
+                continue
+            idx = bisect.bisect_right(range_starts, cyc) - 1
+            if idx >= 0:
+                start, end = valid_ranges[idx]
+                if start <= cyc <= end:
+                    filtered_traces.append(e)
+        traces = filtered_traces
+
+    generate_perfetto_trace(ops, traces, args.output)
 
 
 if __name__ == "__main__":

scripts/sync_vendor.py

@@ -21,7 +21,7 @@ vendor = {
     f"https://raw.githubusercontent.com/yhirose/cpp-httplib/{HTTPLIB_VERSION}/split.py":  "split.py",
     f"https://raw.githubusercontent.com/yhirose/cpp-httplib/{HTTPLIB_VERSION}/LICENSE":   "vendor/cpp-httplib/LICENSE",
 
-    "https://raw.githubusercontent.com/sheredom/subprocess.h/b49c56e9fe214488493021017bf3954b91c7c1f5/subprocess.h": "vendor/sheredom/subprocess.h",
+    "https://raw.githubusercontent.com/sheredom/subprocess.h/8671cee1fc09f11a70ce3782a0ee13177c3aa387/subprocess.h": "vendor/sheredom/subprocess.h",
 }
 
 for url, filename in vendor.items():

src/llama-model-loader.cpp

@@ -4,6 +4,7 @@
 #include "ggml.h"
 #include "gguf.h"
 #include "llama-hparams.h"
+#include "llama.h"
 
 #include <algorithm>
 #include <array>
@@ -522,8 +523,7 @@ llama_model_loader::llama_model_loader(
         const std::string & fname,
         std::vector<std::string> & splits,
         FILE * file,
-        bool use_mmap,
-        bool use_direct_io,
+        llama_load_mode load_mode,
         bool check_tensors,
         bool no_alloc,
         const llama_model_kv_override * param_overrides_p,
@@ -542,6 +542,9 @@ llama_model_loader::llama_model_loader(
 
     tensor_buft_overrides = param_tensor_buft_overrides_p;
 
+    this->use_mmap      = load_mode == LLAMA_LOAD_MODE_MMAP || load_mode == LLAMA_LOAD_MODE_MLOCK;
+    this->use_direct_io = load_mode == LLAMA_LOAD_MODE_DIRECT_IO;
+
     if (!fname.empty()) {
         // Load the main GGUF
         struct ggml_context * ctx = NULL;
@@ -562,20 +565,6 @@ llama_model_loader::llama_model_loader(
         files.emplace_back(new llama_file(fname.c_str(), "rb", use_direct_io));
         contexts.emplace_back(ctx);
 
-        if (use_mmap && use_direct_io) {
-            if (files.back()->has_direct_io()) {
-                LLAMA_LOG_WARN("%s: direct I/O is enabled, disabling mmap\n", __func__);
-                use_mmap = false;
-            } else {
-                LLAMA_LOG_WARN("%s: direct I/O is not available, using mmap\n", __func__);
-                use_direct_io = false;
-
-                // reopen file using std::fopen for mmap
-                files.pop_back();
-                files.emplace_back(new llama_file(fname.c_str(), "rb", false));
-            }
-        }
-
         // Save tensors data offset of the main file.
         // For subsidiary files, `meta` tensor data offset must not be used,
         // so we build a unified tensors index for weights.
@@ -816,13 +805,11 @@ llama_model_loader::llama_model_loader(
         }
     }
 
-    if (!llama_mmap::SUPPORTED) {
+    if (this->use_mmap && !llama_mmap::SUPPORTED) {
         LLAMA_LOG_WARN("%s: mmap is not supported on this platform\n", __func__);
-        use_mmap = false;
+        this->use_mmap = false;
     }
 
-    this->use_mmap = use_mmap;
-    this->use_direct_io = use_direct_io;
     this->check_tensors = check_tensors;
     this->no_alloc = no_alloc;
 }

src/llama-model-loader.h

@@ -126,8 +126,7 @@ struct llama_model_loader {
         const std::string & fname,
         std::vector<std::string> & splits, // optional, only need if the split does not follow naming scheme
         FILE * file,
-        bool use_mmap,
-        bool use_direct_io,
+        llama_load_mode load_mode,
         bool check_tensors,
         bool no_alloc,
         const llama_model_kv_override * param_overrides_p,

src/llama-model.cpp

@@ -16,6 +16,7 @@
 #include "llama-memory-hybrid-iswa.h"
 #include "llama-memory-recurrent.h"
 
+#include "llama.h"
 #include "models/models.h"
 
 #include "ggml.h"
@@ -1243,7 +1244,7 @@ void llama_model_base::load_vocab(llama_model_loader & ml) {
 
 bool llama_model_base::load_tensors(llama_model_loader & ml) {
     const auto & split_mode   = params.split_mode;
-    const auto & use_mlock    = params.use_mlock;
+    const bool use_mlock      = params.load_mode == LLAMA_LOAD_MODE_MLOCK;
     const auto & tensor_split = params.tensor_split;
 
     const int n_layer_all = hparams.n_layer_all;
@@ -1253,8 +1254,8 @@ bool llama_model_base::load_tensors(llama_model_loader & ml) {
 
     this->ml = &ml; // to be used by create_tensor() and load_arch_tensors()
 
-    LLAMA_LOG_INFO("%s: loading model tensors, this can take a while... (mmap = %s, direct_io = %s)\n",
-        __func__, ml.use_mmap ? "true" : "false", ml.use_direct_io ? "true" : "false");
+    LLAMA_LOG_INFO("%s: loading model tensors, this can take a while... (load_mode = %s)\n",
+        __func__, llama_load_mode_name(params.load_mode));
 
     // build a list of buffer types for the CPU and GPU devices
     pimpl->cpu_buft_list = make_cpu_buft_list(devices, params.use_extra_bufts, params.no_host);
@@ -2318,15 +2319,13 @@ llama_model_params llama_model_default_params() {
         /*.tensor_buft_overrides       =*/ nullptr,
         /*.n_gpu_layers                =*/ -1,
         /*.split_mode                  =*/ LLAMA_SPLIT_MODE_LAYER,
+        /*.load_mode                   =*/ LLAMA_LOAD_MODE_MMAP,
         /*.main_gpu                    =*/ 0,
         /*.tensor_split                =*/ nullptr,
         /*.progress_callback           =*/ nullptr,
         /*.progress_callback_user_data =*/ nullptr,
         /*.kv_overrides                =*/ nullptr,
         /*.vocab_only                  =*/ false,
-        /*.use_mmap                    =*/ true,
-        /*.use_direct_io               =*/ false,
-        /*.use_mlock                   =*/ false,
         /*.check_tensors               =*/ false,
         /*.use_extra_bufts             =*/ true,
         /*.no_host                     =*/ false,

src/llama-quant.cpp

@@ -2,6 +2,7 @@
 #include "llama-model.h"
 #include "llama-model-loader.h"
 #include "llama-ext.h"
+#include "llama.h"
 
 #include <algorithm>
 #include <cmath>
@@ -876,15 +877,15 @@ static void llama_model_quantize_impl(const std::string & fname_inp, const std::
     // mmap consistently increases speed on Linux, and also increases speed on Windows with
     // hot cache. It may cause a slowdown on macOS, possibly related to free memory.
 #if defined(__linux__) || defined(_WIN32)
-    constexpr bool use_mmap = true;
+    constexpr llama_load_mode load_mode = LLAMA_LOAD_MODE_MMAP;
 #else
-    constexpr bool use_mmap = false;
+    constexpr llama_load_mode load_mode = LLAMA_LOAD_MODE_NONE;
 #endif
 
     const llama_model_kv_override * kv_overrides = params->kv_overrides;
     std::vector<std::string> splits = {};
     llama_model_loader ml(/*metadata*/ nullptr, /*set_tensor_data*/ nullptr, /*set_tensor_data_ud*/ nullptr,
-        fname_inp, splits, /*file*/ nullptr, use_mmap, /*use_direct_io*/ false, /*check_tensors*/ true, /*no_alloc*/ false, kv_overrides, nullptr);
+        fname_inp, splits, /*file*/ nullptr, /*load_mode*/ load_mode, /*check_tensors*/ true, /*no_alloc*/ false, kv_overrides, nullptr);
     ml.init_mappings(false); // no prefetching
 
     auto mparams = llama_model_default_params();

src/llama.cpp

@@ -46,6 +46,28 @@ const char * llama_flash_attn_type_name(enum llama_flash_attn_type flash_attn_ty
     GGML_ABORT("fatal error");
 }
 
+const char * llama_load_mode_name(enum llama_load_mode load_mode) {
+    switch (load_mode) {
+        case LLAMA_LOAD_MODE_NONE:
+            return "none";
+        case LLAMA_LOAD_MODE_MMAP:
+            return "mmap";
+        case LLAMA_LOAD_MODE_MLOCK:
+            return "mlock";
+        case LLAMA_LOAD_MODE_DIRECT_IO:
+            return "dio";
+    }
+    GGML_ABORT("fatal error");
+}
+
+enum llama_load_mode llama_load_mode_from_str(const char * str) {
+    if (std::strcmp(str, "none")  == 0) { return LLAMA_LOAD_MODE_NONE;      }
+    if (std::strcmp(str, "mmap")  == 0) { return LLAMA_LOAD_MODE_MMAP;      }
+    if (std::strcmp(str, "mlock") == 0) { return LLAMA_LOAD_MODE_MLOCK;     }
+    if (std::strcmp(str, "dio")   == 0) { return LLAMA_LOAD_MODE_DIRECT_IO; }
+    throw std::invalid_argument(std::string("unknown load mode: ") + str);
+}
+
 struct llama_sampler_chain_params llama_sampler_chain_default_params() {
     struct llama_sampler_chain_params result = {
         /*.no_perf =*/ true,
@@ -279,7 +301,7 @@ static bool llama_prepare_model_devices(const llama_model_params & params, llama
 static std::pair<int, llama_model *> llama_model_load(struct gguf_context * metadata, llama_model_set_tensor_data_t set_tensor_data, void * set_tensor_data_ud,
         const std::string & fname, std::vector<std::string> & splits, FILE * file, llama_model_params & params) {
     try {
-        llama_model_loader ml(metadata, set_tensor_data, set_tensor_data_ud, fname, splits, file, params.use_mmap, params.use_direct_io,
+        llama_model_loader ml(metadata, set_tensor_data, set_tensor_data_ud, fname, splits, file, params.load_mode,
             params.check_tensors, params.no_alloc, params.kv_overrides, params.tensor_buft_overrides);
 
         ml.print_info();
@@ -412,7 +434,7 @@ struct llama_model * llama_model_init_from_user(
     GGML_ASSERT(metadata != nullptr);
     std::string path_model;
     std::vector<std::string> splits = {};
-    params.use_mmap = false;
+    params.load_mode = LLAMA_LOAD_MODE_NONE;
     params.use_extra_bufts = false;
     return llama_model_load_from_file_impl(metadata, set_tensor_data, set_tensor_data_ud, path_model, splits, /*file*/ nullptr, params);
 }

tests/test-arg-parser.cpp

@@ -1,6 +1,7 @@
 #include "arg.h"
 #include "common.h"
 #include "download.h"
+#include "llama.h"
 
 #include <string>
 #include <vector>
@@ -102,11 +103,9 @@ static void test(void) {
     argv = {"binary_name", "--draft", "123"};
     assert(false == common_params_parse(argv.size(), list_str_to_char(argv).data(), params, LLAMA_EXAMPLE_EMBEDDING));
 
-    // negated arg
-    argv = {"binary_name", "--no-mmap"};
+    argv = {"binary_name", "-lm", "hello"};
     assert(false == common_params_parse(argv.size(), list_str_to_char(argv).data(), params, LLAMA_EXAMPLE_COMMON));
 
-
     printf("test-arg-parser: test valid usage\n\n");
 
     argv = {"binary_name", "-m", "model_file.gguf"};
@@ -132,6 +131,22 @@ static void test(void) {
     assert(true == common_params_parse(argv.size(), list_str_to_char(argv).data(), params, LLAMA_EXAMPLE_SPECULATIVE));
     assert(params.speculative.draft.n_max == 123);
 
+    argv = {"binary_name", "-lm", "none"};
+    assert(true == common_params_parse(argv.size(), list_str_to_char(argv).data(), params, LLAMA_EXAMPLE_COMMON));
+    assert(params.load_mode == LLAMA_LOAD_MODE_NONE);
+
+    argv = {"binary_name", "-lm", "mmap"};
+    assert(true == common_params_parse(argv.size(), list_str_to_char(argv).data(), params, LLAMA_EXAMPLE_COMMON));
+    assert(params.load_mode == LLAMA_LOAD_MODE_MMAP);
+
+    argv = {"binary_name", "-lm", "mlock"};
+    assert(true == common_params_parse(argv.size(), list_str_to_char(argv).data(), params, LLAMA_EXAMPLE_COMMON));
+    assert(params.load_mode == LLAMA_LOAD_MODE_MLOCK);
+
+    argv = {"binary_name", "-lm", "dio"};
+    assert(true == common_params_parse(argv.size(), list_str_to_char(argv).data(), params, LLAMA_EXAMPLE_COMMON));
+    assert(params.load_mode == LLAMA_LOAD_MODE_DIRECT_IO);
+
     // multi-value args (CSV)
     argv = {"binary_name", "--lora", "file1.gguf,\"file2,2.gguf\",\"file3\"\"3\"\".gguf\",file4\".gguf"};
     assert(true == common_params_parse(argv.size(), list_str_to_char(argv).data(), params, LLAMA_EXAMPLE_COMMON));
@@ -158,13 +173,32 @@ static void test(void) {
     assert(params.model.path == "blah.gguf");
     assert(params.cpuparams.n_threads == 1010);
 
+    setenv("LLAMA_ARG_LOAD_MODE", "blah", true);
+    argv = {"binary_name"};
+    assert(false == common_params_parse(argv.size(), list_str_to_char(argv).data(), params, LLAMA_EXAMPLE_COMMON));
+
+    setenv("LLAMA_ARG_LOAD_MODE", "mmap", true);
+    argv = {"binary_name"};
+    assert(true == common_params_parse(argv.size(), list_str_to_char(argv).data(), params, LLAMA_EXAMPLE_COMMON));
+    assert(params.load_mode == LLAMA_LOAD_MODE_MMAP);
+
+    setenv("LLAMA_ARG_LOAD_MODE", "mlock", true);
+    argv = {"binary_name"};
+    assert(true == common_params_parse(argv.size(), list_str_to_char(argv).data(), params, LLAMA_EXAMPLE_COMMON));
+    assert(params.load_mode == LLAMA_LOAD_MODE_MLOCK);
+
+    setenv("LLAMA_ARG_LOAD_MODE", "dio", true);
+    argv = {"binary_name"};
+    assert(true == common_params_parse(argv.size(), list_str_to_char(argv).data(), params, LLAMA_EXAMPLE_COMMON));
+    assert(params.load_mode == LLAMA_LOAD_MODE_DIRECT_IO);
+
     printf("test-arg-parser: test negated environment variables\n\n");
 
-    setenv("LLAMA_ARG_MMAP", "0", true);
+    setenv("LLAMA_ARG_LOAD_MODE", "none", true);
     setenv("LLAMA_ARG_NO_PERF", "1", true); // legacy format
     argv = {"binary_name"};
     assert(true == common_params_parse(argv.size(), list_str_to_char(argv).data(), params, LLAMA_EXAMPLE_COMMON));
-    assert(params.use_mmap == false);
+    assert(params.load_mode == LLAMA_LOAD_MODE_NONE);
     assert(params.no_perf == true);
 
     printf("test-arg-parser: test environment variables being overwritten\n\n");

tests/test-chat.cpp

@@ -109,6 +109,15 @@ static void assert_contains(const std::string & haystack, const std::string & ne
     }
 }
 
+static void assert_not_contains(const std::string & haystack, const std::string & needle) {
+    if (haystack.find(needle) != std::string::npos) {
+        LOG_ERR("Expected NOT to contain: %s\n", needle.c_str());
+        LOG_ERR("Actual: %s\n", haystack.c_str());
+        common_log_flush(common_log_main());
+        throw std::runtime_error("Test failed");
+    }
+}
+
 static void assert_ends_with(const std::string & str, const std::string & suffix) {
     if (str.size() < suffix.size() ||
         str.compare(str.size() - suffix.size(), suffix.size(), suffix) != 0) {
@@ -2840,6 +2849,17 @@ static void test_template_output_peg_parsers(bool detailed_debug) {
             .expect(message_assist)
             .run();
 
+        // JSON output schema
+        tst.test(
+                "I need to output the invoice details in JSON<|END_THINKING|>"
+                "<|START_TEXT|>{\"amount\": 123.45, \"date\": \"2025-12-03\"}<|END_TEXT|>")
+            .reasoning_format(COMMON_REASONING_FORMAT_DEEPSEEK)
+            .json_schema(invoice_schema)
+            .tools({ special_function_tool })
+            .expect_reasoning("I need to output the invoice details in JSON")
+            .expect_content(R"({"amount": 123.45, "date": "2025-12-03"})")
+                .run();
+
         // Single tool call with reasoning.
         tst.test(
                "I'm\nthinking<|END_THINKING|>"
@@ -4016,6 +4036,132 @@ static void test_template_output_peg_parsers(bool detailed_debug) {
             .run();
     }
 
+    // DeepSeek V4 tests - same DSML markup as V3.2, but the tool call block is named
+    // "tool_calls" and the non-thinking generation prompt ends in a bare </think>
+    // instead of an empty <think></think> pair.
+    {
+        auto tst = peg_tester("models/templates/deepseek-ai-DeepSeek-V4.jinja", detailed_debug);
+
+        // Pure content (non-thinking mode; generation prompt ends with </think>)
+        tst.test("Hello, world!\nWhat's up?")
+            .enable_thinking(false)
+            .reasoning_format(COMMON_REASONING_FORMAT_DEEPSEEK)
+            .expect(message_assist)
+            .run();
+
+        // Thinking + content
+        tst.test("I'm\nthinking</think>Hello, world!\nWhat's up?")
+            .enable_thinking(true)
+            .reasoning_format(COMMON_REASONING_FORMAT_DEEPSEEK)
+            .expect(message_assist_thoughts)
+            .run();
+
+        // Thinking + tool call (single, string param)
+        tst.test(
+               "Let me check the time</think>\n\n"
+               "<｜DSML｜tool_calls>\n"
+               "<｜DSML｜invoke name=\"get_time\">\n"
+               "<｜DSML｜parameter name=\"city\" string=\"true\">Tokyo</｜DSML｜parameter>\n"
+               "</｜DSML｜invoke>\n"
+               "</｜DSML｜tool_calls>")
+            .enable_thinking(true)
+            .reasoning_format(COMMON_REASONING_FORMAT_DEEPSEEK)
+            .tools({ get_time_tool })
+            .expect(message_with_tool_calls_and_reasoning("get_time", R"({"city": "Tokyo"})", "Let me check the time"))
+            .run();
+
+        // Tool call without reasoning (non-thinking mode), integer param (string="false")
+        tst.test(
+               "<｜DSML｜tool_calls>\n"
+               "<｜DSML｜invoke name=\"special_function\">\n"
+               "<｜DSML｜parameter name=\"arg1\" string=\"false\">1</｜DSML｜parameter>\n"
+               "</｜DSML｜invoke>\n"
+               "</｜DSML｜tool_calls>")
+            .enable_thinking(false)
+            .reasoning_format(COMMON_REASONING_FORMAT_DEEPSEEK)
+            .tools({ special_function_tool })
+            .expect(message_assist_call)
+            .run();
+
+        // Multiple parallel tool calls with reasoning
+        tst.test(
+               "Calling both</think>\n\n"
+               "<｜DSML｜tool_calls>\n"
+               "<｜DSML｜invoke name=\"get_time\">\n"
+               "<｜DSML｜parameter name=\"city\" string=\"true\">Paris</｜DSML｜parameter>\n"
+               "</｜DSML｜invoke>\n"
+               "<｜DSML｜invoke name=\"get_weather\">\n"
+               "<｜DSML｜parameter name=\"city\" string=\"true\">Paris</｜DSML｜parameter>\n"
+               "</｜DSML｜invoke>\n"
+               "</｜DSML｜tool_calls>")
+            .enable_thinking(true)
+            .reasoning_format(COMMON_REASONING_FORMAT_DEEPSEEK)
+            .parallel_tool_calls(true)
+            .tools({ get_time_tool, get_weather_tool })
+            .expect(message_with_reasoning_content_and_multiple_tool_calls(
+                "Calling both", "",
+                { { "get_time", R"({"city": "Paris"})" }, { "get_weather", R"({"city": "Paris"})" } }))
+            .run();
+
+        // Tool call with content before tool calls
+        tst.test(
+               "Thinking about it</think>"
+               "Let me call the function.\n\n"
+               "<｜DSML｜tool_calls>\n"
+               "<｜DSML｜invoke name=\"special_function\">\n"
+               "<｜DSML｜parameter name=\"arg1\" string=\"false\">1</｜DSML｜parameter>\n"
+               "</｜DSML｜invoke>\n"
+               "</｜DSML｜tool_calls>")
+            .enable_thinking(true)
+            .reasoning_format(COMMON_REASONING_FORMAT_DEEPSEEK)
+            .tools({ special_function_tool })
+            .expect_reasoning("Thinking about it")
+            .expect_content("Let me call the function.")
+            .expect_tool_calls({
+                { "special_function", R"({"arg1": 1})", {} },
+            })
+            .run();
+
+        // Tool call with multiple params (mixed types)
+        tst.test(
+               "Multi-arg call</think>\n\n"
+               "<｜DSML｜tool_calls>\n"
+               "<｜DSML｜invoke name=\"magic_int\">\n"
+               "<｜DSML｜parameter name=\"ref\" string=\"false\">42</｜DSML｜parameter>\n"
+               "<｜DSML｜parameter name=\"name\" string=\"true\">foo bar</｜DSML｜parameter>\n"
+               "</｜DSML｜invoke>\n"
+               "</｜DSML｜tool_calls>")
+            .enable_thinking(true)
+            .reasoning_format(COMMON_REASONING_FORMAT_DEEPSEEK)
+            .tools({ magic_int_tool })
+            .expect_reasoning("Multi-arg call")
+            .expect_tool_calls({
+                { "magic_int", R"({"ref": 42, "name": "foo bar"})", {} },
+            })
+            .run();
+
+        // Continuation tests
+        tst.test("world!\nWhat's up?")
+            .reasoning_format(COMMON_REASONING_FORMAT_DEEPSEEK)
+            .enable_thinking(true)
+            .messages({ message_user, message_assist_prefill_content })
+            .add_generation_prompt(false)
+            .continue_final_message(COMMON_CHAT_CONTINUATION_CONTENT)
+            .expect_reasoning("I'm thinking")
+            .expect_content("Hello, world!\nWhat's up?")
+            .run();
+
+        tst.test(" thinking</think>Hello, world!\nWhat's up?")
+            .reasoning_format(COMMON_REASONING_FORMAT_DEEPSEEK)
+            .enable_thinking(true)
+            .messages({ message_user, message_assist_prefill_reasoning })
+            .add_generation_prompt(false)
+            .continue_final_message(COMMON_CHAT_CONTINUATION_REASONING)
+            .expect_reasoning("I'm thinking")
+            .expect_content("Hello, world!\nWhat's up?")
+            .run();
+    }
+
     // GLM-4.6 tests - format: <tool_call>function_name\n<arg_key>...</arg_key>\n<arg_value>...</arg_value>\n</tool_call>
     {
         auto tst = peg_tester("models/templates/GLM-4.6.jinja", detailed_debug);
@@ -5918,6 +6064,144 @@ static void test_developer_role_to_system_workaround() {
     }
 }
 
+// Verify reasoning-trace retention rules in the DeepSeek-V4 template:
+// all traces are retained unless drop_thinking is true AND the conversation
+// has no tool calls, in which case only the last (after-final-user) trace is
+// kept and earlier ones are dropped.
+static void test_deepseek_v4_thinking_retention() {
+    LOG_DBG("%s\n", __func__);
+
+    auto tmpls = read_templates("models/templates/deepseek-ai-DeepSeek-V4.jinja");
+
+    common_chat_msg user_q1; user_q1.role = "user"; user_q1.content = "Question 1";
+    common_chat_msg user_q2; user_q2.role = "user"; user_q2.content = "Question 2";
+    common_chat_msg asst_a1 = simple_assist_msg("Answer 1", "thinking A1");
+    common_chat_msg asst_a2 = simple_assist_msg("Answer 2", "thinking A2");
+
+    common_chat_msg tool_assist = message_with_tool_calls("special_function", "{\"arg1\": 1}");
+    common_chat_msg tool_result; tool_result.role = "tool";
+    tool_result.tool_name = "special_function"; tool_result.tool_call_id = "0"; tool_result.content = "result";
+
+    // The template uses U+FF5C as the role separator and literal think tags
+    // for the reasoning block.
+    const std::string asst_marker   = "<\xef\xbd\x9c" "Assistant" "\xef\xbd\x9c>";
+    // Built via concatenation so the thinking tokens are not interpreted by
+    // tooling processing this source file.
+    const std::string think_start   = "<" "think" ">";
+    const std::string think_end     = "</" "think" ">";
+
+    const std::string think_a1      = asst_marker + think_start + "thinking A1" + think_end;
+    const std::string think_a2      = asst_marker + think_start + "thinking A2" + think_end;
+    const std::string asst_no_think = asst_marker + think_end;
+
+    auto render = [&](const std::vector<common_chat_msg> & messages, bool drop_thinking) {
+        common_chat_templates_inputs inputs;
+        inputs.messages = messages;
+        inputs.add_generation_prompt = false;
+        inputs.chat_template_kwargs["thinking"]     = "true";
+        inputs.chat_template_kwargs["drop_thinking"] = drop_thinking ? "true" : "false";
+        return common_chat_templates_apply(tmpls.get(), inputs).prompt;
+    };
+
+    // No tools, drop_thinking=false: all reasoning is retained.
+    {
+        auto prompt = render({ user_q1, asst_a1, user_q2, asst_a2 }, /* drop_thinking = */ false);
+        assert_contains(prompt, think_a1);
+        assert_contains(prompt, think_a2);
+    }
+
+    // No tools, drop_thinking=true: only the last reasoning trace is kept,
+    // earlier ones are dropped (the assistant block emits just the end token).
+    {
+        auto prompt = render({ user_q1, asst_a1, user_q2, asst_a2 }, /* drop_thinking = */ true);
+        assert_not_contains(prompt, think_a1);
+        assert_contains(prompt, think_a2);
+        // The dropped assistant turn still opens with the marker + bare end token.
+        assert_contains(prompt, asst_no_think + "Answer 1");
+    }
+
+    // Single assistant turn, drop_thinking=true: the only trace is the last
+    // one, so it must be retained even with drop_thinking set.
+    {
+        auto prompt = render({ user_q1, asst_a1 }, /* drop_thinking = */ true);
+        assert_contains(prompt, think_a1);
+    }
+
+    // Single assistant turn, drop_thinking=false: reasoning is retained.
+    {
+        auto prompt = render({ user_q1, asst_a1 }, /* drop_thinking = */ false);
+        assert_contains(prompt, think_a1);
+    }
+
+    // With tool calls, drop_thinking=true: tool presence forces all reasoning
+    // to be retained, including the pre-tool-call trace.
+    {
+        auto prompt = render({ user_q1, asst_a1, user_q2, tool_assist, tool_result, asst_a2 },
+                             /* drop_thinking = */ true);
+        assert_contains(prompt, think_a1);
+        assert_contains(prompt, think_a2);
+    }
+
+    // With tool calls, drop_thinking=false: all reasoning retained.
+    {
+        auto prompt = render({ user_q1, asst_a1, user_q2, tool_assist, tool_result, asst_a2 },
+                             /* drop_thinking = */ false);
+        assert_contains(prompt, think_a1);
+        assert_contains(prompt, think_a2);
+    }
+}
+
+// Verify that consecutive tool results are rendered in the tool call order of the
+// preceding assistant message (matched by tool call id), as required by the reference
+// DeepSeek-V4 implementation.
+static void test_deepseek_v4_tool_result_ordering() {
+    LOG_DBG("%s\n", __func__);
+
+    auto tmpls = read_templates("models/templates/deepseek-ai-DeepSeek-V4.jinja");
+
+    common_chat_msg user_q; user_q.role = "user"; user_q.content = "Question";
+
+    common_chat_msg assist_calls;
+    assist_calls.role = "assistant";
+    assist_calls.tool_calls.push_back({ "get_time",    "{\"city\": \"Paris\"}", "call_1" });
+    assist_calls.tool_calls.push_back({ "get_weather", "{\"city\": \"Paris\"}", "call_2" });
+
+    common_chat_msg time_result; time_result.role = "tool";
+    time_result.tool_name = "get_time"; time_result.tool_call_id = "call_1"; time_result.content = "12:00";
+    common_chat_msg weather_result; weather_result.role = "tool";
+    weather_result.tool_name = "get_weather"; weather_result.tool_call_id = "call_2"; weather_result.content = "sunny";
+
+    auto render = [&](const std::vector<common_chat_msg> & messages) {
+        common_chat_templates_inputs inputs;
+        inputs.messages = messages;
+        inputs.add_generation_prompt = false;
+        return common_chat_templates_apply(tmpls.get(), inputs).prompt;
+    };
+
+    // Results sent out of order are reordered to match the tool call order.
+    {
+        auto prompt = render({ user_q, assist_calls, weather_result, time_result });
+        assert_contains(prompt, "<tool_result>12:00</tool_result>\n\n<tool_result>sunny</tool_result>");
+    }
+
+    // Results already in call order stay put.
+    {
+        auto prompt = render({ user_q, assist_calls, time_result, weather_result });
+        assert_contains(prompt, "<tool_result>12:00</tool_result>\n\n<tool_result>sunny</tool_result>");
+    }
+
+    // Without tool call ids there is nothing to match against; order is preserved.
+    {
+        auto no_id_calls = assist_calls;
+        no_id_calls.tool_calls[0].id = "";
+        no_id_calls.tool_calls[1].id = "";
+        auto no_id_weather = weather_result; no_id_weather.tool_call_id = "";
+        auto no_id_time    = time_result;    no_id_time.tool_call_id = "";
+        auto prompt = render({ user_q, no_id_calls, no_id_weather, no_id_time });
+        assert_contains(prompt, "<tool_result>sunny</tool_result>\n\n<tool_result>12:00</tool_result>");
+    }
+}
+
 static void test_reasoning_budget_tokens_per_request() {
     LOG_DBG("%s\n", __func__);
     // Use Qwen3 template which has <think>...</think> reasoning markers.
@@ -6139,6 +6423,8 @@ int main(int argc, char ** argv) {
         test_tools_oaicompat_json_conversion();
         test_convert_responses_to_chatcmpl();
         test_developer_role_to_system_workaround();
+        test_deepseek_v4_thinking_retention();
+        test_deepseek_v4_tool_result_ordering();
         test_template_generation_prompt();
         test_reasoning_budget_tokens_per_request();
         test_reasoning_budget_message_per_request();

tests/test-model-load-cancel.cpp

@@ -16,7 +16,7 @@ int main(int argc, char *argv[] ) {
 
     llama_backend_init();
     auto params = llama_model_params{};
-    params.use_mmap = false;
+    params.load_mode = LLAMA_LOAD_MODE_NONE;
     params.progress_callback = [](float progress, void * ctx){
         (void) ctx;
         return progress > 0.50;

tests/test-quantize-stats.cpp

@@ -312,7 +312,7 @@ int main(int argc, char ** argv) {
 
     {
         auto mparams = llama_model_default_params();
-        mparams.use_mlock  = false;
+        mparams.load_mode = LLAMA_LOAD_MODE_NONE;
 
         model = llama_model_load_from_file(params.model.c_str(), mparams);
 

tools/cli/README.md

@@ -55,9 +55,10 @@
 | `-dt, --defrag-thold N` | KV cache defragmentation threshold (DEPRECATED)<br/>(env: LLAMA_ARG_DEFRAG_THOLD) |
 | `-np, --parallel N` | number of parallel sequences to decode (default: 1)<br/>(env: LLAMA_ARG_N_PARALLEL) |
 | `--rpc SERVERS` | comma-separated list of RPC servers (host:port)<br/>(env: LLAMA_ARG_RPC) |
-| `--mlock` | force system to keep model in RAM rather than swapping or compressing<br/>(env: LLAMA_ARG_MLOCK) |
-| `--mmap, --no-mmap` | whether to memory-map model. (if mmap disabled, slower load but may reduce pageouts if not using mlock) (default: enabled)<br/>(env: LLAMA_ARG_MMAP) |
-| `-dio, --direct-io, -ndio, --no-direct-io` | use DirectIO if available. (default: disabled)<br/>(env: LLAMA_ARG_DIO) |
+| `--mlock` | DEPRECATED in favor of `--load-mode`: mmap + force system to keep model in RAM rather than swapping or compressing<br/>(env: LLAMA_ARG_MLOCK) |
+| `--mmap, --no-mmap` | DEPRECATED in favor of `--load-mode`: whether to memory-map model. (if mmap disabled, slower load but may reduce pageouts if not using mlock)<br/>(env: LLAMA_ARG_MMAP) |
+| `-dio, --direct-io, -ndio, --no-direct-io` | DEPRECATED in favor of `--load-mode`: use DirectIO if available<br/>(env: LLAMA_ARG_DIO) |
+| `-lm, --load-mode MODE` | model loading mode (default: mmap)<br/>- none: no special loading mode<br/>- mmap: memory-map model (if mmap disabled, slower load but may reduce pageouts if not using mlock)<br/>- mlock: mmap + force system to keep model in RAM rather than swapping or compressing<br/>- dio: use DirectIO if available<br/><br/>(env: LLAMA_ARG_LOAD_MODE) |
 | `--numa TYPE` | attempt optimizations that help on some NUMA systems<br/>- distribute: spread execution evenly over all nodes<br/>- isolate: only spawn threads on CPUs on the node that execution started on<br/>- numactl: use the CPU map provided by numactl<br/>if run without this previously, it is recommended to drop the system page cache before using this<br/>see https://github.com/ggml-org/llama.cpp/issues/1437<br/>(env: LLAMA_ARG_NUMA) |
 | `-dev, --device <dev1,dev2,..>` | comma-separated list of devices to use for offloading (none = don't offload)<br/>use --list-devices to see a list of available devices<br/>(env: LLAMA_ARG_DEVICE) |
 | `--list-devices` | print list of available devices and exit |

tools/completion/README.md

@@ -138,9 +138,10 @@ llama-completion.exe -m models\gemma-1.1-7b-it.Q4_K_M.gguf --ignore-eos -n -1
 | `-dt, --defrag-thold N` | KV cache defragmentation threshold (DEPRECATED)<br/>(env: LLAMA_ARG_DEFRAG_THOLD) |
 | `-np, --parallel N` | number of parallel sequences to decode (default: 1)<br/>(env: LLAMA_ARG_N_PARALLEL) |
 | `--rpc SERVERS` | comma-separated list of RPC servers (host:port)<br/>(env: LLAMA_ARG_RPC) |
-| `--mlock` | force system to keep model in RAM rather than swapping or compressing<br/>(env: LLAMA_ARG_MLOCK) |
-| `--mmap, --no-mmap` | whether to memory-map model. (if mmap disabled, slower load but may reduce pageouts if not using mlock) (default: enabled)<br/>(env: LLAMA_ARG_MMAP) |
-| `-dio, --direct-io, -ndio, --no-direct-io` | use DirectIO if available. (default: disabled)<br/>(env: LLAMA_ARG_DIO) |
+| `--mlock` | DEPRECATED in favor of `--load-mode`: mmap + force system to keep model in RAM rather than swapping or compressing<br/>(env: LLAMA_ARG_MLOCK) |
+| `--mmap, --no-mmap` | DEPRECATED in favor of `--load-mode`: whether to memory-map model. (if mmap disabled, slower load but may reduce pageouts if not using mlock)<br/>(env: LLAMA_ARG_MMAP) |
+| `-dio, --direct-io, -ndio, --no-direct-io` | DEPRECATED in favor of `--load-mode`: use DirectIO if available<br/>(env: LLAMA_ARG_DIO) |
+| `-lm, --load-mode MODE` | model loading mode (default: mmap)<br/>- none: no special loading mode<br/>- mmap: memory-map model (if mmap disabled, slower load but may reduce pageouts if not using mlock)<br/>- mlock: mmap + force system to keep model in RAM rather than swapping or compressing<br/>- dio: use DirectIO if available<br/><br/>(env: LLAMA_ARG_LOAD_MODE) |
 | `--numa TYPE` | attempt optimizations that help on some NUMA systems<br/>- distribute: spread execution evenly over all nodes<br/>- isolate: only spawn threads on CPUs on the node that execution started on<br/>- numactl: use the CPU map provided by numactl<br/>if run without this previously, it is recommended to drop the system page cache before using this<br/>see https://github.com/ggml-org/llama.cpp/issues/1437<br/>(env: LLAMA_ARG_NUMA) |
 | `-dev, --device <dev1,dev2,..>` | comma-separated list of devices to use for offloading (none = don't offload)<br/>use --list-devices to see a list of available devices<br/>(env: LLAMA_ARG_DEVICE) |
 | `--list-devices` | print list of available devices and exit |

tools/llama-bench/llama-bench.cpp

@@ -26,6 +26,7 @@
 #include "fit.h"
 #include "ggml.h"
 #include "llama.h"
+#include "log.h"
 
 #ifdef _WIN32
 #    define WIN32_LEAN_AND_MEAN
@@ -339,14 +340,13 @@ struct cmd_params {
     std::vector<int>                 n_gpu_layers;
     std::vector<int>                 n_cpu_moe;
     std::vector<llama_split_mode>    split_mode;
+    std::vector<llama_load_mode>     load_mode;
     std::vector<int>                 main_gpu;
     std::vector<bool>                no_kv_offload;
     std::vector<llama_flash_attn_type> flash_attn;
     std::vector<std::vector<ggml_backend_dev_t>> devices;
     std::vector<std::vector<float>>  tensor_split;
     std::vector<std::vector<llama_model_tensor_buft_override>> tensor_buft_overrides;
-    std::vector<bool>                use_mmap;
-    std::vector<bool>                use_direct_io;
     std::vector<bool>                embeddings;
     std::vector<bool>                no_op_offload;
     std::vector<bool>                no_host;
@@ -384,14 +384,13 @@ static const cmd_params cmd_params_defaults = {
     /* n_gpu_layers         */ { -1 },
     /* n_cpu_moe            */ { 0 },
     /* split_mode           */ { LLAMA_SPLIT_MODE_LAYER },
+    /* load_mode            */ { LLAMA_LOAD_MODE_MMAP },
     /* main_gpu             */ { 0 },
     /* no_kv_offload        */ { false },
     /* flash_attn           */ { LLAMA_FLASH_ATTN_TYPE_AUTO },
     /* devices              */ { {} },
     /* tensor_split         */ { std::vector<float>(llama_max_devices(), 0.0f) },
     /* tensor_buft_overrides*/ { std::vector<llama_model_tensor_buft_override>{ { nullptr, nullptr } } },
-    /* use_mmap             */ { true },
-    /* use_direct_io        */ { false },
     /* embeddings           */ { false },
     /* no_op_offload        */ { false },
     /* no_host              */ { false },
@@ -460,8 +459,9 @@ static void print_usage(int /* argc */, char ** argv) {
     printf("  -nkvo, --no-kv-offload <0|1>                (default: %s)\n", join(cmd_params_defaults.no_kv_offload, ",").c_str());
     printf("  -fa, --flash-attn <on|off|auto>             (default: %s)\n", join(transform_to_str(cmd_params_defaults.flash_attn, llama_flash_attn_type_name), ",").c_str());
     printf("  -dev, --device <dev0/dev1/...>              (default: auto)\n");
-    printf("  -mmp, --mmap <0|1>                          (default: %s)\n", join(cmd_params_defaults.use_mmap, ",").c_str());
-    printf("  -dio, --direct-io <0|1>                     (default: %s)\n", join(cmd_params_defaults.use_direct_io, ",").c_str());
+    printf("  -lm, --load-mode <none|mmap|mlock|dio>      (default: %s)\n", join(transform_to_str(cmd_params_defaults.load_mode, llama_load_mode_name), ",").c_str());
+    printf("  -mmp, --mmap <0|1>                          (DEPRECATED IN FAVOUR OF --load-mode)\n");
+    printf("  -dio, --direct-io <0|1>                     (DEPRECATED IN FAVOUR OF --load-mode)\n");
     printf("  -embd, --embeddings <0|1>                   (default: %s)\n", join(cmd_params_defaults.embeddings, ",").c_str());
     printf("  -ts, --tensor-split <ts0/ts1/..>            (default: 0)\n");
     printf("  -ot --override-tensor <tensor name pattern>=<buffer type>;...\n");
@@ -769,6 +769,34 @@ static cmd_params parse_cmd_params(int argc, char ** argv) {
                     break;
                 }
                 params.split_mode.insert(params.split_mode.end(), modes.begin(), modes.end());
+            } else if (arg == "-lm" || arg == "--load-mode") {
+                if (++i >= argc) {
+                    invalid_param = true;
+                    break;
+                }
+                auto p = string_split<std::string>(argv[i], split_delim);
+
+                std::vector<llama_load_mode> modes;
+                for (const auto & m : p) {
+                    llama_load_mode mode;
+                    if (m == "none") {
+                        mode = LLAMA_LOAD_MODE_NONE;
+                    } else if (m == "mmap") {
+                        mode = LLAMA_LOAD_MODE_MMAP;
+                    } else if (m == "mlock") {
+                        mode = LLAMA_LOAD_MODE_MLOCK;
+                    } else if (m == "dio") {
+                        mode = LLAMA_LOAD_MODE_DIRECT_IO;
+                    } else {
+                        invalid_param = true;
+                        break;
+                    }
+                    modes.push_back(mode);
+                }
+                if (invalid_param) {
+                    break;
+                }
+                params.load_mode.insert(params.load_mode.end(), modes.begin(), modes.end());
             } else if (arg == "-mg" || arg == "--main-gpu") {
                 if (++i >= argc) {
                     invalid_param = true;
@@ -829,15 +857,39 @@ static cmd_params parse_cmd_params(int argc, char ** argv) {
                     invalid_param = true;
                     break;
                 }
+                LOG_WRN("DEPRECATED: -mmp and --mmap are deprecated in favour of --load-mode. Please use --load-mode mmap instead.");
                 auto p = string_split<bool>(argv[i], split_delim);
-                params.use_mmap.insert(params.use_mmap.end(), p.begin(), p.end());
+
+                std::vector<llama_load_mode> modes;
+                for (const auto & m : p) {
+                    llama_load_mode mode;
+                    if (m) {
+                        mode = LLAMA_LOAD_MODE_MMAP;
+                    } else {
+                        mode = LLAMA_LOAD_MODE_NONE;
+                    }
+                    modes.push_back(mode);
+                }
+                params.load_mode.insert(params.load_mode.end(), modes.begin(), modes.end());
             } else if (arg == "-dio" || arg == "--direct-io") {
                 if (++i >= argc) {
                     invalid_param = true;
                     break;
                 }
+                LOG_WRN("DEPRECATED: -dio and --direct-io are deprecated in favour of --load-mode. Please use --load-mode dio instead.");
                 auto p = string_split<bool>(argv[i], split_delim);
-                params.use_direct_io.insert(params.use_direct_io.end(), p.begin(), p.end());
+
+                std::vector<llama_load_mode> modes;
+                for (const auto & m : p) {
+                    llama_load_mode mode;
+                    if (m) {
+                        mode = LLAMA_LOAD_MODE_DIRECT_IO;
+                    } else {
+                        mode = LLAMA_LOAD_MODE_NONE;
+                    }
+                    modes.push_back(mode);
+                }
+                params.load_mode.insert(params.load_mode.end(), modes.begin(), modes.end());
             } else if (arg == "-embd" || arg == "--embeddings") {
                 if (++i >= argc) {
                     invalid_param = true;
@@ -1093,6 +1145,9 @@ static cmd_params parse_cmd_params(int argc, char ** argv) {
     if (params.split_mode.empty()) {
         params.split_mode = cmd_params_defaults.split_mode;
     }
+    if (params.load_mode.empty()) {
+        params.load_mode = cmd_params_defaults.load_mode;
+    }
     if (params.main_gpu.empty()) {
         params.main_gpu = cmd_params_defaults.main_gpu;
     }
@@ -1111,12 +1166,6 @@ static cmd_params parse_cmd_params(int argc, char ** argv) {
     if (params.tensor_buft_overrides.empty()) {
         params.tensor_buft_overrides = cmd_params_defaults.tensor_buft_overrides;
     }
-    if (params.use_mmap.empty()) {
-        params.use_mmap = cmd_params_defaults.use_mmap;
-    }
-    if (params.use_direct_io.empty()) {
-        params.use_direct_io = cmd_params_defaults.use_direct_io;
-    }
     if (params.embeddings.empty()) {
         params.embeddings = cmd_params_defaults.embeddings;
     }
@@ -1164,14 +1213,13 @@ struct cmd_params_instance {
     int                n_gpu_layers;
     int                n_cpu_moe;
     llama_split_mode   split_mode;
+    llama_load_mode    load_mode;
     int                main_gpu;
     bool               no_kv_offload;
     llama_flash_attn_type flash_attn;
     std::vector<ggml_backend_dev_t> devices;
     std::vector<float> tensor_split;
     std::vector<llama_model_tensor_buft_override> tensor_buft_overrides;
-    bool               use_mmap;
-    bool               use_direct_io;
     bool               embeddings;
     bool               no_op_offload;
     bool               no_host;
@@ -1186,10 +1234,9 @@ struct cmd_params_instance {
             mparams.devices = const_cast<ggml_backend_dev_t *>(devices.data());
         }
         mparams.split_mode    = split_mode;
+        mparams.load_mode     = load_mode;
         mparams.main_gpu      = main_gpu;
         mparams.tensor_split  = tensor_split.data();
-        mparams.use_mmap      = use_mmap;
-        mparams.use_direct_io = use_direct_io;
         mparams.no_host       = no_host;
 
         if (n_cpu_moe <= 0) {
@@ -1235,9 +1282,7 @@ struct cmd_params_instance {
         return model == other.model && n_gpu_layers == other.n_gpu_layers && n_cpu_moe == other.n_cpu_moe &&
                split_mode == other.split_mode &&
                main_gpu == other.main_gpu && tensor_split == other.tensor_split &&
-               use_mmap == other.use_mmap && use_direct_io == other.use_direct_io &&
-               devices == other.devices &&
-               no_host == other.no_host &&
+               load_mode == other.load_mode && devices == other.devices && no_host == other.no_host &&
                vec_tensor_buft_override_equal(tensor_buft_overrides, other.tensor_buft_overrides);
     }
 
@@ -1270,12 +1315,11 @@ static std::vector<cmd_params_instance> get_cmd_params_instances(const cmd_param
     for (const auto & nl : params.n_gpu_layers)
     for (const auto & ncmoe : params.n_cpu_moe)
     for (const auto & sm : params.split_mode)
+    for (const auto & lm : params.load_mode)
     for (const auto & mg : params.main_gpu)
     for (const auto & devs : params.devices)
     for (const auto & ts : params.tensor_split)
     for (const auto & ot : params.tensor_buft_overrides)
-    for (const auto & mmp : params.use_mmap)
-    for (const auto & dio : params.use_direct_io)
     for (const auto & noh : params.no_host)
     for (const auto & embd : params.embeddings)
     for (const auto & nopo : params.no_op_offload)
@@ -1295,34 +1339,33 @@ static std::vector<cmd_params_instance> get_cmd_params_instances(const cmd_param
                 continue;
             }
             cmd_params_instance instance = {
-                /* .model        = */ m,
-                /* .n_prompt     = */ n_prompt,
-                /* .n_gen        = */ 0,
-                /* .n_depth      = */ nd,
-                /* .n_batch      = */ nb,
-                /* .n_ubatch     = */ nub,
-                /* .type_k       = */ tk,
-                /* .type_v       = */ tv,
-                /* .n_threads    = */ nt,
-                /* .cpu_mask     = */ cm,
-                /* .cpu_strict   = */ cs,
-                /* .poll         = */ pl,
-                /* .n_gpu_layers = */ nl,
-                /* .n_cpu_moe    = */ ncmoe,
-                /* .split_mode   = */ sm,
-                /* .main_gpu     = */ mg,
-                /* .no_kv_offload= */ nkvo,
-                /* .flash_attn   = */ fa,
-                /* .devices      = */ devs,
-                /* .tensor_split = */ ts,
+                /* .model                 = */ m,
+                /* .n_prompt              = */ n_prompt,
+                /* .n_gen                 = */ 0,
+                /* .n_depth               = */ nd,
+                /* .n_batch               = */ nb,
+                /* .n_ubatch              = */ nub,
+                /* .type_k                = */ tk,
+                /* .type_v                = */ tv,
+                /* .n_threads             = */ nt,
+                /* .cpu_mask              = */ cm,
+                /* .cpu_strict            = */ cs,
+                /* .poll                  = */ pl,
+                /* .n_gpu_layers          = */ nl,
+                /* .n_cpu_moe             = */ ncmoe,
+                /* .split_mode            = */ sm,
+                /* .load_mode             = */ lm,
+                /* .main_gpu              = */ mg,
+                /* .no_kv_offload         = */ nkvo,
+                /* .flash_attn            = */ fa,
+                /* .devices               = */ devs,
+                /* .tensor_split          = */ ts,
                 /* .tensor_buft_overrides = */ ot,
-                /* .use_mmap     = */ mmp,
-                /* .use_direct_io= */ dio,
-                /* .embeddings   = */ embd,
-                /* .no_op_offload= */ nopo,
-                /* .no_host      = */ noh,
-                /* .fit_target   = */ fpt,
-                /* .fit_min_ctx  = */ fpc,
+                /* .embeddings            = */ embd,
+                /* .no_op_offload         = */ nopo,
+                /* .no_host               = */ noh,
+                /* .fit_target            = */ fpt,
+                /* .fit_min_ctx           = */ fpc,
             };
             instances.push_back(instance);
         }
@@ -1332,34 +1375,33 @@ static std::vector<cmd_params_instance> get_cmd_params_instances(const cmd_param
                 continue;
             }
             cmd_params_instance instance = {
-                /* .model        = */ m,
-                /* .n_prompt     = */ 0,
-                /* .n_gen        = */ n_gen,
-                /* .n_depth      = */ nd,
-                /* .n_batch      = */ nb,
-                /* .n_ubatch     = */ nub,
-                /* .type_k       = */ tk,
-                /* .type_v       = */ tv,
-                /* .n_threads    = */ nt,
-                /* .cpu_mask     = */ cm,
-                /* .cpu_strict   = */ cs,
-                /* .poll         = */ pl,
-                /* .n_gpu_layers = */ nl,
-                /* .n_cpu_moe    = */ ncmoe,
-                /* .split_mode   = */ sm,
-                /* .main_gpu     = */ mg,
-                /* .no_kv_offload= */ nkvo,
-                /* .flash_attn   = */ fa,
-                /* .devices      = */ devs,
-                /* .tensor_split = */ ts,
+                /* .model                 = */ m,
+                /* .n_prompt              = */ 0,
+                /* .n_gen                 = */ n_gen,
+                /* .n_depth               = */ nd,
+                /* .n_batch               = */ nb,
+                /* .n_ubatch              = */ nub,
+                /* .type_k                = */ tk,
+                /* .type_v                = */ tv,
+                /* .n_threads             = */ nt,
+                /* .cpu_mask              = */ cm,
+                /* .cpu_strict            = */ cs,
+                /* .poll                  = */ pl,
+                /* .n_gpu_layers          = */ nl,
+                /* .n_cpu_moe             = */ ncmoe,
+                /* .split_mode            = */ sm,
+                /* .load_mode             = */ lm,
+                /* .main_gpu              = */ mg,
+                /* .no_kv_offload         = */ nkvo,
+                /* .flash_attn            = */ fa,
+                /* .devices               = */ devs,
+                /* .tensor_split          = */ ts,
                 /* .tensor_buft_overrides = */ ot,
-                /* .use_mmap     = */ mmp,
-                /* .use_direct_io= */ dio,
-                /* .embeddings   = */ embd,
-                /* .no_op_offload= */ nopo,
-                /* .no_host      = */ noh,
-                /* .fit_target   = */ fpt,
-                /* .fit_min_ctx  = */ fpc,
+                /* .embeddings            = */ embd,
+                /* .no_op_offload         = */ nopo,
+                /* .no_host               = */ noh,
+                /* .fit_target            = */ fpt,
+                /* .fit_min_ctx           = */ fpc,
             };
             instances.push_back(instance);
         }
@@ -1369,34 +1411,33 @@ static std::vector<cmd_params_instance> get_cmd_params_instances(const cmd_param
                 continue;
             }
             cmd_params_instance instance = {
-                /* .model        = */ m,
-                /* .n_prompt     = */ n_pg.first,
-                /* .n_gen        = */ n_pg.second,
-                /* .n_depth      = */ nd,
-                /* .n_batch      = */ nb,
-                /* .n_ubatch     = */ nub,
-                /* .type_k       = */ tk,
-                /* .type_v       = */ tv,
-                /* .n_threads    = */ nt,
-                /* .cpu_mask     = */ cm,
-                /* .cpu_strict   = */ cs,
-                /* .poll         = */ pl,
-                /* .n_gpu_layers = */ nl,
-                /* .n_cpu_moe    = */ ncmoe,
-                /* .split_mode   = */ sm,
-                /* .main_gpu     = */ mg,
-                /* .no_kv_offload= */ nkvo,
-                /* .flash_attn   = */ fa,
-                /* .devices      = */ devs,
-                /* .tensor_split = */ ts,
+                /* .model                 = */ m,
+                /* .n_prompt              = */ n_pg.first,
+                /* .n_gen                 = */ n_pg.second,
+                /* .n_depth               = */ nd,
+                /* .n_batch               = */ nb,
+                /* .n_ubatch              = */ nub,
+                /* .type_k                = */ tk,
+                /* .type_v                = */ tv,
+                /* .n_threads             = */ nt,
+                /* .cpu_mask              = */ cm,
+                /* .cpu_strict            = */ cs,
+                /* .poll                  = */ pl,
+                /* .n_gpu_layers          = */ nl,
+                /* .n_cpu_moe             = */ ncmoe,
+                /* .split_mode            = */ sm,
+                /* .load_mode             = */ lm,
+                /* .main_gpu              = */ mg,
+                /* .no_kv_offload         = */ nkvo,
+                /* .flash_attn            = */ fa,
+                /* .devices               = */ devs,
+                /* .tensor_split          = */ ts,
                 /* .tensor_buft_overrides = */ ot,
-                /* .use_mmap     = */ mmp,
-                /* .use_direct_io= */ dio,
-                /* .embeddings   = */ embd,
-                /* .no_op_offload= */ nopo,
-                /* .no_host      = */ noh,
-                /* .fit_target   = */ fpt,
-                /* .fit_min_ctx  = */ fpc,
+                /* .embeddings            = */ embd,
+                /* .no_op_offload         = */ nopo,
+                /* .no_host               = */ noh,
+                /* .fit_target            = */ fpt,
+                /* .fit_min_ctx           = */ fpc,
             };
             instances.push_back(instance);
         }
@@ -1426,14 +1467,13 @@ struct test {
     int                      n_gpu_layers;
     int                      n_cpu_moe;
     llama_split_mode         split_mode;
+    llama_load_mode          load_mode;
     int                      main_gpu;
     bool                     no_kv_offload;
     llama_flash_attn_type    flash_attn;
     std::vector<ggml_backend_dev_t> devices;
     std::vector<float>       tensor_split;
     std::vector<llama_model_tensor_buft_override> tensor_buft_overrides;
-    bool                     use_mmap;
-    bool                     use_direct_io;
     bool                     embeddings;
     bool                     no_op_offload;
     bool                     no_host;
@@ -1466,14 +1506,13 @@ struct test {
         n_gpu_layers   = inst.n_gpu_layers;
         n_cpu_moe      = inst.n_cpu_moe;
         split_mode     = inst.split_mode;
+        load_mode      = inst.load_mode;
         main_gpu       = inst.main_gpu;
         no_kv_offload  = inst.no_kv_offload;
         flash_attn     = inst.flash_attn;
         devices        = inst.devices;
         tensor_split   = inst.tensor_split;
         tensor_buft_overrides = inst.tensor_buft_overrides;
-        use_mmap       = inst.use_mmap;
-        use_direct_io  = inst.use_direct_io;
         embeddings     = inst.embeddings;
         no_op_offload  = inst.no_op_offload;
         no_host        = inst.no_host;
@@ -1535,8 +1574,8 @@ struct test {
             "n_ubatch",       "n_threads",      "cpu_mask",      "cpu_strict",     "poll",
             "type_k",         "type_v",         "n_gpu_layers",  "n_cpu_moe",      "split_mode",
             "main_gpu",       "no_kv_offload",  "flash_attn",    "devices",        "tensor_split",
-            "tensor_buft_overrides",            "use_mmap",      "use_direct_io",  "embeddings",
-            "no_op_offload",  "no_host",        "fit_target",     "fit_min_ctx",
+            "tensor_buft_overrides",            "load_mode",     "embeddings",
+            "no_op_offload",  "no_host",        "fit_target",    "fit_min_ctx",
             "n_prompt",       "n_gen",          "n_depth",
             "test_time",      "avg_ns",         "stddev_ns",     "avg_ts",         "stddev_ts"
         };
@@ -1554,12 +1593,15 @@ struct test {
             return INT;
         }
         if (field == "f16_kv" || field == "no_kv_offload" || field == "cpu_strict" ||
-            field == "use_mmap" || field == "use_direct_io" || field == "embeddings" || field == "no_host") {
+            field == "embeddings" || field == "no_host") {
             return BOOL;
         }
         if (field == "avg_ts" || field == "stddev_ts") {
             return FLOAT;
         }
+        if (field == "load_mode") {
+            return STRING;
+        }
         return STRING;
     }
 
@@ -1626,8 +1668,7 @@ struct test {
                                             devices_to_string(devices),
                                             tensor_split_str,
                                             tensor_buft_overrides_str,
-                                            std::to_string(use_mmap),
-                                            std::to_string(use_direct_io),
+                                            llama_load_mode_name(load_mode),
                                             std::to_string(embeddings),
                                             std::to_string(no_op_offload),
                                             std::to_string(no_host),
@@ -1806,18 +1847,15 @@ struct markdown_printer : public printer {
         if (field == "split_mode") {
             return 6;
         }
+        if (field == "load_mode") {
+            return 10;
+        }
         if (field == "flash_attn") {
             return 3;
         }
         if (field == "devices") {
             return -12;
         }
-        if (field == "use_mmap") {
-            return 4;
-        }
-        if (field == "use_direct_io") {
-            return 3;
-        }
         if (field == "test") {
             return 15;
         }
@@ -1852,11 +1890,8 @@ struct markdown_printer : public printer {
         if (field == "flash_attn") {
             return "fa";
         }
-        if (field == "use_mmap") {
-            return "mmap";
-        }
-        if (field == "use_direct_io") {
-            return "dio";
+        if (field == "load_mode") {
+            return "lm";
         }
         if (field == "embeddings") {
             return "embd";
@@ -1945,11 +1980,8 @@ struct markdown_printer : public printer {
         if (params.tensor_buft_overrides.size() > 1 || !vec_vec_tensor_buft_override_equal(params.tensor_buft_overrides, cmd_params_defaults.tensor_buft_overrides)) {
             fields.emplace_back("tensor_buft_overrides");
         }
-        if (params.use_mmap.size() > 1 || params.use_mmap != cmd_params_defaults.use_mmap) {
-            fields.emplace_back("use_mmap");
-        }
-        if (params.use_direct_io.size() > 1 || params.use_direct_io != cmd_params_defaults.use_direct_io) {
-            fields.emplace_back("use_direct_io");
+        if (params.load_mode.size() > 1 || params.load_mode != cmd_params_defaults.load_mode) {
+            fields.emplace_back("load_mode");
         }
         if (params.embeddings.size() > 1 || params.embeddings != cmd_params_defaults.embeddings) {
             fields.emplace_back("embeddings");

tools/mtmd/mtmd-helper.cpp

@@ -238,6 +238,29 @@ struct decode_embd_batch {
     }
 };
 
+// Helper class to set non-causal attention via RAII
+class scope_non_causal {
+public:
+    scope_non_causal(llama_context * context, bool enabled) : context_(context), enabled_(enabled) {
+        if (enabled_) {
+            // TODO @ngxson : need to make sure only one image is processed at a time, and n_ubatch must be enough to hold the image
+            llama_set_causal_attn(context_, false);
+        }
+    }
+    ~scope_non_causal() {
+        if (enabled_) {
+            llama_set_causal_attn(context_, true);
+        }
+    }
+
+    scope_non_causal(const scope_non_causal &) = delete;
+    scope_non_causal & operator=(const scope_non_causal &) = delete;
+
+private:
+    llama_context * context_;
+    bool enabled_;
+};
+
 // Helper function for decoding an image whose embeddings have already been calculated
 int32_t mtmd_helper_decode_image_chunk(
         mtmd_context * ctx,
@@ -288,10 +311,7 @@ int32_t mtmd_helper_decode_image_chunk(
     }
 
     const bool use_non_causal = mtmd_decode_use_non_causal(ctx, chunk);
-    if (use_non_causal) {
-        llama_set_causal_attn(lctx, false);
-        // TODO @ngxson : need to make sure only one image is processed at a time, and n_ubatch must be enough to hold the image
-    }
+    const scope_non_causal non_causal(lctx, use_non_causal);
 
     while (i_batch < n_img_batches) { // split into batches
         int pos_offset = i_batch*n_batch;
@@ -304,9 +324,6 @@ int32_t mtmd_helper_decode_image_chunk(
         int32_t ret = llama_decode(lctx, batch_embd_view);
         if (ret != 0) {
             LOG_ERR("failed to decode %s\n", name);
-            if (use_non_causal) {
-                llama_set_causal_attn(lctx, true);
-            }
             return ret;
         }
 
@@ -314,9 +331,6 @@ int32_t mtmd_helper_decode_image_chunk(
             ret = callback(batch_embd_view, user_data);
             if (ret != 0) {
                 LOG_ERR("post-decode callback failed\n");
-                if (use_non_causal) {
-                    llama_set_causal_attn(lctx, true);
-                }
                 return ret;
             }
         }
@@ -329,9 +343,6 @@ int32_t mtmd_helper_decode_image_chunk(
     n_past += mtmd_input_chunk_get_n_pos(chunk);
     *new_n_past = n_past;
 
-    if (use_non_causal) {
-        llama_set_causal_attn(lctx, true);
-    }
     return 0;
 }
 

tools/server/README.md

@@ -72,9 +72,10 @@ For the full list of features, please refer to [server's changelog](https://gith
 | `-ctv, --cache-type-v TYPE` | KV cache data type for V<br/>allowed values: f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1<br/>(default: f16)<br/>(env: LLAMA_ARG_CACHE_TYPE_V) |
 | `-dt, --defrag-thold N` | KV cache defragmentation threshold (DEPRECATED)<br/>(env: LLAMA_ARG_DEFRAG_THOLD) |
 | `--rpc SERVERS` | comma-separated list of RPC servers (host:port)<br/>(env: LLAMA_ARG_RPC) |
-| `--mlock` | force system to keep model in RAM rather than swapping or compressing<br/>(env: LLAMA_ARG_MLOCK) |
-| `--mmap, --no-mmap` | whether to memory-map model. (if mmap disabled, slower load but may reduce pageouts if not using mlock) (default: enabled)<br/>(env: LLAMA_ARG_MMAP) |
-| `-dio, --direct-io, -ndio, --no-direct-io` | use DirectIO if available. (default: disabled)<br/>(env: LLAMA_ARG_DIO) |
+| `--mlock` | DEPRECATED in favor of `--load-mode`: mmap + force system to keep model in RAM rather than swapping or compressing<br/>(env: LLAMA_ARG_MLOCK) |
+| `--mmap, --no-mmap` | DEPRECATED in favor of `--load-mode`: whether to memory-map model. (if mmap disabled, slower load but may reduce pageouts if not using mlock)<br/>(env: LLAMA_ARG_MMAP) |
+| `-dio, --direct-io, -ndio, --no-direct-io` | DEPRECATED in favor of `--load-mode`: use DirectIO if available<br/>(env: LLAMA_ARG_DIO) |
+| `-lm, --load-mode MODE` | model loading mode (default: mmap)<br/>- none: no special loading mode<br/>- mmap: memory-map model (if mmap disabled, slower load but may reduce pageouts if not using mlock)<br/>- mlock: mmap + force system to keep model in RAM rather than swapping or compressing<br/>- dio: use DirectIO if available<br/><br/>(env: LLAMA_ARG_LOAD_MODE) |
 | `--numa TYPE` | attempt optimizations that help on some NUMA systems<br/>- distribute: spread execution evenly over all nodes<br/>- isolate: only spawn threads on CPUs on the node that execution started on<br/>- numactl: use the CPU map provided by numactl<br/>if run without this previously, it is recommended to drop the system page cache before using this<br/>see https://github.com/ggml-org/llama.cpp/issues/1437<br/>(env: LLAMA_ARG_NUMA) |
 | `-dev, --device <dev1,dev2,..>` | comma-separated list of devices to use for offloading (none = don't offload)<br/>use --list-devices to see a list of available devices<br/>(env: LLAMA_ARG_DEVICE) |
 | `--list-devices` | print list of available devices and exit |

tools/server/server-models.cpp

@@ -852,7 +852,7 @@ void server_models::load(const std::string & name, const load_options & opts) {
         //                so that we can use stdout for commands and stderr for logging
         int options = subprocess_option_no_window | subprocess_option_combined_stdout_stderr;
         inst.subproc->sproc.emplace();
-        int result = subprocess_create_ex(argv.data(), options, envp.data(), &inst.subproc->get());
+        int result = subprocess_create_ex(argv.data(), options, envp.data(), nullptr, &inst.subproc->get());
         if (result != 0) {
             throw std::runtime_error("failed to spawn server instance");
         }

tools/ui/.gitignore

@@ -36,3 +36,7 @@ static/favicon*
 *storybook.log
 storybook-static
 *.code-workspace
+
+# Vitest browser mode failure artifacts
+.vitest-attachments/
+tests/**/__screenshots__/

tools/ui/.prettierignore

@@ -16,3 +16,6 @@ build/
 /build/
 /.svelte-kit/
 test-results
+
+# Vendored third party sources, kept byte identical to upstream
+src/lib/vendors/

tools/ui/eslint.config.js

@@ -59,7 +59,8 @@ export default ts.config(
 			'.svelte-kit/**',
 			'test-results/**',
 			'.storybook/**/*',
-			'src/lib/services/sandbox-worker.js'
+			'src/lib/services/sandbox-worker.js',
+			'src/lib/vendors/**'
 		]
 	},
 	storybook.configs['flat/recommended']

tools/ui/scripts/vite-plugin-nerdamer.ts

@@ -0,0 +1,49 @@
+import { build } from 'esbuild';
+import { dirname, resolve } from 'path';
+import { fileURLToPath } from 'url';
+import type { Plugin } from 'vite';
+
+const __dirname = dirname(fileURLToPath(import.meta.url));
+
+const VENDORS_DIR = resolve(__dirname, '../src/lib/vendors');
+const VIRTUAL_ID = 'virtual:nerdamer';
+const RESOLVED_ID = '\0' + VIRTUAL_ID;
+
+/**
+ * Bundle the vendored nerdamer-prime source into a minified IIFE string,
+ * exposed as the `virtual:nerdamer` module. Flags mirror the upstream
+ * build (esbuild --bundle --minify --format=iife --global-name=nerdamer),
+ * so only human readable source lives in the repo and minification is a
+ * build artifact. Vendored under src/lib/vendors/, upstream snapshot:
+ * https://github.com/together-science/nerdamer-prime/commit/1936145f8af306ec0d883b9bfd7730aedd175c24
+ */
+export function nerdamerPlugin(): Plugin {
+	let bundled: string | null = null;
+
+	return {
+		name: 'llamacpp:nerdamer',
+		resolveId(id) {
+			return id === VIRTUAL_ID ? RESOLVED_ID : undefined;
+		},
+		async load(id) {
+			if (id !== RESOLVED_ID) return undefined;
+			if (bundled === null) {
+				const result = await build({
+					entryPoints: [resolve(VENDORS_DIR, 'nerdamer-prime/all.js')],
+					bundle: true,
+					minify: true,
+					format: 'iife',
+					globalName: 'nerdamer',
+					alias: {
+						'big-integer': resolve(VENDORS_DIR, 'big-integer/BigInteger.js'),
+						'decimal.js': resolve(VENDORS_DIR, 'decimal.js/decimal.js')
+					},
+					write: false,
+					logLevel: 'silent'
+				});
+				bundled = result.outputFiles[0].text;
+			}
+			return `export default ${JSON.stringify(bundled)};`;
+		}
+	};
+}

tools/ui/src/lib/components/app/chat/ChatForm/ChatFormActions/ChatFormActionAdd/ChatFormActionAddDropdown.svelte

@@ -71,7 +71,9 @@
 
 <div class="flex items-center gap-1 {className}">
 	<DropdownMenu.Root bind:open={dropdownOpen}>
-		<Tooltip.Root>
+		<!-- ignoreNonKeyboardFocus prevents the tooltip from flashing when the
+		     menu closes and focus returns to the trigger -->
+		<Tooltip.Root ignoreNonKeyboardFocus>
 			<Tooltip.Trigger>
 				{#snippet child({ props })}
 					<DropdownMenu.Trigger

tools/ui/src/lib/components/app/chat/ChatForm/ChatFormActions/ChatFormActionAdd/ChatFormActionAddReasoningSubmenu.svelte

@@ -5,18 +5,18 @@
 	import * as Tooltip from '$lib/components/ui/tooltip';
 	import { useReasoningMenu } from '$lib/hooks/use-reasoning-menu.svelte';
 
-	let subOpen = $state(false);
-
 	const reasoning = useReasoningMenu();
 </script>
 
 {#if reasoning.modelSupportsThinking}
-	<DropdownMenu.Sub bind:open={subOpen}>
+	<DropdownMenu.Sub>
 		<DropdownMenu.SubTrigger class="flex cursor-pointer items-center gap-2">
 			{#if reasoning.thinkingEnabled}
 				<Lightbulb class="{ICON_CLASS_DEFAULT} shrink-0 text-amber-400" />
-			{:else}
+			{:else if reasoning.isOff}
 				<LightbulbOff class="{ICON_CLASS_DEFAULT} shrink-0 text-muted-foreground" />
+			{:else}
+				<Lightbulb class="{ICON_CLASS_DEFAULT} shrink-0 text-muted-foreground" />
 			{/if}
 
 			<span
@@ -27,7 +27,7 @@
 				Reasoning
 
 				<span class="capitalize text-muted-foreground">
-					{reasoning.thinkingEnabled ? reasoning.currentEffort : 'off'}
+					{reasoning.currentEffort}
 				</span>
 			</span>
 		</DropdownMenu.SubTrigger>
@@ -37,14 +37,13 @@
 		>
 			{#each reasoning.levels as level (level.value)}
 				{@const tokenLabel = reasoning.tokenLabel(level)}
-				<button
-					type="button"
-					class="flex w-full cursor-pointer items-center gap-3 rounded-md px-2 py-1.75 text-left text-sm transition-colors hover:bg-accent"
-					class:bg-accent={reasoning.isSelected(level)}
-					onclick={() => {
-						reasoning.select(level);
-						subOpen = false;
-					}}
+				<DropdownMenu.Item
+					class="flex w-full cursor-pointer items-center gap-3 rounded-md px-2 py-1.75 text-left text-sm transition-colors hover:bg-accent {reasoning.isSelected(
+						level
+					)
+						? 'bg-accent'
+						: ''}"
+					onclick={() => reasoning.select(level)}
 				>
 					{#if reasoning.isSelected(level)}
 						<Check class="{ICON_CLASS_DEFAULT} shrink-0 text-foreground" />
@@ -70,7 +69,7 @@
 							</Tooltip.Content>
 						</Tooltip.Root>
 					{/if}
-				</button>
+				</DropdownMenu.Item>
 			{/each}
 		</DropdownMenu.SubContent>
 	</DropdownMenu.Sub>

tools/ui/src/lib/components/app/chat/ChatForm/ChatFormActions/ChatFormActionAdd/ChatFormActionAddSheet.svelte

@@ -116,14 +116,16 @@
 
 							{#if reasoning.thinkingEnabled}
 								<Lightbulb class="{ICON_CLASS_DEFAULT} shrink-0 text-amber-400" />
-							{:else}
+							{:else if reasoning.isOff}
 								<LightbulbOff class="{ICON_CLASS_DEFAULT} shrink-0 text-muted-foreground" />
+							{:else}
+								<Lightbulb class="{ICON_CLASS_DEFAULT} shrink-0 text-muted-foreground" />
 							{/if}
 
 							<span class="flex-1">Reasoning</span>
 
 							<span class="text-xs capitalize text-muted-foreground">
-								{reasoning.thinkingEnabled ? reasoning.currentEffort : 'off'}
+								{reasoning.currentEffort}
 							</span>
 						</Collapsible.Trigger>
 

tools/ui/src/lib/components/app/chat/ChatForm/ChatFormActions/ChatFormReasoningEffortSubmenu.svelte

@@ -1,127 +0,0 @@
-<script lang="ts">
-	import { Check, Info, Lightbulb, LightbulbOff } from '@lucide/svelte';
-	import * as DropdownMenu from '$lib/components/ui/dropdown-menu';
-	import * as Tooltip from '$lib/components/ui/tooltip';
-	import { ReasoningEffort, MessageRole } from '$lib/enums';
-	import { REASONING_EFFORT_TOKENS } from '$lib/constants/reasoning-effort-tokens';
-	import { REASONING_EFFORT_LEVELS } from '$lib/constants/reasoning-effort';
-	import type { ReasoningEffortLevel } from '$lib/types';
-	import { DIALOG_SUBMENU_CONTENT, ICON_CLASS_DEFAULT } from '$lib/constants/css-classes';
-	import {
-		modelsStore,
-		checkModelSupportsThinking,
-		supportsThinking,
-		propsCacheVersion,
-		loadedModelIds
-	} from '$lib/stores/models.svelte';
-	import { chatStore } from '$lib/stores/chat.svelte';
-	import { conversationsStore, activeMessages } from '$lib/stores/conversations.svelte';
-	import { isRouterMode } from '$lib/stores/server.svelte';
-	import type { DatabaseMessage } from '$lib/types/database';
-
-	let thinkingEnabled = $derived(conversationsStore.getThinkingEnabled());
-	let currentEffort = $derived(conversationsStore.getReasoningEffort());
-	let isOff = $derived(!thinkingEnabled);
-	let subOpen = $state(false);
-
-	// Get conversation model from message history
-	let conversationModel = $derived(
-		chatStore.getConversationModel(activeMessages() as DatabaseMessage[])
-	);
-
-	let modelSupportsThinkingFromMessages = $derived.by(() => {
-		const modelId = isRouterMode() ? modelsStore.selectedModelName || conversationModel : null;
-		if (!modelId) return false;
-
-		const messages = conversationsStore.activeMessages;
-
-		return messages.some(
-			(m: DatabaseMessage) =>
-				m.role === MessageRole.ASSISTANT && m.model === modelId && !!m.reasoningContent
-		);
-	});
-
-	let modelSupportsThinking = $derived.by(() => {
-		loadedModelIds();
-		propsCacheVersion();
-
-		if (isRouterMode()) {
-			const modelId = modelsStore.selectedModelName || conversationModel;
-			return checkModelSupportsThinking(modelId ?? '') || modelSupportsThinkingFromMessages;
-		}
-
-		return supportsThinking() || modelSupportsThinkingFromMessages;
-	});
-
-	function isSelected(item: ReasoningEffortLevel): boolean {
-		if (item.isOff) return isOff;
-
-		return thinkingEnabled && currentEffort === item.value;
-	}
-
-	function handleSelection(item: ReasoningEffortLevel) {
-		if (item.isOff) {
-			conversationsStore.setThinkingEnabled(false);
-		} else {
-			conversationsStore.setThinkingEnabled(true);
-			conversationsStore.setReasoningEffort(item.value as ReasoningEffort);
-		}
-		subOpen = false;
-	}
-</script>
-
-{#if modelSupportsThinking}
-	<DropdownMenu.Sub bind:open={subOpen}>
-		<DropdownMenu.SubTrigger class="flex cursor-pointer items-center gap-2">
-			{#if thinkingEnabled}
-				<Lightbulb class="{ICON_CLASS_DEFAULT} shrink-0 text-amber-400" />
-			{:else}
-				<LightbulbOff class="{ICON_CLASS_DEFAULT} shrink-0 text-muted-foreground" />
-			{/if}
-
-			<span class="flex-1">Thinking</span>
-
-			{#if thinkingEnabled}
-				<span class="text-xs text-muted-foreground">{currentEffort}</span>
-			{:else}
-				<span class="text-xs text-muted-foreground">off</span>
-			{/if}
-		</DropdownMenu.SubTrigger>
-
-		<DropdownMenu.SubContent class={DIALOG_SUBMENU_CONTENT}>
-			{#each REASONING_EFFORT_LEVELS as level (level.value)}
-				<button
-					type="button"
-					class="flex w-full cursor-pointer items-center gap-2"
-					class:bg-accent={isSelected(level)}
-					onclick={() => handleSelection(level)}
-				>
-					<span class="flex-1 text-left">{level.label}</span>
-
-					{#if !level.isOff}
-						<span class="text-[11px] text-muted-foreground opacity-60">
-							{REASONING_EFFORT_TOKENS[level.value] === -1
-								? 'Unlimited'
-								: `Max ${REASONING_EFFORT_TOKENS[level.value].toLocaleString()} tokens`}
-						</span>
-					{/if}
-
-					{#if level.hasInfo}
-						<Tooltip.Root>
-							<Tooltip.Trigger>
-								<Info class="h-3.5 w-3.5 shrink-0 text-muted-foreground" />
-							</Tooltip.Trigger>
-							<Tooltip.Content side="left">
-								<p>Maximum thinking effort with extended context usage</p>
-							</Tooltip.Content>
-						</Tooltip.Root>
-					{/if}
-
-					{#if isSelected(level)}
-						<Check class="{ICON_CLASS_DEFAULT} shrink-0 text-foreground" />
-					{/if}
-				</button>
-			{/each}
-		</DropdownMenu.SubContent>
-	</DropdownMenu.Sub>
-{/if}

tools/ui/src/lib/constants/reasoning-effort.ts

@@ -6,6 +6,7 @@ import type { ReasoningEffortLevel } from '$lib/types';
  * Keys match the ReasoningEffort enum values for type-safe lookups.
  */
 export const REASONING_EFFORT_LABELS: Record<string, string> = {
+	[ReasoningEffort.DEFAULT]: 'Default',
 	[ReasoningEffort.OFF]: 'Off',
 	[ReasoningEffort.LOW]: 'Low',
 	[ReasoningEffort.MEDIUM]: 'Medium',
@@ -14,7 +15,8 @@ export const REASONING_EFFORT_LABELS: Record<string, string> = {
 };
 
 export const REASONING_EFFORT_LEVELS: ReasoningEffortLevel[] = [
-	{ value: ReasoningEffort.OFF, label: 'Off', isOff: true },
+	{ value: ReasoningEffort.DEFAULT, label: 'Default' },
+	{ value: ReasoningEffort.OFF, label: 'Off' },
 	{ value: ReasoningEffort.LOW, label: 'Low' },
 	{ value: ReasoningEffort.MEDIUM, label: 'Medium' },
 	{ value: ReasoningEffort.HIGH, label: 'High' },

tools/ui/src/lib/constants/sandbox.ts

@@ -13,27 +13,44 @@ export const SANDBOX_EMPTY_OUTPUT = '(no output)';
 
 export const SANDBOX_TRUNCATION_NOTICE = '[output truncated]';
 
-export const SANDBOX_TOOL_DEFINITION: OpenAIToolDefinition = {
-	type: ToolCallType.FUNCTION,
-	function: {
-		name: SANDBOX_TOOL_NAME,
-		description:
-			'Execute JavaScript in a sandboxed browser worker (no DOM, no page access). ' +
-			'Top level await is supported. Use console.log to print intermediate values; ' +
-			'a top level return statement is captured as the result.',
-		parameters: {
-			type: JsonSchemaType.OBJECT,
-			properties: {
-				code: {
-					type: JsonSchemaType.STRING,
-					description: 'JavaScript source to execute'
+const NERDAMER_DESCRIPTION = `
+Symbolic/numeric math via \`nerdamer\` (pre-loaded, do not require, use it directly).
+nerdamer('diff(sin(x)/x,x)') or nerdamer.diff('sin(x)/x','x') → Expression; convert with .toString()/.text()/.toTeX(), or .evaluate() (→ still Expression, then .toString()).
+nerdamer(expr,{x:2}) substitutes only; chain .evaluate() or pass 'numer' for numeric result.
+solve(expr,var)→Symbol[]; solveEquations([eq1,..])→[[var,val],..] pairs.
+Functions: simplify/expand/factor(expr), diff(expr,var[,n]), integrate(expr,var), defint(expr,from,to,var), limit(expr,var,to), laplace(expr,t,s), ilt(expr,s,t), gcd/lcm(a,b), roots/coeffs/partfrac(expr,var), pfactor(n), numer/decimals/erf(expr), product/sum(expr,var,from,to), mean/median/stdev/variance(...vals).
+Object.keys(nerdamer).filter(k=>typeof nerdamer[k]==='function') lists all available functions. If you need a function not documented above, list them first — do not guess function names.`;
+
+/**
+ * Build the sandbox tool definition. When `includeSymbolicMath` is true,
+ * the description includes nerdamer API documentation; otherwise it
+ * describes a plain JavaScript sandbox.
+ */
+export function buildSandboxToolDefinition(includeSymbolicMath: boolean): OpenAIToolDefinition {
+	return {
+		type: ToolCallType.FUNCTION,
+		function: {
+			name: SANDBOX_TOOL_NAME,
+			description: includeSymbolicMath
+				? `Execute JS in a sandboxed browser worker (no DOM/page access). Top-level await ok; console.log for intermediates; top-level return is captured as result.${NERDAMER_DESCRIPTION}`
+				: 'Execute JS in a sandboxed browser worker (no DOM/page access). Top-level await ok; console.log for intermediates; top-level return is captured as result.',
+			parameters: {
+				type: JsonSchemaType.OBJECT,
+				properties: {
+					code: {
+						type: JsonSchemaType.STRING,
+						description: 'JavaScript source to execute'
+					},
+					timeout_ms: {
+						type: JsonSchemaType.NUMBER,
+						description: `Execution timeout in milliseconds, default ${SANDBOX_TIMEOUT_MS_DEFAULT}, max ${SANDBOX_TIMEOUT_MS_MAX}`
+					}
 				},
-				timeout_ms: {
-					type: JsonSchemaType.NUMBER,
-					description: `Execution timeout in milliseconds, default ${SANDBOX_TIMEOUT_MS_DEFAULT}, max ${SANDBOX_TIMEOUT_MS_MAX}`
-				}
-			},
-			required: ['code']
+				required: ['code']
+			}
 		}
-	}
-};
+	};
+}
+
+/** @deprecated Use {@link buildSandboxToolDefinition} instead. Kept for backward compatibility. */
+export const SANDBOX_TOOL_DEFINITION = buildSandboxToolDefinition(true);

tools/ui/src/lib/constants/settings-keys.ts

@@ -67,6 +67,7 @@ export const SETTINGS_KEYS = {
 	EXCLUDE_REASONING_FROM_CONTEXT: 'excludeReasoningFromContext',
 	SHOW_RAW_OUTPUT_SWITCH: 'showRawOutputSwitch',
 	JS_SANDBOX_ENABLED: 'jsSandboxEnabled',
+	SYMBOLIC_MATH_ENABLED: 'symbolicMathEnabled',
 	// PY_INTERPRETER_ENABLED: 'pyInterpreterEnabled',
 	CUSTOM_JSON: 'customJson',
 	CUSTOM_CSS: 'customCss'

tools/ui/src/lib/constants/settings-registry.ts

@@ -724,6 +724,15 @@ const SETTINGS_REGISTRY: Record<string, SettingsSectionEntry> = {
 					paramType: SyncableParameterType.BOOLEAN
 				}
 			},
+			{
+				key: SETTINGS_KEYS.SYMBOLIC_MATH_ENABLED,
+				label: 'Symbolic math (nerdamer)',
+				help: 'Pre-load nerdamer in the sandbox for symbolic computation: simplify, diff, integrate, solve, and more. Requires "JavaScript sandbox tool" to be enabled.',
+				defaultValue: false,
+				type: SettingsFieldType.CHECKBOX,
+				section: SETTINGS_SECTION_SLUGS.DEVELOPER,
+				dependsOn: SETTINGS_KEYS.JS_SANDBOX_ENABLED
+			},
 			{
 				key: SETTINGS_KEYS.CUSTOM_JSON,
 				label: 'Custom JSON',

tools/ui/src/lib/enums/reasoning-effort.enums.ts

@@ -3,6 +3,7 @@
  * These values are sent to the server and mapped to token budgets.
  */
 export enum ReasoningEffort {
+	DEFAULT = 'default',
 	OFF = 'off',
 	LOW = 'low',
 	MEDIUM = 'medium',

tools/ui/src/lib/hooks/use-reasoning-menu.svelte.ts

@@ -17,6 +17,7 @@ import { isRouterMode } from '$lib/stores/server.svelte';
 export interface UseReasoningMenuReturn {
 	readonly modelSupportsThinking: boolean;
 	readonly thinkingEnabled: boolean;
+	readonly isOff: boolean;
 	readonly currentEffort: ReasoningEffort;
 	readonly levels: ReasoningEffortLevel[];
 	isSelected(level: ReasoningEffortLevel): boolean;
@@ -59,8 +60,10 @@ export function useReasoningMenu(): UseReasoningMenuReturn {
 		return supportsThinking() || modelSupportsThinkingFromMessages;
 	});
 
-	const thinkingEnabled = $derived(conversationsStore.getThinkingEnabled());
 	const currentEffort = $derived(conversationsStore.getReasoningEffort());
+	const thinkingEnabled = $derived(
+		currentEffort !== ReasoningEffort.OFF && currentEffort !== ReasoningEffort.DEFAULT
+	);
 
 	return {
 		get modelSupportsThinking() {
@@ -69,6 +72,9 @@ export function useReasoningMenu(): UseReasoningMenuReturn {
 		get thinkingEnabled() {
 			return thinkingEnabled;
 		},
+		get isOff() {
+			return currentEffort === ReasoningEffort.OFF;
+		},
 		get currentEffort() {
 			return currentEffort;
 		},
@@ -76,20 +82,15 @@ export function useReasoningMenu(): UseReasoningMenuReturn {
 			return REASONING_EFFORT_LEVELS;
 		},
 		isSelected(level: ReasoningEffortLevel): boolean {
-			if (level.isOff) return !thinkingEnabled;
-			return thinkingEnabled && currentEffort === level.value;
+			return currentEffort === level.value;
 		},
 		tokenLabel(level: ReasoningEffortLevel): string | null {
-			if (level.isOff) return null;
+			if (level.value === ReasoningEffort.DEFAULT) return 'Model default';
 			const tokens = REASONING_EFFORT_TOKENS[level.value];
+			if (tokens === undefined) return null;
 			return tokens === -1 ? 'Unlimited' : `Max ${tokens.toLocaleString()} tokens`;
 		},
 		select(level: ReasoningEffortLevel): void {
-			if (level.isOff) {
-				conversationsStore.setThinkingEnabled(false);
-				return;
-			}
-			conversationsStore.setThinkingEnabled(true);
 			conversationsStore.setReasoningEffort(level.value as ReasoningEffort);
 		}
 	};

tools/ui/src/lib/services/chat.service.ts

@@ -271,10 +271,14 @@ export class ChatService {
 		const reasoningBudgetTokens =
 			enableThinking && reasoningEffort ? (REASONING_EFFORT_TOKENS[reasoningEffort] ?? -1) : -1;
 
-		requestBody.chat_template_kwargs = {
-			...(requestBody.chat_template_kwargs ?? {}),
-			enable_thinking: enableThinking
-		};
+		// an explicit user choice injects the kwarg, otherwise it is omitted so
+		// the server default applies (--reasoning flag or chat template)
+		if (enableThinking !== undefined) {
+			requestBody.chat_template_kwargs = {
+				...(requestBody.chat_template_kwargs ?? {}),
+				enable_thinking: enableThinking
+			};
+		}
 
 		if (reasoningBudgetTokens >= 0) {
 			requestBody.thinking_budget_tokens = reasoningBudgetTokens;

tools/ui/src/lib/services/index.ts

@@ -276,7 +276,7 @@ export { MCPService } from './mcp.service';
  * - **toolsStore**: Exposes the tool definition when the sandbox is enabled
  * - **agenticStore**: Dispatches ToolSource.FRONTEND calls here
  *
- * @see SANDBOX_TOOL_DEFINITION in constants/sandbox.ts - tool schema sent to the LLM
+ * @see buildSandboxToolDefinition in constants/sandbox.ts - tool schema sent to the LLM
  * @see agenticStore in stores/agentic.svelte.ts - tool dispatch
  */
 export { SandboxService } from './sandbox.service';

tools/ui/src/lib/services/sandbox-harness.ts

@@ -1,14 +1,25 @@
+import { NEWLINE } from '$lib/constants';
 import WORKER_SHIM from './sandbox-worker.js?raw';
 
+/**
+ * CSP for the harness document, inherited by the blob worker. connect-src
+ * falls back to default-src, removing network egress for model and vendored
+ * code. 'unsafe-eval' is required by the worker's AsyncFunction constructor,
+ * 'unsafe-inline' by the inline script below, worker-src by the blob worker.
+ */
+const HARNESS_CSP = `default-src 'none'; script-src 'unsafe-inline' 'unsafe-eval'; worker-src blob:`;
+
 /**
  * Harness loaded as srcdoc into a sandboxed iframe (allow-scripts only).
  * The opaque origin is the security boundary: no access to the app origin,
  * its storage or its API. The harness spawns a worker so model code never
  * runs on a main thread, which makes the parent timeout enforceable by
- * removing the iframe.
+ * removing the iframe. The prelude runs in the worker before the shim,
+ * exposing globals such as `nerdamer` to model code.
  */
-export const SANDBOX_HARNESS_HTML = `<!doctype html><script>
-const SHIM = ${JSON.stringify(WORKER_SHIM)};
+export function buildSandboxHarness(preludeJs: string): string {
+	return `<!doctype html><meta http-equiv="Content-Security-Policy" content="${HARNESS_CSP}"><script>
+const SHIM = ${JSON.stringify(preludeJs + NEWLINE + WORKER_SHIM)};
 addEventListener('message', (event) => {
 	const respond = (payload) => parent.postMessage(payload, '*');
 	let worker;
@@ -23,3 +34,4 @@ addEventListener('message', (event) => {
 	worker.postMessage({ code: event.data.code });
 });
 </script>`;
+}

tools/ui/src/lib/services/sandbox-worker.js

@@ -21,7 +21,9 @@ self.onmessage = async (event) => {
 	const reply = { logs, result: null, error: null };
 	try {
 		const AsyncFunction = Object.getPrototypeOf(async function () {}).constructor;
-		const value = await new AsyncFunction(event.data.code)();
+		// The prelude bundled ahead of this shim defines self.nerdamer,
+		// passed into the execution scope as the `nerdamer` parameter.
+		const value = await new AsyncFunction('nerdamer', event.data.code)(self.nerdamer);
 		if (value !== undefined) reply.result = fmt(value);
 	} catch (err) {
 		reply.error = err instanceof Error ? err.stack || err.message : String(err);

tools/ui/src/lib/services/sandbox.service.ts

@@ -7,9 +7,32 @@ import {
 	SANDBOX_TOOL_NAME,
 	SANDBOX_TRUNCATION_NOTICE
 } from '$lib/constants';
-import { SANDBOX_HARNESS_HTML } from './sandbox-harness';
+import { buildSandboxHarness } from './sandbox-harness';
+import { config } from '$lib/stores/settings.svelte';
 import type { ToolExecutionResult } from '$lib/types';
 
+/** Cached harnesses keyed by whether nerdamer is included. */
+const harnessCache: Record<string, string> = {};
+
+/**
+ * Build the sandbox harness. When symbolic math is enabled, loads the
+ * nerdamer prelude lazily; otherwise builds a plain harness with an empty
+ * prelude. Cached per variant so toggling the setting is instant.
+ */
+async function getHarness(): Promise<string> {
+	const enabled = !!config().symbolicMathEnabled;
+	const key = enabled ? 'nerdamer' : 'plain';
+	if (!harnessCache[key]) {
+		if (enabled) {
+			const { default: nerdamerJs } = await import('virtual:nerdamer');
+			harnessCache[key] = buildSandboxHarness(nerdamerJs);
+		} else {
+			harnessCache[key] = buildSandboxHarness('');
+		}
+	}
+	return harnessCache[key];
+}
+
 interface SandboxReply {
 	logs?: unknown;
 	result?: unknown;
@@ -45,20 +68,22 @@ export class SandboxService {
 	 * timeout or abort. Removing the iframe terminates the worker
 	 * at the browser level, so runaway code cannot outlive it.
 	 */
-	static executeTool(
+	static async executeTool(
 		toolName: string,
 		params: Record<string, unknown>,
 		signal?: AbortSignal
 	): Promise<ToolExecutionResult> {
 		if (toolName !== SANDBOX_TOOL_NAME) {
-			return Promise.resolve({ content: `Unknown frontend tool: ${toolName}`, isError: true });
+			return { content: `Unknown frontend tool: ${toolName}`, isError: true };
 		}
 
 		const code = typeof params.code === 'string' ? params.code : '';
 		if (!code) {
-			return Promise.resolve({ content: 'Missing required parameter: code', isError: true });
+			return { content: 'Missing required parameter: code', isError: true };
 		}
 
+		const harness = await getHarness();
+
 		const requested = Number(params.timeout_ms);
 		const timeoutMs =
 			Number.isFinite(requested) && requested > 0
@@ -69,7 +94,7 @@ export class SandboxService {
 			const iframe = document.createElement('iframe');
 			iframe.setAttribute('sandbox', 'allow-scripts');
 			iframe.style.display = 'none';
-			iframe.srcdoc = SANDBOX_HARNESS_HTML;
+			iframe.srcdoc = harness;
 
 			let settled = false;
 

tools/ui/src/lib/stores/chat.svelte.ts

@@ -2373,9 +2373,12 @@ class ChatStore {
 
 		if (currentConfig.excludeReasoningFromContext) apiOptions.excludeReasoningFromContext = true;
 
-		apiOptions.enableThinking = conversationsStore.getThinkingEnabled();
+		// an explicit reasoning choice overrides the server default, DEFAULT sends nothing
 		const effort = conversationsStore.getReasoningEffort();
-		if (effort !== ReasoningEffort.OFF) apiOptions.reasoningEffort = effort;
+		if (effort !== ReasoningEffort.DEFAULT) {
+			apiOptions.enableThinking = effort !== ReasoningEffort.OFF;
+			if (effort !== ReasoningEffort.OFF) apiOptions.reasoningEffort = effort;
+		}
 
 		if (hasValue(currentConfig.temperature))
 			apiOptions.temperature = Number(currentConfig.temperature);

tools/ui/src/lib/stores/conversations.svelte.ts

@@ -80,25 +80,17 @@ class ConversationsStore {
 	/** Whether the store has been initialized */
 	isInitialized = $state(false);
 
-	/** Global (non-conversation-specific) thinking toggle default, derived from reasoning effort */
-	pendingThinkingEnabled = $state(false);
-
 	/** Global (non-conversation-specific) reasoning effort default */
-	pendingReasoningEffort = $state<ReasoningEffort | ReasoningEffort.OFF>(
-		ConversationsStore.loadReasoningEffortDefault()
-	);
+	pendingReasoningEffort = $state<ReasoningEffort>(ConversationsStore.loadReasoningEffortDefault());
 
-	/** Last non-off reasoning effort, restored when re-enabling thinking globally */
-	private lastNonOffEffort: ReasoningEffort | null = null;
-
-	/** Load reasoning effort default from localStorage */
-	private static loadReasoningEffortDefault(): ReasoningEffort | ReasoningEffort.OFF {
-		if (typeof globalThis.localStorage === 'undefined') return ReasoningEffort.OFF;
+	/** Load reasoning effort default from localStorage, DEFAULT defers to the server */
+	private static loadReasoningEffortDefault(): ReasoningEffort {
+		if (typeof globalThis.localStorage === 'undefined') return ReasoningEffort.DEFAULT;
 		try {
 			const raw = localStorage.getItem(REASONING_EFFORT_DEFAULT_LOCALSTORAGE_KEY);
-			return (raw as ReasoningEffort | ReasoningEffort.OFF) || ReasoningEffort.OFF;
+			return (raw as ReasoningEffort) || ReasoningEffort.DEFAULT;
 		} catch {
-			return ReasoningEffort.OFF;
+			return ReasoningEffort.DEFAULT;
 		}
 	}
 
@@ -235,17 +227,10 @@ class ConversationsStore {
 		// servers without a per-conversation override to `mcpServers[i].enabled`,
 		// and only explicit toggles are stored on the conversation.
 
-		// Inherit global thinking/reasoning defaults into the new conversation
-		const thinkingEnabled = this.getThinkingEnabled();
-		conversation.thinkingEnabled = thinkingEnabled;
-		conversation.reasoningEffort =
-			this.pendingReasoningEffort === ReasoningEffort.OFF ? undefined : this.pendingReasoningEffort;
+		// Inherit the global reasoning default into the new conversation
+		conversation.reasoningEffort = this.pendingReasoningEffort;
 		await DatabaseService.updateConversation(conversation.id, {
-			thinkingEnabled,
-			reasoningEffort:
-				this.pendingReasoningEffort === ReasoningEffort.OFF
-					? undefined
-					: this.pendingReasoningEffort
+			reasoningEffort: this.pendingReasoningEffort
 		});
 
 		this.conversations = [conversation, ...this.conversations];
@@ -793,63 +778,21 @@ class ConversationsStore {
 		await this.setMcpServerOverride(serverId, undefined);
 	}
 
-	/**
-	 * Gets the effective thinking-enabled state for the active conversation.
-	 * Returns the conversation override if set, otherwise the global default.
-	 */
-	getThinkingEnabled(): boolean {
-		if (this.activeConversation) {
-			if (this.activeConversation.thinkingEnabled !== undefined) {
-				return this.activeConversation.thinkingEnabled;
-			}
-		}
-		return this.getReasoningEffort() !== ReasoningEffort.OFF;
-	}
-
-	/**
-	 * Sets the thinking-enabled state for the active conversation.
-	 * If no conversation exists, stores the global default.
-	 * @param enabled - The enabled state
-	 */
-	async setThinkingEnabled(enabled: boolean): Promise<void> {
-		if (!this.activeConversation) {
-			if (enabled) {
-				const effort = this.lastNonOffEffort ?? ReasoningEffort.LOW;
-				this.pendingReasoningEffort = effort;
-				this.saveReasoningEffortDefaults();
-			} else {
-				if (this.pendingReasoningEffort !== ReasoningEffort.OFF) {
-					this.lastNonOffEffort = this.pendingReasoningEffort;
-				}
-				this.pendingReasoningEffort = ReasoningEffort.OFF;
-				this.saveReasoningEffortDefaults();
-			}
-			return;
-		}
-
-		this.activeConversation = {
-			...this.activeConversation,
-			thinkingEnabled: enabled
-		};
-
-		await DatabaseService.updateConversation(this.activeConversation.id, {
-			thinkingEnabled: enabled
-		});
-
-		const convIndex = this.conversations.findIndex((c) => c.id === this.activeConversation!.id);
-		if (convIndex !== -1) {
-			this.conversations[convIndex].thinkingEnabled = enabled;
-			this.conversations = [...this.conversations];
-		}
-	}
-
 	/**
 	 * Gets the effective reasoning effort for the active conversation.
 	 * Returns the conversation override if set, otherwise the global default.
+	 * DEFAULT means no override is sent and the server decides.
 	 */
-	getReasoningEffort(): ReasoningEffort | ReasoningEffort.OFF {
+	getReasoningEffort(): ReasoningEffort {
 		if (this.activeConversation) {
-			return this.activeConversation.reasoningEffort ?? this.pendingReasoningEffort;
+			if (this.activeConversation.reasoningEffort !== undefined) {
+				return this.activeConversation.reasoningEffort;
+			}
+			// conversations created before the tri-state store an explicit
+			// opt-out only as thinkingEnabled = false
+			if (this.activeConversation.thinkingEnabled === false) {
+				return ReasoningEffort.OFF;
+			}
 		}
 		return this.pendingReasoningEffort;
 	}
@@ -857,7 +800,7 @@ class ConversationsStore {
 	/**
 	 * Sets the reasoning effort for the active conversation.
 	 * If no conversation exists, stores the global default.
-	 * @param effort - The effort level ('low' | 'medium' | 'high' | 'max')
+	 * @param effort - The effort level ('default' | 'off' | 'low' | 'medium' | 'high' | 'max')
 	 */
 	async setReasoningEffort(effort: ReasoningEffort): Promise<void> {
 		if (!this.activeConversation) {

tools/ui/src/lib/stores/tools.svelte.ts

@@ -5,7 +5,7 @@ import { HealthCheckStatus, JsonSchemaType, ToolCallType, ToolSource } from '$li
 import { config } from '$lib/stores/settings.svelte';
 import {
 	DISABLED_TOOL_KEYS_LOCALSTORAGE_KEY,
-	SANDBOX_TOOL_DEFINITION,
+	buildSandboxToolDefinition,
 	TOOL_GROUP_LABELS,
 	TOOL_SERVER_LABELS
 } from '$lib/constants';
@@ -143,7 +143,9 @@ class ToolsStore {
 	}
 
 	get frontendTools(): OpenAIToolDefinition[] {
-		return config().jsSandboxEnabled ? [SANDBOX_TOOL_DEFINITION] : [];
+		return config().jsSandboxEnabled
+			? [buildSandboxToolDefinition(!!config().symbolicMathEnabled)]
+			: [];
 	}
 
 	get customTools(): OpenAIToolDefinition[] {

tools/ui/src/lib/types/reasoning.ts

@@ -1,6 +1,5 @@
 export interface ReasoningEffortLevel {
 	value: string;
 	label: string;
-	isOff?: boolean;
 	hasInfo?: boolean;
 }

tools/ui/src/lib/vendors/big-integer/LICENSE

@@ -0,0 +1,24 @@
+This is free and unencumbered software released into the public domain.
+
+Anyone is free to copy, modify, publish, use, compile, sell, or
+distribute this software, either in source code form or as a compiled
+binary, for any purpose, commercial or non-commercial, and by any
+means.
+
+In jurisdictions that recognize copyright laws, the author or authors
+of this software dedicate any and all copyright interest in the
+software to the public domain. We make this dedication for the benefit
+of the public at large and to the detriment of our heirs and
+successors. We intend this dedication to be an overt act of
+relinquishment in perpetuity of all present and future rights to this
+software under copyright law.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR
+OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+OTHER DEALINGS IN THE SOFTWARE.
+
+For more information, please refer to <http://unlicense.org>

tools/ui/src/lib/vendors/decimal.js/LICENCE.md

@@ -0,0 +1,23 @@
+The MIT Licence.
+
+Copyright (c) 2025 Michael Mclaughlin
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+'Software'), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
+CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
+SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+

tools/ui/src/lib/vendors/nerdamer-prime/Extra.js

@@ -0,0 +1,926 @@
+/*
+ * Author : Martin Donk
+ * Website : http://www.nerdamer.com
+ * Email : martin.r.donk@gmail.com
+ * License : MIT
+ * Source : https://github.com/jiggzson/nerdamer
+ */
+
+// Type imports for JSDoc ======================================================
+// These typedefs provide type aliases for the interfaces defined in index.d.ts.
+// They enable proper type checking when working with the classes defined in this file.
+//
+// Usage patterns:
+// - For return types: @returns {NerdamerSymbolType}
+// - For parameters: @param {NerdamerSymbolType} symbol
+// - For variable declarations: /** @type {NerdamerSymbolType} */
+//
+// Note: When casting local class instances to interface types, use the pattern:
+//   /** @type {InterfaceType} */ (/** @type {unknown} */ (localInstance))
+// This is needed because TypeScript sees local classes and interfaces as separate types.
+
+/**
+ * Core type aliases from index.d.ts
+ *
+ * @typedef {import('./index').NerdamerCore.NerdamerSymbol} NerdamerSymbolType
+ *
+ * @typedef {import('./index').NerdamerCore.Frac} FracType
+ *
+ * @typedef {import('./index').NerdamerCore.Vector} VectorType
+ *
+ * @typedef {import('./index').NerdamerCore.Matrix} MatrixType
+ *
+ * @typedef {import('./index').NerdamerCore.Parser} ParserType
+ *
+ * @typedef {import('./index').NerdamerCore.Settings} SettingsType
+ *
+ * @typedef {import('./index').NerdamerExpression} ExpressionType
+ *
+ * @typedef {typeof import('./index')} NerdamerType
+ *
+ * @typedef {import('./index').NerdamerCore.Utils} UtilsInterface
+ *
+ * @typedef {import('./index').NerdamerCore.Math2} Math2Interface
+ *
+ * @typedef {import('./index').NerdamerCore.Core} CoreType
+ *
+ * @typedef {import('./index').ExpressionParam} ExpressionParam
+ *
+ * @typedef {import('./index').ArithmeticOperand} ArithmeticOperand
+ *
+ * @typedef {import('./index').NerdamerCore.AlgebraModule} AlgebraModuleType
+ *
+ * @typedef {import('./index').NerdamerCore.PartFracSubModule} PartFracSubModuleType
+ *
+ * @typedef {import('./index').NerdamerCore.CalculusModule} CalculusModuleType
+ *
+ * @typedef {import('./index').NerdamerCore.ExtraModule} ExtraModuleType
+ *
+ * @typedef {import('./index').NerdamerCore.LaPlaceSubModule} LaPlaceSubModuleType
+ *
+ * @typedef {import('./index').NerdamerCore.StatisticsSubModule} StatisticsSubModuleType
+ *
+ * @typedef {import('./index').NerdamerCore.UnitsSubModule} UnitsSubModuleType
+ *
+ * @typedef {import('./index').NerdamerCore.DecomposeResultObject} DecomposeResultType
+ */
+
+// Check if nerdamer exists globally (browser) or needs to be required (Node.js)
+let nerdamer = typeof globalThis !== 'undefined' && globalThis.nerdamer ? globalThis.nerdamer : undefined;
+if (typeof module !== 'undefined' && nerdamer === undefined) {
+    nerdamer = require('./nerdamer.core.js');
+    require('./Calculus');
+    require('./Algebra');
+}
+
+/** @returns {ExtraModuleType} */
+(function initExtraModule() {
+    /** @type {CoreType} */
+    const core = nerdamer.getCore();
+    /** @type {ParserType} */
+    const _ = core.PARSER;
+    const {
+        NerdamerSymbol,
+        Vector: _Vector,
+        /** @type {AlgebraModuleType} */
+        Algebra,
+        /** @type {CalculusModuleType} */
+        Calculus,
+    } = core;
+    const { format, isVector, isArray, isSymbol } = core.Utils;
+    const { S, EX: _EX, CP, PL, CB, FN } = core.groups;
+    core.Settings.Laplace_integration_depth = 40;
+
+    /**
+     * Check if a symbol's power is itself a symbol with group S or CB
+     *
+     * @param {NerdamerSymbolType} sym
+     * @returns {boolean}
+     */
+    function hasPowerGroupSOrCB(sym) {
+        return isSymbol(sym.power) && (sym.power.group === S || sym.power.group === CB);
+    }
+
+    /**
+     * Finds a function by name within this symbol's tree.
+     *
+     * @this {NerdamerSymbolType}
+     * @param {string} fname The function name to search for
+     * @returns {NerdamerSymbolType | undefined} The found function symbol clone, or undefined if not found
+     */
+    NerdamerSymbol.prototype.findFunction = function findFunction(fname) {
+        // This is what we're looking for
+        if (this.group === FN && this.fname === fname) {
+            return this.clone();
+        }
+        let found;
+        if (this.symbols) {
+            for (const x in this.symbols) {
+                if (!Object.hasOwn(this.symbols, x)) {
+                    continue;
+                }
+                found = this.symbols[x].findFunction(fname);
+                if (found) {
+                    break;
+                }
+            }
+        }
+
+        return found;
+    };
+
+    /** @type {ExtraModuleType} */
+    const __ = (core.Extra = {
+        version: '1.4.2',
+        // http://integral-table.com/downloads/LaplaceTable.pdf
+        // Laplace assumes all coefficients to be positive
+        LaPlace: {
+            // Using: integral_0^oo f(t)*e^(-s*t) dt
+            /**
+             * @param {NerdamerSymbolType} symbol
+             * @param {NerdamerSymbolType | string} t
+             * @param {NerdamerSymbolType | string} s
+             * @returns {NerdamerSymbolType}
+             */
+            transform(symbol, t, s) {
+                /** @type {NerdamerSymbolType} */
+                symbol = symbol.clone();
+
+                t = t.toString();
+                // First try a lookup for a speed boost
+                symbol = NerdamerSymbol.unwrapSQRT(symbol, true);
+                /** @type {NerdamerSymbolType} */
+                let retval;
+                const coeff = symbol.stripVar(t);
+                const g = symbol.group;
+
+                symbol = /** @type {NerdamerSymbolType} */ (_.divide(symbol, coeff.clone()));
+
+                if (symbol.isConstant() || !symbol.contains(t, true)) {
+                    retval = _.parse(format('({0})/({1})', symbol, s));
+                } else if (g === S && core.Utils.isInt(symbol.power)) {
+                    const n = String(symbol.power);
+                    retval = _.parse(format('factorial({0})/({1})^({0}+1)', n, s));
+                } else if (symbol.group === S && symbol.power.equals(1 / 2)) {
+                    retval = _.parse(format('sqrt(pi)/(2*({0})^(3/2))', s));
+                } else if (symbol.isComposite()) {
+                    retval = new NerdamerSymbol(0);
+                    symbol.each(x => {
+                        retval = /** @type {NerdamerSymbolType} */ (_.add(retval, __.LaPlace.transform(x, t, s)));
+                    }, true);
+                } else if (symbol.isE() && hasPowerGroupSOrCB(symbol)) {
+                    const a = /** @type {NerdamerSymbolType} */ (symbol.power).stripVar(t);
+                    retval = _.parse(format('1/(({1})-({0}))', a, s));
+                } else {
+                    const fns = ['sin', 'cos', 'sinh', 'cosh'];
+                    // Support for symbols in fns with arguments in the form a*t or n*t where a = symbolic and n = Number
+                    if (
+                        symbol.group === FN &&
+                        fns.indexOf(symbol.fname) !== -1 &&
+                        (symbol.args[0].group === S || symbol.args[0].group === CB)
+                    ) {
+                        const a = symbol.args[0].stripVar(t);
+
+                        switch (symbol.fname) {
+                            case 'sin':
+                                retval = _.parse(format('({0})/(({1})^2+({0})^2)', a, s));
+                                break;
+                            case 'cos':
+                                retval = _.parse(format('({1})/(({1})^2+({0})^2)', a, s));
+                                break;
+                            case 'sinh':
+                                retval = _.parse(format('({0})/(({1})^2-({0})^2)', a, s));
+                                break;
+                            case 'cosh':
+                                retval = _.parse(format('({1})/(({1})^2-({0})^2)', a, s));
+                                break;
+                        }
+                    } else {
+                        // Try to integrate for a solution
+                        // we need at least the Laplace integration depth
+                        const depthIsLower = core.Settings.integration_depth < core.Settings.Laplace_integration_depth;
+
+                        let savedIntegrationDepth;
+                        if (depthIsLower) {
+                            savedIntegrationDepth = core.Settings.integration_depth; // Save the depth
+                            core.Settings.integration_depth = core.Settings.Laplace_integration_depth; // Transforms need a little more room
+                        }
+
+                        core.Utils.block(
+                            'PARSE2NUMBER',
+                            () => {
+                                const u = t;
+                                const sym = symbol.sub(t, u);
+                                const integrationExpr = _.parse(`e^(-${s}*${u})*${sym}`);
+                                retval = Calculus.integrate(integrationExpr, u);
+                                if (retval.hasIntegral?.()) {
+                                    retval = _.symfunction('laplace', [symbol, _.parse(String(t)), _.parse(String(s))]);
+                                    return;
+                                }
+                                //                                _.error('Unable to compute transform');
+                                retval = retval.sub(t, 0);
+                                retval = /** @type {NerdamerSymbolType} */ (
+                                    _.expand(_.multiply(retval, new NerdamerSymbol(-1)))
+                                );
+                                retval = retval.sub(u, t);
+                            },
+                            false
+                        );
+
+                        retval = /** @type {NerdamerSymbolType} */ (
+                            core.Utils.block('PARSE2NUMBER', () => _.parse(retval), true)
+                        );
+
+                        if (depthIsLower) // Put the integration depth as it was
+                        {
+                            core.Settings.integration_depth = savedIntegrationDepth;
+                        }
+                    }
+                }
+
+                return /** @type {NerdamerSymbolType} */ (_.multiply(retval, coeff));
+            },
+            /**
+             * @param {NerdamerSymbolType} symbol
+             * @param {NerdamerSymbolType | string} s_
+             * @param {NerdamerSymbolType | string} t
+             * @returns {NerdamerSymbolType}
+             */
+            inverse(symbol, s_, t) {
+                const inputSymbol = symbol.clone();
+                return core.Utils.block(
+                    'POSITIVE_MULTIPLIERS',
+                    () => {
+                        /** @type {NerdamerSymbolType | undefined} */
+                        let retval;
+                        // Expand and get partial fractions
+                        if (symbol.group === CB) {
+                            symbol = /** @type {NerdamerSymbolType} */ (
+                                /** @type {PartFracSubModuleType} */ (Algebra.PartFrac).partfrac(
+                                    /** @type {NerdamerSymbolType} */ (_.expand(symbol)),
+                                    s_
+                                )
+                            );
+                        }
+
+                        if (symbol.group === S || symbol.group === CB || symbol.isComposite()) {
+                            /** @type {number | FracType} */
+                            let p;
+                            /** @type {FracType} */
+                            let denP;
+                            /** @type {NerdamerSymbolType} */
+                            let a;
+                            /** @type {NerdamerSymbolType | string} */
+                            let b;
+                            /** @type {NerdamerSymbolType} */
+                            let d;
+                            /** @type {string} */
+                            let exp;
+                            /** @type {DecomposeResultType} */
+                            let f2;
+                            /** @type {string | number} */
+                            let fact;
+                            // Remove the multiplier
+                            const m = symbol.multiplier.clone();
+                            symbol.toUnitMultiplier();
+                            // Get the numerator and denominator
+                            let num = symbol.getNum();
+                            const den = symbol.getDenom().toUnitMultiplier();
+
+                            // TODO: Make it so factor doesn't destroy pi
+                            // num = core.Algebra.Factor.factor(symbol.getNum());
+                            // den = core.Algebra.Factor.factor(symbol.getDenom().invert(null, true));
+
+                            if (den.group === CP || den.group === PL) {
+                                denP = /** @type {FracType} */ (den.power.clone());
+                                den.toLinear();
+                            } else {
+                                denP = new core.Frac(1);
+                            }
+
+                            // Convert s to a string
+                            const s = s_.toString();
+                            // Split up the denominator if in the form ax+b
+                            /** @type {DecomposeResultType} */
+                            const f = core.Utils.decompose_fn(den, s, true);
+                            // Move the multiplier to the numerator
+                            /** @type {DecomposeResultType} */
+                            const _fe = core.Utils.decompose_fn(
+                                /** @type {NerdamerSymbolType} */ (_.expand(num.clone())),
+                                s,
+                                true
+                            );
+                            num.multiplier = num.multiplier.multiply(m);
+
+                            const finalize = function () {
+                                // Put back the numerator
+                                retval = /** @type {NerdamerSymbolType} */ (_.multiply(retval, num));
+                                retval.multiplier = retval.multiplier.multiply(symbol.multiplier);
+                                // Put back a
+                                retval = /** @type {NerdamerSymbolType} */ (_.divide(retval, f.a));
+                            };
+
+                            // Store the parts in variables for easy recognition
+                            // check if in the form t^n where n = integer
+                            if (
+                                (den.group === S || den.group === CB) &&
+                                f.x.value === s &&
+                                f.b.equals(0) &&
+                                core.Utils.isInt(f.x.power)
+                            ) {
+                                p = /** @type {number} */ (/** @type {unknown} */ (f.x.power)) - 1;
+                                fact = core.Math2.factorial(p);
+                                //  N!/s^(n-1)
+                                retval = /** @type {NerdamerSymbolType} */ (
+                                    _.divide(_.pow(_.parse(String(t)), new NerdamerSymbol(p)), new NerdamerSymbol(fact))
+                                );
+                                // Wrap it up
+                                finalize();
+                            } else if (den.group === CP && denP.equals(1)) {
+                                if (f.x.group === core.groups.PL && Algebra.degree(den).equals(2)) {
+                                    // Possibly in the form 1/(s^2+2*s+1)
+                                    // Try factoring to get it in a more familiar form{
+                                    // Apply inverse of F(s-a)
+                                    /**
+                                     * @type {{
+                                     *     f: NerdamerSymbolType;
+                                     *     a: NerdamerSymbolType;
+                                     *     h: NerdamerSymbolType;
+                                     *     c?: NerdamerSymbolType;
+                                     * }}
+                                     */
+                                    const completed = Algebra.sqComplete(den, s);
+                                    const u = core.Utils.getU(den);
+                                    // Get a for the function above
+                                    a = core.Utils.decompose_fn(completed.a, s, true).b;
+                                    const tf = __.LaPlace.inverse(
+                                        _.parse(`1/((${u})^2+(${completed.c}))`),
+                                        u,
+                                        String(t)
+                                    );
+                                    retval = /** @type {NerdamerSymbolType} */ (
+                                        _.multiply(tf, _.parse(`(${m})*e^(-(${a})*(${t}))`))
+                                    );
+                                    // A/(b*s-c) -> ae^(-bt)
+                                } else if (f.x.isLinear() && !num.contains(s)) {
+                                    t = /** @type {NerdamerSymbolType | string} */ (
+                                        _.divide(_.parse(String(t)), f.a.clone())
+                                    );
+
+                                    // Don't add factorial of one or zero
+                                    p = /** @type {number} */ (/** @type {unknown} */ (denP)) - 1;
+                                    fact = p === 0 || p === 1 ? '1' : `(${denP}-1)!`;
+                                    retval = _.parse(
+                                        format(
+                                            '(({0})^({3}-1)*e^(-(({2})*({0}))/({1})))/(({4})*({1})^({3}))',
+                                            t,
+                                            f.a,
+                                            f.b,
+                                            denP,
+                                            fact
+                                        )
+                                    );
+                                    // Wrap it up
+                                    finalize();
+                                } else if (f.x.group === S && f.x.power.equals(2)) {
+                                    if (num.contains(s)) {
+                                        // A*s/(b*s^2+c^2)
+                                        a = new NerdamerSymbol(1);
+                                        if (num.group === CB) {
+                                            /** @type {NerdamerSymbolType} */
+                                            let newNum = new NerdamerSymbol(1);
+                                            num.each(x => {
+                                                if (x.contains(s)) {
+                                                    newNum = /** @type {NerdamerSymbolType} */ (_.multiply(newNum, x));
+                                                } else {
+                                                    a = /** @type {NerdamerSymbolType} */ (_.multiply(a, x));
+                                                }
+                                            });
+                                            num = newNum;
+                                        }
+
+                                        // We need more information about the denominator to decide
+                                        f2 = core.Utils.decompose_fn(num, s, true);
+                                        const fn1 = f2.a;
+                                        const fn2 = f2.b;
+                                        const aHasSin = fn1.containsFunction('sin');
+                                        const aHasCos = fn1.containsFunction('cos');
+                                        const bHasCos = fn2.containsFunction('cos');
+                                        const bHasSin = fn2.containsFunction('sin');
+                                        if (
+                                            f2.x.value === s &&
+                                            f2.x.isLinear() &&
+                                            !((aHasSin && bHasCos) || aHasCos || bHasSin)
+                                        ) {
+                                            retval = _.parse(
+                                                format(
+                                                    '(({1})*cos((sqrt(({2})*({3}))*({0}))/({2})))/({2})',
+                                                    t,
+                                                    f2.a,
+                                                    f.a,
+                                                    f.b
+                                                )
+                                            );
+                                        } else if (aHasSin && bHasCos) {
+                                            const sin = /** @type {NerdamerSymbolType} */ (fn1.findFunction?.('sin'));
+                                            const cos = /** @type {NerdamerSymbolType} */ (fn2.findFunction?.('cos'));
+                                            // Who has the s?
+                                            if (sin?.args?.[0].equals(cos?.args?.[0]) && !sin?.args?.[0].contains(s)) {
+                                                b = /** @type {NerdamerSymbolType} */ (
+                                                    _.divide(fn2, cos.toUnitMultiplier())
+                                                ).toString();
+                                                const c = sin.args[0].toString();
+                                                d = f.b;
+                                                const e = _.divide(fn1, sin.toUnitMultiplier());
+                                                exp =
+                                                    '(({1})*({2})*cos({3})*sin(sqrt({4})*({0})))/sqrt({4})+({1})*sin({3})*({5})*cos(sqrt({4})*({0}))';
+                                                retval = _.parse(format(exp, t, a, b, c, d, e));
+                                            }
+                                        }
+                                    } else {
+                                        retval = _.parse(
+                                            format(
+                                                '(({1})*sin((sqrt(({2})*({3}))*({0}))/({2})))/sqrt(({2})*({3}))',
+                                                t,
+                                                num,
+                                                f.a,
+                                                f.b
+                                            )
+                                        );
+                                    }
+                                }
+                            } else if (
+                                /** @type {FracType} */ (f.x.power).num &&
+                                /** @type {FracType} */ (f.x.power).num.equals(3) &&
+                                /** @type {FracType} */ (f.x.power).den.equals(2) &&
+                                num.contains('sqrt(pi)') &&
+                                !num.contains(s) &&
+                                num.isLinear()
+                            ) {
+                                b = /** @type {NerdamerSymbolType} */ (_.divide(num.clone(), _.parse('sqrt(pi)')));
+                                retval = _.parse(format('(2*({2})*sqrt({0}))/({1})', t, f.a, b, num));
+                            } else if (denP.equals(2) && f.x.power.equals(2)) {
+                                if (num.contains(s)) {
+                                    // Decompose the numerator to check value of s
+                                    f2 = core.Utils.decompose_fn(
+                                        /** @type {NerdamerSymbolType} */ (_.expand(num.clone())),
+                                        s,
+                                        true
+                                    );
+                                    if (f2.x.isComposite()) {
+                                        /** @type {DecomposeResultType[]} */
+                                        const sTerms = [];
+                                        // First collect the factors e.g. (a)(bx)(cx^2+d)
+                                        /** @type {DecomposeResultType[]} */
+                                        const symbols = /** @type {DecomposeResultType[]} */ (
+                                            num
+                                                .collectSymbols(x => {
+                                                    x = NerdamerSymbol.unwrapPARENS(x);
+                                                    /** @type {DecomposeResultType} */
+                                                    const decomp = core.Utils.decompose_fn(x, s, true);
+                                                    decomp.symbol = x;
+                                                    return decomp;
+                                                })
+                                                // Then sort them by power hightest to lowest
+                                                .sort((x1, x2) => {
+                                                    const p1 =
+                                                        /** @type {DecomposeResultType} */ (x1).x.value === s
+                                                            ? /** @type {number} */ (
+                                                                  /** @type {unknown} */ (
+                                                                      /** @type {DecomposeResultType} */ (x1).x.power
+                                                                  )
+                                                              )
+                                                            : 0;
+                                                    const p2 =
+                                                        /** @type {DecomposeResultType} */ (x2).x.value === s
+                                                            ? /** @type {number} */ (
+                                                                  /** @type {unknown} */ (
+                                                                      /** @type {DecomposeResultType} */ (x2).x.power
+                                                                  )
+                                                              )
+                                                            : 0;
+                                                    return p2 - p1;
+                                                })
+                                        );
+                                        a = new NerdamerSymbol(-1);
+                                        // Grab only the ones which have s
+                                        for (let i = 0; i < symbols.length; i++) {
+                                            const fc = symbols[i];
+                                            if (fc.x.value === s) {
+                                                sTerms.push(fc);
+                                            } else {
+                                                a = /** @type {NerdamerSymbolType} */ (_.multiply(a, fc.symbol));
+                                            }
+                                        }
+                                        // The following 2 assumptions are made
+                                        // 1. since the numerator was factored above then each s_term has a unique power
+                                        // 2. because the terms are sorted by descending powers then the first item
+                                        //    has the highest power
+                                        // We can now check for the next type s(s^2-a^2)/(s^2+a^2)^2
+                                        if (
+                                            sTerms[0].x.power.equals(2) &&
+                                            sTerms[1].x.power.equals(1) &&
+                                            sTerms[1].b.equals(0) &&
+                                            !sTerms[0].b.equals(0)
+                                        ) {
+                                            b = sTerms[0].a.negate();
+                                            exp =
+                                                '-(({1})*({2})*({5})*({0})*sin((sqrt(({4})*({5}))*({0}))/({4})))/' +
+                                                '(2*({4})^2*sqrt(({4})*({5})))-(({1})*({3})*({0})*sin((sqrt(({4})*({5}))*({0}))/({4})))' +
+                                                '/(2*({4})*sqrt(({4})*({5})))+(({1})*({2})*cos((sqrt(({4})*({5}))*({0}))/({4})))/({4})^2';
+                                            retval = _.parse(format(exp, t, a, b, sTerms[0].b, f.a, f.b));
+                                        }
+                                    } else if (f2.x.isLinear()) {
+                                        a = /** @type {NerdamerSymbolType} */ (_.divide(f2.a, new NerdamerSymbol(2)));
+                                        exp =
+                                            '(({1})*({0})*sin((sqrt(({2})*({3}))*({0}))/({2})))/(({2})*sqrt(({2})*({3})))';
+                                        retval = _.parse(format(exp, t, a, f.a, f.b));
+                                    } else if (f2.x.power.equals(2)) {
+                                        if (f2.b.equals(0)) {
+                                            a = /** @type {NerdamerSymbolType} */ (
+                                                _.divide(f2.a, new NerdamerSymbol(2))
+                                            );
+                                            exp =
+                                                '(({1})*sin((sqrt(({2})*({3}))*({0}))/({2})))/(({2})*sqrt(({2})*({3})))+(({1})*({0})*cos((sqrt(({2})*({3}))*({0}))/({2})))/({2})^2';
+                                            retval = _.parse(format(exp, t, a, f.a, f.b));
+                                        } else {
+                                            a = /** @type {NerdamerSymbolType} */ (
+                                                _.divide(f2.a, new NerdamerSymbol(2))
+                                            );
+                                            d = f2.b.negate();
+                                            exp =
+                                                '-((({2})*({4})-2*({1})*({3}))*sin((sqrt(({2})*({3}))*({0}))/({2})))/(2*({2})*({3})*sqrt(({2})*({3})))+' +
+                                                '(({4})*({0})*cos((sqrt(({2})*({3}))*({0}))/({2})))/(2*({2})*({3}))+(({1})*({0})*cos((sqrt(({2})*({3}))*({0}))/({2})))/({2})^2';
+                                            retval = _.parse(format(exp, t, a, f.a, f.b, d));
+                                        }
+                                    }
+                                } else {
+                                    a = /** @type {NerdamerSymbolType} */ (_.divide(num, new NerdamerSymbol(2)));
+                                    exp =
+                                        '(({1})*sin((sqrt(({2})*({3}))*({0}))/({2})))/(({3})*sqrt(({2})*({3})))-(({1})*({0})*cos((sqrt(({2})*({3}))*({0}))/({2})))/(({2})*({3}))';
+                                    retval = _.parse(format(exp, t, a, f.a, f.b));
+                                }
+                            } else if (symbol.isComposite()) {
+                                // 1/(s+1)^2
+                                if (denP.equals(2) && f.x.group === S) {
+                                    retval = _.parse(`(${m})*(${t})*e^(-(${f.b})*(${t}))`);
+                                } else {
+                                    retval = new NerdamerSymbol(0);
+
+                                    symbol = /** @type {NerdamerSymbolType} */ (
+                                        /** @type {PartFracSubModuleType} */ (Algebra.PartFrac).partfrac(
+                                            /** @type {NerdamerSymbolType} */ (_.expand(symbol)),
+                                            s_
+                                        )
+                                    );
+
+                                    symbol.each(x => {
+                                        retval = /** @type {NerdamerSymbolType} */ (
+                                            _.add(retval, __.LaPlace.inverse(x, s_, t))
+                                        );
+                                    }, true);
+                                }
+                            }
+                        }
+
+                        retval ||= _.symfunction('ilt', [inputSymbol, _.parse(String(s_)), _.parse(String(t))]);
+
+                        return /** @type {NerdamerSymbolType} */ (retval);
+                    },
+                    true
+                );
+            },
+        },
+        Statistics: {
+            /**
+             * @param {NerdamerSymbolType[]} arr
+             * @returns {Record<string, number>}
+             */
+            frequencyMap(arr) {
+                /** @type {Record<string, number>} */
+                const map = {};
+                // Get the frequency map
+                for (let i = 0, l = arr.length; i < l; i++) {
+                    const e = arr[i];
+                    const key = e.toString();
+                    map[key] ||= 0; // Default it to zero
+                    map[key]++; // Increment
+                }
+                return map;
+            },
+            /**
+             * @param {NerdamerSymbolType[]} arr
+             * @returns {NerdamerSymbolType[]}
+             */
+            sort(arr) {
+                return arr.sort((a, b) => {
+                    if (!a.isConstant() || !b.isConstant()) {
+                        _.error('Unable to sort! All values must be numeric');
+                    }
+                    return /** @type {number} */ (/** @type {unknown} */ (a.multiplier.subtract(b.multiplier)));
+                });
+            },
+            /**
+             * @param {NerdamerSymbolType[]} arr
+             * @returns {NerdamerSymbolType}
+             */
+            count(arr) {
+                return new NerdamerSymbol(arr.length);
+            },
+            /**
+             * @param {NerdamerSymbolType[]} arr
+             * @param {NerdamerSymbolType} [x_]
+             * @returns {NerdamerSymbolType}
+             */
+            sum(arr, x_) {
+                /** @type {NerdamerSymbolType} */
+                let sum = new NerdamerSymbol(0);
+                for (let i = 0, l = arr.length; i < l; i++) {
+                    const xi = arr[i].clone();
+                    if (x_) {
+                        sum = /** @type {NerdamerSymbolType} */ (
+                            _.add(_.pow(_.subtract(xi, x_.clone()), new NerdamerSymbol(2)), sum)
+                        );
+                    } else {
+                        sum = /** @type {NerdamerSymbolType} */ (_.add(xi, sum));
+                    }
+                }
+
+                return sum;
+            },
+            /**
+             * @param {...NerdamerSymbolType} args
+             * @returns {NerdamerSymbolType}
+             */
+            mean(...args) {
+                // Handle arrays
+                if (isVector(args[0])) {
+                    return __.Statistics.mean(.../** @type {NerdamerSymbolType[]} */ (args[0].elements));
+                }
+                return /** @type {NerdamerSymbolType} */ (_.divide(__.Statistics.sum(args), __.Statistics.count(args)));
+            },
+            /**
+             * @param {...NerdamerSymbolType} args
+             * @returns {NerdamerSymbolType}
+             */
+            median(...args) {
+                /** @type {NerdamerSymbolType} */
+                let retval;
+                // Handle arrays
+                if (isVector(args[0])) {
+                    return __.Statistics.median(.../** @type {NerdamerSymbolType[]} */ (args[0].elements));
+                }
+                try {
+                    const sorted = __.Statistics.sort(args);
+                    const l = args.length;
+                    if (core.Utils.even(l)) {
+                        const mid = l / 2;
+                        retval = __.Statistics.mean(sorted[mid - 1], sorted[mid]);
+                    } else {
+                        retval = sorted[Math.floor(l / 2)];
+                    }
+                } catch (e) {
+                    if (/** @type {Error} */ (e).message === 'timeout') {
+                        throw e;
+                    }
+                    retval = _.symfunction('median', args);
+                }
+                return retval;
+            },
+            /**
+             * @param {...NerdamerSymbolType} args
+             * @returns {NerdamerSymbolType}
+             */
+            mode(...args) {
+                /** @type {NerdamerSymbolType} */
+                let retval;
+                // Handle arrays
+                if (isVector(args[0])) {
+                    return __.Statistics.mode(.../** @type {NerdamerSymbolType[]} */ (args[0].elements));
+                }
+
+                const map = __.Statistics.frequencyMap(args);
+
+                // The mode of 1 item is that item as per issue #310 (verified by Happypig375).
+                if (core.Utils.keys(map).length === 1) {
+                    retval = args[0];
+                } else {
+                    // Invert by arraning them according to their frequency
+                    /** @type {Record<number, string | string[]>} */
+                    const inverse = {};
+                    for (const x in map) {
+                        if (!Object.hasOwn(map, x)) {
+                            continue;
+                        }
+                        const freq = map[x];
+                        // Check if it's in the inverse already
+                        if (freq in inverse) {
+                            const e = inverse[freq];
+                            // If it's already an array then just add it
+                            if (isArray(e)) {
+                                e.push(x);
+                            }
+                            // Convert it to and array
+                            else {
+                                inverse[freq] = [x, /** @type {string} */ (inverse[freq])];
+                            }
+                        } else {
+                            inverse[freq] = x;
+                        }
+                    }
+                    // The keys now represent the maxes. We want the max of those keys
+                    const keyNums = core.Utils.keys(inverse).map(k => Number(k));
+                    const maxKey = Math.max.apply(null, keyNums);
+                    const max = inverse[maxKey];
+                    // Check it's an array. If it is then map over the results and convert
+                    // them to NerdamerSymbol
+                    if (isArray(max)) {
+                        retval = _.symfunction(
+                            'mode',
+                            max.sort().map(v => _.parse(v))
+                        );
+                    } else {
+                        retval = _.parse(/** @type {string} */ (max));
+                    }
+                }
+
+                return retval;
+            },
+            /**
+             * @param {NerdamerSymbolType} k
+             * @param {NerdamerSymbolType[]} args
+             * @returns {NerdamerSymbolType}
+             */
+            gVariance(k, args) {
+                const x_ = __.Statistics.mean(...args);
+                const sum = __.Statistics.sum(args, x_);
+                return /** @type {NerdamerSymbolType} */ (_.multiply(k, sum));
+            },
+            /**
+             * @param {...NerdamerSymbolType} args
+             * @returns {NerdamerSymbolType}
+             */
+            variance(...args) {
+                // Handle arrays
+                if (isVector(args[0])) {
+                    return __.Statistics.variance(.../** @type {NerdamerSymbolType[]} */ (args[0].elements));
+                }
+                const k = /** @type {NerdamerSymbolType} */ (
+                    _.divide(new NerdamerSymbol(1), __.Statistics.count(args))
+                );
+                return __.Statistics.gVariance(k, args);
+            },
+            /**
+             * @param {...NerdamerSymbolType} args
+             * @returns {NerdamerSymbolType}
+             */
+            sampleVariance(...args) {
+                // Handle arrays
+                if (isVector(args[0])) {
+                    return __.Statistics.sampleVariance(.../** @type {NerdamerSymbolType[]} */ (args[0].elements));
+                }
+
+                const k = /** @type {NerdamerSymbolType} */ (
+                    _.divide(new NerdamerSymbol(1), _.subtract(__.Statistics.count(args), new NerdamerSymbol(1)))
+                );
+                return __.Statistics.gVariance(k, args);
+            },
+            /**
+             * @param {...NerdamerSymbolType} args
+             * @returns {NerdamerSymbolType}
+             */
+            standardDeviation(...args) {
+                // Handle arrays
+                if (isVector(args[0])) {
+                    return __.Statistics.standardDeviation(.../** @type {NerdamerSymbolType[]} */ (args[0].elements));
+                }
+                return /** @type {NerdamerSymbolType} */ (
+                    _.pow(__.Statistics.variance(...args), new NerdamerSymbol(1 / 2))
+                );
+            },
+            /**
+             * @param {...NerdamerSymbolType} args
+             * @returns {NerdamerSymbolType}
+             */
+            sampleStandardDeviation(...args) {
+                // Handle arrays
+                if (isVector(args[0])) {
+                    return __.Statistics.sampleStandardDeviation(
+                        .../** @type {NerdamerSymbolType[]} */ (args[0].elements)
+                    );
+                }
+                return /** @type {NerdamerSymbolType} */ (
+                    _.pow(__.Statistics.sampleVariance(...args), new NerdamerSymbol(1 / 2))
+                );
+            },
+            /**
+             * @param {NerdamerSymbolType} x
+             * @param {NerdamerSymbolType} mean
+             * @param {NerdamerSymbolType} stdev
+             * @returns {NerdamerSymbolType}
+             */
+            zScore(x, mean, stdev) {
+                return /** @type {NerdamerSymbolType} */ (_.divide(_.subtract(x, mean), stdev));
+            },
+        },
+        Units: {
+            table: {
+                foot: '12 inch',
+                meter: '100 cm',
+                decimeter: '10 cm',
+            },
+        },
+    });
+
+    nerdamer.register([
+        {
+            name: 'laplace',
+            visible: true,
+            numargs: 3,
+            build() {
+                return __.LaPlace.transform;
+            },
+        },
+        {
+            name: 'ilt',
+            visible: true,
+            numargs: 3,
+            build() {
+                return __.LaPlace.inverse;
+            },
+        },
+        // Statistical
+        {
+            name: 'mean',
+            visible: true,
+            numargs: -1,
+            build() {
+                return __.Statistics.mean;
+            },
+        },
+        {
+            name: 'median',
+            visible: true,
+            numargs: -1,
+            build() {
+                return __.Statistics.median;
+            },
+        },
+        {
+            name: 'mode',
+            visible: true,
+            numargs: -1,
+            build() {
+                return __.Statistics.mode;
+            },
+        },
+        {
+            name: 'smpvar',
+            visible: true,
+            numargs: -1,
+            build() {
+                return __.Statistics.sampleVariance;
+            },
+        },
+        {
+            name: 'variance',
+            visible: true,
+            numargs: -1,
+            build() {
+                return __.Statistics.variance;
+            },
+        },
+        {
+            name: 'smpstdev',
+            visible: true,
+            numargs: -1,
+            build() {
+                return __.Statistics.sampleStandardDeviation;
+            },
+        },
+        {
+            name: 'stdev',
+            visible: true,
+            numargs: -1,
+            build() {
+                return __.Statistics.standardDeviation;
+            },
+        },
+        {
+            name: 'zscore',
+            visible: true,
+            numargs: 3,
+            build() {
+                return __.Statistics.zScore;
+            },
+        },
+    ]);
+
+    // Link registered functions externally
+    nerdamer.updateAPI();
+})();
+
+// Added for all.min.js
+if (typeof module !== 'undefined') {
+    module.exports = nerdamer;
+}

tools/ui/src/lib/vendors/nerdamer-prime/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2023 together-science
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

tools/ui/src/lib/vendors/nerdamer-prime/all.js

@@ -0,0 +1,16 @@
+/*
+ * Author : Martin Donk
+ * Website : http://www.nerdamer.com
+ * Email : martin.r.donk@gmail.com
+ * Source : https://github.com/jiggzson/nerdamer
+ * Can be used to load all add-ons with one require
+ */
+
+const nerdamer = require('./nerdamer.core.js');
+require('./Algebra.js');
+require('./Calculus.js');
+require('./Solve.js');
+require('./Extra.js');
+
+// Export nerdamer
+module.exports = nerdamer;

tools/ui/src/lib/vendors/nerdamer-prime/constants.js

@@ -0,0 +1,261 @@
+/*
+ * Mathematical constants for nerdamer
+ * This file contains precomputed values and mathematical constants
+ * used throughout the library.
+ */
+
+/**
+ * Container of pregenerated prime numbers up to 2083 This array is used as a cache and can be extended at runtime by
+ * functions like generatePrimes()
+ */
+const PRIMES = [
+    2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89, 97, 101, 103, 107, 109,
+    113, 127, 131, 137, 139, 149, 151, 157, 163, 167, 173, 179, 181, 191, 193, 197, 199, 211, 223, 227, 229, 233, 239,
+    241, 251, 257, 263, 269, 271, 277, 281, 283, 293, 307, 311, 313, 317, 331, 337, 347, 349, 353, 359, 367, 373, 379,
+    383, 389, 397, 401, 409, 419, 421, 431, 433, 439, 443, 449, 457, 461, 463, 467, 479, 487, 491, 499, 503, 509, 521,
+    523, 541, 547, 557, 563, 569, 571, 577, 587, 593, 599, 601, 607, 613, 617, 619, 631, 641, 643, 647, 653, 659, 661,
+    673, 677, 683, 691, 701, 709, 719, 727, 733, 739, 743, 751, 757, 761, 769, 773, 787, 797, 809, 811, 821, 823, 827,
+    829, 839, 853, 857, 859, 863, 877, 881, 883, 887, 907, 911, 919, 929, 937, 941, 947, 953, 967, 971, 977, 983, 991,
+    997, 1009, 1013, 1019, 1021, 1031, 1033, 1039, 1049, 1051, 1061, 1063, 1069, 1087, 1091, 1093, 1097, 1103, 1109,
+    1117, 1123, 1129, 1151, 1153, 1163, 1171, 1181, 1187, 1193, 1201, 1213, 1217, 1223, 1229, 1231, 1237, 1249, 1259,
+    1277, 1279, 1283, 1289, 1291, 1297, 1301, 1303, 1307, 1319, 1321, 1327, 1361, 1367, 1373, 1381, 1399, 1409, 1423,
+    1427, 1429, 1433, 1439, 1447, 1451, 1453, 1459, 1471, 1481, 1483, 1487, 1489, 1493, 1499, 1511, 1523, 1531, 1543,
+    1549, 1553, 1559, 1567, 1571, 1579, 1583, 1597, 1601, 1607, 1609, 1613, 1619, 1621, 1627, 1637, 1657, 1663, 1667,
+    1669, 1693, 1697, 1699, 1709, 1721, 1723, 1733, 1741, 1747, 1753, 1759, 1777, 1783, 1787, 1789, 1801, 1811, 1823,
+    1831, 1847, 1861, 1867, 1871, 1873, 1877, 1879, 1889, 1901, 1907, 1913, 1931, 1933, 1949, 1951, 1973, 1979, 1987,
+    1993, 1997, 1999, 2003, 2011, 2017, 2027, 2029, 2039, 2053, 2063, 2069, 2081, 2083,
+];
+
+/** Set representation of PRIMES for O(1) lookup This object is used as a cache and can be extended at runtime */
+/** @type {Record<number, boolean>} */
+const PRIMES_SET = {};
+for (const p of PRIMES) {
+    PRIMES_SET[p] = true;
+}
+
+/** High precision value of Pi (200 decimal places) Used for high-precision calculations */
+const LONG_PI =
+    '3.14159265358979323846264338327950288419716939937510582097494459230781640628620899862803482534211706798214' +
+    '808651328230664709384460955058223172535940812848111745028410270193852110555964462294895493038196';
+
+/** High precision value of Euler's number e (200 decimal places) Used for high-precision calculations */
+const LONG_E =
+    '2.718281828459045235360287471352662497757247093699959574966967627724076630353547594571382178525166427427466' +
+    '39193200305992181741359662904357290033429526059563073813232862794349076323382988075319525101901';
+
+/**
+ * Precomputed high-precision fraction values for the bigLog function. These are used for arbitrary-precision logarithm
+ * calculations. Each entry is a string representation of a high-precision rational number.
+ */
+const BIG_LOG_CACHE = [
+    '-253631954333118718762629409109262279926288908775918712466601196032/39970093576053625963957478139049824030906352922262642968060706375',
+    '0',
+    '24553090145869607172412918483124184864289170814122579923404694986469653261608528681589949629750677407356463601998534945057511664951799678336/35422621391945757431676178435630229283255250779216421054188228659061954317501699707236864189383591478024245495110561124597124995986978302375',
+    '369017335340917140706044240090243368728616279239227943871048759140274862131699550043150713059889196223917527172547/335894053932612728969975338549993764554481173661218585876475837409922537622385232776657791604345125227005476864000',
+    '24606853025626737903121303930100462245506322607985779603220820323211395607931699126390918477501325805513849611930008427268176602460462988972957593458726734897129954728102144/17750092415977639787139561330326170936321452137635322313122938207611787444311735251389066106937796085669460151963285086542745859461943369606018450213014148175716400146484375',
+    '399073568781976806715759409052286641738926636328983929439450824555613704676637191564699164303012247386095942144825603522401740680808466858044/247958349743620302021733249049411604982786755454514947379317600613433680222511897950658049325685140346169718465773927872179874971908848116625',
+    '1468102989495846944084741146947295378041808701256909016224309866143294556551407470861354311593351276612463858816796714569499021375899793849136855085849133702029337910502448189055357182595424959360/819363879309286303497217527375463120404739098260200279520788950777458900438307356738082930586032462601215802636320993648007907724899611296693997216938989854861043298494990214825163523387600982777',
+    '5896704855274661767824574093605344871722790278354431422729640950821239030785642943033153793245906863203822369276271050164634206965056233097479117980782641839669/3030306850569309344013726745100070601277982132543905537366562638553198167007159067544789592089960911065181606283478843359856123992707598685058297067179343872000',
+    '76631772943534985713873427262830314617912556928476573358548256872141516989538374761909611879922349479420014771499018155447198112155515453671128814488139633810493264352294560043912066253026059140653027326566801398784/36852092933388988649396042883218509607503204211148493545892849595498822817623842579026942621098851631842754395231561679671400197056377380063233740202370686144673585955581403046886083948450136247134308381940165804875',
+    '3159076083816399509754948610929467278257473888282947311280653574634802580912280940686954763313882823327077171624015737719617373932318151594325834524000275847475866299387913048/1437757485694188822758304467756419845842037623148461107362957994816554782989250555362514354661961482939226272309026092009962414616417412938087494467254146002233028411865234375',
+    '22266067259907364984531611601870291368272674573653403965630628996687370994139884833897773468149149664829922302484782423514167405397665098388400450149078982462318781750661005833037235183394221496186539779712428265837926417581952/9670030144664428565128962309657100138096047028794689249320859276197340398920725569428532293373676415359965773460364494998334259893079003125373872108770534788283842907318071170285038777091588292539102269617376180390982915567375',
+    '14604654564989239958569331443385369522850975185358647132770022716433280072271007767111036877803328768910274400515590151934676819262085211828028638417329558229123989556376108454497813055/6090614019162516693013973409650613208227889078878781039105047015752493519149314227721984436973374032279421344818329285207124280297611253861173835238379831004010748379874393292231671808',
+    '1901241885407696031217292877862925220917660047127261026827869027159993239567933534052663335498281439239753018507182016153657409777749792228538380379703411298411623469292891476969894084838876001545818141543890273256985768690847587711270930688/765116019778838839812655402103512685695769161212360553099732689795578904762091216998790589926057819838537805856579109910198553330075924857419395160755642371550113347465300208422126945265887065434116781678702741657275181694851670325469434625',
+    '139459806786604751793737926146840623607010208216289543036026206208962059593900745886202214788747453279179283344350478734275973878932538430194363355795823581315329311220701640235653288975569812161436/54371368534412517053056101353618694718215711767266376573138772968257303578467926450212293233332401067673270853953399269852376592855992724934941173346260129257754416412476202526978443681584633116375',
+    '1045669091124493070709683241190022970908640501171378776604126771144008324358233819560649021940145166254659028524319517244711645162132513416238958170819347361185944945680269442845829390112062101255500836072082817820950448463314034677353723256969344/396228259004446234921310936915931611736815598535963504660076315228798989932959459406702091180060429080345146735173591749448509810270759531977278642135591672189002006272326131885315743181289970885337574780897529347356567086535505950450897216796875',
+    '9912919238915437302006264477931031611447467070103973106567538528951878797932559935860738745374437522819124347510590800370471910492338584284092534264608801221235029062881964101996762011296996851893455828946521/3660537472668264151218961634689665210933936249986285290553357254224360417386515311493310199319523687171757653216994741150377508234317025158302057758196429623723072084157928224798322861732880034847243894784000',
+    '9263710175433181746575186369318246002919895649622127410824041370079225200282403368319370743363303164313395723904510539050157032684710468364067204876434546848634842333436957245275217583248805993142227630297924119330553308466662488683624783307023014909360640/3341177182697517248552428837661919299725031035849865632511882688786226888137634168024976033652753689210700218163621739078534353578510364301481093730054725078138658805025014615651043313990684347632166030359086885561104034510990826655289288319840595753002771',
+    '5116082230713622171832327542439052727465114322479570603905499496221224653983960598946033081212909066917137546065542953865612718836914393275681318667667521726785633638189373998191090501201427906618075889744489190209584/1805752553736060443820406101277706970767657006346276183748749630179442318063568286372320188433843729960294965366346522303898609655762491623098453269916163621089005711823488749297418113474056676109581110715068124438875',
+    '246569125619713282434448566970352231845414317018379160824176638351574938993535464763890962336882760882398479702237564384291290459961036068916857265499633061660562532011248501476114401629839742058389195725393702000011860799793778295606988057303225493814005789533570432/85307063020836305797178273029353623060860009152114361453434032434699636078115114412588719432277441055049132559782203988387794711585368296817222565434951256788867244687081233632650953850383220864394261763844194948389861147622944651546912394593164406926489862036343375',
+    '133672026303452911046163998480860917119290576658330909785707604886881155606725822685088929236266583416708668502760907677019598002175122453170574729028452721476464728566191464897928696630979863154661704374206171469014225143/45398130975270785045482567762871405072140548998125471025451666500000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000',
+    '6041693953360002800224091673336562508913199995987479264605216252220579740134601435770085920869376641180763419907442721705887169884230643795126568815123647603047739799302562095542459344811429882053086550900803768964612193941424128649976704727183797495759082741166938351872/2016766992122395667828553277997478570503475626107286343497917705446132017125079612756035254750822860815515899557855166824523851779156336235294914777307802256439645525835223691751931866188957324792276149549076500784191791380803500156776088683900346065830066370370083309875',
+    '705868391597244582764749229356331441978820024796066870551110486625729826111158236686696326058778874201639006234449557592353247542995871491078308187261304930042019640830629526023972693107193897009168955674240659026247094657679060/231848642748474339277532000336338632910990823562381469441716922006107433404523316252618490265927265734670539384485699132080062215196462178933963957679882342083893417545858074378754089719547920901917516016346211301054206383643383',
+    '101832160604157943093944673541651013907278188571533075311673249923948856034633446617630054761681006062910980371900782781226979391765818325065031889334563981235894369036439929651260587335544056975715076598739977065390678221999918899003881778449092038750712969437519295878491018112/32944277910571666002449086492515464541550138004002141571670657643770713783329063548790202120805341989608877739811787937782240802963962520261844114327432160788193314874913687387269408387417806176202979244637915812905426565263196954203487934225589622864145960079736633434831996625',
+    '10655703119271468913597640479490594180964700448340778168715956712130636958373270202484276402718566314881119559090842449610957974112230306343486091910217340665146602598568991520563987490686996746558858366002301982443029430290679385551/3398412687366638541233365137084722368200311117891192348532156645374786104142009695796409107380345795998400850838706661851176885183144928701608654514812261697598380070746520197171576610572921007069104300695592751543563472456384512000',
+    '1903039332876763837419920240543738799531131775028971323439870868730321221615515008394327723508670975623498588291298064320786970626232668956372004004897872810230178526101184611242511193415796638694370503100219710864543168952682617801833318493436174387568067811938490953495819438108686336/598806534367503338307287246320963280558134937382149405305466709787179429317914803617527827862441615350396864359976273212272586892074799651088317544101755361439294687323233086696182687664637422796995789967075271448560870681210580691574924544896656175563265378514188341796398162841796875',
+    '525573915563826130963525826191411949262846916750432019596028344808298471293378917508549164993368392834023782480702893643486699787870059946429810070222126260200026332874480239090370088123833491499400991181659445914352500247596757005142623368/163278727324937389095822405034435687776345799835442022795533783889356344755225815267819508608559076191292900367982490827396869405536484846115955581043091229202052407483776587687967125885665493681707461345895999542381476164157058393971431375',
+    '9263815657177858787273494705338516861045771674838057329170239610953039987023429736752079544014780707408666628475997291124805562998227296677616204140605356257712022384368492575381355563976330347792504605666631512343447560301417325154003481040250148561839861837778597346623630046623751094400/2843321709948499955095590862256744532227698001408929142548057792217790532624003190447363578048562448168721539177458065482170148482375585867230123873178100117094533143052886527452665480614620123764036974180917207421482431983407742154634391264619615289225747664532332469783301704643254076601',
+    '407959339726114455622180187758753007349209016396248763075759257357925636039752474207685682218422721827857994768023399625060206708378433960993946156803948655098667156937949174400873748557248801874735834957795040139401560494087476967548060208243867/123780218751812156744401121690996305978134694678934447237402511116731459214498784497436358160964198336874043702652746834763131444030185151143987331404604087778514863973633941401826334750268416015224906056576641018962863645043976537664227639296000',
+    '2547676391598917379516698439971914695230548782904479778605691338364453606537643088857116141939170899135026552016969320061900926954008522781162186995856580955090548471448276736878300717869625651893741316530109438876067419826217901657017506157997588944233677467357220316084583383623602865379325184/764562034757392298786420374672266498815021229519853724850874576419885380830752931701831256959159800764672605004880389358601658343203513177084389490286723240185146570925957286083025676875197029662038213216541352875570101363668917766225709569356861275434470568767077844675593176178611021135573625',
+    '186545352286463730559933346565311535598243666022232037054735807289501173444103692309735768703898330430135399033529355360391658728987379385732098960609744313878477967971557204207043802935782878745271859468248704012618254203101767841517569443555143252/55399179641621656233589820996143825959365789093262978988289445625153099592463372579496245442338653053662134699646413817866770218574795378644415019944304868289119443774932782235638737888469746745621382139263856603239588594078668393194675445556640625',
+    '664884440164786473344854955309049113269357314957985265728106924238588705533437169796551912202931185746193155801905841712503407258166135075966280435780812714252670362202091663287095423712596462690753468682634261029392794173636943978404002804413009590005984736612421172979101972556772005594499779860608/195485517776407145286424460448995460754674039560651791192647586550615878988380153730602665795647187884543361218962125172808792176382956599256188706636727418572541254480798303566840010217729386905041217793614214518363859058348249961790104618910877813067510758225302884815410347238200133693756493703875',
+    '2614957283934314904315471338485451166053664494383241929385424599389309215073267052860464009981063483440201193771607520572077231889699858482582363845275452280606276949653970992719332472370351170732899676316967244504534154616036371979031399425846100527685/761493664432749089312665480773496290658029971027686543404885407644062485746072719559288231362060149626237939029641098328278650939665665969011529293869562636656650999759724704272743235210867676873525147820749560155294022488994426729939894753293900972032',
+    '124843380518493746761140367283007507854364503961156704095198010255465940085534099747297600085903814014415830785663764373057896014399822131175202342399536439284123918855893825207202244831315575594886675813256448846863723093240955901916229136393454605455444105444987028391748121054399538064686074523506176/36022228212051654395480210378626648518430280334458144892889271272122662467638331091863215146548048144675657239846337165813938424387499358852301016926312083940212100001220180762189978024821166744964908871443681332664798940660421469519997746775275873085770018269706847741064037876137315001228315806659875',
+    '827992369063043155578730871896750570951766628472810506926098505028264552046829097082095665194000002802661600196840639204300804225352337632259980703832713031790922485730615305441309917696044954289187837653933158950774246017223571461858939407386087081525130831392/236805932823686534991153393869288530368011574665859226704279685567723830696754821658770176385138917722808377962346690757191122309876922069867472518117628639913077442806147910884267694879089753138429767401700283014143248445966474839193628309668702223994071394625',
+    '17347276886878323736540051321582548724378497839789943634071026331001588645519865992773157565595886250230140452154269197770615097377486013097979087647774513500701793885978192218455687078883766086309728287172567466406449372659680040183273634701092561727514713494914793425407149186041796935055187281744386432/4919325621804683623339606849970832094714371903709195539440424738973575902329797546592497378000858196173718145883783709223158260700365224756081275272021856393735663399552166737690038832550853145831185979094979556715294990257315369124065787473707136464772247917156232366320267601622617803514003753662109375',
+    '137984231830526866236186357461458917020538108058615632801298091031540729111527734872044790487396302545910108285921421417358113055522725197998483383380192391312304647004240060970929072498293210057120617332323445379424867965764749534125081131327565507524502163460761/38810445792642817561168950890315210470940006613819790543653745327778579787694809782601777514116858514049585074667085399925278459138508514838268321349069481334967221455722811414399738756151414906092225265355449011152267068726417045644222323488445626292574879744000',
+    '746567120547823334914136339633766098626636643449144032626270358619125402826113269699709721071135471625588981126637674402048519990010499180844665151971356149292818375448504122545400227696621572263621729512461528550588108384619064912224884465737417596190735966915167530332762203074440688676123756162572829692160/208334337057923929636884170505570363171441147899816815785150954417598643614152856767186132467069365605496210036171429712485182162940460120834349006784956522600679357307849981862006710239311750261522832996877712350330290831638640913932265004107623954913155144975252743257846945609734368518424172846119306643431',
+    '64649371728330695076928013661001819989330953381731372450140483779536126948957993261299287753791770622512248630224724990234903928056275080682537641377393210728546364176267034339221558641084730052304770498929958838997239635790469536857863963589118888238069738647239076/17903951498200212327802847425913723358452100686246224008745414214690047078122925247086521362329833307849817944645647750649290248110509395628305970523384831671737569872597295947593410067364379687588919135621621162007748635920864926867870502568935739725312687094047375',
+    '2454918942158003099688922026016393688092399295166304634317616773083386087532869193458590448918958337530406410803840837646465522656670050113548208618655070231274778592766244282964463702354872753657766121825196898916725498553882689210280080206627916046484942827487726300822318764058084323314109595329304407466188383616/674880185931325925966586583820010578979699141814417326552629206140252348822939845006845669570885271576698771404162512001549922909048916000017837898649100825976232784446638776021483802989797501705685620612986771521390439936066527738682396560462899753657942715306792783283782238662155922082005591512296007820682995125',
+    '74018558041066162916454010680594042518462756234254788158141115244349044958441521749277686851928706433556285971088455226217644009628399441967508838553345152310730562224910795446341601049647392069373970101491741830623078126344928804029524181578945586663110848142571149861/20204153620006780689923328634586091101021423979622170579036140596085566172775051595588438592742563923428900864000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000',
+    '2127032036758045513335690185608563023954009095206088224487365541995326714285119384743928987635752931664240752323937321097955456543854943206092931247498833001499955456190701695430459583885125382086777607021670447795321669948733328973350279846928613949120929250312666393359442423066212311060931469017737106028339882830848/576612418511902928757340062840968526862381326698309578771238715462180282212422302261044980131594522407066369222998903808960617461164985318633518680304995784614308979881735537678182134128319596636920719106506829571072447362052319438091347699720147003209417806230149598345068078717948025207635448205253184540936478445125',
+    '35289653975561083576641954928762116897061274899517309102784750384002335187117263273488751066569234386120759866204372398611196356888479036949053282301027789530999737306501029700128744408015642211359442183943916106790666114870974212159410284751571905275610921784716184508440/9503006066880728386808143045924119024212377150217533250562188228062174064693375135306438120385877320162710918716613546077156389583384656340709638430674364232343609717735574035535102953482366914421205216675248471695111720986346092738728929878538430662191272737183832556131',
+    '102633551023964794485575491065909467125458972250222581133681080524371507544152979467328048718122409841060527545925136196267751819689935599599321090571687632103850847605493223603751038996548520557330016046032671961857623066292962260173840972332108111505971231021442896036760967107060309991355545554631003681544611731245475968/27459658121882266328752886605529964804078316737648012166874496015808620265471203512606463219297059547428855195782384236337998738233668399173746663289852416697917397644234441300570212555870401420579737973722145663287124151049692290432756231390864184491891697469874600345958989433125942336757049639797225309327019275689074625',
+    '10034393558388390065766795008210457368713365491566387292163814915435906649268119060550511145023450790393353937124495488860451123302412204483570913557762460385297770427946219119911920640306914453207097103853766023934602534502476962159682750262143380527529536498215384467975023/2667919902603322771586358077760621955455470781865624844984169443739075976572061827709528710108877015489050369589117491611045518221354793418884447632063538994046714401229510497599783726376490260140723032102883617341970952663947646017489439179953454964374887388652792446976000',
+    '248528145263843375390386172800048509380966183384567983242213959113927668429802237067505890436957693495616107089384741585283620097982859345081736730899912519273262934785992235852866637878831878448348444611412764161078458068549719800733237024285525816723480868704742804077255242682077291713092790250511567621735004237450946304/65676865669148624809340872151906045781446981664561196686217551358486802274698228825404698950974939545099727242259547145392352658637333562345477931951890984276718673618736565926663528625796412420753961231404680876558659735251469326707567479071881966875336951133475135427640218972722939427821842173216282390058040618896484375',
+    '7805448718805635696495809414501206964843262114470109146341305656318015059743127114324245035489577134938579856003956861881125856595981500593426840968087618241785931128978516340812066502964561231235073012672356530509663384739132686548934288703179479011016719045530855033205271548/2050525178024039744126592505352202216905491833360272553169520915020715464206141942151086176509423406413311520838568324134077402841030113427309725873344806030836314500267104070131451720947531994814710189000076651895520222646974590481497382830325485174899169093049299764813276375',
+    '3355325071293197839434119105039673324264765809771192815982246040415580387729382404624613875653005261578877047405365032178619450963731719777167015959920645055600439987161800547901539269321100559393048973255388860193948274255340335876890491746900991668165565729269698196233805991206691196045182214641935483083662356666996922240/876402579119117579582569839757462461050855174353108858954282915644790659429341853404829661899850841645529640454766173209897510988090318303454542547519850473808789222552969933222203420847859171250332350076509996295844203965564448154484566493395403967626596213792922784509892086361572955175655987334882030766001799867659814117',
+    '218871061991045868372866381545267589365410350294028138778572466235486397478028823720846191998825628156716190463263492304639890659254282445466806224943413446008645087186307985343574807361972238230520975439736199291019544576443791916302825193643774360055545186783819367378492631806297/56849560726416896431557940314760680962653658127458002233782028041537121216487790008085876994020812492987733987414743604239935223783349870516284048368761617736127892160849065895223288023531930411718807065209903593668117085505482007061969339237404945180379460053180570404846043136000',
+    '5008685108365226931582937964451700746853986170633433728409171904803795018146152804690759530990140552460596075588463394200510044617816085275660078502126507209302951286606953039953843685800941558212440519542602092919776366067720586295390886070120828199562643208637974347390938772070049344991272621102622931576339988103674070876518912/1293888539680354282541277646947380627241979967611883341823378331667976045287311988103163380651334828012840330710760757271860219584371109472132211215957402251594055009937397184768184517621978947384029376766290498101728971145633139541827544539988344772578184316843734267915665730981857376872622787627370859411909330227080697966353375',
+    '15388340113525711660227566446101909585796746979396093776960989868457211684028149502578116456785221720682202816140911944661051001675127262774824593420825587319436537346311831003212424497488485098543512314062112948777572038731823948224734505930748371522309451168088057190162878224801232/3954220582960831691377435160890656173654063611768428458807273708040518769541211737927975894584024448193835165167801976423275767590502552964407494549049777006346189436817215329891530811451811864579644894987864267389290848598289794977382504890216219362031324635609053075313568115234375',
+    '5099039333987561374222193551155323470675617979816941646196895589439391685938046865391119484510329634015275893520725135141878751153360264368353595348921951280561029028912953500944814771064409611917475818956659775131751121312316084465321917769679881052144364834485866477379437705913911371481828140817759401117780199246301705600020671104/1303503600297679371136943454060319958680553228879031326679449263682048703103464872914972900105569835004878963701599765030590097739639045890060548760692125546754294514068052902543220382104483822438283040090444827980927544440984823535260277595466339403795403200720622852069244768910603820007632395190204569927612348189089161551951106625',
+    '4902837141334073026145827027361937996261324349722726869116185158777439337041263482852376194988371853413467559557923410949898048139830183335197992754748294810838187068126867611615800383834975563313220497573778480109264178673389149671194149749735833378557143135481387904961537942569904075/1247045310545991266291285730016853118981099516935251861146038369985109288084420528171217942065832292739130145780833406014673689119563698528225048800794718789218267628507713621235056538202070171596177775095071513194885568843375526804796016261173388452184505503341132236719484809714335744',
+    '38114743522716832107917466438257616720476488812538316101658139632867788464381862291240727309611460187159930652186486096300862388591521625093237019662273764387591494074792574929490381910446287947994150655077877204446864004067956087975012773988833339521775463977233068498404144221045837190392670308437391686081418318624745039402145439223552/9647001083383999453668111809775451078976046488746916070976218645431946648087171586252172936600115032316383427265217993193444199863138429602138841976586190525451324093772097241349417938578878934577091671046050326087898259692917931230974174799815198493279413438192301437068820185757869608523761456160341754512329264442115351926967120404125',
+    '573695055225225727008803730767518906490704995929177617646275646884555707960986625481944101622708415415988844740028718027554452662358957933526173824325955904005404113684003841990198157072540659184995738719040024647370869010473254071681533880576462368600901824622431045529064651675640055917092/144509482511118816399089096021290587489594541280398871255876563615464628718527634679330291741479135415168539765887291789615790513527330600394937614433502341116068305347468133950204152174094704092402978083370792135432486240914953928188835819767755172666693219213868545854371103120604946200875',
+    '23876960329653589647925126180903391687666378233201794403339630995420215267415575142266707357255726330536094448314199602616026935251126469221925945960901748679919435908556550271504767784553484434363646489174587463466333864577705745452492395785557425904735048180164697040313528831173448025400634629163795223739061661461986923675833880378496/5986312408594306954013526197465608559068621248896320652512228238115589875514604632230098997609482248000888567135685167138762172475788060284232459813998201719590208742091697294562538265829954186149162974972471533202880368317237508987477069872431064075005305838801862900501819963793062041081601844759452202282545840716920793056488037109375',
+    '58168289917567723171226992383559866214094157894992327555495441698028867727845766488121900626912848698952863438654895252811583144479300382761129433911280049009362667380001406579175563745824368613319103673817094498117944856004415812877213722455299491145649879676787079744410765053845551958756701/14517067289347903655500020160671113450349743650636953726251191692074385521975132268313263723831804150872238173602847065423463131917373356798750100313145228608894881457107689499956903046984443545789053438946050974567665049237414588435796381674590098629779384355275820782532479708807512981504000',
+    '728621890568281859295409481422447012528302594365693410763821707074444799793690738137592101239862736313347273167450056625929591960610208335290882047413011571781161008296084630072829079783328937418641417642857196346026366370059522990813537731394823630207433267854616768658990289454635793326766697884798538576055949457122067828153655416688640/181030730759516991863708593747964787874073354051675597050399087612142539517308720603687322924426591889179726492403913356461908748733972707460063017057809060190437917851790767968877215795679844983288935075688219234885360839984681619084834228226744165610073685719017596630302462070188937998558312507638434329299017584329479516410907786681093',
+    '86855946923438322218622470067224691860808273886184997065663554841573982963995340977083049132518812923329423480393306918856650577072525633920456721265953575424233701929892019410099166322511413146891121248381648145391642571638857576890568882512129960291171866772665863159474602604647289052079991768/21485753507365901947528588896402264670781310878547726104482740647554738151100954835784115119035980523529677083504495839730499664052882400915208251594384038810917282207449860876251558307288700200910747338758723324686939379138206117634546981163355060740270734146780942696291669461182599512320099625',
+    '2158989152301022938148680102142188531448821359505188055264665167313418619665693092337665573150374231484840948447637297247277576415460889296724813940128955070240137590073233263168835678714131062764247434144994737610229909964847568491446606012581370840699582055341626266533733744293929658949697805855362114229666626620766245630122333733703618176/531794915405164005613733454597931482878479882704956110685223892325074211694837836221759995948610212818642789132749082430059593652854659130217225506942675608692701447738732031302987802196501895840510235161825501235133794449421919927396142470196961877376701957829921152848178076410141813926924749057304222282687697297216661687583257901415465125',
+    '139432548574396829074586704387656697097760057897628994548358619815052936481650396157428747411173567801047221928593253479330480454469358220685854351236980383914223693722868233819483137401339800304943891968050399345430243790898955416907228948287367356990263740207046902209563417267686591994743547621/34201151688775214071963206765436083445901621442002061707492082843232231754829227303539041286301398668437202547003300396162741375435703188500000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000',
+    '11008517174872833286150985180322584448162884832099344969609291070844193524816852920942383850580217443209402836100467940651581092350600329145627967515818684442171571156446321228596914355704205623857871497315955269266498229823278800717909321269179839084452384509142712677235552103459737790674103994445173074670347080506698168482564009465276165824768/2689223396936080856855299215659204161946704205931885125148201643087176556822542895325191478283706585400237901215485150928036895428721912118467760766508162631903585126377676412573187912443878232521444786090510891599171741773242011017926658231638022943018461086517502584854390836347781674626615709751386455292026775663545470794167629144456268750125',
+    '16108638074211260588800537540680707641986073914251424878121255234668558067988171568946079848860335948991834525552515669040163026131919804987340113244760738846884911038097907756220945883750502673899084880578229601870882631165510396775126850307838505063922101682333806284668762825609556049426829531780/3919363961344261777100658318137884299575193089462944554282218278496298610828757650104922583359642384253066896538203596057302203635134833545580869871333892935330950583664400555463557735723364497947986885146043017010159347046389604172186788902608216894094289769850517098027486468084407618748895626853',
+    '34420755849180279597302103726180110022640946692592540634353734157479505420320000324260530767186132260970572450489530034440214259559325114511265075416512316229177952140217732655405289808326341696986755141965043719344169685611217958619102774617224847284122901023774956887687026904767714958090256282893003000752947427857703259704682455375442735857024/8342030311716679826889917494957593165464748884572298173556257652389845294530325764837124998293398445804458613956489096007564811101361266196542129764287084823604897187311540561857741285793447174119667215803837719660675298308873496219385226998078648428368061868944322478384684509466965129972030932418920415308276430355882329457342937549162000252625',
+    '345888075261020004071220843714060353763382280664960929903544964118831237876694384053904571498830068831026644303797377762345709976595360421502594656308937649239978525713471393570536680412814805076323426256584504251728507416368609420882442293831684681071553766603478479006495757222912500012444787804577811/83505703731469734628961395063481893801938371516752417759131774530720075262459158384433785006689548434701904106312038822969658455364219435022841597243178757423598248565463985786213156556523685666430799283870548238467817226915680747412191245046634279766450629886904716776719219698922088211154187845632000',
+    '1061717830619177527082296723099890392273896386613997004874669053445943252046748251883532634529759169500795452576392700472771365240996842610207274128102329096619028487369622001737128463631016494371635687841733644339636164570819431573829173533941056258744442930643735587780907310433371453992062647737259587563398111688659657406089003293576961475848704/255359631537215747979895955806995352799574790340218399351168178555478073997876110889483456972687438702262017800167048243754141722496276537685853311434069991222324039005160057724073156957530106623908696241268268096879569794431919729620178375212905203484165745866913773304319069321426245521467122472046370356725530914587807274074293673038482666015625',
+    '274122944106300296738399632684955400761495830361663966466225652918683099779465438024846903286816813856490888796372134557295699980528187779624865098445756013563535339056233912394908544185885547842235097677765325396255649207317018754967666450708249125316192200151505568416495274671679500594656671785202496/65687592621976546250581560102201535533608158256953087745856906437400149205693427285162333502528793675585022025602144243543064185647792948495372442630333800126269123531636800213405254045262127593759539706750242430153456891792533267948231185296091297979933562727112487057234422009426868531651634706262125',
+    '7842680480716516803148821198697967237136721860017131244266974996267074742248599085253569637183007740566941125452215834642683053334607896723447140851344501084122965014242091312411884985569341166545074688756440728922408743841592658677792796881188604773469108807869960161395759837407978596679911066586626885830991556090978327508459276025943279064965688960/1872528612245648675720382138045071131304652050696842872529163720558126655075937845539792108048310219395746259570506175902206215101518698490144716531697689534559827422735649881381597761684154409796315455445459537515308174919488497154409643876490472215352056502193150125644288086294418253309947229151074464928874881827227706992859640236086417889990541889',
+    '5300824422251242070074569186825929119848111723012841627275830216301188228660779008353049603527567784119877706984722171178137272986345560485784907345500893648715341273841147320288851034078863843374665850852481747000237834238703248634174397792745914847774297223176674917912406659831206869442510948965571661/1261140476013707338477604677428573831791396352814802149994640617701773078174882455512668089072441176857892331468691160991310474734143842336092636848492066592397892638052212250229129355009939118431643425836944282456647571558383755315238500832868535816144280088644939696339160092963629012001958205063168000',
+    '2220223718762215584659309059880106334425515875615107369399767892051551634000614327272260081056973863669004224981561870246078120862256383581012183852291444462730018546753183156982897386563561418424093883164027305254176874653780425452987066512563140531367766900610414277825262239199580925879453806414860409441845631158680721091621460775043562065815179617536/526383206607841251253861841374779803798480623722760367843070466043030228662340154304405180907941079883976168609082254331465595267209149963786388600028701073430773581228212441424400748220833542964971495005714483235359479470452593264280645360131482713147116366500300066771223383007216182988263355451923333319170174755334598973202740108032097242475554128875',
+    '521891797109626296684891455959263713257353500867652268541535940159815152120871142196535233326890353914761242025931373491906127275561002910157909306979093246574207104081108188995072105948138299097848175016082947174156278439986705241571619793059501724269644447572323501261424770743329858038040685313621446524/123310256826873923765604825413207481739886340225713108649758575106598510022338480189649787216845041382860899099250547657534972156328080736149239332330143771138115695598493059325064119176038137294863053148618656356436332991079150723235214278848602671333076219529535123842212129829931654967367649078369140625',
+    '66086044538329677372986118727999622900471937619891337714357792768200341519193500393739322894033303245376225584865369486696276607060432449792893028061817203932068085863800494054274423512956136695211796751845295921015953538329385253280866669403169919614982155350899648626481405781514434761541281229159396787287553493046927448595964103589100429722948913403008/15562137339474350565671240515273666798063901504051979980452491653975250630723677279081058884163396938548780856293034775459223871281049026140999055923743471466471830572672766633086347312178711643724485955576579988182546105048041649947277672869613992334541438784737993706482731696809943027528882927942967419447250586964258807454003775693567366165507144866375',
+    '335377615394100148751647837967017467711612297170079949298328061159559939969228226474615711044891085626519877634842694983669611974807129333052471799687426665556738316626171408219730853872410792831871526174987402129691897433888027072807302411474690613948951673562473758814664346259109886876538510453475290967835/78715592752271462306588358880337347638000605031000575876214116610339827495261512281635361568951675037834544811575026718101166562072917855004822606752296233435017284127594847656529606648345533195437635894948829857913798336356647286032372695130461573940500785137424365840081503133157308796505622439791698116608',
+    '20090879701618729602554170716780970848925039917987945471322994867171660307998603515745066411687983450400412739285577269751603921163835619296822801840348319742203974023505186187060251544248644338412667631232247108675504629538319425769464277309915502144443973397371136256151336255138506001292355330875114245901820438821732843540725116728866301271466614762497024/4700223519410528857298732096729483544820841497820611795617923063440946097326817340637303431283005509904481323205480729806879570430868897342398783028649633951362398196137429076844504529051072393709154483678349272930361110568616112723747726853614661953537957117231900032044221535502745676310313569997665352252492568100075191900969170979460298189170486601502625',
+    '2904778979985524171206573028445379872240558084236464200857594814631031581387804621371822074061289363372523364167184697785570324832815972970658633551879143187709707164796663015180877412717910872234647704536817108676736661804878068078543241390828229923424191204586313620612539678930999769543756218765870513049986792/677408099044823641581658869221044375312077929976719183424865834811543737800956896926637625166844372424044003929341361734886232742770909683021563822987505236295727478159938135467975522336774471915167606673489722102077041330652185811196423400701795791669780695158730756241178262962515917389382302757366325768069625',
+    '31270155809329751863885224732454397292230969002004953832354065319735530624996254695453061851449600345977646455072512400760539747054003851289540339425848681804190284451253462663731135337775088379954403740058084949675460445909826322297817535400604180338201322667139062500269285493417563095365899631360901732684124930296643108551710704785906431324876072470231424/7269578038000504017073007978844992319987411732848567116655821196644382777088703228960020894756722675887473977480537577509061256138261063926845643360849217556370868752909531088361229374467207196928745673402380473721018157327193509586295879051411183657185176812738231456253321187419224704301236205478184115996135940848503487199394612616742961108684539794921875',
+    '432538822079707760382094121020421735679118830363764570640789368235407853152380328891350816400541189148550353337874309885334920995713154225799660601389784410911658967499100610376065640785585342035058364676314084595283850213942576431310823836792440218271879354669291052589804956435743500204185107215929849054782893113/100246574739326291035824954677502591279343311051719151327066341370995390423713403739043396503785261917771859220535505691760472395306543276314938287868734009582906895763073519374099272340577921671298878837400921045252035507925021904954445172372479744465666760762909731237634082051855588025732494461939980856983552000',
+    '1914333673689206389116942789116917579088664511118582610293383428712902211612554212779880638065888518488492298586641997844141510832940409501694726851666478650414191249534733087933879981733561565249818572204932715347752949087800778646065986244657260832234479202201129845117955957242616947361016603702640821256099895469088229339240402478576285854783063612307200/442333801076281757298117784528962837115323835962460661601905131618341609371649347131724700192551574625400701487125984359494804828935577124602622582550626336986871791407398609915208114339012374456785705161994343348351473385887949809051796407340988735853810174589261300681030826392672282630613354461927208579993042968520650313621522933214063366558703422757071',
+    '5704691626402072213006354545292364761246893919997531024861408248746241619817955824682639582830486790618668221530365426203424888737658778881721063941495350237419723855000515747725926735319471480027293210991869255971365522170749568996651406002311020883635577590045650037569906001924971041810547543163363976464780729932/1314240362076792592671773873754757443276256223533339004339330559325754574023619698171225777585408160438834449576526997055649849875516310105297894855264038450585295422343454458568152668980131977005808840141079502436391909349182185596381509091427752151860204235071122788499996991078935216686010888734471173124487937875',
+    '451219364084386208718456142329444023337343409261545444643031014769484085278440612677813682892926852469540118625738238137458321112005189595703619178533263706811689687213128887082197330137502064260105387500552856851972124172206996205919556553246133218441769325133725631665593372188755090094541462474970525820334385058333756591222492801647110594428922046641945259392/103648250172203340865458115839764297558925693061798169434516001775068769911768281084188883278842546791612199025413573394167639925287068809631958006622842716869211374513136766899877504136929177267362862319748507372147243911522667591375015611312165464514308451222180272935398828092646393830572838772085366567154646425598388620105539619174837489536378605144891769625',
+    '1591076564577634575701791393842535460875733974464805197283632670013516183281542903377750304419996681222758401497321278555686661981435637461350320471258386388843198706277657208526372100698700615835733712519332548607115875288787602084336341594576426630670911478276101702119972195558314357975365863803265163991961173/364433108410193393847203348728981296285742202617988970384277162225847256295865554341611171460436362780497179090329831885853324392923449579538286804729856000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000',
+    '5307507148709435807261229345132535134848030343901300324623409721698217134932866488937772986386501538026693956121121771089740095760486362654754229904633476234090792827930433598453200536551779789049352131005222659995845427680686677324963223653949277037520711609343166926627713758990384011274777087968799793410506391884872456420041494598273703914689586637995139580794368/1212244290381524115082005575105703496583315188540177702780216570265159923654197746388568151706509243638302707511238539845531608313334248788980188054651601536067740709977678222790481348003085034888244447626347312094586389347535167903408519024105748523384932942441843118813819781347089702286481924493361058339731949844620763272778960860509570622673048786968660129650125',
+    '3046577098843580578619955603029073328361298436129791931304665777036152915858575282362742008136721284817899542475666054101957899858138590963789072530710775790433466847100422875906866496318804986202089528198912098377828580031852152997907433335176267300286466072997014023120087988748396598176622765923059385876855303940720/693911859077752895978833241338902695755528613729508555938416419864772385336777924892434701804078893822446727762077537955240605927550548046309823051841326355655848406646248595628715185413852040295631448250459076043816328082561248420925930431777502622148019371383254316636979070731427737764160145097293260938978413488819',
+    '155310926743873343426312607182060072939030765297630534544899230213054714482456829198485999782086944271490229412707980997432528653509945301574794242118433868747172279224118534460563400440670015323324602117517342201640016853927923976796453577048995243600967202462466681112804476672348206491490513419336308575442086511015101999847896197546900512764233674747415844045184/35278162986589659300679088538176070649273991865663041616088851923111175518405117391134863644540911005782372831496121355954470744169969774540892096320978686548284501139861783292226140413769665461494668479430833892857992401391262903582109993838728281915806394747833588629467613308837196269424421131934859079260185052081536487462257809987002198437182539441349474426375',
+    '46729591025621874782758519074451728476386657576036360734358719976400940301493939192083339293779149127132651616972817165172116269307276487158069293114860391502484125554406945249728802484128756924044633825692779783425628292959170597009935305429239702926898931975023598456207165067568919757902764815108751735753431153581849/10585868084079030838651390738371141142245086465033459640458366146849314274285871375459898014414833295804139979016362796357043372316321872357817727821559232353993714062535883074661734509440994664726425399880995133711038483607773782532430879196405793694658185175583691180757783597895616920432527125993118171361116684288000',
+    '3736372348124144720852190769710129461145889011598636925228657393934132828633132357232883470466940330848177425542748100211498184494252714616379450272611850068867816250209867530921278645286769418080018709947826876461419654782341309127709703626401211996255743831998918894661053669189312375159058718767082163156988766821194002596331826150321864927832618126580509732359424/844176851007504003627016945212023239308348428094023437269532743221937069345682900884618378849283002998220513307273333096775669556093005637615012750733216460458689282791632437851364674879152695438589692227159038555722539345972833161146108367329370564838671791241431125003401861887477969954740544092178721327724620450947646209914621007186497081420384347438812255859375',
+    '1258774755828991281578968023382624723772927642002016270484090409043454336040857926581316994594109169123354553321469500848146015719851609220423736153365139804086413284787598253618361769125996755159571523632747129480387254164008968993734442164892486441152227433281625391753702577143985047832519062595123255569172968685060844/283659859661671181526547833415653453506477950678651675193210969173130116121017723360258249430884213011988678011357458727603413521688184521573094783291496368005697481333739504490647194454695504722542014845706216224432373442438242355188461951883454664693262684873988061018976711201351382163527093784753398257094429403691625',
+    '84932063355292829988908961192574710493098897148701473172754949846455626381329456661808566365329266898990829247446356970454502007127269708487563279536825277374133681167235811080298134899629580318813382668399644553111080625918213250223197440426147821225593304993621451053135332451997633132772608233430131400186571793929377129211228689703376067763625568623535588709576320/19089978133324852910950469658566458037096027722326716800113107848115231563787455584278193954518442601810776347091253561956877155673550458955562102935555510392954425196165785410319126098393353878286400877305164869548380670204577544630353859009177051698096136470072137579698473017257397949994722015089768745013713383769765609613514021200888647472804720456494757423299627',
+    '802638881530832431828249604040579750916118423833791608589560402449036920165704012070349537114920882938466635598602387718300074733476150548724726460209016834416094317724261857969955414000155807312852092720310159572547644569797512233899495300028159721348599816083166712365215075728968005941610056018023633235372936903015771583/179951197386119079732438617407921535065140503043429174394605652913879982486051627760652197484142547447000508189455126493868229565647284332735552462525598465192073558793335913005459266977086104359621022691931002488052727597513413492393525660272900161375677499228252863529934576881596384036401784035248649026076581302370304000',
+    '25208742399375362881099811032135575360109715964024747212026245529087599633280142314962581193303683759605084995818253124445773115574470717199218828756449187055537877478033129862600982068782249943150019637186466260707552416433010545437321814115233841687700051830170191107127799355485920046505591193770164750886037885397478191534797655616745528343172318318678405576430544896/5637713398995569614196397857525646325234056219513202928587580534159596897880731043336790273040813044077153051260989730372846713618900145444802234629922717464041261370803598799826604841654608724727320798324006129524610666235998113655193642594744544226880944882342204407750193512869672849910003246504052298022468012594459974209940607450480609190841893267203392880360823875',
+    '732332637178584560220688900268566130246820235956768724845747830959547501950765063982943061181526237061809052444110437930100210105274824607344902764184151030827266142225894655531497849161692760385938786736436977268616029862577293984376170905024712952813234949508186306774257035535805330366742322777611063402455261391021097128/163377988152179636922409938163005948596822656868040662831003991395905185823836089887990201522673196719628247106777881741843786365701135555917263285268753605514812568258179391272113405920369587922702002732667842511108732068683407168090725712401954314293193572654347237716691784386690948494003094992876867763698101043701171875',
+    '22391374854299462107923583267570593886002658786775211597896252879708753450794332301142909715845151746786016535157797023153041007263258732991465037648536702217273897876864351559736449481285518249846264600935543582562018874574655740660800634883403597181876364712521253650744197321080655028374602772696770227741610874399454362583980465225235674816785988164157351243193265401728/4983275997188967758382167867656806681565521964135131710678447245984239733544941881418957694612909599261843693554043504855207170782744518943778096768859337980470219783210191765678908537645270392500777411134164918803949155037581108678968701961544979451081836872425574123683574475790997414488641534074873248802450108535183413970057319247033379016606809226547805269752123746375',
+    '21252344995592269775107236774689012760750850598567799560343535731251766780016530978663079386453430272698006977668063208502408607227544322945446079708000304179073184745623740466334127867464883762843346619853325415963062181018736410570113171387891518398846617197097407612852037905687899800932343156979591740310928257766081697645/4718515378484509142377558412184183991357747235892194234377932213613746008373918923611236346330189287336493794499188640380975364991548794341177060325509698404571318037298112293926833877103554546466055612498927474225619680186119313129604319096374924037870803554153458104225741255753314955115645304948564151765516183663634546688',
+    '76703085666560609319365659209445044957052359500745196718269665234646121134787807938680787341023203786904308047981099228357850016602203539979246579454229078497537148799349353250167621935384193502383187510928609818011142642954550330069991570983902841318203794113898871515702445720207144863877026526025447819537396146822201819435536435835285098485990260677737192230985069824/16989892821104122916312992616665764943723222199277412857053896319814438961475117951904867066845412639236790762432996309717924037467863024360211163971190006272168845197000304328480920483651558672879967350982199768255256753110375467976246339260326422891913564151453729285191351273342206198366624882195135056176502724912849575385576554208844814085228696838834665339811027625',
+    '40153831166521391225489894857551838468150576827129133168972859441122728710165233865309084872159583787083130713473702296122465319276461991457173085312407612757280915853584698420083436946976844240063731333638150599017886937783470942007376523947840104246927580254612991191040951001539641947153847776050138532219595746056076776809812/8873576113581065493273519627544307418139908640325227196912114520903505426722086265723900326735989771660987609604507503750212298754414472822477243293638486047313990878348820573874809636084574108909615044524663712495422565151340037564156235745628465172219286664653343193162657374410926171153516871765680561505768223660055617934625',
+    '200736374277835272485186523480177159453030082779872429648449412313247640312479214261681899286513818912991945046297334418102711629933437654377760028476767482162031641781499114859553677502002901248124750539270235098492377826240319766677856465093871367817683734079621359186477306173620331748496369122143019303740252461824523590096923561928354020960176605008889579578495073283712/44259440065125442964014453739391594153290923937737136823355312892557975399035370631521014554980129291582486968083228888083079118116286312583989433037097397252506140224404130371160608365777625538148303917306340620019882928744151294738308047800711028158277500317134537540772759486365658024194428796846364486187105343266398338060080611643110071184992193593643605709075927734375',
+    '488459049325494693259159444507437983381645757291858092983371672334043029615965882574409808932509285079401239403272414271652617474184321852388397021836909585659327974611273820676128650810907598106588433939541654215766888212287311943387232664417347883717203611092450971786083806323404432266197250919880225892099146722474124874256523/107455855466267410923480140898552598306699341366032095904938430084768624148829473848763761936703072253841751966906142283640379336131083461646777390874405323298406517250231389493084918065243079819898691146841695350589992171102939046740169198394671965069747042621265775948217054519855346617967866614303497258042439052681849864192000',
+    '8430374068596413768975326329313648683222744787097373111477788794692418932056901235174019333392845529821722488982447683794746518712070635063397464904489452840169472596494433874412726723065560358405027764015273789053064830528919091414531400589850704395887250756457780868677553068164933299802944396576360356896758745030955907258232257358056931680617626606715393885378887657377280/1850480358582748412767893656294669486390769246349349706200869446679212812264400876685764762211659843658541567071977813110496329850010288179702589501255367345935389050373021591572539260951149696801665953123597521983257852207280970577287472932167362521740230973547070264273850381896786822127290397075758780174083415923482640313277341120938690594859118603017930500014640169211239',
+    '58273615882491925540881784328370957720539797010816130007322211512070634295154626813477498607761260613353633996451797686143131893884559909668805628269959901044961666535533876340116728982072626875010391195372759135469446048394835148654407846242721443845351855234347422256700864370273092971804207514736924787577569355949601385705517152/12763172138328432984573837529764902730448666594097466544652830703737195804747682450548488162227364538052585484418491914323658215406327363805353540266048943197822838925184942600367647278764952826664684839953902567694832690683467635727409976388280092262839126779730168889686628630302519063225007497707028600865624133968363921915016625',
+    '2685278694947152969468407055109959900160804835228466214479525891103128459065131221736713585038130737099067040196326815283171857977706979990467444622550497565521863441617001097354649972660183038850033950647594754644997436624623325003537308428828750008305464174443210079059253920383411910911594636045517662796811001285984447671371943267419151220518726940455756833040934506414133888/586864816044968996825907488721678304211296267371936463272955488077666796280028127333229453342355554450708284934221461946991556216095213627059047984199091625921432110913096253352065238765240063555347292393427726758010888546427415276046238297382474708612526770055488323003698641493939145624184943388159614111350839555141202024991876221874110644879676602924730506882384603409121625',
+    '27700345710264347957758638741952394530538598225904772664391173119594616680996031886665218221392453628824570256438960349220263575741495285780845399187013582169907518462437966962923592601721119258663490655013419675469864809004562272799012227293974254329929404036071055528766397079743200179924243479370594973626764330855493789112414191/6040997839051213541001279276287478343874107660287651711609067205111574718442229224212245187655300348484144001298428958025052162253854676210451657425437588244676589965820312500000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000',
+    '80160269787574270953020489212619791839643207793889009503234879683494928396231769167082355143564723274576166638869430572977442020236299319718643894871736539109822173538266278165276591962215927816541165548699380129014840956895931285215480350321259054359835472320394089646764361802445068798451796525897304221529850133401768027469126408090577121172238802281079916258951770135485696/17444957084936455555074876721808241311651668802091828589911310253709517226715880722948267977560381436307152342821304525073651379763144281678911047276389231713860392794975453518107055568765009486973017341479560154792340888780337029481646046233967923289468141293699236815196553819373828280398488966665994092877680860643266914861837639322607761804740256920429518434353839878280875',
+    '2352816897072623416220002134476921108405735955266703519252095150412419264317091785317803024028565851487861978477208755211589867803009700996828082850796103789009194767813611798769297617674631277847910238088040257037678126316866517601388289837575778165301828137259948173292658462765645645868094197736382683775415645307647043205988394700/510973411316690313485681833991645423802776865720580280058344111363363091616558230182669376841383375321912553771027107891991318313938505988903735114191313214327683449514676737796942389784957658395806951715454372568523597420866050825822028817422805046287798054682484158011143949412956697442991898914560191411624040490122951328364833797',
+    '6403295584873165688372907494046202150046769667837790834896334486679541887567517050446119511695248926941383207478170182650927368177009669717288184903306689332746127845953193587519575304974203099873732502605739219028995266139383163062837007982999189114810534856227848568800302527760100163350814120545587074865568436789021082619398126713943637898657861949091545516403987546145915409024/1387794272010111535893205703999712437783041553900341112488641528986385462810448493444968969845352401058333929711237978223214186693177251566069419805757440174840170213159651962686467523533938145629444468366235554597245713128812532716616087753947246800626006504878203666972651384731498770435755225220796872155249202960801768048854869001310722927230237083418017482134588865244642658875',
+    '82993247683514419570466529457059660634483860665557779709153549045427987672829778520201315148149878525274005978368939092115193636113741972236218502664881450367443614971109677363668874484696543982239492409231870942414193419634675024621942196087473557914167832058111113476295926250739099284241826553737074679953551191767148712684157318697/17950946423927357725787689855263532224005643859095168852729513034456208872420513601894508438640531171097082516559962755244698695622824386001219435651555513795509616906355363573638916895074349491513539093024980575852693293474288638209680085037286354050958859425647536735341886663074581909148323105020337857959651624760873736590065664000',
+    '584891611376763781852144397260140844977346305541197362434227194779766612939978629636198589818106137319267243431810481928639442343946346034433828599323416877248326356345631611148749005937144684862502198147087702668524450709118588741606955966569427636630159793409544653944608958808602551061186799401212712216156799273254257486955348236914237644151956226336750212957092204830385441792/126256710861549838395499078249922986417488985866522660521294610280820007499099176190820291214370699289977888813109514854178180265382471972583921477022411657285850911270389720508719377071949050253246021921916685716353393018411683757067093259101151481189635423399611625009617884983318141463140199839724797412514785751549277231259345923662499623640886881048572831787168979644775390625',
+    '4743054867460856425399742072925732465660626340183690464743217147109403130730445842673866624947360862438925194786600531200056131309608642363389833474026007798643235346104937733349791667694862514383520689594596660275306247615314272223660862212527346572811422915223417783887717426641317921972456913534338708745549252254865615725859881457906376/1021840037832289788284691535543138164288462770384961802287250236519983887262771944174557087207566030730743075334169628971336358708580124427857321953981475719503574867471090607795242192417162791762511940848493176847925838738242153177311649130718886794249673254183907621449520415060660496225939963471753122766421338241291756892571824984344625',
+    '3784664074155769467702999785016514468281913375341134899878893061325465790589101335015569840325786070795267055386681356241209412947116340524588831510768864231937929260236754881829005065056310226407358204278658699999612596866156294195316867934035877283950841910726224355461522065773816109849107487214275801829843762482082803559183694631856772777313673086715631547326400170962722842240/813783281473223559981291694175087508812520505931454895884442580280342455516154674683217348039336712901850738745276445107477919518905155156380620466135946654952837573797479076027688866326359448543065305071605591497778941561986579230698384305536224430794233462949056326864499827444363206162251104552740175503996670997705423888382328014210171324376593090187604268086334796245552762333',
+    '1946671258536842642381655747294621776070051525209940130115769153666368932042152311477746728678182920842238801763565726086459485050237860693394471644023999467670017470376746609892484192072297938254898880354014176373253875722410186683852253828299669530022166361304397631667133689128358365296701757782382475692465977794960109690362462330857/417774039698408581013003883929127512062321623871486379101498968145670269174833505080260389860863417408848209525427705249526516766731427603641806256289098209429110794311660844125377702016056984573671024035213045475132134896835814746425864304907974005950155657789157496389234919107772201305672421982198984613601511388413381215220924416000',
+    '328578487723377153600821813410631465225159589727320893988991729306688663770697528156625398176929288628930496338036815910669579019719633124832162035588583242320537435554101841406566009219059042888412893352417446437227440076869427577701706713564212185163005644118869496869980925214424591264870388237079776503547314587137721633918824664946974571838634860386893404364499977966899068777016064/70382695927096628347637455030970644630942872888311870124896575877585692281624151103204551947776906510366521972834357031777287506848887607457751667816514216907338309372900327919413372044625602555866294337672022362865161523684052940126509854895320132828009551012639156341878073682501185688461720664801829933689280752623773789373199210473995823367199793407048741191786377993390965297664875',
+    '2617201476921368517857942326432090876874414269689140439151907982631768946799200744678055980827789859579196832718849393046147656447672531861353456343821196812881882023188898815179947651274131166835133965629115749368441605680383605331300030886676081418867305170012032824011912531673468215561506423833911621270680104083533917562622600478548/559567121085534865189976875600841717161617153776904752351231210970240323460800785728739412474960421770806162360125774000625035522428160837855944215417821324758873127567654244465281354654299068546570789547691342609793050301258532586131013585959164719533123826201937801986942606844854123769532447570107525081084531848318874835968017578125',
+    '636225736038986537559880265988431731529837451289737542395494683393492040808565905376235074534207026537145408562785279823547657299565440309510931336394031904920056464959974743525662459433889398003683078967642651812081450227654478095420306880762753401111260630654049389197602389949892636943971690212484981672010398108426002137114819838399222096538921225458913266993881960732663394566024064/135776510176793971074115131648637508758953050390591773574951317807919051619690313331192027871176160424663811116849856489187562728496099757910540362703888937768555824513740118941387831822900198029266206334350448626733139136083404404120210893986654422850183837974770675600952078956326317698998103770833069712616832650406225828969036781514645731022616236082175582937900731419575337473384125',
+    '266695771933124633677367149389643417608461366874310588884377151539325854547826373711099517873721616543570605935954334944030816383858485296542260152894035979141266909050267414072982042090341712035518685997484257326212454742816979806460287972757626105526907510197321350895873473656215941034605746494172316089636216915825022339855304925515685/56811706665210352283362623728191218698295056176625217939528332247537278605113496147630185544004654583441448319260578659922931798845493756189402805173037491645434052737405379674607517658118427614090338938517963215812444779184193933749520313676564187507594274551791388039139409235056119788261689087831209441779870873305232021728002651979776',
+    '3511806683161697708497547617957719390189982761002154386881580160856792742952365159764830433511949678304281539875366378131195670004345568047690216126001067194904446295336734931691743477531830892015690816210752795806120303198745685394015161323982229908105397857791180342330098240702332072396030780386362735967021055250450666535422528637737695053315137238368787607412459874094164027214953984/746741569878639983491390741637813989978804202898438708743258000150996080386381281307609038830698579358879333079215327202911977568726258527646560497079622703052765164031089590965199628534477381843079892123440214378949632707668935001371616475282883095939750704292616758568964786737752325652839226013335092148777788733453702438432279149298482004122593243860829060557386699231448957232420125',
+    '499543951252504651717279461487337168721376180441322735807713499521878076780205427598086756686061009718016175215146305489885835839673419698751530207404115002383180054704927695269921072232395828258826213554806570423602966743337801741633869267620843626804016742412046844770601322513184123515405692795346790813502805238635003099976693786012816/106032986203682550514602969462803214831559442358811656484036951559991322084935692953211120289352600484987931812632505499612220494311324679562152078883141464406938524087376964935494245010636163705698220308243980789514815579319533341929989455211613890905485981353837321342730307819060825125980500975023479813757024395422041501208805339176375',
+    '8540849722242122835873311629952985285477986765819584558233324868363310302206006773828897816547299245727087876984857726652932481899766510637403577175623031467933486823994040576431755172220921921877192006685053572215922347418846423419683723609799729359551828522978186208983138345476801247634773975311405913274552616037005854836040162212761099347518242200807692224460514057530656658616850816/1809694575992816440924165741094996511361288430727981159314194000585536832192004274726651828760263522962868944612215633976203596150606503849595633388319338251977160257979846235995654357082321611719654136058257458753105122024859078420173515189641515736029725847315662215778251370398007468665986604529263071981040014321717598469319370651877296085250322599247141397427185438573360443115234375',
+    '51377057693118720457387330519321684810411289582445982078756989158447711080414032055843805733348558621949380554029375294451303430269197115810494758213980833379214402452992657502777067850631862453239349835215260705131864911194740832694498014048717871429797341104227759557199716944554347879767916801106279081864127192611546757627038037780244683/10867246748205139797826516105458406878398263495890048637741584969144406206902037615496936724683237700974333817527350121106320991054058406504571347680049945745432432384570515723033245430148177097144850158758462968940525568041926860856763884474605057056550146491001458649244094211354153171337463406192651150028767064600270112838159624568832000',
+    '2517199821548153657910904242290029026229621935918771922146425373057248090467388430999870036608278542449661971830076845113363443421757278612394167195431759807908413353743377586893872930828548256216510734912027450642648789925142482023713336937258932978503310551945630359646777080073329099268529064437120002567758664419443374110941824106727218341795100740316967386497091058915066147604481280/531524260324016969370728057738851340792702640911631807178654723224511108065633433027773388768889035083734041289308124943130099580765995149865288328550751289346866467755881013217287175392101334538392596956248952430438673292186779006015549928829953979591250274457658042926539541422697852390294886719770034058916886894408015624288115732154492554105512936468520265305162358665491880968652687',
+    '937159485027553069020805864547349554547661693803137696125140510201297417077233061462068930061915381468160677500445535971818609631674361074587732183297775728389124966713749450996643183965795829780345678634356741756033039293590264355732129789110416943789326482663579413229893878422643110852069225671783017610387570935863228248464396615988222968188/197552642195166614912991396771845374656891854357948006221100585931656016640982345159207601253599235265353060942313995345483862152489766993442419668492877390774016299878943454249209917002169480134131612530977452579520564400124908392339263545911443803619972765231094502745980083350567509014991735227390978850048088378721210219321877150756855372125',
+    '5119346675109082499980756672052066247676641510661024682574296075659671743397880591881419516307292610449161239233139088348510165442597278965718490070997173182184417648987611422446347235659085092530927312760229499083348217392125906851921500360553758600756698605047197529446594606336369425040236883171542367713951500007600355406492128741649090501989105696576697387613035366083536629358444229248/1077349190930018641197987339365056771667455373500846354778079878558146320193384166982231171317419356136237346389570422332278447217033773023556383420372685049620361481309067017697564760747047839930422553324681218253359586320952896352245366885171583221864112040038531116947187482413031483309534553461613717398523132176948321964066833901215878133594739833910960843185464241840039390204295097875',
+    '58469980853327028628854378052548435225264353747414624620847476036979290604604988637469977624105451959796142112200285366199247075503424204044884512269071061176877225315884188402954249683107639777479663275428465242905314418930053869461904934138752031124685541853599407727446896558064252696813370692668496452783003306386669962110734257851955001481/12284458784412533668960387046583440199646081362484321078872891438551639470826582933879934711861551014258493898870867541349962531447724393381011633785884743889445407749238183080941846528000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000',
+    '632694338036291086868292541736272151517212971647680989553388199374739841494700315711180961760384484042159533979873604395621016272772109619401112452105475906812313548587880899058905419220497456502543971131226430775692744506047039958551641270070996110661036371362526401525207493270590999387476275629412687473362310414634469788979236895300129227082421021999210841384596617452097273320509348159488/132710874087075221626157136110978536488168313127228703025408943153782802265472978945703742152954881438058373686972477195846162708986982742647038014147903817539418442869011995988024638278746664230046452545499722614320284478925059527086663455478374200352375121910000967044366831275681254530914306064535110528723482282501358749001657608122307686482720926189537227492041135055090784059994118192625',
+    '2022422468242151190347511883185841333968390991430363660248687284021524206079162212007263606819387821055183192871951076644707111802972354128011904720586729919826758651167984507804867954970064938022106484656560866447290934136610214175516487579122328055482392137619829739560923807880779865442016530712986750711199998190148885726144596663143651240/423528009127070458603837580844559347493775236970253513203186912091140251858605354218715508091394564000304400858761758838182654201653901408046974668599216804223721114331085507752600832390227068985693006017591304916519540245902645712532993889000048992026931475754508248265356181099817467608863068068953756055334213584649136541490530849577363647',
+    '150542201776735231618709653388506394887777837646651406023174643241783006776019388336318216053045300417086929061565868663031237502137150880313922488106751241006715449458736723129545400133450376339156062387859536289694727635468338235084533552501618659272047966040838509286942825414463511175531001410923014751124732748099355476332649458471135027773371541572374016038525336728276024676601778299264/31475618522527975728853391146702172347290819252268551219771201846701658848754300838383637247014705706066050673096278897933831272607774110183488615338021656923433335661890414740664480671162459271850877855521869741833030384682549648626038314686632195158507693321505500888988183270276134667071014075017317692121986144742473139078857816578133693776368882596780120407949076129685719314703426278875',
+    '162664351499283182546788129866259011542529017155637405328005084357078048074065548349602626724265307733005745244338982661090506394708170105916177679714664432754153795772811389612700051002336719842784098507498286260450939289109123759859710248621544296008356071415713931721102824434964336667980062728653562369697289761913925215639430496443820411811651/33956433298509232632327667206936379248172049619640091186698677954874297837764235300806836350019565970550533206216590639008637925266405524715041372407701947953283803854153876844320283735281209725651036513491246238622967031979233265136280687995181375582844780830517825595245506321879177965868747546127685256867456319112247572348416655367667187712000',
+    '27228583713734183629451808889147559240254570554044667607519238865891084165382075658707675405673276615039702916443152954157535195508033284355580742761569101934821088036084480297442018005972653124248838420021440899368869744141698556359295489470499243402799483082406238128875386939389123336303631993557844565927960009744780823166462314116039469622216263221278402673028225214534859076857413557466368/5675134787340359276387085910732530678163604187009567576895331134521132071092293139167824957253050637085885426100206370974479279292804959049999647539684766632278449357893560561608633329561385765035738071726214510832066885156767060324427330401967034889966727638276371320401053302413136551142103678412692111973028587922399670456670380189570979581674699102934733563330382821732200682163238525390625',
+    '1340937726818688387636018677960518622106989311577445614347061615417831160296369500704524022869612973331444242000788110133157689786976834913297544087249593601420972111789552905846168933014570399562460789075871112224533723259660967481288525220689577290386781169316008987012566216235876842757183233207943071523447855623001920854571597886491163165150556/279054368814522483241446594911968519760869180429079780837994812436395792594449929617293503975169489945136244602350941695478861584261665582123038340824700224583170270427163469762789603657025958695551448477050958557681299495041994050951692916032501884523150972841782658389836059236040091605838583126400660344344601879969354053137568588338096589095875',
+    '8910951660339249479517731530773509037034977353457185453617569046885132434552771722727442649095007828468878374139001808825741829728375370140505088133244952288239619141544661695065318568074863317054989982620944602245967155400712702259180443238473885690297281525960618905625084639870407113029712510345262279507055197024677292523214467306898573956604928730377318210588837126450942086527893437954864000/1851573255256476362977604759655284263358595254392797078730401113550522214182030016660458780180636930591149034499142464357511158868625309442843792189394292787657774627425816694628839987456477799348874853555586931850788645677998004186437300173080088839930384424797706381282430208961565555953254145002318893188197561315855286173276564036402546798318351229978103276843822345004757661005876638005274037',
+    '84997663352987536417038496128111159210386455419431137931308301989414462311135708114322046234708053448098538629166672456703651524480738219822858469100454933583774404032968304328859365484249897913676382909050450855222567147661842184538302991526635974158862033287154321908483007506326807469239745851379559259262425675035790574846229470041596801343087041/17634687575122715507915388096857069366146981156897230620371714518707518105157483283253374797127075689236298114335734874553183291806268149747826050838089931351158483980735336772879045027213982661634865556129548356487722537193914986994283729746578044621115589195066212043187229606921062007047940293972381121200723292297716621231320254309060321476608000',
+    '1648849257486312935416274009474520589799750757877252390543329851029437452988499485179026845292068377957697884739052491455807688429630524210823560524122596610276957537756142278585091315141096146341773982302888007751632401090642472506538817998043326415693840862261701364114923327286462650797963969289743571514890723145251583881752876340628073395708716735192690808352188867334842743895738955762067968/341580411902962140529547622267381834425264893226447658723654120920549166416114849731588552578626041597847002946105459812230979900480135423787717825697955152193526724029890829615845314765605770060975538542976523177976182401820926854406945392167140347444433885786340415437006775075196031433365827874254477351374236932624514469241634422312990555534789607377023339236417434574755566856489013828206625',
+    '4899626659231633406569362199187192352933567344960498139854460545644210119722856870634652562532083546344965166340063297647520613455037240144734014575196764021102614748218834054157165781057405100578668000290493431869797883155764963238760612333048336953434663867580862692267574574465371615023999531030446250351305046572267686566999209229202587258144318592/1013522377571209303732207048597964555261512342860326827024786500686692475697570284120359840724596444209277948648863014479460188496587721578075151793081823795161404110811480269106846359964436993190293615381593756076515012514100289406862487971220020752628653845345780766244842386136740768474703009091552043945843632199999007070800871588289737701416015625',
+    '362903847358088423032098939589019598036593525722040887883164115981767777873860799955715587436566233010543937498922435963385091400670303956612881974450548904906949032005878241638546734597308786514086678561900713741746905742866635779432216761799371793260025549698865319760786921444505323758235164807966277024031605673481480590193724980659448692487108892154809490199414370023696404259691299614147712/74959603173756091922435708504751461448318819548943586676159545028063979695880930926805574951275602951047845904770828900645717477098988124553474616764040938121171776556999069351772286262529729056288451037717198273521810585868307402693261479298923209587143468206907278220909447483738730657438961435587752873879244136801168836767273934997417156543866722171766794171623224855358640156858606843576125',
+    '8521348154958613550574095745596657249924328336262127244827955886474742319277424381360677655239017583606851483318951440691757875270001283229585075137091617073509041644097526047127618954293881264113694362598157456878784814696577083997902588926421675843291157049153701100218784481406086810264243143337016042301806376864424632290226377881206765051896745/1757584673254145109614914862134217345456065873908262753169404848658913774363650863904627183104777337461445063704655164325930891269843669860409763569362863600932267319335889540453302785099532688264425988677082044905948179153286276609910158854349711054144968131934713419772852663860893241364433872853338630216232313607969790083448433666334742108176384',
+    '1802461420562646993856730082999823508145602238125054717836501201545920604020389361370931345491160549787411668288359013059160331370751496329806488246135100776263777863399096485894306306621852596694700845918608199329091852956315870664531614358379176680326508877329862713333336188556181470928613423972314983964578645688876556351433429494008513812136152576866650152623510296911708111085518974142728903424/371238439252064016214448115231139360835481920731625321084601566992497057371416121407293286027832357816257507287295719261141426130159269433199862002841433235955835172774430690352481806933860390015899973299268361513643524088242973968537595085600335021120523852420135081643310663869327371821064862449426404044951571585406092523478861879148545678639697679215289523234342337347529225543852787599016265125',
+    '88795280670112240977945082069219541902481768504536167808816453021962616596410396813316064685579412429940019071114917828928080181638058444302439626425155946562362550070187433083541414569447612195370911498321149367969974762244140788494955146280201020345849385865084095189982291190135293114489407801749533448443222584092575861096946605418438181411489276/18262596223069549313969288693970246663726147886364354584825624311486557881239003216971630599389642752648815377795018130449384513824927138566203714891090017787258920529990523519479383032564209629983810866231367438270859421572641494420831435563580050954867101292568399107965982704216434467243223964964535464624988344342084084237484262612100980519602625',
+    '752911409358158070688133336918078236438086521781731735123294741731669530734909563155732092406099003582749182967246236657021875488130304108082404134004381196925042617909108654109138702993903561036987511410396567636331465803949049946947888490287210013788802287247422993384874670368649616782837096233384279817294778916419067404863458318451722971117669642134894906235066332205740872938406767767019265664/154635332883086377656813458754577288834216641577439356891585125701020579768315520324293435008561626926046988056302745604248220408154454267643222328696606367409715369481567213640103076112146615938180089303294709009813772509602954514391675152597458428326644988225555987883250642389550531163153338541870029415077982539614784206119074702022117790679351778993541426110436276530890609137713909149169921875',
+    '15286089077439918584953144558775765002061832952090847117286473868694909713791678395221438112006475047633028543501632631679316850512797978594476229228325228403849089079675681042224057234415951253954044235099139983386056610384489202589484570852048157272551098909455253037561994944665563291526597323199530997923272894119350512727802414680772513760081360291/3135161418037836259442831302977219750614726139657718770206960776514822107155928800694375522572523712277389679035632531727155957120566617736817225541673148354829212969778664144907503411589126111888917929495106029890532891919001138770210977012708107496187227751496980000137223047079127096028939624830654227758737339725345681745847598157538483308068864000',
+    '17723482381737693269787076798246423310802126092500438681864673375704464394105734049099094818738115345706100191580982712146882034618103161268453808819471603805345990162762722549964406042295355916458624836894291267553138041035528315839108252422478610879301656207772491548742981990469454529537049123948604931661837539943536878260665617524824604291547373872716194154198824107266275716083848262625068958720/3630091565725887087605600771358216927099748013831912080380983545872077197366172619225751217619097657405660172801849926220948413919823038425519006413005998509677472226729295278652284861834178791332630676772284971762683579047508016687969040658592436230062049051143001484384767836353233324570686249824980508520186643477313941054900197268584495128610811273834192099099592687485176400539748616360799518357',
+    '39160514032490258389003214587901781721548011632821053230881239001436341012396166857515197251499460842954863342641015306304854226194551189224721257664420778416733889030680295206179109679627522113592373938602737416822981698362363553492610842201005420727750505681358595340127099855823746410663750133019743170700888375554918213182862789028531284364877435896/8009919337434786244380818390213546866794958389603666737562053423775386173149395943310276429895991903753441584557917565343673605019989336264748588735390182130334278887642569515202100870042209261561001627386525182096090194076839370589780431326097965428311786635366819094633651297179987340498215947144870164066409654464551017761658775207392803825996394875',
+    '18188388167811476762477659006849121912679763597132233588170406666718758511478154418948855160838212151370453213943784067796172102916618102024199200086478021533171984254799008541948973652219533819511807681148179806051835318645591102296256347927018942348574774821478499092054497390790734798143983945096982240517058861467440217558586822663985584775920517755909444876443211404222400335500608733284461685888/3715278241795087610941547133490827711133909145414878217965273061493740719983191775105550559905283733134189195190321643965858771461560130864714477448011225816210219450040558581571206711791272284197614810026220745160693898421068168630870850392739817340370146186703867097982858560347971281142766235850971837330222970900661653800041795020382847588362774677568059355238174402415384788280852059532958121625',
+    '46353430636874284402376008361176880938798775506236714098123916668545331718677407145199311191108199195405139575147933283523305343027297808443653460575141799729554209181454676876263582758919969948546918471128299522715691694855904535448703118221685887195383025133924852224568922999317583667985459047407473419196367812949579858081181660610155317911318937349/9455890179897829052705408931064120820559254220394152447502395298827357144081378513080398823188243663943177539817982481713456798757563952162598464929784370305696420245648035222083207596797327498500000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000',
+    '13607414211126412689435920170832779523235279308039712294681570127241403406455725064605210473923965685858012114332021572088551127762628892611459785625407107300355287623761046438724706918300976102581928643740596336865698485838431972050267114940079353996100620311097561405192365496385584164725035842307146351211183443267583872067352881387294766804298681841402861626441205037637282007810811602289501477350144/2772205947240095684743358245950494400934136973845543049938212403703005150954425087866758842625041409060217562878157900197951382737615822655863548356735553650755342108183641353142157728364469706919993519071329069805875834057743793434164720370026073143626055293058601124362130539040404491770346043977106988100969901297669523170508321703496903672333781306743488573086901583022550356345156457496396749797875',
+    '23264350093315690507131500726914401771633259602904894387312742275462601574788485574913195967854009886824436237740789910851843045306673524324611130536719802717912615609109285261163220007241492964833933975789257681755002054890453747803418465119567389315754801791950553304034666399677630030683759466627797123971742061866928213780452325847848901682792886840980/4733430508316851445525799359992627142007028326865116523033957869609125212251136452396410236930025876982265915963089597825393317909648054808274488356887197933393799241941120691826916741647173843634076299630884655997401370400734954098249519111198509015387869462198674657171808841005212058822254513429836457539629035675822719720295352508916898476137111981397',
+    '538960934807826161927564251064896730663918579817797131801326848839524373936888402762331254883505157846659830555799909822861853145306547699817986010353774500545305596630924815657016569444824389893996399550343022444937690798126189162954188680257730893424775303801562327757655350002001080811866451425665301674688928515041429654624970707803336800125648742168380676815769457467038849213225206824671025803392/109517448833211208909213522579666376007937648875042608355810687327424621469323788799169718267715808580775296460944943658428295481347915131091297305260674731651980547668762317138753250458639043477453950774629862540017034566407286963562574188172753764978963685535201028286359818524519980885827473904574840285411260037264057197155527809916623487144493019982419606308144576939532055057697567269576800542875',
+    '24451481554350251656598350124163983489544445991992661049829014268366250968046589127810466483349378560701491445658725080497078590986743512397927747885285095846398852574219367580068049729894667374214861939897557930527646930793431512683750733092448862551580494245486308114129295467976874843852138144539250235632985676953662288536669286915983261820949386845806781/4962240683851186225340479316959605207278948397833849899236383273145853045488131674290332609876751222112917095096150531825255611911377071887145019702433883343849119017643117746273523770961304609989779231048095665388209125939879595509516490230961558353074751312876252070889983946768175488341370453053495319823381943793679003096504200440997908071027962806272000',
+    '166591964685609920525429719961758335887941461052357297159523350448241484106515158310298216150030768812873781356326923214637847449893575074813991141092681107583183275003570804134461187213143832123628701876041315736048451374824101256980832991214945762266748755723551899001458245164317581617731332466039757786839492869211248064662857987111284518155462148044812626055842061283907063784492325693994911811072/33765943466859756347231737285768516240749714264564410814839966011490233598006227246396923594069488496368994857718734549787000213944990695920130591750377483911116845697231074543526982580529937081378303555262933800375576494067959129682770065443667091668104132884375840812635624862495548888467575848428463895756394050252457592540570490061561616914443009224917652440212112452400106121785938739776611328125',
+    '142464729519206642943985715465196065680101634264787375740459792619754842396453535646509170066091478069799080172299369994438836484938378991283020978342879057271045033903518255546705211647900945143546293360496661466657600494787228476009216508124736254967761901224547269496682078470378676865981855568197705507111970901389105723589666709580851611656838285777566736/28839643075889294346635430272525349651555039807615358795334663734940126053968249608601725991946342449660829847322517564370606746964714098508298705743689729306507554363954348780144675937510900983622836770223945538844314468550644543502501484521561567788992156632892028591678742014134130878101553633195551605796158103033794695984958987367799607407626319024034625',
+    '380457154857422076784416625436182734840012277442496600178664323821064977485271288502806124740204295811759917468598315932331219906040253463960961995849761235911145436552991677050565022181516596464968316292789048809559129516930574340710998611088996508613850407294743457197519091160170604426002924022923388338999324182261954685794522778833029950214443271526469607532589775916381786492089432407550224137344640/76922142291839859196033215395526448408989799722090330866256930898223012427332209054699494033511478050536463051459450844520020426942553107946282832072826069342120835351421637970462054707566481086202287686453371483126873188020538661871147778312986131920021929455276187425940666580158173676284545586944833678819622895550345083175685846944633982680908505900099388037050135210936852210263717185637166725605577',
+    '79999800708369666670693340813609265988600461251844239982904928629083975986385235816587497657678297663151298709076884516168798694292862559083730259839524968813932151813233366175091317046758260934357096757322369138747731224414176608247605314668045838769433811781490030635374856042786947109101585097035794969502111354126773165391577217071956928171336060247633607/16154850265413610119312596299635391433380319311682208755390341659152160578366244774029302281068833963848811596545791104919556346810193385001196395489740571188884873246131813864782581356887110499639544662322008824704830818136660544673293865921754569356583698951612797219776290835161780326945625621758204931298948625885081325268123344406261740843197232316416000',
+    '26517014004241498798848003208133111307300592620947908933072661812816683085261359696887064436611747292468717739609776825664344064024535028242737616786238200200498291226984541942836459188328346609549239724653950302145388877472274074665166631509490297630344351573245265440164082334092997307729007429218385933622623935704482371360985836659823269511115937443161532987660855490906546130131716829909388620539153152/5348257529530073524280989257577050878033239486885664777340947102417077875628238501791711336803418503616042922880218860266611595594400020206192599559925169589023339295256084579966533260861249529670141854632965231468533448709363477120437277477668173149958267956654411822360053335695956270719680086524322530514249891157439878407417971839744392044064720476417844056346174795601710447583861127956455406902139625',
+    '71197348290771978405602732839537012310626009318133570890482364371056186523989330157966045289127980687764218363064941391483781008265266963537623770685390148810455244349859354857421474277963009857011653865476081389968122418691934354122230780300833008753686234433642072919923583176333668289998661499146617277981691372057042930131717336250309680764289256080895092/14342680993892886303954853579883363225921109635079605392320944523109187641283117181461662846411912306480691901515655220490850586434947504504317080144378185989596578890875561336496479576651516902048606328604696610314441229321219627021610788398730173161473456954861916116391687748151443335513249810808309201830214153565966339609616397865465842187404632568359375',
+    '342401018754023891059352629509715357053454506037284788725127073168445189422181540408896452651232759821766616850264122591225331120588918865882061344322996688880669665450231982002597398732201789017495427566260651180470635467002453249479483900268320570004293679361696197905823690145326531180324429318916092471863347575533714871355651558610515931373080997536792242278529575700808878786732127630024064418468916608/68894647682461956089382461406499759613691148650738957016015844830333168307020284281532026442272618372290646283794122772122665531108484704223311663270836645399554940741928358317884620420886609705267248656765357348477857852684140561714900614303731786621656041881975328169543884661780531057279582784036689401206706069028229798913082362331334839997133771880514515972024899408556517159759907883793675672065360125',
+    '190286571838805495686895463752492174798162827967490343625540320679760700551286395956241059969689020885714190883323266375067607282890063444652926151862650426323339187596278760962390404656342309737428214493422028545912504503974763216230927196187743439226243880933947417760147785405444976177822143722569696182520826218120595298059020305548681823408268556482981325/38242608291912408815838081993045123864617058390583394828737640036167893859507494091864963383596867701436244007905770284939991912884465101575490548813894148831966132970730263562113961685364216989999844961688441914455435934703797359360780356482629495214859917249465567846934284121032247250332634283689409883499295469118485912489058982596725212975311140618764288',
+    '15345379701153714436938289567639102307887273898834051456693981514956329731572600026866162286097662958712922836383881840211072389867463987895894067288699955858337488812343674818565432780224654423872618380928462939482247900055572079914753652915184313685270722583926261499687728814560680092231526501970431280783177357735398580700300402036873780385926225005561766822479878280891130232036389707452005183370152749056/3080426623087819678602049171756961824196425548529619576972793653585089380165862889092832375934535464160425580449193072125955685099151500755278763306599307218302635233595329082702148677300945556130106197481236122437978375963261172524503625593066102601042060541617791008722569799735202779005407381267021641545644264334109052258081225159748043792288699674153722046659880085850980843460715080965607434114275256125',
+    '36443601662144279337033484452272105547257065463167427418525882851507989449375452076227652634440476502702584889057376829320548464677623038648843251719168810701744338735202853412651160681259533015293997501525430222870696235141468274274064390273214484922080219643095978640300874086585386991338440415203863901714217818222666715642467237183267303214471854472286478344/7307237990148312451155090506264642657872224684879860008436735430879314349554023096366846856470375025555139219961666754780864835035902856314981482268319886395881809659368275720375576152008289949085453326722156553603358879420820012964006978808113307340321824660484441618487292138631579616351777191144073820686595597986548374588839763719082323114663844043046087375',
+    '1575294930860663922843149113097927168024893739962464327016763741314068438447271326302129681931564068796631337571133444163024224071270063174829664848259482066723104251718808936762938249028582992557822551950212098797718627418782376027133576209718376784834456568694813577764300716139817031545408608110880973380162736536313173329973171844528083756524804574267826691156368696436273087171467254161930265910972517504/315498546654512047637461608750018349421492986465373840347519942494904200733840189483934245529561231604050629653992944140351106887747105945984325466452136415067807666316202346895657914393512001212656936450425268507235863153258769360296536050028695266363553141327896064944506216017933135251875087069031098099192325817395958019524956320840925176589616439477134480262100237057953933117460110224783420562744140625',
+    '7692057599553133417225997786980128299372439542315125030077404519679450372805989755158000864378924877036225189784045171045139667691524763609840822628706422918926410377185494048355128046783294954373197082851501927410350313585640237550107070936498164597231431242858771457898152684298047446530220873884022213114819556822050178579707210264243087257050468000092842761/1538814487241112760739561704846381585063002784795057066438330325566843108114787083741788078680861495340026117682606689156065008672344707615420501964123121620401073675430690541617930520947170306763541251152854304975131871537939618684186766168376514664494488691311397307418298556185724993104760692216619178358268934740746932420562664234738576394742559349407744000',
+    '15190836470550557926140012263310230378455340797095100474720776113765300081463960235506950799869073894127884189857505951314428810423734978025409205413789025720642609035117121385453856877706229778165533034683130064865256448444765902482146676668197546098809055341050873494446231793312407516463576380102073171090898973046611572020564659042146465345334273595086941562425006763960197801018796906114925005810069684480/3035556598829526968124942916297025416522606357010637652635799096498208643970273828496655227572100159414352020909846298817521082538898702393916708804775798552206401718323929157863173504795127180177622667376867235806208021329981136637175570409791442328468473111768140746895005561471574356889047981458002257872092408771959478275534098789982899056118498417649167444480913552548249005268380838457505944462494874581',
+    '2923754549090941424546974281011770689534961442264350984393235983189168024079617848190236639328676436485460311093391982825027512234467853098757486086492333410753919927528867817406389678461354257841007722472106464785557179844602277350411593959136520732862935740433163383885526249897183805981219416420390137516812715245569037561966929572448247077578419534651990827044/583599691615378545100950777601104007812792435102945248411439102112424251478512952087517463400054567832353967731887062349496616445419782499817548928078665212077449420216892583997505908858360011413290038645573504472422610780073975450391707084016931661068694024023083417126286468909117254531307085545768412513447197205482285022417611018126886491416133665625891075125',
+    '441922815213568908489589193556560586318864326425397702047965372289295962228254658331201274291697626694859414786292266008476137667843874780506886929181835469302142365772763129838217316953835831059616268288399119642314758261677639945004547833810080700127847214666763863338091523753898362699222880772298183760217731228628442782431573977490761665091005756534420789258276075174436091798475838370152192077494553263232/88113527373573049332749294663402406454524640221866407165839162576979477138694653404281889227416977618913249275712917582743426231744980461918772793673202077716234026090308791893409200158969108910565467273725064769890443408324386897017825489615600787634985397885230455168591240328585041836771921929567914891379235573982442311269392541694564675053636078990836491104116280492254795616055818375774141667319774630875',
+    '91197133767962483852278456285591810579974783077874307589555559715577599604626819245000672929518645004771085106770784762994476973842713035791487204122653245797126906189270204029735872508462106527240320932458036231707311992868347221176460407003487373754014365881209649632547685657666691589606571865740595829983849941049450471212387808617247938941417000965198123059467/18163679611214677813643455199979098261849405189805364986036911945012016248297645596270602508015864371788974684317239728239847888960155365342460843559658233069613218442910226128113196984801126248862273503232000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000',
+    '733396724434554481470656964167233690851279923085197144879644750437402627936723531635898542414136943003236478432848965516925891956615111678444586384493883113385335463639674699443808084827715994137410743992130712903997815471369201406683862985677774155704267039224546352414417632475628136460919900347977754644214725932476856018335195927604771408949980634142255497812566866795799230501026066145434755838336990365184/145912495644159810313026420025053989609942265686388630266226856650796088768164358961644113076809511566125145395699271530802891523741636707532676065833327082092521989958971154798465784847092870457944857590623097781956884297103961713362268264159015928554707158239398721416218953003215109202771651993622244788505649098445109438104466256408060944298437033058380804126574878377874459371251056800895394904459469832875',
+    '167606423853658713615749101123336326908381619586838606922565378505211386185130407423587264776519589295420758009764678759213587906216543160801059029133457062067529468464636376639307690657945597847080706144898920305379653754920571797771178341550808041652360635584401789901525709166172283390177673009483341826911179861584213314395893323035527116283758445038413646757280/33310364848744488727470761889861069993071965180756187311178124887286133554324042422503263994717993129539448310743598138364172889305532401505890384806078790034557049074169245078416704866867380089103064821732673967120808611636618526737138435792818828990403374766228037003927569643972203993263329403469747342204808360947840103843317533210523853631711601450617095736473',
+    '22466735478643771803624909352181817912387732117378538284107485621001246177152644141853604633676939694208131243081227668237716194316732125445041260568403752325416203858329469477774720998849334961751541191761521224644520742691981372782009394826739249604257098248751626957528543558542443122470657230835176462612530464044652140581367666271771882198747182114685116336512604038833327707163107207664705120728068987996560512/4460342260599110071819142535903990817893424192972341872394014748315090430999224107274777695306032783105944721337053156074851240201179902835353097683269298594190456286888326067932777720561181760369783486872666974623910515795208353605697065447942804414120030678603538761538385793829787734830129764152263691296719726909633580318897787729004972323599773548689272008712427926734258913271101899780538635213826239542884125',
+    '1972862002066252798837977155272535342727415329632252213508267781390870646909686724955861608046325765650407340102472833428905279473185818898693854344138403744757857928883601637909604082588338869612221777263582365745419719805587946558326048087138197694016224613118304623864709183681172362693106460535117459192992832969130895829916210933710820035479169562970982784580861/391262854245936198649509275719624152305206807261382621390946060636032376057385723427601113233050260359684153963520782285097664628279596618521963444574543409076428660874564026651742234728997339844161514273669735686516761239076529088145208958938384732800586298943281728435207452253170935715239392534815683367114290676180839217919539381651841151674859715992309727232000',
+    '25408691426286653276004556426513401767718328640048945917745783225459709487487356095164164652218100925888847689099153320777389381851598236046402124684511968385325107497047025485302333503075085562533472482645023963717347867696861885363417595298751490212495932766872748125111115039311507374363000062513977436921772096443948159289707996749061007112458126301618769295162649600673247799961118662602756889448152517364992/5033870770817104909025512331238419776407331281944565644457345507262881100618908405924951559853352841154730802006750688581603270849880756026262217651685813584900300357685874044482329601894134443542753690048962260914642678445058891387041919581376195543193199307524572671344323361282633828144708683536721118910549933336800461248338059209853766305058943888965279863852107507075916981165164543199352920055389404296875',
+    '616565940481577765011174617619124841631603944267256178093555235591208387588894068005162766513720546626399308168923904625210748051788652328072241494243040185611373035059619136712559505023703153194815611268503291295059648269772663939137446253437248541496631852986499688052678961495116210133458596866684413931188348798244969797973846568160999564540652111599082181871436/122025682149627484528651507368051367620578469392233875493556828623437869650284356703295836998178256875766319508797903923638967073687991229708051285605187358616531926112736402511567721889845693991963610192161413692928378708223712791930113386840109729368772998347036090902364801014045328348830748307179590755723160142468563977011700764517880467116766150322736003147125',
+    '979882056834583241477188686550561625289337025747787132773853690143359657519481397575561812727394470125172272246774578404543257210124412718725081536309818383310896102958167337333430555168671327992395916735022711297435263157670501117734412587691271084981140167017421900852550822579410636184333142944411060788723653596377015359036101170054828623669169047118551010102258802465554033173968468183492058271365801755986560/193732066620504775981759812703028757608600125742477178751189607602983198559778039141192492589734277739017909889924488699372940787977794555519474807920198944109549023547259912324279741006975129518191081373554817584214941629021088206977532599055014150433581544599721903881284581257812850697423350805715586436114842183295730680490110793225514602133304729484547181740290534056445482186430729398701311563285494964645419',
+    '1138059441890428903336718712159907364975959482163004276901676063186939296217569849990951930955270849692775077813359573394435924934540269310629731801376134549169322946310756738187834128956671195358413024062230774111935109203207768898525882417078119314003162592087920755572050993355843681931139257303484526223461840939880759908449181135029327442039666513325916729112853/224777393704985303260433439401503507705148576727364388720387174977154204629336985298082893443205223017966145098064861566754244624732671470935725326043491911142467178027013112527668183783450932992534517827863293825061296257420183669383719043873666284378647952385401303733274771042385666436214517205678674213118374889121542369996042313879258622144464565403138392064000',
+    '614262617634904509410555695398092190166323289787703954329113412647342230855826568873946233136303566178604275296873234926652080462259552775254755109931312516616838091852915397846984381183732773411263828600108443070778861986849272794442105107285234729077818614332155890946804257724112143631642663084660012362691100872465148864413343598067567142977134846266425075195546265101720326272502009234689388355365794889444257557504/121200854655310345878069155190734381476914416893031941761756379148622591310152793328910334357724762079139509813381997287954319983165971628429295803162563856219211627796200599196263868781937234015821367575512548522790158101892507996780675863234181700253545912643548466256938936835469646114659598727592390628209362096738343417775009361678599075671154405137663077798419897932404104579113056038271753988669813988076515476125',
+    '106893445531995430131837569895703697604978445859412739331042151546083234373443601923983714252801103551874986033841637079479794895452077803182698291822629713500021995241940133037200268381793282318691568234728329378829301282121562652452936297422136961147432559766896143081059518238234777055931999714973999553456547746825617168005650221542417216291412324174772893635832/21070287546583154895102927029203125577385528350171457634203261197137238923923245382493868134869009844094435068284335909266152042903157700510534904847257436666980671067329084409346149729385625108233234542133838924281113454526471855573271066865534480974484196644173270141056470857666632902827625475719738076524047736123182255306429677688129231682978570461273193359375',
+    '1112742798776748551928744983308584493817112400536084744850969025063360679785378789764875076529575583792571198510306186417773101346425704969671256564494135750783642910394213727189639661184379341961737135764570942961692159177189753139209638929302928699705042832922774012622017270172331722295881412111812933593370743210214549038212395941675257558442878012136484052827439007483777859391215222647499741572327725689840850048/219121905514652575155579185322367175400669386131891843382447432851122462422992147756333729893028362013318264155489537138133372703295804002840920402482110260590055749843170064921843460498824128274420925357547023876488939786863662801268798987685503391058250902634999352652063668663245829496836853509616873246554292796823779397317819134891873583857621623143253788854530146303641400986028503534408870194989478421253057625',
+    '983995170259637497940121707857673268620603982844439610076334749792119502557380361624096960070148627663305171311646851566811683351232268240169848810658089698606409195744471343416197434926513684882439596141368794175162315633192028977211909698595778839497844847281262559899914504080092844816319298286511436690429175757804604673752838786852277255064434365401871903307643705/193579499709739777577893639665643417521924510242119561765932078835121983190401762561436208758210519231103767232899529457440602166918701550386152070813526760823513884391055054076485295945972185402977095815609448651316824455250005505410192101148149284756928250258632031981439345164037318129987999546683113483700660879431801728391911161900686707231543216283304182063038464',
+    '30691013076836380319652884848585431917439358375118437035944950498156836885409110087616602043242747413728866747975631727566613593257623099549583001362246649928075859944695060947434885707102684568632559339383409378997568182445019183928058902445738702610436187907199482604867148445903921474846103586253320950728989228898324829144701782792767763794895137313050381112872972806712570559942583906619373159053654481250809168128/6031936190594376585238407905469035598668211679521553144958053099463931217856780171872253258738635531385711702176347575597387389685526098889278278029314982721284691198967204664771178761416891317672705719303088746953644978471289001577698554025410820041497108311270431130911021653860874238212040640737171103823631718483572464361276703378587106311153455557896527384426327053165214814954154328483257657424873929502112904625',
+    '33633002641218571027682690377239300567277297301791434890431645909447040830457253186262970747405514897340315490970846553666201190386457221846694126637998462873786334837901001833771510353365825087187582807843940479192544772559118293851131234910638705258664901225494129023244916125545283621607920983287799648992632814461828998517949449887522866259724675318233522060729654796/6603800575190864815125307817934327053530311915572323544225173935830559756077637251538096164499112411672913910767032482188570876526606118068929913515497400397562722669603861813453418621246538114872330536044519242663538662076583511078523573320035084744399535581216815034837553165853576616422592790103728141443697362408727056813977234078214386771115586436985105038873998875',
+    '9090550743395453237608373761422705922989356346696677314277463900103283296889488490217663400349361962695279888433677856471792856853109469433355422802663947305503479561493033851081993247556828544308258821557994512133708656174864544457143793106703578476914924328960109390350040954158797845288786321215010324565010420734044059968931558099077932776525974187717655105951506223649947806568984848883258144616897526257038550656/1783219994432430133950472014582130883873515063182143523712063859964133099770763874744121895885645346884371273930727450069430704145915261127061538288702855011220545530326588799412076879985026006593893648911637986337776174555198865379578424838210466703683642106934154603710587408512357351455891557594977191641529451749437270208258072920514602243985184744955476557012910446023911358037761232253615162335336208343505859375',
+    '396226087129511787616470716244996965529391571019620342657409150252952330537366537352182763477820160355931003174946109480036915624974500187179367618457008802667089004678925595775249725629495848146669773574395699032785024522882892752987825063933573399691764866816284097232869198405981830106568826923407996367321353003569052419999880089180418759265274331299787468134974811/77651132902382550540201641628906752945078964002822585000619340261128555048628863996784740469908853602590877726845475670110127527874088625214912598995012971630852593597027649154487264595100300180591016581254965295280089902497540043916128232871685159131211213497239568483141031137909208031392464815187505714066235868748950184065923236670418822911614780061752905170944000',
+    '1288523130900767412477856952035690867153496584747510253984686336382980221603917538996138294311274357713893956725672062832264871415025750136690143411884111580335458539824573239771267043414225271729141508235839982033182605445173050590151912297720800168687369852280323356691345317649445843078291369394324862039143247087442309135074453265658730028145500871559458519282433012494054724678302957276371418936282902434203225600/252284450515080370603949213084529993395050575951188044032908270470514999880554862449223347761507020354321422785760279154913310488296485989979706208254206974075438570521939826124860483842758137231373182496438568202110337751291106165654945352183167578675508602832478752400011567482434301614269579368882858973850914513870297071866544875650189106576672289734326492492059816669288907018095832842249738081022227281715919357',
+    '559506782175368748187145067236900803925823792088293851453005882621173124590368101213652371965740365119375069053747415415684193092607523211636912074219215053809881814859129515591479780669511533619258835199715146655774919418434080807837587396115955105126806762961292429904380132116986984435528307613054828929294602185927322238461491059740342387266084431988727321711184308176/109446188216864405154936682374609448082436347600301765944233278303643744736544109262794383069197271409398615846380701459882784547857821831816863216080069944818453908760476475874319791512782445408136169878207358663250499628002971644430164185578966399908622640732411885568638054229180673755690889471388585521113862492432946110902070139679164884208988489955874131206343597375',
+    '365842109776085256207829452385781129402656684569922320812625438296958174120084800064669048017227033185969596458778213306152397766619692644694641677223213185977626940193107608138827771259334379895020377700759789774994800008933137569543592829220611543060189849601828342989023692876182148818363689293009174358168826340010491728715549904988196347316703985894355217235339548078187164176255405596310602921647203428200269666176/71497266937783826306774367225149833789788300609530680399994804704196212252148324552193149332755057982662844988588678159610516634667597707204818296457616315484973288944637139939912166034691155272078609876888919170894915224022982785138928008133003473320018151370486895637936102520353951511525996197267572355782608352075229547764732602557133494648237258900273354364050662921889937355682486287799239363976556848320046933875',
+    '59378684894605773397070783936113244261297966863668546282887298999811950599772732377979382620766180662159460666220894896896564357533604962977968843474062092083279165837424210591042655271577974272394203253875965589611493384384257117088518521598379301169916060951350650621757237206805037261878212424267780312614942173870651684587166691539909068452932891649554575072205080763/11593916877920131260484148322810317629326850763057953025674154928920977965095141485536566217873796019006709247938019404840440511433467040918770165456925609124166970862430104329357419669175323763143377917913760028500000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000',
+    '440706629842402075137926797732405268015915199334341565218443748798479563505125605478061664327658137598204192554352688830679475840386882350034150077506804799859586772904929402683276898098601814091430598259347980573084686932949502809068565420937301632494387691284902026088950093888426872222813857928092871267130505744095697172494119689779474840037266295586359244899477781345249657455186847452249040895151288114085352974933708544/85971876318334010542313048056773419567285701852816309711738644074302467763075671531248793080425349570584102078764464196653843973936119037322000695949209283327606280423406302517302874806092835694196353778161663338667091195199032831587897410329509487165187753418615356836636352319310103639079123698118997500068644500893282438137614420288839230827096109835206605385673197123994357393257304049093683934224826006704198171592304125',
+    '13220371178312284248107863849287558696210741535280184003301555188086761767176560823174196915070378763091889333203635784037294769747641959860810039816533609140662043493962707445564057197150603226797446933532133956054671103942520142568674339411250001585991743608723150284270101326419464088290151483823061613403673956918458456165552482913079283537536214967294211244361353060/2576683013797055224377707713704899315377379548630511095136568220145808780162671473907260623503552385384482723173913562423316997528913651044277418227616447815609873991028073011738113927961824632868956802271260292075825561646156680740323686126811928372672670917332613490241615850953660246271541592056143611581223295045062157404996958100136358649109693659623847024011265147',
+    '1538372578072246705266078247950089219881197139399081643650271348770110178217959856819170325400476655944115553238940541695152110949081675985299189913515975242840473616169775880272489805123124548256666658584129873597192213080038937409046114908719298170078476325423466734429870048697383304265342125425087105698152643145531760853127801123272389134514333752173173161562273487752064850461823726948588221913540799816489566883798912/299566102446745552813544808169323270315327122249327394279467999308669285416179486075255023523726449428151807672528368606071047812907935426406886253284580296349349558797128593446316786243666290649442956134448480598303527795996222703837663605600624271807461605295956674925277658331760000117761013381199880905748188151556184620779667670278558839116983636972680749836497346863752662521197700717702790359162307617024566689271625',
+    '281696937222363685333688097915539146272117417024070728031063821781398985795813333650547982965341298894634762289669256129805545549280355643122501181852136332529515756074502902204877664395456278205770206019663082511098956468724829475220996427130295010430751942126263620206886404244432409164651524408198286831329682605959881993845189585640874220032515465014097345759490750307207/54806274927935495596891312029173862473687610099325998583092091915101265393157631244775387832621032230434007486602816972765210205596482942091518459849775458089575743699797096697542306049628939919466378189656065461708686421130956996965004776108499273871055615413566043083634090083096967567965373203683231171594922347812966905149873265654962896456451681332216735396249206784000',
+    '228506238533410805238253122961471637501498406104430046727793420497895852412513560203475440156216063574654372726194215013407325207763369075250819386897086702757125418327084312933061738392406321003231853349865003322697360659712771167066453430309997639548320673279964210730701373707765051246595806673865646607841876463608039111252607452843372970797479959213447237540830563469297144446069689149549515457318899981763489792/44418759813188470246596822974281932219088697005123285598909329328419879212164564561558656547678211265529803086702843380128457744445765859393426675396393053094588240457569380155526179040891009717323059641035189017918495485630657360525410506453554255734005459614858972904374739673834145083512160883402151636664208253116362089211658645312778404886460203600762546810897889629330022309705583438699250109493732452392578125',
+    '2396110297187570506257836426251930521245117960980667897194680897478996322590925380310917400763581787981992131769075961259393685331523600287377841990678117113566991100720996059187144934569703045966210306282348527729533806882408323961336166190162075154579344185244957673138372347997134388801085788353482276257401005734929059700346687783400772331280644647089520646562669857652648/465370154262863030800729180763091967573522779260407615567563274096037297593764955820626213827962697601334935773802201919483657310282236100250543992334865461800976838791035335861420892967699843608855185710753433852279737724446294443337363478679601049432404435672469098850676302728116369979460222148206944323730586138087862719386327396334468302608499235112520318322763857676375',
+    '1188272888075247520453568790651819187486389546755745961952033150576243575862120569399808467067800830890874391487822723879717525264875722720416173276963170672622214121308038530968291758445510982186014289745026439477197148445087614813262170975056866881909056450023324743409114115141145340480846088869807540798073417035168139476153897066325127540609756310068558610112731607522096096821938163134080426356894899245575410494080/230586762395914863625709048789913840511134004337945700368117795195769576856434588144361459740833007125516905017415184787078710001969815416281381427387949263792200577791644278234661952852149216540491852757174842146376788280677956350901082838474705227642420261641783722891948710833898248566893322144435457967452449965772543077421407816225704952270697016152452032506898311260979594550143344971416786595888053149375287120733',
+    '1072184299258468249004114536023934869518152618369895613271618698908405514630154819453979477077738526673716264830106745031385951736108376866253955319711026042389282009348594845497252059906901748860111986811345742998968284077119165082435598381810306691078037018432766116865792677955520465319184311529667642995217214507658693259694586107847099911341219913576038605872218680562047/207882129535607118520818565811162643811811563516006836924341123976848022970024793004700710992938987476728640943834012473687892481059618823520721737479364352887178017866425027953255856440099446112508663603319745066750522936408781591578321031173620445865113295407603090322283539974563690157418931060565698315406021569491751612761888641141446406531645379264720684248398299136000',
+    '158094023904374464057702010126993231151433474361820266584020400101533651587072432728050776506672527718975042616651177034635188178967462982926261516748842073587981953038445977514446838733557926696939591069712397947055261777416704028506441173540338327591514650398349069348381309688677335132100441905603664282735310299686268721634369502296867409514777359935022301425165750780988231639828675424239931533849896744198012747670272/30626384366513923671063955469537776291182594080083494737341604579575446319190916924682809546024369330706012653799783122328775875399363759308193432327358837772825609356553955350401725689682838701996446404559005896479618311357573203081153855961484398360670837599853294627153740224543175497655414585918231215639906924313779933211774291565425360415684556655021031131349317390382320286092667852173815259422737956767922154798375',
+    '12216278924286401897470540169822542417611356797676160128176706721343511598951866107858729405615631246003458062949723290192242594609994613705183460814797527214549142043573863947658832694800184691222426521703728394203722420336773310908687071670107794543022567943478843065246011544888390757810534211554666736239506503613750756246364621509359804902243487113673014588325806249988332/2364584127734680580481708368927921125934330000162935118371174269642831409084634375654239735054363928748162229294829526339927907778510530444646754603322373712788147193691037688552857183116332292048731726305451516954679277320777955342295564000586923227775208580772868590742136150431152495718132142742153643081187691849828217592954519485869135930755646768375299870967864990234375',
+    '578939195017447864385837341975558679292903716580114421030538728778491606282153588930983966100514147976321305616196463174070345884292672842497715966933722088536619333912687108737657068701352918582656105418491672377271304970407730786932913490869797471081081431072836814581099271832696990064828717287258860166842791984158634684078346453202928137987930328826212011774670396356151265781053547412633310327737900786639842674288716416/111966291649651600701875186827360807558960817589719096407887999108024557254221008849510152469119211567152616745769256123968832250104006560061907741567034087109512483357851919402017950812795426393415409198268246337542639470834168040295134130657073201020340432236723824670803498949071870597039309004562024033656488150406105175401372709011082725359382691270659245312133919518760328422827067578306005644825927050994860723405473375',
+    '8041514821545833865350873212853873042112343327573364523948780030849543753072688526717886590013028227714585965095216381829914996539132359247488639692828639170075726301411777307168737667077730792892326979446426349813642680537699905651125068091362135055902152994216233622733424128982011248923051410141033146604688083721413402395290842738021365861670415833356403364696759583009635/1553937833499559104910551811592138833189892464446761247731330672284191313590708439680460540329836344767557599746234213006493842842879398684745864919973051635127080738612248150108793397855113122368507483429209127035263504194847798809612278864438287815470711036548695032421148125849669175729331459149833945183228413961985381338162155596008680906676498435954178226973272371625984',
+    '12059659399309169167301353842123191409306995788633965680414620302285606896146667751472818541537183787935352700725982053536032930866708413010971856856768384974849695774437178790890015242323873769697407411558706626105140596160049735605389301414460825674631122462760014451298494875244754354054447697502521388381253293175367558014453865905104467382184997528879171136492277222127399012368677604910497789360212571238146627048080568400384/2328494054002703067219004484782931471919751585969487160602269605916676223315055450845942763690581877771159615428854802721943114463834992106300185328876910271612375509166975628540839858771078495267566008071532876816826751069408153323840515037497664871689085370699991860738368371510578076683395101062439493831773194498579620513073693022191702398601471237166387045790480861932005012807234071429628725981935147069738988682858417182625',
+    '683009426705008850682549700382901603742691705123356866338951397347368059105140826655486518416578697931244347554322654428162554903861783227680935341690740579867651545205562690172425016836559678937794312287639193745517811234257062356826622207475918293015140472942579713388671782202262396642735640315948612572908444501108579457520714632371056288874351015666995549139003946314085696/131769214246522869780429817236150406548549044658474416909806179734873399109656411633488099249198882165279745627038984487365589625562901914030850178330133080409420340227179447548044572594184601425691821617928130423410997853384425373999967759499571890326740433081063424903098252502791477854302204200941376983295622430177445507316706609802292121282957519571162158620971952084486375',
+    '1418116259749091420309271913909337143756589314597557128553117870573928055109322864794169592355058370891251425125187770724828108988587888087023982451205475980371440423992838481547236638456717520835668859093413956339261802305597694403381021863258753441889536361061407579332570493634254542013337999844407181346815549594836374497512447969751514466481466717590255369810506813543577735080228718660747940956248427356785784199073569408/273368882980516556922532339653303822104510203916024760371844491705599093435133738774975538364241630079468712981056336655766117378160822393876348755370711942273876284944422610974149712212944211513462921649872232656449940987552227525504529710373144228818606127541937831103642415534318303187393593236209298242574677293352798419428712873685798636474667204083938875672934235816396371477146089168375198141802684403955936431884765625',
+    '546411077581845999248238069684194831621691193366466972898628141511422317954477211865167402284211971296286346492698354214043840080181702652132835681239302798339667835439189275457116051279366194551632712819028177402221815140045210690908631767786711675029194359998826252332434573121740830827417293557234891406021102083555472155479148332697839972702631641910213821827820374586459319/105246945889799140338072754365901705544530661781295362563214285953881890233987516230568663143342952799036341701952267011236663118111226623815975847494372076420199418712941018288738590264475520229651370015377608638075866459684692488079223686296541020897904588347083045723167762682959422858996673894180805702299907478685374065593161226849447131100462022089783265346597553176576000',
+    '2398330640958841474772606439916070050977544535580605737383995160447105736276950196885906408317628083110923322157113892928963237845914017845444295040924101784423382681801754191301860383927129006953354739240926643562987838836997453985855576402628166875869041032631651591871962852884189548538272285387092843044669499688035134181859376665409767886188304314888753894905317929877238322615838524354191263502347881033855441181420399360/461588070868590122892265681879734295007029130965626060552783760068897000195207878227714842617470320231527222074701444349530952699708435668339712860464533455345665068841333232359698449088497137068713309811942968433868609329301082001752617420002377892756821532220676085014874112083615054550278903960627185675459015343606391094523511117705747842645927349130302549554534056269331809016770715819934970200483161548527932617036185253',
+    '6041015879424725383006424536130409209607854044642113747266098198777011981328765528361630516108680392500990580908509403483891763219659726090675140672989657743882183951954294745396417829943469201306594018454995862321821016087416840247422350906412007336103086620396467456181771583200365740253389107968122850063607085957109965406634738740996318415514360956028575560979203447735121436/1161752799109428422288020947061281540989708937450568100764830251908850596717606701047413407636907934320789870175907792017513896999208892282137299070761467096211814586909598705615312819596495636017728313513520193786266452836805291464826226833593878504804389728477191170027729963773716267868284479768397603444919008915279522376004326398403851684761808785381609370767169521034383625',
+    '13240077436443988749179508462267267187169441948722358165090554769250505713747934643200804819418670147225695324432684266924694524337920816452346599774452681831320005286326986675907899608537972384924882996757503264622991355949039882526389342174307168805166215838138277557052303430492669193939212362638263582899713198716541723383138016564027766560215944409353427176135895982596327685665844815618402881202645610620284792793420780517248/2544223084468158291883698813309541801455311468982232546872485444308211415529998472787377800559884210837213042932180479090277285630234238711851480232520137856848809986631784843528381778520727465146661792797924458540957133423665746229799675650290296217658444899605236550972043549278128087645211909479009099766619355677984218929672461506691980442071860591767266913041147587815452007726513853820116629482732060593116624596368806566625',
+    '1953999166296955830935495158735359200362904181792947794529339487489730042568305997099959302322956898299616194932283060554261566410988618045107398092345476532371402134206635235570281738377188438407703089325315446371127042537576093536896282955524842632708645655481028161471313608974238110718242273935956977555610147714316158486553633871312187084618154014921190595222799283957140353/375191165084882521037046014569185165885459082629136124177286500000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000',
+];
+
+if (typeof module !== 'undefined') {
+    module.exports = {
+        PRIMES,
+        PRIMES_SET,
+        LONG_PI,
+        LONG_E,
+        BIG_LOG_CACHE,
+    };
+}

tools/ui/src/lib/vendors/package.json

@@ -0,0 +1,3 @@
+{
+	"type": "commonjs"
+}

tools/ui/src/virtual-nerdamer.d.ts

@@ -0,0 +1,4 @@
+declare module 'virtual:nerdamer' {
+	const code: string;
+	export default code;
+}

tools/ui/tests/client/sandbox.service.svelte.test.ts

@@ -0,0 +1,59 @@
+import { beforeEach, describe, expect, it } from 'vitest';
+import { SandboxService } from '$lib/services/sandbox.service';
+import { SANDBOX_TOOL_NAME } from '$lib/constants';
+
+const run = (code: string, timeoutMs?: number) =>
+	SandboxService.executeTool(SANDBOX_TOOL_NAME, {
+		code,
+		...(timeoutMs !== undefined ? { timeout_ms: timeoutMs } : {})
+	});
+
+describe('sandbox service', () => {
+	beforeEach(async () => {
+		const { settingsStore } = await import('$lib/stores/settings.svelte');
+		settingsStore.config = {
+			...settingsStore.config,
+			symbolicMathEnabled: true
+		};
+	});
+
+	it('executes plain JavaScript', async () => {
+		const reply = await run('return 1 + 1;');
+		expect(reply.isError).toBe(false);
+		expect(reply.content).toContain('=> 2');
+	});
+
+	it('exposes nerdamer for symbolic computation', async () => {
+		const reply = await run("return nerdamer.diff('sin(x)/x', 'x').toString();");
+		expect(reply.isError).toBe(false);
+		expect(reply.content).toContain('cos(x)');
+	});
+
+	it('computes exact rational arithmetic', async () => {
+		const reply = await run("return nerdamer('1/3 + 1/6').toString();");
+		expect(reply.isError).toBe(false);
+		expect(reply.content).toContain('=> 1/2');
+	});
+
+	it('proves a polynomial identity symbolically', async () => {
+		const reply = await run(
+			"return nerdamer('expand((1+x*y)^3 - (1 + 3*x*y + 3*x^2*y^2 + x^3*y^3))').toString();"
+		);
+		expect(reply.isError).toBe(false);
+		expect(reply.content).toContain('=> 0');
+	});
+
+	it('blocks network egress in the worker via CSP', async () => {
+		const reply = await run(
+			"try { await fetch('https://example.com/'); return 'leaked'; } catch { return 'blocked'; }"
+		);
+		expect(reply.isError).toBe(false);
+		expect(reply.content).toContain('=> blocked');
+	});
+
+	it('enforces the timeout on runaway code', async () => {
+		const reply = await run('while (true) {}', 500);
+		expect(reply.isError).toBe(true);
+		expect(reply.content).toContain('timed out');
+	});
+});

tools/ui/tsconfig.json

@@ -25,7 +25,7 @@
 		".storybook/**/*.ts",
 		".storybook/**/*.svelte"
 	],
-	"exclude": ["src/lib/services/sandbox-worker.js"]
+	"exclude": ["src/lib/services/sandbox-worker.js", "src/lib/vendors/**"]
 	// Path aliases are handled by https://svelte.dev/docs/kit/configuration#alias
 	// except $lib which is handled by https://svelte.dev/docs/kit/configuration#files
 	//

tools/ui/vite.config.ts

@@ -9,6 +9,7 @@ import { storybookTest } from '@storybook/addon-vitest/vitest-plugin';
 import { splashScreenPlugin } from './scripts/vite-plugin-splash-screen';
 import { buildInfoPlugin } from './scripts/vite-plugin-build-info';
 import { relativizeBasePlugin } from './scripts/vite-plugin-relativize-base';
+import { nerdamerPlugin } from './scripts/vite-plugin-nerdamer';
 import { playwright } from '@vitest/browser-playwright';
 import { SVELTEKIT_PWA_OPTIONS } from './src/lib/constants/pwa';
 
@@ -46,6 +47,7 @@ export default defineConfig({
 		SvelteKitPWA(SVELTEKIT_PWA_OPTIONS),
 		splashScreenPlugin(),
 		buildInfoPlugin(),
+		nerdamerPlugin(),
 		relativizeBasePlugin()
 	],