* chat : fix reasoning leak with force-opened bare <think> templates
The reasoning start tag inferred from prior turns can carry trailing
whitespace (e.g. <think>\n) while a force-open template prefills a bare
<think>. Trim the tag used for the prefix split so the bare prefill is
matched instead of being swallowed into content.
* chat : fix Nemotron Nano v2 regression
---------
Co-authored-by: Alde Rojas <hello@alde.dev>
* fix: drop MCP recommendations auto-popup and silent preloads
* feat: Add consent-driven MCP recommendations inside Add New Server dialog
* refactor: Drop mcpDefaultServerOverrides for mcpServers[i].enabled
* feat: Center the empty state on the MCP settings page
* fix: keep existing MCP cards intact when adding a new server
* fix: keep MCP cards stable when a new server is added
* refactor: keep MCP server list in config insertion order
* feat: shrink the recommended-MCP cards to two tools each and fit them in one row
* feat: make recommended MCP cards click-to-fill and tighten copy
* feat: highlight the selected MCP recommendation and stop auto-focus on dialog open
* feat: derive MCP recommendation selection from the form URL
* fix: make recommendation MCP cards fully non-focusable
* fix: redirect focus from first card to the URL input on consent
* chore: Formatting
* refactor: Remove Recommended MCP Servers completely
* fix: Preserve legacy mcpDefaultServerOverrides key after merge migration for downgrade compatibility
* server: honour per-request reasoning_budget_tokens in chat completions
The reasoning-budget block in oaicompat_chat_params_parse read only the
server-level default (opt.reasoning_budget, typically -1) and the
Anthropic-style alias thinking_budget_tokens, but never the canonical
reasoning_budget_tokens field from the request body. Because the key
was then written into llama_params before the generic body-copy loop
ran, the copy loop found the key already present and silently skipped
the caller-supplied value. Any per-request override (e.g. 0 to
suppress thinking entirely) was therefore discarded.
Fix: read reasoning_budget_tokens from the request body first, so the
value that reaches the sampling layer is the one the caller intended.
Add a unit test in test-chat.cpp that exercises this path via
oaicompat_chat_params_parse with a Qwen3 template (which the autoparser
detects as a thinking-capable model) and asserts the returned
llama_params carries reasoning_budget_tokens == 0.
* server: honour per-request reasoning_budget_message in chat completions
The reasoning-budget block in oaicompat_chat_params_parse wrote
reasoning_budget_message into llama_params straight from the server-level
default (opt.reasoning_budget_message) and never read the canonical
reasoning_budget_message field from the request body. Because the key
was written before the generic body-copy loop ran, that loop found the
key already present and silently skipped the caller-supplied value. Any
per-request override of the message injected before the end tag when the
budget is exhausted was therefore discarded, even though server-task.cpp
already reads reasoning_budget_message from that data.
This mirrors the reasoning_budget_tokens bug fixed in the previous commit.
Fix: read reasoning_budget_message from the request body first, falling
back to the server default, so the value that reaches the sampling layer
is the one the caller intended.
While here, collapse the adjacent reasoning_budget_tokens override to a
single json_value() call; json_value already falls back to the default on
a missing/null/wrong-type key, so the explicit body.contains() guard was
redundant. No behavioral change.
Add a unit test in test-chat.cpp that exercises this path via
oaicompat_chat_params_parse with a Qwen3 template (which the autoparser
detects as a thinking-capable model) and asserts the returned
llama_params carries the per-request reasoning_budget_message rather than
the server default.
* cleanup
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
If mmproj is explicitly disabled via the model preset or command-line
parameters then the model won't be able to handle image/audio inputs and
this shouldn't be declared as supported input modality on the /v1/models
endpoint.
* mtmd: fix silent prompt truncation on embedded NUL
mtmd_input_text carried the prompt as a bare const char* with no
length, so a NUL byte in message content cut the prompt at the
tokenizer boundary and dropped every later message plus the assistant
marker, with no log. Add an explicit text_len and thread it through,
matching llama_tokenize and the text only path.
* cleanup
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* server : fix image blocks in tool_result being dropped during Anthropic→OpenAI conversion
server_chat_convert_anthropic_to_oai() silently discarded image blocks
inside Anthropic tool_result content. This broke multimodal tool outputs
(e.g. a tool that returns an image) because the model never received the
image.
When tool_result contains image blocks, convert them to OpenAI
multimodal content parts (text + image_url array). Plain-text results
remain simple strings for backwards compatibility.
* server : add test for image blocks in Anthropic tool_result conversion
If a Cuda device has no or limited available memory, the actual call
to cudaMemGetInfo() itself can cause a fatal crash due to a cuda out
of memory error (there is not enough memory to actually query memory)
This causes an issue because we query memory for all devices at
startup even if the user isn't trying to use the device for inference.
Fix this by making the error non-fatal and assigning zero total/free
memory to the device. This will have the downstream effect of the fit
algorithm not trying to put any layers on it, which is desired outcome
vs hard crashing.
this also prevents crashes in cuda enabled builds when user explicitly
passes '-dev none'
* [Vulkan] Fixes llama-cli breaking over longer promts sizes
The llama-cli was breaking for longer promts sizes for q4_0 quantized networks. Causing due to insufficient shared memory.
* Removed the un-used Adreno device
* Updated matmul for small pipeline.
* server: accept null sampling params
Extend the schema validation to treat a null value as absent, so
clients can send null on nullable params (temperature, top_p, ...)
to request the server default. This matches the OpenAI spec and the
json_value convention used elsewhere.
Add has_field() to skip null in the field eval guards.
* has_field -> has_value
llama_meta_device_get_split_state() recompiled 29 std::regex on every call.
In -sm tensor mode the callback runs once per tensor per token, so this
dominated the decode thread in profiling. Mark them static const so they are
compiled once. Kept inside the function (local statics are thread-safe since
C++11). Patterns are literal and stateless, so behavior is unchanged.
* hex-sort: add efficient bitomic sort in hvx regs up to 1024 elements
* hex-sort: fix inverted vrors
* hex-sort: specialize sort functions for the common cases
* hex-sort: add tracing and local context
* mtmd: deepseek-ocr v1 multi-tile dynamic resolution + unified image-preprocessors for both versions (ds-ocr v1 and v2)
* remove hacky API
* fuse row into a long image
* almost working
* adapt to new preprocessor api
* rm debugging printf
* improve
* mtmd: dsocr-tiles fixes (#25481)
* ds-ocr img-preproc fuse_row tile-drop fix for multi rows and columns images
* mtmd drop the duplicate redundant img_end
* deepseekocr graph simplify CLS broadcast cleanup
* test-deepseek-ocr: relax v1 single-view tolerance; drop trailing prompt space; make DRY opt-in and n_predict model-specific (#25486)
---------
Co-authored-by: Saba Fallah <10401143+sfallah@users.noreply.github.com>
Co-authored-by: Saba Fallah <sabafallah@gmail.com>
* llama-cli: fix crash on wrong server base url by catching exceptions and graceful exit
* review: leaner catch group: json error and standard exception
downloadConversation serialized activeMessages, the root -> currNode
path, so exporting a conversation with edited or regenerated messages
dropped every alternate version and kept only the selected one.
Fetch the whole message tree via getConversationMessages so the export
carries all message versions, matching the multi-conversation export
path which already did this. Keep the active conversation as the header
source to preserve an up-to-date currNode.
Forks are separate conversations, each with its own convId, and are
exported on their own.
* llama : make all KQ masks (except the lightning indexer one) f16 if FA is used and remove zero attention bias in DeepSeek V4
* llama : remove dead code that repeats unified raw_k cache for each stream in DeepSeek V4 - no longer needed as raw_k is always non-unified.
---------
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
* ggml : process data in smaller chunks in CUDA ggml_top_k() implementation to reduce temporary buffers memory usage
* ggml : allocate tmp_dst only only once before the loop
* chore : whitespaces
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* ggml : use chunked processing in both CUDA CUB top-k and argsort implementations
* chore : separate argsort_f32_i32_cuda_bitonic() call from return statement
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* chore : replace ternary operators with min/max
---------
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
A model whose chat template parses at init but fails parser generation
at apply time (e.g. uses {% call %}) throws std::invalid_argument from
common_chat_templates_support_enable_thinking(), which ran outside the
try/catch guarding common_chat_templates_init(). The throw was uncaught
and llama-cli aborted (SIGABRT) instead of failing to load. Moved the
probe inside that try/catch so an apply-time error fails load the same
way an init parse error does.
Signed-off-by: Jesse LaRose <jesse@taey.ai>
* meta: add hard emphasis on agents not writing descriptions/comments
Add a block in AGENTS.md to emphasize that agents are forbidden, under any circumstances, to post comments or pull request descriptions on behalf of the user.
* Add example
* Move examples to examples
* White space
* Use smart pointers in test_case::eval
This makes it consistent with other methods of `test_case`.
* Use smart pointer in show_test_coverage also
* Also use smart pointers for backends
* metal : add CONV_2D_DW (depthwise 2D convolution) support
* test : add perf cases for CONV_2D_DW
* metal : use 3D dispatch for CONV_2D_DW kernel
* metal : add channel-tiled CONV_2D_DW kernel for non-contiguous layouts
* metal : simplify CONV_2D_DW dispatch and trim comments
* metal : merge duplicate CONV_2D_DW pipeline getters
* tests : add F16 CONV2D_DW tests
* cpu : fix F16 kernel support for CONV_2D_DW
* tests : remove commented-out CONV_2D_DW test block
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
CUDA is compiled with fast math and AMD/HIP is not — this flag lets AMD use fast math too.
We can't use -ffast-math: it implies -ffinite-math-only, which won't compile (ggml uses INFINITY for masking) and produces NaNs. -funsafe-math-optimizations gives the speedup without the NaN problems.
Co-authored-by: Mark Caldwell <mark@cloudhands.ai>
* cuda: fix snake fusion type predicate, a and inv_b are F32
The matcher required a->type == x->type while launch_snake reads both
as const float *, matching the CPU and Metal contract where a and inv_b
stay F32. F16/BF16 chains never fused and fell back to the naive path,
and a hypothetical all F16 chain would have read F16 bits as float.
Aligns the predicate and the comment with ggml-cpu.c
* cuda: reject snake fusion on non-contiguous operands
The kernel reads x[idx] and a[c] / inv_b[c] linearly, so a
non-contiguous view passing the matcher would silently read wrong data.
Mirror the contiguity guard already present in the CPU, Vulkan and
Metal matchers.
