41 Commits

Include renames

Author	SHA1	Message	Date
Thireus ☠andGitHub	6d78a87c4c	perplexity: signal-driven hot-swap mode for persistent per-tensor PPL/KLD benchmarking (extends #1989) (#2131) ... * perplexity: add signal-driven hot-swap mode for persistent KLD/PPL benchmarking llama-perplexity can now stay resident and be driven through control/status files (reload/compute/exit): it reloads only the tensors that changed on disk and recomputes PPL/KLD without ever reloading the full model. File-based signalling works on Windows, macOS and Linux. The reload returning-to-original path now refreshes tensor data from disk instead of reattaching stale weights. * Not Cygwin specific	2026-07-14 12:56:03 +03:00
3 people	b90939934a	model: add openPangu-2.0-Flash (92B-A6B) with MLA-latent cache, DSA/SWA, mHC, and multi-head MTP (#2065) ... * openpangu: Stage-1 converter probe for openPangu-2.0-Flash Add OpenPanguV2ForCausalLM conversion support (converter-only; runtime graph is Stage-2). Registers a new LLM_ARCH_OPENPANGU on the Python/gguf-py side: - gguf-py/constants.py: MODEL_ARCH.OPENPANGU + name, indexer KV keys, 22 new tensor enums (DSA indexer x4, MoME convs x3, param-sink x2, mHC/Hyper- Connections x12, block-post-norm), and the full MODEL_TENSORS list reusing the deepseek MLA + MoE + NextN bricks. - tensor_mapping.py: arch-specific block mappings that disambiguate the sandwich norms (post_attention/pre_mlp/post_mlp) and pin every Pangu-only tensor; non-block global mHC merge module. - convert_hf_to_gguf.py: OpenPanguV2Model (subclasses DeepseekV2Model) with set_gguf_parameters (MLA/MoE/indexer/mHC/param-sink/DSA+SWA metadata), modify_tensors (expert merge, kv_b split, no MTP skip), and the OpenPanguV2Tokenizer pre-tokenizer hash. Validated offline against the real 50-shard safetensors index: all 37,587 tensors map to a GGUF target (0 unmapped), and set_gguf_parameters reads only hparams present in config.json. No weights downloaded; no GPU. Pinned on the ik/dsa_loop_hadamard_blend DSA substrate. * openpangu: Stage-2 arch scaffold (LLM_ARCH_OPENPANGU) — loadable, compiles New arch on main (DSA-decoupled). Declares openPangu-2.0-Flash to the runtime so the model loads into memory; the compute graph is the next step. - llama-arch.{h,cpp}: LLM_ARCH_OPENPANGU + name; 3 KV keys (mhc_num_stream, mhc_recur_norm, param_sink_number); 18 tensor enums (mHC x12, MoME conv x3, param-sink x2, block-post-norm). - llama-model.cpp: OPENPANGU tensor-name block, strings matched to the converter. - llama-model.h: layer + model struct fields (mHC / conv / sink / block-post / merge). - llama-hparams.{h,cpp}: reader (MLA + MoE + sigmoid gate + indexer + mHC + param-sink + NextN); n_layer_kv_from_start = n_layer - nextn (MTP skipped). - llama-load-tensors.cpp: create_openpangu_tensors (GLM-DSA MLA/MoE base + Pangu tensors; indexer loaded-but-unused for dense fallback); dispatch + is_mla_attn. Builds clean (CPU-only libllama). Dense-fallback design: no DSA indexer / SWA windowing / MTP for first generation (exact <=512 tokens). Graph is Stage-2b. * openpangu: fix compresskv_conv dim (kv_lora_rank, not +rope); pin attention order in spec * openpangu: end-to-end runtime — build_openpangu graph runs, generates (garbled) First full forward pass of openPangu-2.0-Flash on ik_llama. Pipeline works end to end: new LLM_ARCH_OPENPANGU loads the Q4 GGUF, the graph executes, and llama-cli generates 40 tokens (EXIT=0). Output is currently garbled (tensor-layout bug to debug), but the structure is proven. graphs/build_openpangu.cpp: dense decompressed-MHA attention + 4-stream mHC (Hyper-Connections) with 20-iter Sinkhorn + MoE(sigmoid+shared) + sandwich norms + entry stream-repeat/tail-merge + inp_out_ids selection. Bring-up fixes to load+run: - llama-vocab.cpp: register 'openpangu' pre-tokenizer (QWEN2 family) - llama.cpp: OPENPANGU -> LLAMA_ROPE_TYPE_NORM (was defaulting to NONE=-1) - llama-load-tensors.cpp: full wkv_b load; k_b/v_b as flattened 2D; block_post_norm dim = S*H (10240); conv weights 2D {3,C}; mHC alpha/beta/gamma + param_sink + merge params use bare (no-.weight) tensor names - llama-model.cpp: OPENPANGU is NOT is_mla_attn (decompressed MHA, standard KV cache) - graph loop bounded to base layers (skip NextN/MTP) v0 deferrals (need conv-state cache / manual attention path, all documented): MoME convs (passthrough), o_conv, param_sink. Next: fix the layout bug to coherence. * openpangu: COHERENT generation — NEOX rope, Sinkhorn orientation, MoME convs, param_sink Four correctness fixes on top of the end-to-end scaffold, verified checkpoint-by- checkpoint against a Python golden reference running on the GGUF's own dequantized weights (block-0 activations now match to rounding at full fidelity): - rope: NORM -> NEOX. Pangu config rope_interleave=false; the Infer source maps it as is_neox_style = not rope_interleave (rotary_mode='half'). - mHC Sinkhorn: the flat h_res block is torch-[r,c] row-major, so a bare ggml reshape lands column-fastest; the doubly-stochastic iteration ran transposed (Sinkhorn is not transpose-symmetric). One transpose at input fixes the whole chain including the mhc_post application. - MoME convs (qa/compresskv/o): were passthrough stubs. Implemented as out = x + causal_conv1d(x) (every Infer call site uses residual_connection=1; tap stats confirm the perturbation form). Taps cast f16->f32 for ggml_mul. Batch-local v0: exact for fresh-sequence prefill; decode steps miss the t-1/t-2 taps until a conv-state cache exists. - param_sink: 128 learned latent-KV entries prepended per layer via a manual attention path (kv_store + explicit soft_max over [sinks ++ cache]); huge effect at short context. o_conv now applied pre-o_proj on the same path. flash_attn forced off for OPENPANGU (FA kernel cannot see the sinks). - converter: add_bos_token=true (HF prepends <|pangu_text_start|> via the post-processor; the key was absent so ik dropped BOS). Greedy Q4_K_M smoke, chat template + <think>: coherent CoT reasoning and a correct answer. Layer-0 instrumentation (opg0_* names) kept for now. * openpangu: MoME conv-state cache — decode steps get real t-1/t-2 taps Allocate a per-layer cache_s_l tensor for OPENPANGU base layers holding the last two pre-conv latents of the three MoME sites, packed [qa 2*1024 | compresskv 2*512 | o 2*6144] f32 (~60KB/layer). The conv helper reads the [C,2] history window (zeros at sequence start, kv_head==0), builds xx = [hist ++ x], and writes the last two columns back each ubatch — the concat naturally handles both prefill chaining and the T==1 shift. Read precedes write in graph order; the fixed-offset copy is graph-reuse safe. Verified: prefill anchors unchanged (bit-path identical, zero-history branch); -ub 1 token-by-token run matches the golden reference at t4 (qlora_conv 0.084, R_block 0.008 rel; attn_out 0.15 on one channel = f16 KV-cache rounding, washes out by post-norm); final logits differ from full-batch only by a common-mode shift that softmax cancels. Chat-template greedy smoke: think-block repetition is gone — clean structured CoT and correct answer. v0 limits documented in the helper: one state slot (single sequence); cache rewinds leave the state stale. * openpangu: NextN/MTP speculative decoding — 1.7-1.8x TG on CPU Wire the three NextN layers (46-48) into ik's MTP speculative framework (--spec-type mtp). v0 drafts with head 1 (layer 46), self-chained by the framework. - llama.cpp: add OPENPANGU to the cparams.mtp arch allowlist (it was silently zeroed, which left the target context without a logits buffer once the server enabled embeddings -> GGML_ASSERT(lctx.logits) in speculative_is_compat). - load-tensors: MTP layers carry no mHC tensors (tail_use_mhc=false in the reference) — create them only for base layers. nextn.* tensors were already wired by the Stage-1 probe. - build_openpangu: extract the attention sublayer into build_openpangu_attention (shared base/MTP); add build_openpangu_mtp: eh_proj(cat(enorm(embed), hnorm(prev_hidden))) -> one plain-residual Pangu block (sandwich norms, convs+param_sink, MoE+shexp, no mHC/block_post_norm) -> shared_head norm+head. MTP branch returns the draft graph when mtp_op_type != NONE; main graph keeps all-token outputs under cparams.mtp. MTP convs run batch-local (no conv-state slot) — affects acceptance only. A/B (Q4_K_M, CPU, greedy, 192-token chat CoT completion, warm back-to-back, medians of 3, bracketed B/A/B): no-spec: 2.44 t/s (2.34-2.86) --spec-type mtp:n_max=3: 4.23 / 4.49 t/s (brackets) => ~1.7-1.8x Draft acceptance 34% on CoT prose (46% on repetitive text); spec and no-spec greedy outputs are byte-identical. Headroom: conv-state for MTP drafts, n_max tuning, true 3-head chaining (spec_step_idx). * server: include draft_n/draft_n_accepted in /completion timings get_formated_timings() (the /completion path) omitted the speculative counters that get_timings() (the OAI path) already reports; add them, guarded by n_draft_total > 0 like the OAI path. * openpangu: position-indexed MoME conv-state ring — rollback-safe spec decoding + MTP draft chaining The v0 single-slot conv state held the last-2 pre-conv latents of the most recent batch, so any speculative draft rejection left latents of REJECTED positions in the state and every later decode ran with wrong t-1/t-2 taps (3 conv sites x 46 layers). At 192-token greedy runs every spec config diverged from no-spec, each differently (rejection-pattern dependent). Replace it with a per-layer ring cache_s_l [n_lora_q+n_lora_kv+n_head*v_dim, 16]: column pos%16 holds position pos's pre-conv latents ([qa|ckv|o] packed). Invariant: reads target only positions before the first batch token, which are committed, and committed latents depend only on the committed prefix - rollback-safe by construction, no checkpointing. Writes cover the last min(T,16) batch positions in <=2 contiguous cpy segments; the copy sources are views of the [hist ++ x] concat so the history read is an ancestor of every write (read-before-write by graph dependency). The ring is also allocated for the NextN/MTP layers, so the draft head chains real conv taps across WARMUP -> sequential DRAFT_GEN steps (was batch-local zero-history per draft token). graph_reuse is forced off for the arch: ring view offsets are position- baked and the reuse patcher only updates the standard K/V-store copies. Measured cost on the CPU server path: none visible. ggml_set_rows driven by an input index tensor is the future reuse-safe shape. Verified (Q4_K_M, CPU, greedy 192-tok chat-CoT, warm single process): - no-spec output byte-identical to pre-ring build - spec output byte-identical to no-spec below the n_predict cap, for all of n_max in {1,2,3,4,6} x p_min in {0,0.3,0.6} (old build: all diverged) - acceptance n3-p0: 33.9% -> 60.9%; n3-p0.3: 58.1% -> 68.9% - TG medians: no-spec 3.19-3.32 t/s; mtp:n_max=3,p_min=0.3 6.97 t/s (~2.1x) * openpangu: DSA lightning indexer + SWA schedule — long-context correctness past the dense fallback The dense fallback was exact only <=512 tokens (SWA window). This wires the real DSA/SWA hybrid schedule, self-contained from GGUF keys the converter already writes (openpangu.swa_layers + sliding_window_list; absent keys keep the old dense fallback): - SWA layers (30 base @512): the generic inp_KQ_mask_swa path, per-layer mask choice in the builder. The NextN/MTP layers are SWA @2048 in the checkpoint schedule; MTP graphs run in their own context, so the mask fill picks hparams.n_swa_mtp when built with an MTP op type. - DSA layers (16, every 3rd): lightning indexer implemented in-graph from the Infer reference semantics (jointfix _pangu_torch_calib): q_idx = wq_b on the post-conv post-norm q-lora latent (24x128), k_idx = rms-normed wk(x) shared across heads, both NEOX-roped on the FIRST n_rot channels; score = sum_g w_g * relu(q_g . k) in f32, causal-masked, exact top-k via argsort + ggml_set_rows scatter into a -1e30 base -> additive selection mask on the existing manual soft_max seam. Selection engages only when the causal window exceeds index_top_k (2048); below that the layer is exactly dense. - Indexer keys cached per position (cache_idx_l, f32 [128, kv_size], DSA layers only) with the same committed-position invariant as the conv-state ring, so speculative rollbacks stay safe. - param sinks remain outside both the window and the selection budget, matching the reference. Verified (Q4_K_M, CPU): - <=512 tokens: byte-identical to the dense build (96/160-token greedy) - indexer scores vs a GGUF-dequant golden reference at 2101 tokens: 1e-3 rel (f16 weight rounding); top-3 selection indices exact on all compared queries - >512 coherence clean; 3.4K-token needle retrieval through active selection (needle outside every SWA window, ~1300 positions pruned) answers exactly * openpangu: MLA-latent KV cache — attention absorbed into the 512-latent, 14x smaller cache, ~2.2x TG Store per position only [ckv_norm 512 | roped k_pe 64] (f32, k_l) plus the transposed 512-latent (f32, v_l, v_trans layout); per-head K/V are never materialized. q_nope is absorbed through attn_k_b (loaded 2D from the converter split for base layers; derived at load via llm_prepare_mla for the NextN layers - now guarded for layers without attention weights, e.g. the idle NextN heads 2/3). The value side is the latent itself, up-projected through attn_v_b after the weighted sum, matching the Infer _forward_dsa reference. param sinks are native latent-space entries, which removes the per-step full-cache concat+cast that dominated long-context decode. llama_state row sizes now come from llama_kv_k_row_embd/llama_kv_v_row_embd (arch-aware), fixing an out-of-bounds crash in the server prompt-cache save path (hparams-derived 9216-wide rows vs actual 576-wide latent rows). Verified (Q4_K_M, CPU): layer-0 attention output matches an f32 golden reference computed from the same GGUF weight encodings (~1e-2 on O(1) values); MTP spec output byte-identical to no-spec; 3.4K needle retrieval through active DSA selection exact under greedy. Output differs from the materialized build at the token level because attn_k_b/attn_v_b are independently quantized tensors - both are legitimate Q4-fidelity encodings. Perf (CPU, warm): no-spec TG 3.2-3.3 -> 6.9-7.1 t/s; mtp:n_max=3,p_min=0.3 -> 11.1 t/s (byte-exact, 67% acceptance); prefill 30.5 t/s at 3.4K; KV self size at 4K ctx: 5.5 GiB -> 391 MiB. Not yet supported on the latent cache: K-shift/defrag (context shifting) - unreached in current usage. * openpangu: fence unsupported serving modes, truth-pass comments, drop dead weight/keys Post-audit hardening. The cache's position-indexed side state (MoME conv ring, DSA indexer keys) made several generic serving paths silently unsound; they are now fenced loudly instead of documented as unsupported: - s_l_position_ring flag on llama_kv_cache: the qnext-state predicate no longer claims the conv ring, so per-seq state save, seq_cp and the s_copy graph skip it - state save/restore refused for the arch at every llama_state_* entry (the ring and idx_l are not in the state format; restoring without them diverges silently) - K-shift/self-extend assert, defrag skips with a warning, server ctx_shift off via new llama_model_supports_ctx_shift() - single sequence enforced at context creation (n_seq_max > 1 refused) - server prompt-cache reuse limited to pure extension via new llama_model_supports_partial_kv_reuse(): mid-cache divergence reprocesses from scratch (the 16-column ring cannot rewind); multi-turn continuation stays fast - MTP draft length clamped to 13 via new llama_model_max_draft_tokens() so a rejected draft can never overwrite the ring columns the next decode reads - K/V cache types forced to f32 for the arch so the KV size log reports the truth - cache_size(): real latent-cache branch (was falling through to the ~14x larger materialized estimate used for offload planning) - unused fused wkv_b no longer loaded (TENSOR_SKIP; the graph runs entirely on the pre-split k_b/v_b), llm_prepare_mla openPangu special-case removed (it was a no-op) - stale v0 comments rewritten to describe the shipped graph; converter stops writing dead keys (dsa_layers, block_post_layernorm_idx) and the tokenizer pre-hash is registered in convert_hf_to_gguf_update.py Gates on this build: greedy spec output byte-identical to no-spec (EOS-terminated, sha-equal); 3.4K needle retrieved exactly; -np 2 / state save / n_max=20 / stale prefix reuse all refused or clamped with clear messages. * openpangu: assert kv_head == first batch position at graph build The ring, indexer and latent stores are addressed by absolute position through kv_head; the fences make append-only decode the only reachable mode, but the invariant was unchecked. Assert it at both graph entries (base and MTP) so any future cache plumbing that breaks it fails at build instead of corrupting output. Worst-case measurement builds pass pos = null and are exempt. * openpangu: cont h_pre before the mHC broadcast mul (CUDA binbcast misreads strided views) h_pre is a row-slice view of the fused mixes tensor. The CPU mul handles the strides; the CUDA broadcast path reads the view as if contiguous, so token 0 mixes correctly and every later token gets h_post/h_res rows instead. First divergent node in the whole graph (oracle rel 0.36 at opg0_attn_mhcpre_x, fixed to 7.5e-5). Sibling views h_post/h_res were already cont-wrapped, which is why only h_pre was exposed. * openpangu: keep DSA zero-trick sources finite (CUDA clamp propagates the 0*(-inf) NaN) The selection-mask base and zeros were built by scaling the MASKED scores by zero, but post-mask sc contains -inf and 0 * -inf = NaN. The CPU clamp launders NaN back to -1e30 (fminf/fmaxf ignore NaN); the CUDA clamp propagates it, so every DSA layer emitted NaN masks at n_kv > top_k and logits collapsed (observed: eval-callback CLAMP sum -1.3e36 on CPU vs nan on CUDA, 11748 NaNs downstream). Scale the pre-mask finite scores instead, which is correct on any backend regardless of clamp NaN semantics. Also defensively cont the strided KQ_mask slice feeding the score add (same strided-view kernel class as the mHC h_pre fix; unproven here but cheap). Gates after fix: 2600-token probe coherent, 3.4K needle exact ('7391') with and without MTP speculation, PP ~120 t/s. * openpangu: f16 latent KV cache option (explicit -ctk/-ctv f16 halves cache memory, f32 stays default) Track explicit cache-type requests through CLI/env; openPangu resolves no-request to f32 (unchanged), accepts explicit f32/f16, warns and falls back to f32 for BF16/quantized. Sink and cached-token KQ paths stay separate until after KQ so the latent cache is read directly without the f32-only concat; value is the sum of the sink and cache matmuls. Ring and DSA indexer caches stay f32; cache_size() follows the resolved types. * openpangu: enable graph reuse * openpangu: wire multi-head MTP drafting * openpangu: add MTP heads override * openpangu: keep MTP update logits last * openpangu: scope MTP warmup heads * speculative: apply per-request MTP heads before warmup * openpangu: fix multi-head MTP warmup computing on unwritten inputs Each chained head called the build_inp_* helpers itself, so the warmup and update graphs held one inp_tokens/inp_pos/inp_out_ids/KQ_mask tensor per head while llama_set_inputs only fills the tensors the lctx pointers reference, i.e. the last head's copies. Every head but the last read unwritten compute-buffer memory: with heads=3 active even head 1's ring, latent cache, and cached one-token draft were computed from garbage, which is why depth-1 acceptance measured 4% against 98% for the heads=1 control. Create the batch inputs once in build_openpangu and pass them to every build_openpangu_mtp call, and fix the two chaining errors that were hiding behind the garbage inputs: - Shift the chained hidden: head k+1's row at position p consumes head k's output row at p-1, the same convention head 1 uses for the target's conditioned hidden rows. The predecessor of a batch's first row lives in the previous warmup/update, carried across decodes through a new inp_mtp_carry input backed by lctx.mtp_carry (written back per ubatch, zeroed when a prompt warmup restarts from position 0). - Fill head 3's cache row at draft step 2: each draft step runs one head, so head 3's own decode at step 3 attended over a never-written row at the step-2 position. Pre-write it from the committed carry. Also include the active head count in the graph-reuse key next to the existing step index (reuse stays forced off for this arch). * speculative: default MTP drafting to a single head A stage without an explicit heads= override previously resolved to 0, meaning all model heads, so multi-head drafting was silently on by default for models that carry more than one NextN layer. Keep it opt-in (heads=N or heads=0 for all) until multi-head measures a win over the single-head config; single-head models are unaffected either way. * speculative: fence MTP head upshift over a warmed prefix Deeper NextN heads only hold valid cache rows for spans that were warmed with them. A request drafting with more MTP heads than the cached prefix was warmed with (e.g. a heads=1 conversation continued with heads=3, a pure extension the divergence fence deliberately allows) would read never-written deeper-head rows: verification keeps the output correct, but acceptance quietly collapses and any measurement taken there is misleading. Track the minimum head count the committed context has been warmed with since position 0 and have the server reprocess from scratch when a request asks for more. Also announce the model's NextN head count and the single-head default once at MTP context setup. * openpangu: skip dead MTP chain compute and stall-free carry readback The update chain's last head and the draft-time row fill only matter for their latent-cache and conv-ring writes; their FFN, norms, and shared head fed nothing. Add a cache-writes-only mode to the MTP block builder that returns after the attention block, and use it at both sites. The multi-head carry readback previously synchronized the scheduler after every warmup/update decode, a hard stall on CUDA. Issue the device-to-host copy async on the backend stream instead (stream order protects the source buffer from later graphs) and synchronize lazily when the host buffer is next consumed or resized. * openpangu: stop emitting fused kv_b tensor * openpangu: default latent cache to f16 * openpangu: refuse unsupported latent cache types * Window OpenPangu SWA cache reads * Gather OpenPangu DSA decode reads Gather DSA decode attention over the selected latent rows for OpenPangu base-model decode and verify graphs. The gathered branch now uses ggml_top_k order directly, runs maskless softmax over sinks plus selected rows for T <= 14, and derives values from the gathered k_l rows instead of the transposed latent cache. * Chunk OpenPangu indexer prefill scoring * Chunk OpenPangu prefill attention * Gather OpenPangu sparse prefill attention * Drop OpenPangu value cache * Add OpenPangu indexer cache type flag * Add OpenPangu q8_0 latent cache type Store the OpenPangu MLA latent K cache as q8_0 via -ctk q8_0 (about 0.53x of f16); the default stays f16 so behavior is unchanged without the flag. Latent V stays f16/f32. The q8 latent cache is a storage format only: it is dequanted to F32 before all compute. K reads go through openpangu_build_k_latent_for_read, V derivation through openpangu_build_v_latent_from_k (full 576-wide row to F32, then slice), and the DSA gather paths already dequant via get_rows. Feeding a q8 latent view directly into the KQ mul_mat corrupts large-context prefill, so that path is removed for quantized caches. The cache write stages ckv and kpe through F32 and writes one full 576-wide q8 row per token. Verified on a small discriminator model: the default f16 path is byte-identical to the prior code; the first-DSA-layer attention envelope is within 0.6% of the f16 cache (linf_rel 0.0057); top-k selection is bit-identical between cache types; the q8 latent cache is 0.531x the f16 size at 8K and 32K context; and generation stays coherent on both the dense and DSA-gather paths at all tested context lengths. * Remove OpenPangu debug trace env knobs and redundant DSA_TOPK override Drop the five LLAMA_OPENPANGU_*_TRACE debug-logging knobs (DSA_GATHER_TRACE, IDX_CHUNK_TRACE, ATT_CHUNK_TRACE, PREFILL_GATHER_TRACE, SWA_WINDOW_TRACE) and the LLAMA_OPENPANGU_DSA_TOPK override, which duplicated the -dsatk / --dsa-top-k CLI flag; top-k now comes solely from cparams.dsa_top_k. The five perf-tuning knobs (DSA_GATHER, IDX_CHUNK, ATT_CHUNK, ATT_KQ_MAX_MIB, PREFILL_GATHER) are retained pending the perf battery. No change to default behavior. * Subchunk OpenPangu DSA prefill gather to fit CUDA grid limit The prefill gathered-attention ggml_get_rows produced dst rows = topk * token_chunk (2048 * 256 = 524288) mapped to the CUDA grid.y dimension, which caps at 65535, crashing with GET_ROWS invalid argument at long context (N_KV around 10.5K with the natural topk of 2048). Split the prefill gather into token subchunks so topk * subchunk_tokens stays within the grid limit, and guard the decode gather with the same fit check (falling back to the dense masked path if a pathological topk would not fit). The subchunking is over the token dimension only, so per-token attention is unchanged and the result is numerically identical. Verified: the GPU sweep runs past the old crash boundary to 22K+ with zero CUDA errors; CPU and -ctk q8_0 paths unaffected. * openpangu: fix scheduler node budget for chunked DSA prefill; drop unused attn_kv_b; remove env tunables - Size the scheduler graph node budget for the chunked DSA prefill so 32K/ub2048 no longer trips the hash-set reservation assert; derive the extra budget from the builder's chunk/top-k/window structure with a fixed safety margin. - Remove LLAMA_OPENPANGU_* environment tunables from both the node-budget estimator and build_openpangu.cpp; use fixed constants in both so they stay in sync. - Converter: emit only the split attn_k_b/attn_v_b projections and drop the unused fused attn_kv_b tensor. * openpangu: restore DeepSeek converter kv_b; drop trace env + dead code; fix dense-fallback node budget - convert_hf_to_gguf.py: restore fused attn_kv_b in DeepseekV2Model (shared parent); openPangu subclass keeps split-only k_b/v_b. Stops newly-converted DeepSeek GGUFs from failing to load. - src/llama.cpp: remove LLAMA_GRAPH_REUSE_TRACE getenv, hit/miss counters, and the unconditional destructor log (no getenv or behavior change for any arch); node-budget estimator now covers the dense-fallback (n_swa==0) attention-chunk loop while skipping absent idx/top-k terms, preserving a strict overcount; remove unreachable openPangu split-cache block. - src/llama-context.h: drop now-dead graph_reuse_hits/misses members. - include/llama.h: move type_k/type_v/idx_type_k *_explicit bools to struct end to avoid a mid-struct ABI shift for out-of-tree consumers. - src/graphs/build_openpangu.cpp: replace vestigial env-struct singletons with the OPENPANGU_* constants; drop a redundant Sinkhorn permute round-trip (one transpose; greedy output verified byte-identical). Decode output unchanged (byte-identical greedy generation verified); shared-file changes are openPangu-gated or restore the pre-PR baseline. * openpangu: chat-parser support (reasoning split + thinking toggle) Two openPangu-only fixes, both gated on the arch-unique token <|pangu_text_start|> so no other model's parsing changes. - chat-diff-analyzer: add a workarounds entry that force-sets TAG_BASED reasoning with an empty start and a </think> end. openPangu prefills <think> in the generation prompt, so the output is delimited only by </think>; the differential detector otherwise learns start="<think>" from the assistant-history form and fails to split, leaking reasoning into content. Same shape as the existing Laguna prefill patch. - chat.cpp: bridge enable_thinking to the template's `thinking` variable. openPangu's template gates reasoning on `thinking` rather than the ecosystem-standard `enable_thinking`, so the standard toggle was inert. An explicit `thinking` chat_template_kwarg still overrides via the extra_context merge. Blast radius: test-chat-auto-parser 437/437 unchanged; the sole test-chat-template diff is a pre-existing GLM trailing-newline. * openpangu: use ggml_cast for latent dequant reads Replace ggml_cpy(view, ggml_new_tensor_2d(F32, ...)) with ggml_cast in the MLA latent V-from-K and K-read helpers. ggml_cast emits the identical GGML_OP_CPY node into a fresh f32 tensor, so behavior is unchanged; it is the idiomatic form. Per review. * openpangu: narrow SWA reuse-key fields to 32-bit The openpangu_swa_window_view reuse key stored n_kv/n_tokens/window/pad as int64_t, but these are bounded well under 2^31 (window/pad are uint32_t at source; n_kv/n_tokens <= context length). Narrow to int32_t/uint32_t and drop the widening casts. w_view/win_off stay int64_t: they feed ggml view dims/offsets. Per review. * openpangu: precompute param_sink derived tensors at load The per-layer attention-sink block (sink_blk [576,NS]) and its transposed latent (s_lat_t [NS,512]) are pure functions of the layer weights, yet were rebuilt every eval across all 49 layers (RMS-norm + cast + concat + transpose). Compute them once at load, mirroring the wk_b derived-weight precompute, and read the stored tensors in build_openpangu_attention. Numerically identical; removes per-token work at decode. * openpangu: replace conv position-ring with ggml_ssm_conv + spec-rollback checkpoint Migrate the MoME depthwise causal conv (three sites per attention sublayer: qa-lora, compressed-kv, attn-out) from the bespoke 16-column position-indexed ring onto the core ggml_ssm_conv op with a recurrent conv-state slot. Cache: s_l becomes [2*conv_col_ne, qnext_state_slots], holding the (d_conv-1)=2 history taps per channel for the three sites (float offsets 0 / 2*n_lora_q / 2*(n_lora_q+n_lora_kv)). Drops the conv_hist_idx / conv_write_idx graph inputs and their fill in llama_set_inputs; adds one single-sequence sq input for ggml_ssm_conv shared across the three sites and the MTP head. Speculative rollback: the position ring self-healed rejected draft columns by absolute position; a recurrent slot does not, since seq_rm is a no-op for recurrent state. openPangu is admitted at the three spec-checkpoint save/init gates so the whole-slot shadow checkpoint (gpu-fallback) snapshots the conv slot before drafting and restores it before the accepted-token replay. The restore path is already keyed on ckpt.valid, so no gate change is needed there. Per-step checkpoint mode is declined for openPangu, which has no SSM recurrent term, so auto mode resolves to the whole-slot shadow. Gated: non-spec needle unchanged; MTP-spec needle correct with healthy draft acceptance (rollback verified via the acceptance canary). * openpangu: single ggml_concat copy for the latent cache store The non-quantized latent store split the [ckv | roped k_pe] row into two views and two cache copies, with a base_offset field on the CacheCopy struct to place the second one. Match the quantized path: concat the two parts and do one copy into the cache row. This drops the second cache-copy slot (OPENPANGU_COPY_K_KPE) and removes base_offset from CacheCopy entirely. Cache contents are unchanged: the concat writes the same [ckv 512 | k_pe 64] bytes to the same row. Gated on the needle for both the f16 latent path (the one that changed) and the q8 latent path, plus coherence. * openpangu: reuse the shared kr_l indexer cache instead of a separate idx_l The DSA lightning indexer stored its per-position keys in an openPangu-only idx_l cache, parallel to the kr_l indexer cache GLM-DSA already uses. Both have the same storage contract: [indexer_head_size, kv_size], idx_type_k dtype, one row per KV cell, written at kv_head and read [dim, n_kv] from zero. openPangu now allocates its indexer keys into kr_l and shares the dsa_cache_copies graph-reuse fixup. The fixup patch is factored into a helper that both the generic path and the openPangu update_cache_copies branch call, so the openPangu indexer copy is repointed to the current kv_head on graph reuse like every other cache write. This drops the idx_l vector, its allocation and memory accounting, and the openPangu third cache-copy slot (now one latent copy per layer). Per-arch allocation predicates stay separate (GLM uses indexer_is_full, openPangu uses the window==0 DSA schedule); only the kr_l storage and the copy fixup are shared. openPangu keeps its no-shift/no-defrag/no-state-I/O behavior, and the GLM Hadamard/k-shift logic stays GLM-gated. Gated: needle correct on f16 and q8 latent caches and under MTP speculation (acceptance unchanged at 0.67), plus coherence. * openpangu: discard pos-0 graphs from reuse; retire stale conv-state comments The ggml_ssm_conv refactor bakes the pos-0 conv-state reset into graph topology (a scale-by-zero node on the state view). A graph built at pos 0 could be reused at pos > 0 when the batch shape and padded n_kv match (a 1-token prompt followed by TG is the concrete case), zeroing the conv history on every reused decode. Admit openPangu at the existing reset_previous gate so pos-0 graphs are discarded from reuse, the same guard the qnext recurrent state relies on. Also retire the internal phase-plan comments the conv refactor left behind: they claimed the spec-checkpoint wiring had not landed in the commit that landed it, and misdescribed the s_l slot as awaiting rollback support. Gated: needle 8457 on f16 and q8 latent, MTP-spec needle (drafts fully accepted), coherence. * openpangu: drop the _explicit cache-type plumbing; validate unconditionally Review follow-up (item 1 of the second review). The explicit/default distinction carried less than claimed: the latent K/V fallback was f16, which is already the -ctk/-ctv and API default, so distinguishing unset from set-to-the-default bought nothing, and the two bools were behaviorally redundant. The only load-bearing use was the indexer cache, where openPangu defaulted to f32 while -ictk defaults to f16. Gating the f16 indexer directly (needle on f16 and q8 latent paths, MTP speculation, coherence) shows no quality difference, so openPangu now takes the standard f16 indexer default and the f32 special case is gone. Default indexer cache memory halves (64 -> 32 MiB at c 8192). Removes type_k_explicit/type_v_explicit/idx_type_k_explicit from llama.h, the cparams/mparams plumbing, and common; the resolve helpers become plain unconditional validators, so -ctk q8_0 is honored and an unsupported type errors out at load instead of silently coercing. Gated: needle 8457 on the new f16-indexer default, on q8 latent with MTP speculation, and with -ictk f32 explicitly honored (64 MiB f32 buffer in the load log); -ctk q4_0 and -ictk q4_1 refused with a clear error. * openpangu: keep MTP draft decodes position-contiguous under speculation The MTP framework's one-token draft shortcut caches a prediction one row past the accepted prefix during the accepted-token update, then skips re-decoding the last sampled token at the next draft round. A mask-addressed cache tolerates the resulting position gap; openPangu's position-addressed append-only cache (cell == position) does not: after a rollback the next draft decode lands one cell behind its position, and after a full acceptance the cache head sits one row ahead of the next draft base, either way tripping the kv_head == pos[0] invariant and aborting the server. The checkpoint admission in the conv refactor made this the standard openPangu speculative flow; the needle-first gates never generated enough draft rounds against a short prompt to reach it. Decline the shortcut re-seed for openPangu in mtp_accept_batch (restoring the drafting behavior all measured acceptance numbers were taken on) and trim rows at or beyond the draft base in mtp_speculative_gen_draft, so every draft decode stays position-contiguous with the cache head. Gated: the crashing flow (short prompt, 512-token spec generation, then a second request) completes with acceptance 0.60 prose / 0.87 code, matching the pre-checkpoint baseline profile; needle 8457 plus coherence on f16+spec and q8+spec. * openpangu: remove stale ring limits and fix MTP graph reuse * cli: preserve speculative carry on fallback Decode an already-emitted pending token when a draft cannot be used instead of sampling unchanged logits and duplicating output. Document single-head MTP as the default and multi-head modes as experimental. --------- Co-authored-by: Joel Farthing <262452229+joelfarthing@users.noreply.github.com>	2026-07-11 12:29:20 +03:00
Marian M.andGitHub	70ea15226f	Update docs (#2104) ... * Update README.md - models - dsa - tensor reload * Update parameters.md - typos - recent parameters, dsa and dflash - env var and build argument	2026-07-10 09:27:44 +03:00
magikRUKKOLAandGitHub	72440a19fc	on-demand tensor reload (#1989) ... * host-swap tensor loop the host-swap functionality is only triggered when the certain env. variables are declared * target_include_directories tweak * hot-swap tensor support two intrusions: 1.) at the model loading to collect the snapshot 2.) the modification of the `/health` HTTP endpoint to be able to trigger the hot-swap via sending the `llama-server` the HTTP-request. *both a braced by the specific env. variables * hot-swap tensor support; graph invalidation ggml_backend_cuda_invalidate_graphs export * hot-swap tensor support graph invalidation implementation; extended debug output (commented out) * llama_reload_changed_tensors export * tensor hot-swap on-demand reload cpu-only/hybrid/gpu-only with split mode layer/graph full support implementation * docs * reuse the gguf parsing from llama.cpp gguf_init_from_file, gguf_find_tensor, ggml_get_tensor * remove the manual scheduling for hybrid inference * update docs * tensor shape validation * update docs * update docs accidentally wiped the previous changes; so recovered them * revert the GGML_CUDA_MAX_DEVICES to 16 * update llama_reload_changed_tensor update llama_reload_changed_tensor, revert CMakeLists.txt * update llama_reload_changed_tensor * GGML_MAX_SRC GGML_MAX_SRC compile-time definition support * GGML_MAX_SRC GGML_MAX_SRC compile-time definition support * GGML_MAX_SRC GGML_MAX_SRC compile-time definition support * llama_reload_changed_tensor update llama_reload_changed_tensor definition * refactory move the tensor-reloading implementation to llama-reload.cpp, llama-reload-info.h; some bugfixes and code reduction * revert added back the missing newline * update docs * reload_info constructor * bugfix: cpu-only TODO: improve the working environment by compiling for multiple hardware configurations; possibly make a test pipeline * cpu-only bugfix set the fix again after unsuccessful sync with main * windows os compilation fix #include <string> * fix windows os build error C2039: 'string': is not a member of 'std' * remove dead file * implement perplexity in server * Revert "implement perplexity in server"	2026-06-22 16:36:34 +02:00
Marian M.andGitHub	5fb707d19b	Update docs (#1956) ... * Update README.md Models, MTP, fit * Update parameters.md Disclaimer, terms, new flags, graph split list.	2026-06-12 08:24:22 +02:00
3 people	71d5aa21f7	common: handle Laguna chat delimiters (#1943) ... * common: handle Laguna chat delimiters * common: limit tool parser changes to end-delimited content --------- Co-authored-by: Joel Farthing <262452229+joelfarthing@users.noreply.github.com>	2026-06-10 07:46:19 +02:00
Samuel Oliveira AlvesandGitHub	007d640098	Standardize speculative decoding arguments on the server (#1908) ... * refactor spec args * add shell-safe quoting of string-valued stage keys in speculative decoding	2026-06-04 15:44:57 +02:00
Samuel Oliveira AlvesandGitHub	f4f4b3ff26	Allow dual speculative decoding (#1789) ... * wip: test logic to use multiple specs * feat: introduce composite speculative decoding stages * handle MTP context and draft invalidation * fix: allow gemma mtp for speculative stages * fix: normalize spec stage keys * refactor: remove enable_mtp flag and improve speculative stage handling * fix: update cached text tokens handling for stage chains * feat: implement sync for external MTP after non-MTP accept	2026-05-15 10:10:40 +03:00
Marian M.andGitHub	b2e7f7f6cd	Update docs (#1800) ... * Update README.md - New model - New features * Update parameters.md - Recent new parameters	2026-05-14 08:44:58 +03:00
Henrik BerglundandGitHub	75f0ab300e	Update repository clone instructions in build.md (#1753)	2026-05-07 12:57:06 +03:00
Andrew MoryakovandGitHub	485c431b9d	docs : restructure AVX-512 build flags section, recommend GGML_AVX512_*=ON first (#1733) ... Per @ikawrakow follow-up suggestion in #1729 to "offer the original version at the beginning and note that in case that does not work, they can use GGML_ARCH_FLAGS in that way". Restructured the docs/build.md AVX-512 section so that the recommended high-level CMake options come first, with GGML_ARCH_FLAGS as the fallback for cases where the high-level options don't propagate the necessary macros (older MSVC, ARM cross-compile, exotic toolchains). Empirical confirmation that GGML_AVX512_*=ON activates HAVE_FANCY_SIMD: on MSVC 2022, the resulting compile line (read from build/.../flags.make) contains both `/arch:AVX512` (from GGML_AVX512=ON) and explicit `-D__AVX512VNNI__` / `-D__AVX512VBMI__` / `-D__AVX512BF16__` (added by the matching GGML_AVX512_*=ON options via add_compile_definitions(...) at ggml/src/CMakeLists.txt:1361-1372). The runtime banner prints `HAVE_FANCY_SIMD is defined` and `system_info: AVX512_VNNI = 1`. Also added a brief note about the separate HAVE_VNNI256 gate in iqk_config.h:52-54, which gives meaningful speedups on AVX2-only CPUs with the VNNI extension (some Alder/Raptor Lake parts). Documentation only — no code changes.	2026-05-04 15:32:38 +03:00
Andrew MoryakovandGitHub	418d60a909	docs : add AVX-512 build flags reference for Zen4 / Sapphire Rapids+ (#1729) ... The IQK quantized GEMM kernels (ggml/src/iqk/iqk_gemm_*.cpp) are gated by HAVE_FANCY_SIMD in iqk_config.h, which requires five AVX-512 macros to be defined: __AVX512F__, __AVX512VNNI__, __AVX512VL__, __AVX512BW__, __AVX512DQ__. If they are not defined, the AVX-512 quantized matmul path is skipped silently — no build warning, no runtime symptom, just lower performance than the hardware can deliver. Surprises users on Windows/MSVC where -march=native semantics are not propagated. Adds a docs/build.md section that documents: - Which macros gate which path (HAVE_FANCY_SIMD for quant GEMM, __AVX512F__ alone for f16/f32, __AVX512BF16__ for bf16, __AVXVNNI__ for AVX2+VNNI-only CPUs). - Linux/GCC: GGML_NATIVE=ON (default) handles this automatically on Zen4 / Sapphire Rapids; just verify with objdump. - Windows/MSVC and cross-compile: explicit GGML_ARCH_FLAGS with -D__AVX512* defines is required. - Note on Zen4 implementing AVX-512 as 256-bit double-pumped. Documentation only — no code changes, no behavioural changes, no new CMake options introduced.	2026-05-03 17:35:01 +03:00
4 people	e0596bf614	Autoparser - complete refactoring of parser architecture (#1376) ... * Autoparser - complete refactoring of parser architecture Autoparser: add optional argument reshuffle capability Autoparser: True streaming (#20177) * Relax atomicity constraint for nicer, more pleasent, True Streaming parsing * Whitespace * Remove redundant atomics Revert to OAI-compatible args (#20213) * Revert to OAI-compatible args * Apply workaround::func_args_not_string Fix structured outputs (#20223) * Fix structured outputs * Update common/chat-auto-parser-generator.cpp Co-authored-by: Aldehir Rojas <hello@alde.dev> --------- Co-authored-by: Aldehir Rojas <hello@alde.dev> Fix compile bug (#20203) * Fix compile bug * Update common/chat-auto-parser-helpers.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> # Conflicts: # common/chat-auto-parser-helpers.cpp common : gracefully handle incomplete output (#20191) * common : handle incomplete UTF-8 at end of input in PEG parser * cont : if reached end prematurely, emit needs_more_input to propagate partial output * cont: refactor peg parse context to add lenient flag * cont : remove partial flag, keep lenient flag PEG parser for LFM2 (#20251) * PEG parser for LFM2 * Simplify using python_value() common: map developer role to system (#20215) * Map developer role to system * Simplify common: consolidate PEG string parsers (#20263) * common : consolidate PEG string parsers * cont : fix json_string_content() examples : fix empty items in json_schema_to_grammar.py [no ci] (#19968) * Fix logic for retrieving schema items in `json_schema_to_grammar.py` If `schema['items']` is `{}` and `prefixItems not in schema', as `{}` is Falsy, the original code here will raise an error. I think if `schema['items']` is `{}`, them items should just be `{}` * Apply suggestion from @CISC Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Add tests for arrays with empty items Add two unit tests to `tests/test-json-schema-to-grammar.cpp` that validate handling of arrays when 'items' is an empty schema and when 'prefixItems' is present alongside an empty 'items'. Both tests expect the same generated grammar, ensuring the JSON Schema->grammar conversion treats an empty 'items' schema (and the presence of 'prefixItems') correctly and covering this edge case. --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> Reduce level of content parser warning message to avoid log spam on non-debug verbosity (#20347) do not return if template parse failed add arg to enable parallel tool call common : fix incorrect uses of stoul (#20313) # Conflicts: # common/arg.cpp # src/llama-grammar.cpp examples : fix empty items in json_schema_to_grammar.py [no ci] (#19968) * Fix logic for retrieving schema items in `json_schema_to_grammar.py` If `schema['items']` is `{}` and `prefixItems not in schema', as `{}` is Falsy, the original code here will raise an error. I think if `schema['items']` is `{}`, them items should just be `{}` * Apply suggestion from @CISC Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Add tests for arrays with empty items Add two unit tests to `tests/test-json-schema-to-grammar.cpp` that validate handling of arrays when 'items' is an empty schema and when 'prefixItems' is present alongside an empty 'items'. Both tests expect the same generated grammar, ensuring the JSON Schema->grammar conversion treats an empty 'items' schema (and the presence of 'prefixItems') correctly and covering this edge case. --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> Add support for MiroThinker with new jinja template common/parser: handle reasoning budget (#20297) * v1 * Finished! * Handlie cli * Reasoning sampler * Apply suggestions from code review Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Less explosive terminology :) * Add utf-8 case and tests * common : migrate reasoning budget sampler to common * cont : clean up * cont : expose state and allow passing as initial state * cont : remove unused imports * cont : update state machine doc string --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> Co-authored-by: Alde Rojas <hello@alde.dev> common/parser: use nlohmann::ordered_json to preserve parameter order (#20385) common/parser: add GigaChatV3/3.1 models support (#19931) Co-authored-by: Mishusha <pmv26021975@gmail.com> common/parser: gracefully handle undetected tool parser, print error message. (#20286) fix: prevent nullptr dereference (#20552) common : fix iterator::end() dereference (#20445) # Conflicts: # common/regex-partial.cpp jinja : add capability check for object args (#20612) common/parser: add `--skip-chat-parsing` to force a pure content parser. (#20289) * Add `--force-pure-content` to force a pure content parser. * Update common/arg.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> common : rework gpt-oss parser (#20393) * common : rework gpt-oss parser * cont : fix gpt-oss tests * cont : add structured output test * cont : rename final to final_msg common : fix gpt-oss content removal (#20745) common/parser: add proper reasoning tag prefill reading (#20424) * Implement proper prefill extraction * Refactor cli parameters, update docs, move reasoning budget sampler part to common/reasoning-budget.cpp * Update tools/server/server-task.cpp * refactor: move grammars to variant, remove grammar_external, handle exception internally * Make code less C++y Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> chat : handle tool calls with no required args in TAG_WITH_TAGGED format (#20764) * chat : handle tool calls with no required args in TAG_WITH_TAGGED format * Update tests/test-chat.cpp [no ci] Co-authored-by: Aldehir Rojas <hello@alde.dev> --------- Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com> Co-authored-by: Aldehir Rojas <hello@alde.dev> common/parser : fix out_of_range crash in throw path (#20424 regression) (#20777) * chat : fix out_of_range crash in throw path (#20424 regression) #20424 introduced effective_input = generation_prompt + input, but the throw path uses input.substr(result.end) where result.end is a position within effective_input. Every thinking model with a non-empty generation_prompt crashes with std::out_of_range instead of the intended error message. Test crashes on unpatched master, passes with fix: cmake -B build -DLLAMA_BUILD_TESTS=ON -DLLAMA_BUILD_TOOLS=OFF cmake --build build --target test-chat ./build/bin/test-chat * Update test-chat.cpp * Update test-chat.cpp * Update test-chat.cpp --------- Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com> jinja : fix heap OOB read in value equality comparison (#20782) Address GHSA-q9j6-4hhc-rq9p and GHSA-2q4c-9gq5-5vfp. The three-iterator overload of std::equal in value_array_t::equivalent() and value_object_t::equivalent() reads past the end of the shorter container when comparing arrays or objects of different lengths. Use the four-iterator overload (C++14) which checks both range lengths. Found-by: Pwno common : fix typo in debug log ('extracft' -> 'extract') (#20807) common/parser: fix nasty bug causing subtle corruption of generation prompt (#20825) jinja : refactor token advancement (#20864) * refactor token advancement * exercise sub-expressions common/autoparser : detect reasoning markers when enable_thinking changes system prompt (#20859) common : replace wrap_for_generation with a prefix convenience function and fix gpt-oss (#20912) jinja: fix macro with kwargs (#20960) * jinja: fix macro with kwargs * Apply suggestions from code review Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * fix newline problem --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> common : inhibit lazy grammar sampler while reasoning is active (#20970) * common : inhibit grammar while reasoning budget is active * cont : update force_pos in accept * cont : fix tests * cont : tweak should apply logic * cont : return early not using grammar sampler * Add tests * cont : prevent backend sampling when reasoning budget enabled * cont : fix typo --------- Co-authored-by: Piotr Wilkin <piotr.wilkin@syndatis.com> # Conflicts: # common/reasoning-budget.h # common/sampling.cpp # tools/cli/cli.cpp # tools/server/server-common.cpp # tools/server/server-task.cpp common/parser: fix reasoning whitespace bugs + extra parser tests (#21085) * fix whitespace reasoning issues + add reconstruction tests * Proper fix * fix Nemotron autoparser test expectations to include newline in marker common : add reasoning_format = none support to gpt-oss (#21094) common/json-schema: fix: handle non-capturing groups (?:...) in JSON schema pattern converter (#21124) The regex-to-grammar converter in _visit_pattern() crashes with SIGSEGV when a JSON schema "pattern" field contains a non-capturing group (?:...). Root cause: when the parser sees '(' followed by '?', it pushes a warning but does not advance past '?:'. The recursive transform() call then interprets '?' as a quantifier and calls seq.back() on an empty vector, causing undefined behavior. This commonly occurs when serving OpenAI-compatible tool calls from clients that include complex regex patterns in their JSON schemas (e.g., date validation patterns like ^(?:(?:\d\d[2468][048]|...)-02-29|...)$). The fix: - Skip '?:' after '(' to treat non-capturing groups as regular groups - For unsupported syntax (?=, ?!, etc.), skip to matching ')' safely, handling escaped characters to avoid miscounting parenthesis depth - Adjust the ')' unbalanced-parentheses check using direct char comparisons instead of substr - Add test cases for non-capturing groups (C++ only, as the JS/Python implementations do not yet support this syntax) common/parser: fix handling of tool definition with missing properties key (#21128) jinja : handle empty expressions correctly (#20913) * Reject empty computed member expressions before returning slices[0] from parse_member_expression_arguments(). * Treat empty computed member expressions with Jinja2 undefined semantics Treat empty computed member expressions like `a[]` as undefined instead of raising a parser error, to match Jinja2 behavior. - return a noop expression for empty computed member arguments - return undefined when a computed member key evaluates to undefined - add Jinja tests covering `a[]|default('fallback')` and `a[] is undefined` * Handle undefined computed member properties Move undefined-property handling to the common member access path, and add a test covering `a[undefined] is undefined`. * Use default undefined value in member access Initialize val and then return it when property is undefined. Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * empty statement parses to blank_expression instead of noop_statement --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> common : gpt-oss handle builtin and unsolicited tool calls (#21213) fix: tool call parsing for LFM2 and LFM2.5 models (#21242) * fix: tool call parsing for LFM2 and LFM2.5 models' * refactor: add test / break out lfm2 and lfm2.5 parsing logic # Conflicts: # common/chat.cpp Relax prefill parser to allow space. (#21240) * Relax prefill parser to allow space. * Move changes from prefix() to parser generation * Only allow spaces if we're not having a pure content parser next common : add commentary rules for gpt-oss-20b (#21286) add reasoning budget model, mtmd: fix gguf conversion for audio/vision mmproj (#21309) * fix gguf conversion for audio/vision mmproj * fix test # Conflicts: # convert_hf_to_gguf.py # examples/eval-callback/eval-callback.cpp # examples/mtmd/CMakeLists.txt # examples/mtmd/clip-impl.h # examples/mtmd/mtmd.cpp # gguf-py/gguf/constants.py # gguf-py/gguf/gguf_writer.py # gguf-py/gguf/tensor_mapping.py # src/CMakeLists.txt # src/llama-arch.cpp # src/llama-arch.h # src/llama-model.cpp # src/llama-model.h # src/llama-vocab.cpp # src/models/models.h # tests/test-llama-archs.cpp # tools/mtmd/clip-graph.h # tools/mtmd/clip-model.h # tools/mtmd/clip.cpp # tools/mtmd/models/models.h fix: gemma 4 template (#21326) chat : avoid including json in chat.h (#21306) jinja: coerce input for string-specific filters (#21370) common : fix tool call type detection for nullable and enum schemas (#21327) * common : fix tool call type detection for nullable and enum schemas * common, tests : fix grammar delegation for nullable/enum schemas and add tests Fix enum type inference to scan all enum values (not just index 0) so schemas like {"enum": [0, "celsius"]} correctly detect string type. Fix schema_delegates in peg-parser to handle nullable type arrays (["string", "null"]) and typeless enum schemas in raw mode, allowing the tagged parser to use raw text instead of JSON-formatted strings. Add test cases for Qwen3-Coder (TAG_WITH_TAGGED format): - nullable string ["string", "null"] - nullable string with null first ["null", "string"] - nullable integer ["integer", "null"] - enum without explicit type key common/parser: fix call ID detection (Mistral parser mostly) + atomicity for tag-json parsers (#21230) * Fix call ID detection (Mistral parser mostly) + atomicity for tag-json parsers * Rename * Update common/chat-auto-parser-generator.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> common : add gemma 4 specialized parser (#21418) * common : add gemma4 dedicated parser * cont : add '<|tool_response>' as eog * cont : emit JSON from Gemma4 tool call AST * cont : more fixes * cont : refactor convert function * cont : refine rules and mapping * cont : add more tests * cont : clean up * cont : remove autoparser gemma4 implementation * cont : more cleanup * cont : rename gemma4.jinja to match the others * cont : add custom template to support interleaved thinking * cont : preserve reasoning in model turns * cont : fix initializer error * cont : fix unused vars * cont : fix accidental static * cont : fix specialized_template signature * fix extra semicolon * remove debug line and extra space [no ci] fix reasoning budget parser: fix MiniMax handling (#21573) jinja : support ensure_ascii=true, string repetition and int/float self-filtering (#21623) * feat: jinja engine improvements for reka-edge Port three Jinja engine improvements needed for the reka-edge model: 1. Python-style string repetition ("ab" * 3 → "ababab") 2. ensure_ascii=true support for tojson filter (escapes non-ASCII to \uXXXX) 3. int() builtin on value_int_t (identity, needed for Reka Edge template) * fix: escape invalid utf8 bytes when ensure_ascii=true The json_ensure_ascii_preserving_format function does not correctly handle an edge case where if UTF-8 parsing fails, it adds the non-ascii character back to the output as a raw byte. This commit fixes that by adding the unicode standard replacement character \\ufffd to the output instead. This is the standard behavior for various programming languages like Python, Rust, Go, etc. * chore: address PR comments 1. Add todo comment for supporting string repetition for array/tuples 2. Add support for float identity operation 3. Move invalid ascii test case to test_fuzzing * chore: accept suggestion for common/jinja/value.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> common : simplify autoparser tagged parser rules (#21216) * common : simplify autoparser tagged parser rules * cont : remove upper limit on optional args * cont : revert changes to parsing at the end * cont : undo arbitrary ordering of optional args * cont : fix uninitialized required parameters * revert to simplify merge * re-apply patches * restore flexible optional arg ordering tests common : fix ambiguous grammar rule in gemma4 (#21661) * common : fix ambiguous grammar rule in gemma4 * cont : fix missing comma... common : enable reasoning budget sampler for gemma4 (#21697) * fix: enable reasoning budget sampler for gemma4 Add thinking_start_tag and thinking_end_tag to common_chat_params_init_gemma4(). Without these, the reasoning budget sampler never activates for gemma4. Make the newline after "thought" optional in the PEG parser to handle budget=0 (sampler forces end tag before the newline). Add test case for empty thinking block. Fixes #21487 * use p.space() instead of p.optional(p.literal("\n")) in gemma4 thought parser common : better align to the updated official gemma4 template (#21704) fix: Fix broken structured output when using $refs in json_schema (#21699) chat: dedicated DeepSeek v3.2 parser + "official" template (#21785) Hide render_message_to_json warning common/gemma4 : handle parsing edge cases (#21760) common: skip reasoning budget sampler when no budget is requested (#21870) * common: skip reasoning budget sampler when no budget is requested After I added thinking_start_tag / thinking_end_tag for gemma4 in #21697, the reasoning budget sampler gets unconditionally created even when no budget is configured (the default -1). The same applies to kimi_k2, lfm2, lfm2_5, and ministral_3 which also set these tags. The budget gets converted to INT_MAX, so the sampler never actually forces any tokens but still runs per-token checks (start tag matching in IDLE state, token-to-piece conversion + UTF-8 checks in COUNTING state). More importantly, the mere existence of the sampler (non-null rbudget) disables backend sampling. Backend sampling lets the GPU select tokens directly, avoiding a full logits transfer from GPU to CPU every token. This could explain the 30% speed regression reported in #21784 (98 t/s to 70 t/s on Vulkan). So I added a reasoning_budget_tokens >= 0 check to the sampler creation condition. When the budget is unlimited, the sampler is not created, backend sampling stays enabled, and no per-token overhead is added. When a budget is explicitly set (0, 128, 1024, etc.), the sampler is created and works as before. * common: preserve rbudget when grammar is lazy Following up on the review feedback on #21870: keep the reasoning budget sampler when grammar_lazy is true, so the thinking-block grammar suppression from #20970 still works when tools are in use. This way, we only skip the sampler when both no budget is set AND grammar is not lazy. autoparser: support case of JSON_NATIVE with per-call markers (test case: Reka-Edge) (#21892) * fix grammar * fix add sampled token --------- Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com> Co-authored-by: firecoperana <firecoperana>	2026-04-22 10:04:13 +02:00
mcm007andGitHub	5720a4131a	Update docs (#1606) ... * Update parameters.md - list sm graph architectures - gpu tips - build options and parameters * Update README.md - Gemma4	2026-04-10 18:20:28 +02:00
mcm007andGitHub	d557d6c098	Update docs (#1574) ... * Update README.md - Model support - KV cache improvements * Update parameters.md - KV Q4_0 improvements - wgt, with notice - mtmd-kq-type	2026-04-03 08:30:29 +02:00
mcm007andGitHub	028fc79710	Update README.md and parameters docs (#1550) ... * Update parameters.md withe recent changes * Update README.md with recent changes - Hadamard for V cache - AVX-VNNI optimizations - Auto-fit	2026-03-29 18:52:08 +02:00
KawrakowandGitHub	4b1a6560a8	Update parameters documentation with new options (#1511) ... Added descriptions for `--fit` and `--fit-margin`.	2026-03-25 18:23:44 +01:00
KawrakowandGitHub	49778a6ce0	Update Docker documentation with important notice ... Clarify the status of Docker support in ik_llama.cpp.	2026-03-15 12:35:04 +01:00
mcm007andGitHub	bfef07d10b	Update README.md and parameters.md with recent improvements (#1423) ... * Improve text formatting * Update README.md with recent models and features * Update parameters.md with recent additions * Remove deprecated from parameters.md	2026-03-14 18:14:20 +01:00
KawrakowandGitHub	666ea0e983	Revise build instructions for ik_llama.cpp ... Updated documentation to reflect changes from 'llama.cpp' to 'ik_llama.cpp' and clarified build instructions.	2026-03-09 11:23:39 +01:00
3 people	f67fd9a452	Update README.md with build instructions for Windows (#1372) ... * Fix compilation on clang-cl.exe Fixes https://github.com/ikawrakow/ik_llama.cpp/issues/1169 See bitwise ariphmetics here: https://clang.llvm.org/doxygen/avx512fintrin_8h_source.html Clang (and GCC) supports a language feature called Vector Extensions. To Clang, `__m512i` is not just a "struct" or a "bag of bits"; it is recognized by the compiler as a native vector type. Because it is a native vector type, Clang automatically maps standard C operators to the corresponding hardware instructions. When you write `a | b`, Clang sees that a and b are 512-bit integer vectors. It implicitly understands that the bitwise OR operator (|) applies to these vectors. It automatically generates the VPORQ (or VPORD) instruction without needing any helper function. MSVC follows a stricter, more traditional C++ model regarding intrinsics. In MSVC, __m512i is defined in the header files (<immintrin.h>) as a struct or union (e.g., typedef struct __m512i { ... } __m512i). To the MSVC compiler, it is essentially a user-defined data type, not a fundamental language primitive like int or float. Standard C++ does not define what `|` means for a user-defined struct. MSVC does not have the same "Vector Extensions" that automatically apply operators to these structs. When you write `a | b` in MSVC, the compiler looks for a definition of `operator|` for the __m512i struct. Since the standard headers don't provide one, the compiler throws an error. You must use the explicit intrinsic function provided by Intel/MSVC: _mm512_or_si512(a, b). To get the nice syntax `(a | b)` in MSVC, you have to manually "teach" the compiler what `|` means by defining the `operator|` overload yourself. * Update README.md with build instructions for Windows Current README lacks any guide for Windows users, whereas build process on that platform is quite compicated * Update build.md with instruction about clang-cl.exe Brings step-by-step build instruction for Windows * Apply suggestions from code review Co-authored-by: Kawrakow <iwankawrakow@gmail.com> * Polish build.md for Windows usage Added example of use for Windows * Apply suggestions from code review --------- Co-authored-by: Kawrakow <iwankawrakow@gmail.com>	2026-03-09 11:17:26 +01:00
3 people	ab1d74074b	common : introduce composable PEG parser combinators for chat parsing and new jinja template engine (#1369) ... --------- Co-authored-by: Piotr Wilkin <piotr.wilkin@syndatis.com> common : add nemotron 3 parsing (#18077) common : add parser for ministral/mistral large 3/devstral 2 (#17713) common : default content to an empty string (#18485) chat: make tool description and parameters optional per OpenAI spec (#18478) Per the OpenAI API specification, both 'description' and 'parameters' fields in tool function definitions are optional. Previously, the parser would throw an exception if these fields were missing. Attempts to fix #17667 common : implement new jinja template engine (#18462) --------- Co-authored-by: Alde Rojas <hello@alde.dev> Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> jinja: correct member access rule (#18905) jinja : fix lexing of float literals with sign (#18901) jinja : add missing tojson filter for bool (#18900) jinja : attribute support for join, map and sort (#18883) jinja : fix object item order (and properly implement dictsort) (#18904) tests : add test-jinja -py option for cross-checking (#18906) Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> ci : run test-jinja -py on high perf [no ci] (#18916) jinja : fix undefined keys and attributes and int/float as bool (#18924) jinja: support none|string (#18995) Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> jinja : implement mixed type object keys (#18955) --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co> jinja : undefined should be treated as sequence/iterable (return string/array) by filters/tests (#19147) `tojson` is not a supported `undefined` filter keep it DRY and fix some types jinja : do not pass empty tools and add some none filters (#19176) jinja : add unordered_map include to value.h [no ci] (#19205) jinja : add missing 'in' test to template engine (#19004) (#19239) The jinja template parser was missing the 'in' test from global_builtins(), causing templates using reject("in", ...), select("in", ...), or 'x is in(y)' to fail with "selectattr: unknown test 'in'". This broke tool-calling for Qwen3-Coder and any other model whose chat template uses the 'in' test. Added test_is_in supporting array, string, and object containment checks, mirroring the existing 'in' operator logic in runtime.cpp. Includes test cases for all three containment types plus reject/select filter usage. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Sid Mohan <sidmohan0@users.noreply.github.com> Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> Co-authored-by: Xuan Son Nguyen <son@huggingface.co> Add Jinja support for "indent" string filter (#19529) Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> add vendor refactor chat server : support preserving reasoning_content in assistant message (#18994) chat : fix translategemma crash on common_chat_format_example (#19019) chat: fix language input for translategemma (#19052) Co-authored-by: Aldehir Rojas <hello@alde.dev> --------- Co-authored-by: Aldehir Rojas <hello@alde.dev> chat: fix case where template accepts type content only (#19419) mtmd : chat : Fix extra \n between text and media marker (#19595) Thanks to @tugot17 for detecting and reporting the issue. For vision models (e.g. LFM2.5-VL-1.6B and Qwen/Qwen3-VL-4B-Instruct) `llama-mtmd-cli` produces identical output to HF implementation. However `llama-server` doesn't. I traced it down to extra newline inserted after `<__media__>`. This happens in `to_json_oaicompat`, that treats media markers as text and joins all parts with `\n` separator. PR introduces new type `media_marker` and uses it for media markers. Extra logic is added to prevent insertion of newlines before and after media markers. With this change number of input tokens is identical to HF implementation and as a result the output is also identical. I explored other ways to address the issue * remove completely `\n` between text parts in `to_json_oaicompat` * merge text messages in server-common.cpp before sending them to `to_json_oaicompat` Please propose alternative ways of fixing this issue. Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com> --------- Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com> common : merge qwen3-coder and nemotron nano 3 parsers (#19765) common : fix improper trimming in XML parser on complete message (#19805) Co-authored-by: Jules LEIDELINGER <11395311+julio75012@users.noreply.github.com> jinja: correct stats for tojson and string filters (#19785) jinja : correct default size for string slices (#19913) common : handle unicode during partial json parsing (#16526) common : fix json schema with '\' in literals (#17307) add back qwen_coder_xml and mirothinker Co-authored-by: Aldehir Rojas <hello@alde.dev>	2026-03-09 11:03:33 +01:00
mcm007andGitHub	b2cb4512c5	Create parameters overview (#1269) ... * raw parameters.md * fix small typos in common.cpp * Update build args in parameters.md * Update parameters.md - format as table - sections * Update README.md - quickstart - build and run * Update parameters.md other tools examples * add PR links * multiple updates to parameters.md - description - add jargon section - add suggestions from feedbacks * don't imply that only linux is supported in README.md * add alias to parameters.md * Update README.md with recent models and features * Update parameters.md with latest features * address suggestions - no-ooae - placeholder for common commands - no-kv-offload - llama-sweep-bench - placeholder for unique parameters * specify Linux distro in README.md	2026-02-20 07:20:56 +01:00
4 people	1cb7e1bf39	spec : add self speculative decoding, ngram and refactor (#1261) ... * spec : add self speculative decoding and ngram-mod and refactor common : use common_ prefix for common library function llama : use LLAMA_TOKEN_NULL spec : add self speculative decoding (no draft model required) + refactor spec : add ngram-mod spec : various improvements ton ngram-map + docs spec : fix the check-rate logic of ngram-simple common : add common_speculative_is_compat() spec : simplify time measurement using common_time_meas refactor common_sampler_init refactor common_token_to_piece refactor and fix cur_p bug clean up * spec : remove check rate * spec: show warnings instead of abort --------- Co-authored-by: firecoperana <firecoperana> Co-authored-by: Sascha Rogmann <59577610+srogmann@users.noreply.github.com>	2026-02-13 19:04:55 +01:00
4 people	d7882c3cf8	Tool calls support from mainline (#723) ... * Tool calls support from mainline * update cmake * revert api for /completions * Fix broken thinking process for gpt-oss * add missing args and fix webui bugs * add missing args and fix webui bugs2 * Fix reasoning format error * add usage * change default post_sampling_probs to true * add back generated_text * Remove server endpoints tests * add log * Chat fixes * Remove logs * webui: revert extra handling of thinking process --------- Co-authored-by: firecoperana <firecoperana> Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>	2025-09-01 08:38:49 +03:00
3 people	1a4cfbcc53	Merge mainline - Aug 12 2024 (#17) ... * Merge mainline * Fix after merge * Remove CI check --------- Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>	2024-08-12 15:14:32 +02:00
3 people	0ceeb11721	Merge mainline llama.cpp (#3) ... * Merging mainline - WIP * Merging mainline - WIP AVX2 and CUDA appear to work. CUDA performance seems slightly (~1-2%) lower as it is so often the case with llama.cpp/ggml after some "improvements" have been made. * Merging mainline - fix Metal * Remove check --------- Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>	2024-07-27 07:55:01 +02:00
3 people	b267b997c5	build: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809) ... * `main`/`server`: rename to `llama` / `llama-server` for consistency w/ homebrew * server: update refs -> llama-server gitignore llama-server * server: simplify nix package * main: update refs -> llama fix examples/main ref * main/server: fix targets * update more names * Update build.yml * rm accidentally checked in bins * update straggling refs * Update .gitignore * Update server-llm.sh * main: target name -> llama-cli * Prefix all example bins w/ llama- * fix main refs * rename {main->llama}-cmake-pkg binary * prefix more cmake targets w/ llama- * add/fix gbnf-validator subfolder to cmake * sort cmake example subdirs * rm bin files * fix llama-lookup-* Makefile rules * gitignore /llama-* * rename Dockerfiles * rename llama|main -> llama-cli; consistent RPM bin prefixes * fix some missing -cli suffixes * rename dockerfile w/ llama-cli * rename(make): llama-baby-llama * update dockerfile refs * more llama-cli(.exe) * fix test-eval-callback * rename: llama-cli-cmake-pkg(.exe) * address gbnf-validator unused fread warning (switched to C++ / ifstream) * add two missing llama- prefixes * Updating docs for eval-callback binary to use new `llama-` prefix. * Updating a few lingering doc references for rename of main to llama-cli * Updating `run-with-preset.py` to use new binary names. Updating docs around `perplexity` binary rename. * Updating documentation references for lookup-merge and export-lora * Updating two small `main` references missed earlier in the finetune docs. * Update apps.nix * update grammar/README.md w/ new llama-* names * update llama-rpc-server bin name + doc * Revert "update llama-rpc-server bin name + doc" This reverts commit e474ef1df481fd8936cd7d098e3065d7de378930. * add hot topic notice to README.md * Update README.md * Update README.md * rename gguf-split & quantize bins refs in **/tests.sh --------- Co-authored-by: HanClinto <hanclinto@gmail.com>	2024-06-13 00:41:52 +01:00
GalunidandGitHub	81e37e4303	Move convert.py to examples/convert-legacy-llama.py (#7430) ... * Move convert.py to examples/convert-no-torch.py * Fix CI, scripts, readme files * convert-no-torch -> convert-legacy-llama * Move vocab thing to vocab.py * Fix convert-no-torch -> convert-legacy-llama * Fix lost convert.py in ci/run.sh * Fix imports * Fix gguf not imported correctly * Fix flake8 complaints * Fix check-requirements.sh * Get rid of ADDED_TOKENS_FILE, FAST_TOKENIZER_FILE * Review fixes	2024-05-30 21:40:00 +10:00
BrianandGitHub	f0f87a2f2b	Added a single test function script and fix debug-test.sh to be more robust (#7279) ... * run-single-test.sh: added a single test function script and fix debug-test.sh to be more robust * debug-test.sh: combined execute and gdb test mode via -g flag * debug-test.sh: refactor * debug-test: refactor for clarity * debug-test.sh: comment style changes * debug-test.sh: fix gdb	2024-05-17 22:40:14 +10:00
Josh RamerandGitHub	aa01842c75	Scripting & documenting debugging one test without anything else in the loop. (#7096) ... * A little documentation that shares my quick tips for working in the repository. * Update startup-testing-debugging.md * script that shows a menu of tests to pick from & run the debugger on * debug-test.sh: Refactor CLI help message * debug-test.sh: documentation update * debug-test.sh: CLI Help output corrections * debug-test.sh: minor doc fix --------- authored-by: Josh Ramer <ubuntu@ip-172-31-32-53.ec2.internal> Assisted-by: brian khuu <mofosyne@gmail.com>	2024-05-12 03:26:35 +10:00
omahsandGitHub	9478c760a8	docs: fix typos (#7124) ... * fix typo * fix typos * fix typo * fix typos * fix typo * fix typos	2024-05-07 18:20:33 +03:00
4 people	b65aae6a1d	eval-callback: Example how to use eval callback for debugging (#6576) ... * gguf-debug: Example how to use ggml callback for debugging * gguf-debug: no mutex, verify type, fix stride. * llama: cv eval: move cb eval field in common gpt_params * ggml_debug: use common gpt_params to pass cb eval. Fix get tensor SIGV random. * ggml_debug: ci: add tests * ggml_debug: EOL in CMakeLists.txt * ggml_debug: Remove unused param n_batch, no batching here * ggml_debug: fix trailing spaces * ggml_debug: fix trailing spaces * common: fix cb_eval and user data not initialized * ci: build revert label * ggml_debug: add main test label * doc: add a model: add a link to ggml-debug * ggml-debug: add to make toolchain * ggml-debug: tests add the main label * ggml-debug: ci add test curl label * common: allow the warmup to be disabled in llama_init_from_gpt_params * ci: add curl test * ggml-debug: better tensor type support * gitignore : ggml-debug * ggml-debug: printing also the sum of each tensor * ggml-debug: remove block size * eval-callback: renamed from ggml-debug * eval-callback: fix make toolchain --------- Co-authored-by: slaren <slarengh@gmail.com> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>	2024-04-11 14:51:07 +02:00
3 people	7e0ca3abb2	docs : how to add a model (#6565) ... * docs: how to add a model * docs: model: typo and docs * docs: model: add prevision on RoPE * docs: model: rephrasing README.md * docs: model: rephrasing README.md * docs: model: README.md fix trailing spaces * docs : some fixes * Update README.md --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>	2024-04-10 09:58:48 +03:00
slarenandGitHub	f6b28778af	cuda : rename build flag to LLAMA_CUDA (#6299)	2024-03-26 01:16:01 +01:00
Georgi Gerganov	d490717332	docs : add llama-star arch idea	2023-11-23 11:35:04 +02:00
3 people	a05fccf374	Fix some documentation typos/grammar mistakes (#4032) ... * typos * Update examples/parallel/README.md Co-authored-by: Kerfuffle <44031344+KerfuffleV2@users.noreply.github.com> --------- Co-authored-by: Kerfuffle <44031344+KerfuffleV2@users.noreply.github.com>	2023-11-11 23:04:58 -07:00
uint256_tandGitHub	dc9c5a37a3	docs : fix typo GOMP_CPU_AFFINITY (#3597)	2023-10-12 16:36:16 +03:00
Kevin JiandGitHub	478aeb62b9	docs : mark code as Bash (#3375)	2023-09-28 09:11:32 -04:00
7 people	d62137b150	gguf : new file format with flexible meta data (beta) (#2398) ... * gguf : first API pass * gguf : read header + meta data * gguf : read tensor info * gguf : initial model loading - not tested * gguf : add gguf_get_tensor_name() * gguf : do not support passing existing ggml_context to gguf_init * gguf : simplify gguf_get_val * gguf : gguf.c is now part of ggml.c * gguf : read / write sample models * gguf : add comments * refactor : reduce code duplication and better API (#2415) * gguf : expose the gguf_type enum through the API for now * gguf : add array support * gguf.py : some code style changes * convert.py : start a new simplified implementation by removing old stuff * convert.py : remove GGML vocab + other obsolete stuff * GGUF : write tensor (#2426) * WIP: Write tensor * GGUF : Support writing tensors in Python * refactor : rm unused import and upd todos * fix : fix errors upd writing example * rm example.gguf * gitignore *.gguf * undo formatting * gguf : add gguf_find_key (#2438) * gguf.cpp : find key example * ggml.h : add gguf_find_key * ggml.c : add gguf_find_key * gguf : fix writing tensors * gguf : do not hardcode tensor names to read * gguf : write sample tensors to read * gguf : add tokenization constants * quick and dirty conversion example * gguf : fix writing gguf arrays * gguf : write tensors one by one and code reuse * gguf : fix writing gguf arrays * gguf : write tensors one by one * gguf : write tensors one by one * gguf : write tokenizer data * gguf : upd gguf conversion script * Update convert-llama-h5-to-gguf.py * gguf : handle already encoded string * ggml.h : get array str and f32 * ggml.c : get arr str and f32 * gguf.py : support any type * Update convert-llama-h5-to-gguf.py * gguf : fix set is not subscriptable * gguf : update convert-llama-h5-to-gguf.py * constants.py : add layer norm eps * gguf.py : add layer norm eps and merges * ggml.h : increase GGML_MAX_NAME to 64 * ggml.c : add gguf_get_arr_n * Update convert-llama-h5-to-gguf.py * add gptneox gguf example * Makefile : add gptneox gguf example * Update convert-llama-h5-to-gguf.py * add gptneox gguf example * Update convert-llama-h5-to-gguf.py * Update convert-gptneox-h5-to-gguf.py * Update convert-gptneox-h5-to-gguf.py * Update convert-llama-h5-to-gguf.py * gguf : support custom alignment value * gguf : fix typo in function call * gguf : mmap tensor data example * fix : update convert-llama-h5-to-gguf.py * Update convert-llama-h5-to-gguf.py * convert-gptneox-h5-to-gguf.py : Special tokens * gptneox-main.cpp : special tokens * Update gptneox-main.cpp * constants.py : special tokens * gguf.py : accumulate kv and tensor info data + special tokens * convert-gptneox-h5-to-gguf.py : accumulate kv and ti + special tokens * gguf : gguf counterpart of llama-util.h * gguf-util.h : update note * convert-llama-h5-to-gguf.py : accumulate kv / ti + special tokens * convert-llama-h5-to-gguf.py : special tokens * Delete gptneox-common.cpp * Delete gptneox-common.h * convert-gptneox-h5-to-gguf.py : gpt2bpe tokenizer * gptneox-main.cpp : gpt2 bpe tokenizer * gpt2 bpe tokenizer (handles merges and unicode) * Makefile : remove gptneox-common * gguf.py : bytesarray for gpt2bpe tokenizer * cmpnct_gpt2bpe.hpp : comments * gguf.py : use custom alignment if present * gguf : minor stuff * Update gptneox-main.cpp * map tensor names * convert-gptneox-h5-to-gguf.py : map tensor names * convert-llama-h5-to-gguf.py : map tensor names * gptneox-main.cpp : map tensor names * gguf : start implementing libllama in GGUF (WIP) * gguf : start implementing libllama in GGUF (WIP) * rm binary commited by mistake * upd .gitignore * gguf : calculate n_mult * gguf : inference with 7B model working (WIP) * gguf : rm deprecated function * gguf : start implementing gguf_file_saver (WIP) * gguf : start implementing gguf_file_saver (WIP) * gguf : start implementing gguf_file_saver (WIP) * gguf : add gguf_get_kv_type * gguf : add gguf_get_kv_type * gguf : write metadata in gguf_file_saver (WIP) * gguf : write metadata in gguf_file_saver (WIP) * gguf : write metadata in gguf_file_saver * gguf : rm references to old file formats * gguf : shorter name for member variable * gguf : rm redundant method * gguf : get rid of n_mult, read n_ff from file * Update gguf_tensor_map.py * Update gptneox-main.cpp * gguf : rm references to old file magics * gguf : start implementing quantization (WIP) * gguf : start implementing quantization (WIP) * gguf : start implementing quantization (WIP) * gguf : start implementing quantization (WIP) * gguf : start implementing quantization (WIP) * gguf : start implementing quantization (WIP) * gguf : quantization is working * gguf : roper closing of file * gguf.py : no need to convert tensors twice * convert-gptneox-h5-to-gguf.py : no need to convert tensors twice * convert-llama-h5-to-gguf.py : no need to convert tensors twice * convert-gptneox-h5-to-gguf.py : simplify nbytes * convert-llama-h5-to-gguf.py : simplify nbytes * gptneox-main.cpp : n_layer --> n_block * constants.py : n_layer --> n_block * gguf.py : n_layer --> n_block * convert-gptneox-h5-to-gguf.py : n_layer --> n_block * convert-llama-h5-to-gguf.py : n_layer --> n_block * gptneox-main.cpp : n_layer --> n_block * Update gguf_tensor_map.py * convert-gptneox-h5-to-gguf.py : load model in parts to save memory * convert-llama-h5-to-gguf.py : load model in parts to save memory * convert : write more metadata for LLaMA * convert : rm quantization version * convert-gptneox-h5-to-gguf.py : add file_type key * gptneox-main.cpp : add file_type key * fix conflicts * gguf : add todos and comments * convert-gptneox-h5-to-gguf.py : tensor name map changes * Create gguf_namemap.py : tensor name map changes * Delete gguf_tensor_map.py * gptneox-main.cpp : tensor name map changes * convert-llama-h5-to-gguf.py : fixes * gguf.py : dont add empty strings * simple : minor style changes * gguf : use UNIX line ending * Create convert-llama-7b-pth-to-gguf.py * llama : sync gguf-llama.cpp with latest llama.cpp (#2608) * llama : sync gguf-llama.cpp with latest llama.cpp * minor : indentation + assert * llama : refactor gguf_buffer and gguf_ctx_buffer * llama : minor * gitignore : add gptneox-main * llama : tokenizer fixes (#2549) * Merge tokenizer fixes into the gguf branch. * Add test vocabularies * convert : update convert-new.py with tokenizer fixes (#2614) * Merge tokenizer fixes into the gguf branch. * Add test vocabularies * Adapt convert-new.py (and fix a clang-cl compiler error on windows) * llama : sync gguf-llama with llama (#2613) * llama : sync gguf-llama with llama * tests : fix build + warnings (test-tokenizer-1 still fails) * tests : fix wstring_convert * convert : fix layer names * llama : sync gguf-llama.cpp * convert : update HF converter to new tokenizer voodoo magics * llama : update tokenizer style * convert-llama-h5-to-gguf.py : add token types * constants.py : add token types * gguf.py : add token types * convert-llama-7b-pth-to-gguf.py : add token types * gguf-llama.cpp : fix n_head_kv * convert-llama-h5-to-gguf.py : add 70b gqa support * gguf.py : add tensor data layout * convert-llama-h5-to-gguf.py : add tensor data layout * convert-llama-7b-pth-to-gguf.py : add tensor data layout * gptneox-main.cpp : add tensor data layout * convert-llama-h5-to-gguf.py : clarify the reverse permute * llama : refactor model loading code (#2620) * llama : style formatting + remove helper methods * llama : fix quantization using gguf tool * llama : simplify gguf_file_saver * llama : fix method names * llama : simplify write_header() * llama : no need to pass full file loader to the file saver just gguf_ctx * llama : gguf_file_saver write I32 * llama : refactor tensor names (#2622) * gguf: update tensor names searched in quantization * gguf : define tensor names as constants * gguf : initial write API (not tested yet) * gguf : write to file API (not tested) * gguf : initial write API ready + example * gguf : fix header write * gguf : fixes + simplify example + add ggml_nbytes_pad() * gguf : minor * llama : replace gguf_file_saver with new gguf write API * gguf : streaming support when writing files * gguf : remove oboslete write methods * gguf : remove obosolete gguf_get_arr_xxx API * llama : simplify gguf_file_loader * llama : move hparams and vocab from gguf_file_loader to llama_model_loader * llama : merge gguf-util.h in llama.cpp * llama : reorder definitions in .cpp to match .h * llama : minor simplifications * llama : refactor llama_model_loader (WIP) wip : remove ggml_ctx from llama_model_loader wip : merge gguf_file_loader in llama_model_loader * llama : fix shape prints * llama : fix Windows build + fix norm_rms_eps key * llama : throw error on missing KV paris in model meta data * llama : improve printing + log meta data * llama : switch print order of meta data --------- Co-authored-by: M. Yusuf Sarıgöz <yusufsarigoz@gmail.com> * gguf : deduplicate (#2629) * gguf : better type names * dedup : CPU + Metal is working * ggml : fix warnings about unused results * llama.cpp : fix line feed and compiler warning * llama : fix strncpy warning + note token_to_str does not write null * llama : restore the original load/save session implementation Will migrate this to GGUF in the future * convert-llama-h5-to-gguf.py : support alt ctx param name * ggml : assert when using ggml_mul with non-F32 src1 * examples : dedup simple --------- Co-authored-by: klosax <131523366+klosax@users.noreply.github.com> * gguf.py : merge all files in gguf.py * convert-new.py : pick #2427 for HF 70B support * examples/gguf : no need to keep q option for quantization any more * llama.cpp : print actual model size * llama.cpp : use ggml_elements() * convert-new.py : output gguf (#2635) * convert-new.py : output gguf (WIP) * convert-new.py : add gguf key-value pairs * llama : add hparams.ctx_train + no longer print ftype * convert-new.py : minor fixes * convert-new.py : vocab-only option should work now * llama : fix tokenizer to use llama_char_to_byte * tests : add new ggml-vocab-llama.gguf * convert-new.py : tensor name mapping * convert-new.py : add map for skipping tensor serialization * convert-new.py : convert script now works * gguf.py : pick some of the refactoring from #2644 * convert-new.py : minor fixes * convert.py : update to support GGUF output * Revert "ci : disable CI temporary to not waste energy" This reverts commit 7e82d25f40386540c2c15226300ad998ecd871ea. * convert.py : n_head_kv optional and .gguf file extension * convert.py : better always have n_head_kv and default it to n_head * llama : sync with recent PRs on master * editorconfig : ignore models folder ggml-ci * ci : update ".bin" to ".gguf" extension ggml-ci * llama : fix llama_model_loader memory leak * gptneox : move as a WIP example * llama : fix lambda capture ggml-ci * ggml : fix bug in gguf_set_kv ggml-ci * common.h : .bin --> .gguf * quantize-stats.cpp : .bin --> .gguf * convert.py : fix HF tensor permuting / unpacking ggml-ci * llama.cpp : typo * llama : throw error if gguf fails to init from file ggml-ci * llama : fix tensor name grepping during quantization ggml-ci * gguf.py : write tensors in a single pass (#2644) * gguf : single pass for writing tensors + refactoring writer * gguf : single pass for writing tensors + refactoring writer * gguf : single pass for writing tensors + refactoring writer * gguf : style fixes in simple conversion script * gguf : refactor gptneox conversion script * gguf : rename h5 to hf (for HuggingFace) * gguf : refactor pth to gguf conversion script * gguf : rm file_type key and method * gguf.py : fix vertical alignment * gguf.py : indentation --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * convert-gptneox-hf-to-gguf.py : fixes * gguf.py : gptneox mapping * convert-llama-hf-to-gguf.py : fixes * convert-llama-7b-pth-to-gguf.py : fixes * ggml.h : reverse GGUF_MAGIC * gguf.py : reverse GGUF_MAGIC * test-tokenizer-0.cpp : fix warning * llama.cpp : print kv general.name * llama.cpp : get special token kv and linefeed token id * llama : print number of tensors per type + print arch + style * tests : update vocab file with new magic * editorconfig : fix whitespaces * llama : re-order functions * llama : remove C++ API + reorganize common source in /common dir * llama : minor API updates * llama : avoid hardcoded special tokens * llama : fix MPI build ggml-ci * llama : introduce enum llama_vocab_type + remove hardcoded string constants * convert-falcon-hf-to-gguf.py : falcon HF --> gguf conversion, not tested * falcon-main.cpp : falcon inference example * convert-falcon-hf-to-gguf.py : remove extra kv * convert-gptneox-hf-to-gguf.py : remove extra kv * convert-llama-7b-pth-to-gguf.py : remove extra kv * convert-llama-hf-to-gguf.py : remove extra kv * gguf.py : fix for falcon 40b * falcon-main.cpp : fix for falcon 40b * convert-falcon-hf-to-gguf.py : update ref * convert-falcon-hf-to-gguf.py : add tensor data layout * cmpnct_gpt2bpe.hpp : fixes * falcon-main.cpp : fixes * gptneox-main.cpp : fixes * cmpnct_gpt2bpe.hpp : remove non-general stuff * Update examples/server/README.md Co-authored-by: slaren <slarengh@gmail.com> * cmpnct_gpt2bpe.hpp : cleanup * convert-llama-hf-to-gguf.py : special tokens * convert-llama-7b-pth-to-gguf.py : special tokens * convert-permute-debug.py : permute debug print * convert-permute-debug-master.py : permute debug for master * convert-permute-debug.py : change permute type of attn_q * convert.py : 70b model working (change attn_q permute) * Delete convert-permute-debug-master.py * Delete convert-permute-debug.py * convert-llama-hf-to-gguf.py : fix attn_q permute * gguf.py : fix rope scale kv * convert-llama-hf-to-gguf.py : rope scale and added tokens * convert-llama-7b-pth-to-gguf.py : rope scale and added tokens * llama.cpp : use rope scale kv * convert-llama-7b-pth-to-gguf.py : rope scale fix * convert-llama-hf-to-gguf.py : rope scale fix * py : fix whitespace * gguf : add Python script to convert GGMLv3 LLaMA models to GGUF (#2682) * First pass at converting GGMLv3 LLaMA models to GGUF * Cleanups, better output during conversion * Fix vocab space conversion logic * More vocab conversion fixes * Add description to converted GGUF files * Improve help text, expand warning * Allow specifying name and description for output GGUF * Allow overriding vocab and hyperparams from original model metadata * Use correct params override var name * Fix wrong type size for Q8_K Better handling of original style metadata * Set default value for gguf add_tensor raw_shape KW arg * llama : improve token type support (#2668) * Merge tokenizer fixes into the gguf branch. * Add test vocabularies * Adapt convert-new.py (and fix a clang-cl compiler error on windows) * Improved tokenizer test But does it work on MacOS? * Improve token type support - Added @klosax code to convert.py - Improved token type support in vocabulary * Exclude platform dependent tests * More sentencepiece compatibility by eliminating magic numbers * Restored accidentally removed comment * llama : add API for token type ggml-ci * tests : use new tokenizer type API (#2692) * Merge tokenizer fixes into the gguf branch. * Add test vocabularies * Adapt convert-new.py (and fix a clang-cl compiler error on windows) * Improved tokenizer test But does it work on MacOS? * Improve token type support - Added @klosax code to convert.py - Improved token type support in vocabulary * Exclude platform dependent tests * More sentencepiece compatibility by eliminating magic numbers * Restored accidentally removed comment * Improve commentary * Use token type API in test-tokenizer-1.cpp * py : cosmetics * readme : add notice about new file format ggml-ci --------- Co-authored-by: M. Yusuf Sarıgöz <yusufsarigoz@gmail.com> Co-authored-by: klosax <131523366+klosax@users.noreply.github.com> Co-authored-by: goerch <jhr.walter@t-online.de> Co-authored-by: slaren <slarengh@gmail.com> Co-authored-by: Kerfuffle <44031344+KerfuffleV2@users.noreply.github.com>	2023-08-21 23:07:43 +03:00
3 people	739a917df4	docs : add performance troubleshoot + example benchmark documentation (#1674) ... * test anchor link * test table * add benchmarks * Add performance troubleshoot & benchmark * add benchmarks * remove unneeded line --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>	2023-06-05 23:32:36 +03:00