mirror of
https://github.com/ikawrakow/ik_llama.cpp.git
synced 2026-07-21 02:05:35 +00:00
b90939934a
* openpangu: Stage-1 converter probe for openPangu-2.0-Flash
Add OpenPanguV2ForCausalLM conversion support (converter-only; runtime graph
is Stage-2). Registers a new LLM_ARCH_OPENPANGU on the Python/gguf-py side:
- gguf-py/constants.py: MODEL_ARCH.OPENPANGU + name, indexer KV keys, 22 new
tensor enums (DSA indexer x4, MoME convs x3, param-sink x2, mHC/Hyper-
Connections x12, block-post-norm), and the full MODEL_TENSORS list reusing
the deepseek MLA + MoE + NextN bricks.
- tensor_mapping.py: arch-specific block mappings that disambiguate the
sandwich norms (post_attention/pre_mlp/post_mlp) and pin every Pangu-only
tensor; non-block global mHC merge module.
- convert_hf_to_gguf.py: OpenPanguV2Model (subclasses DeepseekV2Model) with
set_gguf_parameters (MLA/MoE/indexer/mHC/param-sink/DSA+SWA metadata),
modify_tensors (expert merge, kv_b split, no MTP skip), and the
OpenPanguV2Tokenizer pre-tokenizer hash.
Validated offline against the real 50-shard safetensors index: all 37,587
tensors map to a GGUF target (0 unmapped), and set_gguf_parameters reads only
hparams present in config.json. No weights downloaded; no GPU. Pinned on the
ik/dsa_loop_hadamard_blend DSA substrate.
* openpangu: Stage-2 arch scaffold (LLM_ARCH_OPENPANGU) — loadable, compiles
New arch on main (DSA-decoupled). Declares openPangu-2.0-Flash to the runtime so
the model loads into memory; the compute graph is the next step.
- llama-arch.{h,cpp}: LLM_ARCH_OPENPANGU + name; 3 KV keys (mhc_num_stream,
mhc_recur_norm, param_sink_number); 18 tensor enums (mHC x12, MoME conv x3,
param-sink x2, block-post-norm).
- llama-model.cpp: OPENPANGU tensor-name block, strings matched to the converter.
- llama-model.h: layer + model struct fields (mHC / conv / sink / block-post / merge).
- llama-hparams.{h,cpp}: reader (MLA + MoE + sigmoid gate + indexer + mHC +
param-sink + NextN); n_layer_kv_from_start = n_layer - nextn (MTP skipped).
- llama-load-tensors.cpp: create_openpangu_tensors (GLM-DSA MLA/MoE base + Pangu
tensors; indexer loaded-but-unused for dense fallback); dispatch + is_mla_attn.
Builds clean (CPU-only libllama). Dense-fallback design: no DSA indexer / SWA
windowing / MTP for first generation (exact <=512 tokens). Graph is Stage-2b.
* openpangu: fix compresskv_conv dim (kv_lora_rank, not +rope); pin attention order in spec
* openpangu: end-to-end runtime — build_openpangu graph runs, generates (garbled)
First full forward pass of openPangu-2.0-Flash on ik_llama. Pipeline works end to
end: new LLM_ARCH_OPENPANGU loads the Q4 GGUF, the graph executes, and llama-cli
generates 40 tokens (EXIT=0). Output is currently garbled (tensor-layout bug to
debug), but the structure is proven.
graphs/build_openpangu.cpp: dense decompressed-MHA attention + 4-stream mHC
(Hyper-Connections) with 20-iter Sinkhorn + MoE(sigmoid+shared) + sandwich norms
+ entry stream-repeat/tail-merge + inp_out_ids selection.
Bring-up fixes to load+run:
- llama-vocab.cpp: register 'openpangu' pre-tokenizer (QWEN2 family)
- llama.cpp: OPENPANGU -> LLAMA_ROPE_TYPE_NORM (was defaulting to NONE=-1)
- llama-load-tensors.cpp: full wkv_b load; k_b/v_b as flattened 2D; block_post_norm
dim = S*H (10240); conv weights 2D {3,C}; mHC alpha/beta/gamma + param_sink +
merge params use bare (no-.weight) tensor names
- llama-model.cpp: OPENPANGU is NOT is_mla_attn (decompressed MHA, standard KV cache)
- graph loop bounded to base layers (skip NextN/MTP)
v0 deferrals (need conv-state cache / manual attention path, all documented):
MoME convs (passthrough), o_conv, param_sink. Next: fix the layout bug to coherence.
* openpangu: COHERENT generation — NEOX rope, Sinkhorn orientation, MoME convs, param_sink
Four correctness fixes on top of the end-to-end scaffold, verified checkpoint-by-
checkpoint against a Python golden reference running on the GGUF's own dequantized
weights (block-0 activations now match to rounding at full fidelity):
- rope: NORM -> NEOX. Pangu config rope_interleave=false; the Infer source maps it
as is_neox_style = not rope_interleave (rotary_mode='half').
- mHC Sinkhorn: the flat h_res block is torch-[r,c] row-major, so a bare ggml
reshape lands column-fastest; the doubly-stochastic iteration ran transposed
(Sinkhorn is not transpose-symmetric). One transpose at input fixes the whole
chain including the mhc_post application.
- MoME convs (qa/compresskv/o): were passthrough stubs. Implemented as
out = x + causal_conv1d(x) (every Infer call site uses residual_connection=1;
tap stats confirm the perturbation form). Taps cast f16->f32 for ggml_mul.
Batch-local v0: exact for fresh-sequence prefill; decode steps miss the
t-1/t-2 taps until a conv-state cache exists.
- param_sink: 128 learned latent-KV entries prepended per layer via a manual
attention path (kv_store + explicit soft_max over [sinks ++ cache]); huge
effect at short context. o_conv now applied pre-o_proj on the same path.
flash_attn forced off for OPENPANGU (FA kernel cannot see the sinks).
- converter: add_bos_token=true (HF prepends <|pangu_text_start|> via the
post-processor; the key was absent so ik dropped BOS).
Greedy Q4_K_M smoke, chat template + <think>: coherent CoT reasoning and a
correct answer. Layer-0 instrumentation (opg0_* names) kept for now.
* openpangu: MoME conv-state cache — decode steps get real t-1/t-2 taps
Allocate a per-layer cache_s_l tensor for OPENPANGU base layers holding the last
two pre-conv latents of the three MoME sites, packed
[qa 2*1024 | compresskv 2*512 | o 2*6144] f32 (~60KB/layer). The conv helper
reads the [C,2] history window (zeros at sequence start, kv_head==0), builds
xx = [hist ++ x], and writes the last two columns back each ubatch — the concat
naturally handles both prefill chaining and the T==1 shift. Read precedes write
in graph order; the fixed-offset copy is graph-reuse safe.
Verified: prefill anchors unchanged (bit-path identical, zero-history branch);
-ub 1 token-by-token run matches the golden reference at t4 (qlora_conv 0.084,
R_block 0.008 rel; attn_out 0.15 on one channel = f16 KV-cache rounding, washes
out by post-norm); final logits differ from full-batch only by a common-mode
shift that softmax cancels. Chat-template greedy smoke: think-block repetition
is gone — clean structured CoT and correct answer.
v0 limits documented in the helper: one state slot (single sequence); cache
rewinds leave the state stale.
* openpangu: NextN/MTP speculative decoding — 1.7-1.8x TG on CPU
Wire the three NextN layers (46-48) into ik's MTP speculative framework
(--spec-type mtp). v0 drafts with head 1 (layer 46), self-chained by the
framework.
- llama.cpp: add OPENPANGU to the cparams.mtp arch allowlist (it was silently
zeroed, which left the target context without a logits buffer once the server
enabled embeddings -> GGML_ASSERT(lctx.logits) in speculative_is_compat).
- load-tensors: MTP layers carry no mHC tensors (tail_use_mhc=false in the
reference) — create them only for base layers. nextn.* tensors were already
wired by the Stage-1 probe.
- build_openpangu: extract the attention sublayer into
build_openpangu_attention (shared base/MTP); add build_openpangu_mtp:
eh_proj(cat(enorm(embed), hnorm(prev_hidden))) -> one plain-residual Pangu
block (sandwich norms, convs+param_sink, MoE+shexp, no mHC/block_post_norm)
-> shared_head norm+head. MTP branch returns the draft graph when
mtp_op_type != NONE; main graph keeps all-token outputs under cparams.mtp.
MTP convs run batch-local (no conv-state slot) — affects acceptance only.
A/B (Q4_K_M, CPU, greedy, 192-token chat CoT completion, warm back-to-back,
medians of 3, bracketed B/A/B):
no-spec: 2.44 t/s (2.34-2.86)
--spec-type mtp:n_max=3: 4.23 / 4.49 t/s (brackets) => ~1.7-1.8x
Draft acceptance 34% on CoT prose (46% on repetitive text); spec and no-spec
greedy outputs are byte-identical. Headroom: conv-state for MTP drafts, n_max
tuning, true 3-head chaining (spec_step_idx).
* server: include draft_n/draft_n_accepted in /completion timings
get_formated_timings() (the /completion path) omitted the speculative
counters that get_timings() (the OAI path) already reports; add them,
guarded by n_draft_total > 0 like the OAI path.
* openpangu: position-indexed MoME conv-state ring — rollback-safe spec decoding + MTP draft chaining
The v0 single-slot conv state held the last-2 pre-conv latents of the most
recent batch, so any speculative draft rejection left latents of REJECTED
positions in the state and every later decode ran with wrong t-1/t-2 taps
(3 conv sites x 46 layers). At 192-token greedy runs every spec config
diverged from no-spec, each differently (rejection-pattern dependent).
Replace it with a per-layer ring cache_s_l [n_lora_q+n_lora_kv+n_head*v_dim, 16]:
column pos%16 holds position pos's pre-conv latents ([qa|ckv|o] packed).
Invariant: reads target only positions before the first batch token, which
are committed, and committed latents depend only on the committed prefix -
rollback-safe by construction, no checkpointing. Writes cover the last
min(T,16) batch positions in <=2 contiguous cpy segments; the copy sources
are views of the [hist ++ x] concat so the history read is an ancestor of
every write (read-before-write by graph dependency).
The ring is also allocated for the NextN/MTP layers, so the draft head
chains real conv taps across WARMUP -> sequential DRAFT_GEN steps (was
batch-local zero-history per draft token).
graph_reuse is forced off for the arch: ring view offsets are position-
baked and the reuse patcher only updates the standard K/V-store copies.
Measured cost on the CPU server path: none visible. ggml_set_rows driven
by an input index tensor is the future reuse-safe shape.
Verified (Q4_K_M, CPU, greedy 192-tok chat-CoT, warm single process):
- no-spec output byte-identical to pre-ring build
- spec output byte-identical to no-spec below the n_predict cap, for all
of n_max in {1,2,3,4,6} x p_min in {0,0.3,0.6} (old build: all diverged)
- acceptance n3-p0: 33.9% -> 60.9%; n3-p0.3: 58.1% -> 68.9%
- TG medians: no-spec 3.19-3.32 t/s; mtp:n_max=3,p_min=0.3 6.97 t/s (~2.1x)
* openpangu: DSA lightning indexer + SWA schedule — long-context correctness past the dense fallback
The dense fallback was exact only <=512 tokens (SWA window). This wires the real
DSA/SWA hybrid schedule, self-contained from GGUF keys the converter already
writes (openpangu.swa_layers + sliding_window_list; absent keys keep the old
dense fallback):
- SWA layers (30 base @512): the generic inp_KQ_mask_swa path, per-layer mask
choice in the builder. The NextN/MTP layers are SWA @2048 in the checkpoint
schedule; MTP graphs run in their own context, so the mask fill picks
hparams.n_swa_mtp when built with an MTP op type.
- DSA layers (16, every 3rd): lightning indexer implemented in-graph from the
Infer reference semantics (jointfix _pangu_torch_calib): q_idx = wq_b on the
post-conv post-norm q-lora latent (24x128), k_idx = rms-normed wk(x) shared
across heads, both NEOX-roped on the FIRST n_rot channels; score =
sum_g w_g * relu(q_g . k) in f32, causal-masked, exact top-k via
argsort + ggml_set_rows scatter into a -1e30 base -> additive selection mask
on the existing manual soft_max seam. Selection engages only when the causal
window exceeds index_top_k (2048); below that the layer is exactly dense.
- Indexer keys cached per position (cache_idx_l, f32 [128, kv_size], DSA layers
only) with the same committed-position invariant as the conv-state ring, so
speculative rollbacks stay safe.
- param sinks remain outside both the window and the selection budget, matching
the reference.
Verified (Q4_K_M, CPU):
- <=512 tokens: byte-identical to the dense build (96/160-token greedy)
- indexer scores vs a GGUF-dequant golden reference at 2101 tokens: 1e-3 rel
(f16 weight rounding); top-3 selection indices exact on all compared queries
- >512 coherence clean; 3.4K-token needle retrieval through active selection
(needle outside every SWA window, ~1300 positions pruned) answers exactly
* openpangu: MLA-latent KV cache — attention absorbed into the 512-latent, 14x smaller cache, ~2.2x TG
Store per position only [ckv_norm 512 | roped k_pe 64] (f32, k_l) plus the
transposed 512-latent (f32, v_l, v_trans layout); per-head K/V are never
materialized. q_nope is absorbed through attn_k_b (loaded 2D from the
converter split for base layers; derived at load via llm_prepare_mla for the
NextN layers - now guarded for layers without attention weights, e.g. the
idle NextN heads 2/3). The value side is the latent itself, up-projected
through attn_v_b after the weighted sum, matching the Infer _forward_dsa
reference. param sinks are native latent-space entries, which removes the
per-step full-cache concat+cast that dominated long-context decode.
llama_state row sizes now come from llama_kv_k_row_embd/llama_kv_v_row_embd
(arch-aware), fixing an out-of-bounds crash in the server prompt-cache save
path (hparams-derived 9216-wide rows vs actual 576-wide latent rows).
Verified (Q4_K_M, CPU): layer-0 attention output matches an f32 golden
reference computed from the same GGUF weight encodings (~1e-2 on O(1)
values); MTP spec output byte-identical to no-spec; 3.4K needle retrieval
through active DSA selection exact under greedy. Output differs from the
materialized build at the token level because attn_k_b/attn_v_b are
independently quantized tensors - both are legitimate Q4-fidelity encodings.
Perf (CPU, warm): no-spec TG 3.2-3.3 -> 6.9-7.1 t/s; mtp:n_max=3,p_min=0.3
-> 11.1 t/s (byte-exact, 67% acceptance); prefill 30.5 t/s at 3.4K; KV self
size at 4K ctx: 5.5 GiB -> 391 MiB. Not yet supported on the latent cache:
K-shift/defrag (context shifting) - unreached in current usage.
* openpangu: fence unsupported serving modes, truth-pass comments, drop dead weight/keys
Post-audit hardening. The cache's position-indexed side state (MoME conv ring,
DSA indexer keys) made several generic serving paths silently unsound; they are
now fenced loudly instead of documented as unsupported:
- s_l_position_ring flag on llama_kv_cache: the qnext-state predicate no longer
claims the conv ring, so per-seq state save, seq_cp and the s_copy graph skip it
- state save/restore refused for the arch at every llama_state_* entry (the ring
and idx_l are not in the state format; restoring without them diverges silently)
- K-shift/self-extend assert, defrag skips with a warning, server ctx_shift off
via new llama_model_supports_ctx_shift()
- single sequence enforced at context creation (n_seq_max > 1 refused)
- server prompt-cache reuse limited to pure extension via new
llama_model_supports_partial_kv_reuse(): mid-cache divergence reprocesses from
scratch (the 16-column ring cannot rewind); multi-turn continuation stays fast
- MTP draft length clamped to 13 via new llama_model_max_draft_tokens() so a
rejected draft can never overwrite the ring columns the next decode reads
- K/V cache types forced to f32 for the arch so the KV size log reports the truth
- cache_size(): real latent-cache branch (was falling through to the ~14x larger
materialized estimate used for offload planning)
- unused fused wkv_b no longer loaded (TENSOR_SKIP; the graph runs entirely on the
pre-split k_b/v_b), llm_prepare_mla openPangu special-case removed (it was a no-op)
- stale v0 comments rewritten to describe the shipped graph; converter stops
writing dead keys (dsa_layers, block_post_layernorm_idx) and the tokenizer
pre-hash is registered in convert_hf_to_gguf_update.py
Gates on this build: greedy spec output byte-identical to no-spec (EOS-terminated,
sha-equal); 3.4K needle retrieved exactly; -np 2 / state save / n_max=20 / stale
prefix reuse all refused or clamped with clear messages.
* openpangu: assert kv_head == first batch position at graph build
The ring, indexer and latent stores are addressed by absolute position through
kv_head; the fences make append-only decode the only reachable mode, but the
invariant was unchecked. Assert it at both graph entries (base and MTP) so any
future cache plumbing that breaks it fails at build instead of corrupting
output. Worst-case measurement builds pass pos = null and are exempt.
* openpangu: cont h_pre before the mHC broadcast mul (CUDA binbcast misreads strided views)
h_pre is a row-slice view of the fused mixes tensor. The CPU mul handles the
strides; the CUDA broadcast path reads the view as if contiguous, so token 0
mixes correctly and every later token gets h_post/h_res rows instead. First
divergent node in the whole graph (oracle rel 0.36 at opg0_attn_mhcpre_x,
fixed to 7.5e-5). Sibling views h_post/h_res were already cont-wrapped, which
is why only h_pre was exposed.
* openpangu: keep DSA zero-trick sources finite (CUDA clamp propagates the 0*(-inf) NaN)
The selection-mask base and zeros were built by scaling the MASKED scores by
zero, but post-mask sc contains -inf and 0 * -inf = NaN. The CPU clamp launders
NaN back to -1e30 (fminf/fmaxf ignore NaN); the CUDA clamp propagates it, so
every DSA layer emitted NaN masks at n_kv > top_k and logits collapsed
(observed: eval-callback CLAMP sum -1.3e36 on CPU vs nan on CUDA, 11748 NaNs
downstream). Scale the pre-mask finite scores instead, which is correct on any
backend regardless of clamp NaN semantics. Also defensively cont the strided
KQ_mask slice feeding the score add (same strided-view kernel class as the mHC
h_pre fix; unproven here but cheap). Gates after fix: 2600-token probe coherent,
3.4K needle exact ('7391') with and without MTP speculation, PP ~120 t/s.
* openpangu: f16 latent KV cache option (explicit -ctk/-ctv f16 halves cache memory, f32 stays default)
Track explicit cache-type requests through CLI/env; openPangu resolves no-request
to f32 (unchanged), accepts explicit f32/f16, warns and falls back to f32 for
BF16/quantized. Sink and cached-token KQ paths stay separate until after KQ so
the latent cache is read directly without the f32-only concat; value is the sum
of the sink and cache matmuls. Ring and DSA indexer caches stay f32; cache_size()
follows the resolved types.
* openpangu: enable graph reuse
* openpangu: wire multi-head MTP drafting
* openpangu: add MTP heads override
* openpangu: keep MTP update logits last
* openpangu: scope MTP warmup heads
* speculative: apply per-request MTP heads before warmup
* openpangu: fix multi-head MTP warmup computing on unwritten inputs
Each chained head called the build_inp_* helpers itself, so the warmup and
update graphs held one inp_tokens/inp_pos/inp_out_ids/KQ_mask tensor per
head while llama_set_inputs only fills the tensors the lctx pointers
reference, i.e. the last head's copies. Every head but the last read
unwritten compute-buffer memory: with heads=3 active even head 1's ring,
latent cache, and cached one-token draft were computed from garbage, which
is why depth-1 acceptance measured 4% against 98% for the heads=1 control.
Create the batch inputs once in build_openpangu and pass them to every
build_openpangu_mtp call, and fix the two chaining errors that were hiding
behind the garbage inputs:
- Shift the chained hidden: head k+1's row at position p consumes head k's
output row at p-1, the same convention head 1 uses for the target's
conditioned hidden rows. The predecessor of a batch's first row lives in
the previous warmup/update, carried across decodes through a new
inp_mtp_carry input backed by lctx.mtp_carry (written back per ubatch,
zeroed when a prompt warmup restarts from position 0).
- Fill head 3's cache row at draft step 2: each draft step runs one head,
so head 3's own decode at step 3 attended over a never-written row at
the step-2 position. Pre-write it from the committed carry.
Also include the active head count in the graph-reuse key next to the
existing step index (reuse stays forced off for this arch).
* speculative: default MTP drafting to a single head
A stage without an explicit heads= override previously resolved to 0,
meaning all model heads, so multi-head drafting was silently on by
default for models that carry more than one NextN layer. Keep it opt-in
(heads=N or heads=0 for all) until multi-head measures a win over the
single-head config; single-head models are unaffected either way.
* speculative: fence MTP head upshift over a warmed prefix
Deeper NextN heads only hold valid cache rows for spans that were warmed
with them. A request drafting with more MTP heads than the cached prefix
was warmed with (e.g. a heads=1 conversation continued with heads=3, a
pure extension the divergence fence deliberately allows) would read
never-written deeper-head rows: verification keeps the output correct,
but acceptance quietly collapses and any measurement taken there is
misleading.
Track the minimum head count the committed context has been warmed with
since position 0 and have the server reprocess from scratch when a
request asks for more. Also announce the model's NextN head count and
the single-head default once at MTP context setup.
* openpangu: skip dead MTP chain compute and stall-free carry readback
The update chain's last head and the draft-time row fill only matter for
their latent-cache and conv-ring writes; their FFN, norms, and shared
head fed nothing. Add a cache-writes-only mode to the MTP block builder
that returns after the attention block, and use it at both sites.
The multi-head carry readback previously synchronized the scheduler
after every warmup/update decode, a hard stall on CUDA. Issue the
device-to-host copy async on the backend stream instead (stream order
protects the source buffer from later graphs) and synchronize lazily
when the host buffer is next consumed or resized.
* openpangu: stop emitting fused kv_b tensor
* openpangu: default latent cache to f16
* openpangu: refuse unsupported latent cache types
* Window OpenPangu SWA cache reads
* Gather OpenPangu DSA decode reads
Gather DSA decode attention over the selected latent rows for OpenPangu base-model decode and verify graphs. The gathered branch now uses ggml_top_k order directly, runs maskless softmax over sinks plus selected rows for T <= 14, and derives values from the gathered k_l rows instead of the transposed latent cache.
* Chunk OpenPangu indexer prefill scoring
* Chunk OpenPangu prefill attention
* Gather OpenPangu sparse prefill attention
* Drop OpenPangu value cache
* Add OpenPangu indexer cache type flag
* Add OpenPangu q8_0 latent cache type
Store the OpenPangu MLA latent K cache as q8_0 via -ctk q8_0 (about 0.53x of
f16); the default stays f16 so behavior is unchanged without the flag. Latent V
stays f16/f32.
The q8 latent cache is a storage format only: it is dequanted to F32 before all
compute. K reads go through openpangu_build_k_latent_for_read, V derivation
through openpangu_build_v_latent_from_k (full 576-wide row to F32, then slice),
and the DSA gather paths already dequant via get_rows. Feeding a q8 latent view
directly into the KQ mul_mat corrupts large-context prefill, so that path is
removed for quantized caches. The cache write stages ckv and kpe through F32 and
writes one full 576-wide q8 row per token.
Verified on a small discriminator model: the default f16 path is byte-identical
to the prior code; the first-DSA-layer attention envelope is within 0.6% of the
f16 cache (linf_rel 0.0057); top-k selection is bit-identical between cache
types; the q8 latent cache is 0.531x the f16 size at 8K and 32K context; and
generation stays coherent on both the dense and DSA-gather paths at all tested
context lengths.
* Remove OpenPangu debug trace env knobs and redundant DSA_TOPK override
Drop the five LLAMA_OPENPANGU_*_TRACE debug-logging knobs (DSA_GATHER_TRACE,
IDX_CHUNK_TRACE, ATT_CHUNK_TRACE, PREFILL_GATHER_TRACE, SWA_WINDOW_TRACE) and the
LLAMA_OPENPANGU_DSA_TOPK override, which duplicated the -dsatk / --dsa-top-k CLI
flag; top-k now comes solely from cparams.dsa_top_k. The five perf-tuning knobs
(DSA_GATHER, IDX_CHUNK, ATT_CHUNK, ATT_KQ_MAX_MIB, PREFILL_GATHER) are retained
pending the perf battery. No change to default behavior.
* Subchunk OpenPangu DSA prefill gather to fit CUDA grid limit
The prefill gathered-attention ggml_get_rows produced dst rows = topk *
token_chunk (2048 * 256 = 524288) mapped to the CUDA grid.y dimension, which
caps at 65535, crashing with GET_ROWS invalid argument at long context (N_KV
around 10.5K with the natural topk of 2048). Split the prefill gather into token
subchunks so topk * subchunk_tokens stays within the grid limit, and guard the
decode gather with the same fit check (falling back to the dense masked path if
a pathological topk would not fit). The subchunking is over the token dimension
only, so per-token attention is unchanged and the result is numerically
identical. Verified: the GPU sweep runs past the old crash boundary to 22K+ with
zero CUDA errors; CPU and -ctk q8_0 paths unaffected.
* openpangu: fix scheduler node budget for chunked DSA prefill; drop unused attn_kv_b; remove env tunables
- Size the scheduler graph node budget for the chunked DSA prefill so 32K/ub2048 no
longer trips the hash-set reservation assert; derive the extra budget from the
builder's chunk/top-k/window structure with a fixed safety margin.
- Remove LLAMA_OPENPANGU_* environment tunables from both the node-budget estimator
and build_openpangu.cpp; use fixed constants in both so they stay in sync.
- Converter: emit only the split attn_k_b/attn_v_b projections and drop the unused
fused attn_kv_b tensor.
* openpangu: restore DeepSeek converter kv_b; drop trace env + dead code; fix dense-fallback node budget
- convert_hf_to_gguf.py: restore fused attn_kv_b in DeepseekV2Model (shared
parent); openPangu subclass keeps split-only k_b/v_b. Stops newly-converted
DeepSeek GGUFs from failing to load.
- src/llama.cpp: remove LLAMA_GRAPH_REUSE_TRACE getenv, hit/miss counters, and
the unconditional destructor log (no getenv or behavior change for any arch);
node-budget estimator now covers the dense-fallback (n_swa==0) attention-chunk
loop while skipping absent idx/top-k terms, preserving a strict overcount;
remove unreachable openPangu split-cache block.
- src/llama-context.h: drop now-dead graph_reuse_hits/misses members.
- include/llama.h: move type_k/type_v/idx_type_k *_explicit bools to struct end
to avoid a mid-struct ABI shift for out-of-tree consumers.
- src/graphs/build_openpangu.cpp: replace vestigial env-struct singletons with
the OPENPANGU_* constants; drop a redundant Sinkhorn permute round-trip
(one transpose; greedy output verified byte-identical).
Decode output unchanged (byte-identical greedy generation verified); shared-file
changes are openPangu-gated or restore the pre-PR baseline.
* openpangu: chat-parser support (reasoning split + thinking toggle)
Two openPangu-only fixes, both gated on the arch-unique token
<|pangu_text_start|> so no other model's parsing changes.
- chat-diff-analyzer: add a workarounds entry that force-sets TAG_BASED
reasoning with an empty start and a </think> end. openPangu prefills
<think> in the generation prompt, so the output is delimited only by
</think>; the differential detector otherwise learns start="<think>"
from the assistant-history form and fails to split, leaking reasoning
into content. Same shape as the existing Laguna prefill patch.
- chat.cpp: bridge enable_thinking to the template's `thinking` variable.
openPangu's template gates reasoning on `thinking` rather than the
ecosystem-standard `enable_thinking`, so the standard toggle was inert.
An explicit `thinking` chat_template_kwarg still overrides via the
extra_context merge.
Blast radius: test-chat-auto-parser 437/437 unchanged; the sole
test-chat-template diff is a pre-existing GLM trailing-newline.
* openpangu: use ggml_cast for latent dequant reads
Replace ggml_cpy(view, ggml_new_tensor_2d(F32, ...)) with ggml_cast in the MLA
latent V-from-K and K-read helpers. ggml_cast emits the identical GGML_OP_CPY
node into a fresh f32 tensor, so behavior is unchanged; it is the idiomatic
form. Per review.
* openpangu: narrow SWA reuse-key fields to 32-bit
The openpangu_swa_window_view reuse key stored n_kv/n_tokens/window/pad as
int64_t, but these are bounded well under 2^31 (window/pad are uint32_t at
source; n_kv/n_tokens <= context length). Narrow to int32_t/uint32_t and drop
the widening casts. w_view/win_off stay int64_t: they feed ggml view
dims/offsets. Per review.
* openpangu: precompute param_sink derived tensors at load
The per-layer attention-sink block (sink_blk [576,NS]) and its transposed
latent (s_lat_t [NS,512]) are pure functions of the layer weights, yet were
rebuilt every eval across all 49 layers (RMS-norm + cast + concat + transpose).
Compute them once at load, mirroring the wk_b derived-weight precompute, and
read the stored tensors in build_openpangu_attention. Numerically identical;
removes per-token work at decode.
* openpangu: replace conv position-ring with ggml_ssm_conv + spec-rollback checkpoint
Migrate the MoME depthwise causal conv (three sites per attention sublayer:
qa-lora, compressed-kv, attn-out) from the bespoke 16-column position-indexed
ring onto the core ggml_ssm_conv op with a recurrent conv-state slot.
Cache: s_l becomes [2*conv_col_ne, qnext_state_slots], holding the (d_conv-1)=2
history taps per channel for the three sites (float offsets 0 / 2*n_lora_q /
2*(n_lora_q+n_lora_kv)). Drops the conv_hist_idx / conv_write_idx graph inputs
and their fill in llama_set_inputs; adds one single-sequence sq input for
ggml_ssm_conv shared across the three sites and the MTP head.
Speculative rollback: the position ring self-healed rejected draft columns by
absolute position; a recurrent slot does not, since seq_rm is a no-op for
recurrent state. openPangu is admitted at the three spec-checkpoint save/init
gates so the whole-slot shadow checkpoint (gpu-fallback) snapshots the conv
slot before drafting and restores it before the accepted-token replay. The
restore path is already keyed on ckpt.valid, so no gate change is needed there.
Per-step checkpoint mode is declined for openPangu, which has no SSM recurrent
term, so auto mode resolves to the whole-slot shadow.
Gated: non-spec needle unchanged; MTP-spec needle correct with healthy draft
acceptance (rollback verified via the acceptance canary).
* openpangu: single ggml_concat copy for the latent cache store
The non-quantized latent store split the [ckv | roped k_pe] row into two views
and two cache copies, with a base_offset field on the CacheCopy struct to place
the second one. Match the quantized path: concat the two parts and do one copy
into the cache row. This drops the second cache-copy slot (OPENPANGU_COPY_K_KPE)
and removes base_offset from CacheCopy entirely.
Cache contents are unchanged: the concat writes the same [ckv 512 | k_pe 64]
bytes to the same row. Gated on the needle for both the f16 latent path (the one
that changed) and the q8 latent path, plus coherence.
* openpangu: reuse the shared kr_l indexer cache instead of a separate idx_l
The DSA lightning indexer stored its per-position keys in an openPangu-only idx_l
cache, parallel to the kr_l indexer cache GLM-DSA already uses. Both have the same
storage contract: [indexer_head_size, kv_size], idx_type_k dtype, one row per KV
cell, written at kv_head and read [dim, n_kv] from zero. openPangu now allocates
its indexer keys into kr_l and shares the dsa_cache_copies graph-reuse fixup.
The fixup patch is factored into a helper that both the generic path and the
openPangu update_cache_copies branch call, so the openPangu indexer copy is
repointed to the current kv_head on graph reuse like every other cache write.
This drops the idx_l vector, its allocation and memory accounting, and the
openPangu third cache-copy slot (now one latent copy per layer).
Per-arch allocation predicates stay separate (GLM uses indexer_is_full, openPangu
uses the window==0 DSA schedule); only the kr_l storage and the copy fixup are
shared. openPangu keeps its no-shift/no-defrag/no-state-I/O behavior, and the GLM
Hadamard/k-shift logic stays GLM-gated.
Gated: needle correct on f16 and q8 latent caches and under MTP speculation
(acceptance unchanged at 0.67), plus coherence.
* openpangu: discard pos-0 graphs from reuse; retire stale conv-state comments
The ggml_ssm_conv refactor bakes the pos-0 conv-state reset into graph
topology (a scale-by-zero node on the state view). A graph built at pos 0
could be reused at pos > 0 when the batch shape and padded n_kv match (a
1-token prompt followed by TG is the concrete case), zeroing the conv
history on every reused decode. Admit openPangu at the existing
reset_previous gate so pos-0 graphs are discarded from reuse, the same
guard the qnext recurrent state relies on.
Also retire the internal phase-plan comments the conv refactor left
behind: they claimed the spec-checkpoint wiring had not landed in the
commit that landed it, and misdescribed the s_l slot as awaiting rollback
support.
Gated: needle 8457 on f16 and q8 latent, MTP-spec needle (drafts fully
accepted), coherence.
* openpangu: drop the _explicit cache-type plumbing; validate unconditionally
Review follow-up (item 1 of the second review). The explicit/default
distinction carried less than claimed: the latent K/V fallback was f16,
which is already the -ctk/-ctv and API default, so distinguishing unset
from set-to-the-default bought nothing, and the two bools were behaviorally
redundant. The only load-bearing use was the indexer cache, where openPangu
defaulted to f32 while -ictk defaults to f16. Gating the f16 indexer
directly (needle on f16 and q8 latent paths, MTP speculation, coherence)
shows no quality difference, so openPangu now takes the standard f16
indexer default and the f32 special case is gone. Default indexer cache
memory halves (64 -> 32 MiB at c 8192).
Removes type_k_explicit/type_v_explicit/idx_type_k_explicit from llama.h,
the cparams/mparams plumbing, and common; the resolve helpers become plain
unconditional validators, so -ctk q8_0 is honored and an unsupported type
errors out at load instead of silently coercing.
Gated: needle 8457 on the new f16-indexer default, on q8 latent with MTP
speculation, and with -ictk f32 explicitly honored (64 MiB f32 buffer in
the load log); -ctk q4_0 and -ictk q4_1 refused with a clear error.
* openpangu: keep MTP draft decodes position-contiguous under speculation
The MTP framework's one-token draft shortcut caches a prediction one row
past the accepted prefix during the accepted-token update, then skips
re-decoding the last sampled token at the next draft round. A
mask-addressed cache tolerates the resulting position gap; openPangu's
position-addressed append-only cache (cell == position) does not: after a
rollback the next draft decode lands one cell behind its position, and
after a full acceptance the cache head sits one row ahead of the next
draft base, either way tripping the kv_head == pos[0] invariant and
aborting the server. The checkpoint admission in the conv refactor made
this the standard openPangu speculative flow; the needle-first gates
never generated enough draft rounds against a short prompt to reach it.
Decline the shortcut re-seed for openPangu in mtp_accept_batch (restoring
the drafting behavior all measured acceptance numbers were taken on) and
trim rows at or beyond the draft base in mtp_speculative_gen_draft, so
every draft decode stays position-contiguous with the cache head.
Gated: the crashing flow (short prompt, 512-token spec generation, then a
second request) completes with acceptance 0.60 prose / 0.87 code,
matching the pre-checkpoint baseline profile; needle 8457 plus coherence
on f16+spec and q8+spec.
* openpangu: remove stale ring limits and fix MTP graph reuse
* cli: preserve speculative carry on fallback
Decode an already-emitted pending token when a draft cannot be used instead of sampling unchanged logits and duplicating output. Document single-head MTP as the default and multi-head modes as experimental.
---------
Co-authored-by: Joel Farthing <262452229+joelfarthing@users.noreply.github.com>
ik_llama.cpp: llama.cpp fork with better CPU performance
TL;DR
This repository is a fork of llama.cpp with better CPU and hybrid GPU/CPU performance, new SOTA quantization types, first-class Bitnet support, better DeepSeek performance via MLA, FlashMLA, fused MoE operations and tensor overrides for hybrid GPU/CPU inference, row-interleaved quant packing, etc.
Important
If you are running hybrid CPU/GPU inference for MoE models with all or some experts left on the CPU, do not use -rtr unless you know what you are doing. The
-rtroption causes all tensors left in RAM to be repacked to row-interleaved format while loading the model. As not all quantization types have a CUDA implementation, this will result in matrix multiplications with these tensors to be always done on the CPU, even when it would have been much better to offload the computation to the GPU, typically resulting in much lower prompt processing speed. Most notably, k-quants (K2_K, Q3_K, Q4_K, Q5_K, Q6_K) do not have CUDA row-interleaved implementation.
Note
The only fully functional and performant compute backends are CPU (
AVX2or better,ARM_NEONor better) and CUDA (Turing or newer). Please do not enter issues related to ROCm, Vulkan, Metal, old Nvidia GPUs,AVXCPUs, etc. They will not get resolved unless you roll up your sleeves and help bring your favorite backend up to speed. With the current regular contributors this project simply does not have the bandwidth to work on all backends available inllama.cpp.
Important
Do not use quantized models from Unsloth that have
_XLin their name. These are likely to not work withik_llama.cpp.The above has caused some stir, so to clarify: the Unsloth
_XLmodels that are likely to not work are those that containf16tensors (which is never a good idea in the first place). All others are fine.
Note
Some users have reported issues with graph parallel (a.k.a. split mode
graph) and partial GPU offload (using--cpu-moeor--n-cpu-moeor tensor overrides). If you are using/want to use split mode graph and observe gibberish/incoherent responses, try adding-cuda graphs=0to your command line.
Quickstart
Prerequisites
git clone https://github.com/ikawrakow/ik_llama.cpp
cd ik_llama.cpp
On Debian/Ubuntu Linux, install the required packages (if using another Linux distro, you need to find the corresponding packages and adapt):
apt-get update && apt-get install build-essential git libcurl4-openssl-dev curl libgomp1 cmake
Build for CPU
cmake -B build -DGGML_NATIVE=ON
cmake --build build --config Release -j$(nproc)
For AVX-512-capable CPUs (AMD Zen4 / Intel Sapphire Rapids+), see
docs/build.md section "CPU build flags for AVX-512" for the
additional flags that activate the IQK quantized GEMM kernels (the
HAVE_FANCY_SIMD path). Without those flags, a vanilla Release build
silently falls back to the AVX2 path on this hardware.
Build for GPU
Install Nvidia Drivers and CUDA Toolkit.
cmake -B build -DGGML_NATIVE=ON -DGGML_CUDA=ON
cmake --build build --config Release -j$(nproc)
Step-by-step instructions for a case of a successful Windows build
https://github.com/ikawrakow/ik_llama.cpp/blob/main/docs/build.md
Run
Download .gguf model files (e.g. bartowski/Qwen_Qwen3-0.6B-IQ4_NL.gguf) to your favorite directory (e.g. /my_local_files/gguf).
Start the server with one of the commands (CPU or GPU):
./build/bin/llama-server --model /my_local_files/gguf/Qwen_Qwen3-0.6B-IQ4_NL.gguf --ctx-size 4096
./build/bin/llama-server --model /my_local_files/gguf/Qwen_Qwen3-0.6B-IQ4_NL.gguf --ctx-size 4096 -ngl 999
That's all! Open http://127.0.0.1:8080 in Browser start chatting.
Step by step guide for ik_llama.cpp in podman/docker container including llama-swap
Common parameters and options
Latest News
Model Support
LlaMA-3-Nemotron PR 377, Qwen3 PR 355, GLM-4 PR 344, Command-A PR 341, bitnet-b1.58-2B-4T PR 337, LLaMA-4 PR 321, Gemma3 PR 276, DeepSeek-V3 PR 176, Kimi-2 PR 609, dots.llm1 PR 573, Hunyuan PR 565, GLM-4.5 PR 668 (4.5/4.6/4.7/AIR), Ernie 4.5 MOE and 0.3B PR 759, grok-2 PR 782, Ling/Ring (Bailing-MoE2) PR 833, Qwen3-VL PR 883, SmolLM3 PR 934, GigaChat3 PR 995, ministral3 PR 1030, Mimo-V2-Flash PR 1096, GLM-4.7-Flash PR 1168, Seed-OSS PR 1218, Step-3.5-Flash PR 1231, GLM-5 PR 1268, Qwen3-Next PR 1266, Qwen3.5-MoE PR 1288 and dense Qwen-3.5 1326, Mistral 4 PR 1450, Bonsai 1-bit PR 1570, Gemma4 PR 1581 including assistant, Mimo-2.5 PR 1723, JetBrains Mellum2 PR 1919, Poolside Laguna XS.2 PR 1911, Cohere2-MoE North Mini Code PR 1945, MiniMax-M3 PR 1963, Laguna M.1 PR 2003
Quantization
Quantization additions
Trellis quants (IQ1_KT, IQ2_KT, IQ3_KT, IQ4_KT)
Information and the original CUDA implementation in PR 113. Additional implementations: Metal PR 475, Neon PR 471, CPU PR 441. IQ1_KT was added more recently in PR 616. Note: these are base on a novel, integer-base trellis, which allows to achieve reasonable CPU performance, see PR 529 and PRs quoted there for details.
IQK quants
Information can be found in Discussion 8.
Initial implementations (Zen4, AVX2, NEON): IQ5_KS_R4 PR 426, IQ5_KS PR 422, IQ4_KS_R4 PR 150, IQ5_K_R4 PR 149, IQ2_K_R4 PR 146, IQ3_K_R4 PR 145, IQ4_K_R4 PR 138, IQ4_KSS PR 89, IQ2_KS PR 85, IQ4_KS PR 83, IQ6_K PR 14, IQ2_K, IQ3_K and IQ5_K PR 7, IQ4_K PR 6
Cuda implementations: IQ4_KS_R4 and IQ5_KS_R4 PR 493, IQ1_S_R4 PR 492, IQ1_M_R4 PR 494. IQ4_KS_R4 and IQ5_KS_R4 PR 462, IQ2_K_R4, IQ3_K_R4, IQ4_K_R4, IQ5_K_R4 PR 461, IQ4_K, IQ5_K, IQ6_K PR 417, IQ2_KS, IQ2_K, IQ3_K PR 418
IQ2_KL is a more recent addition in PR 602
Hadamard transforms for K-cache
Hadamard transforms for V-cache
MXFP4 as used in gpt-oss models
Implemented for Zen4, AVX2, ARM_NEON, Metal, CUDA PR 682
Quantization improvements
IQ1_MPR 327,IQ2_XSPR 312,Q2_K, Q4_K, Q5_K, Q4_1, Q5_1PR 302,Q4_0, Q5_0, Q6_0, Q3_K, Q6_K, IQ4_XS, IQ4_NLPR 295- Low perplexity
Q4_0KV cache PR 1547 PR 1556 - MTP: option to use re-quantized output tensor
--mtp-requantize-output-tensor new_typePR 1809
Quantization performance improvements
- Much faster CPU prompt processing for all non-interleaved quants. Initial idea in PR 515 and PR 531, with many follow up PRs to apply to all quantization types for the 3 supported CPU platforms.
- All quantization types now have quantized matrix multiplication CUDA kernels, see PR 557 and several others
- Faster CPU prompt processing for Trellis quants and MoE models. PR 488
- Trellis quants: faster CPU prompt processing PR 482.
- Minor (~2%)
iq2_ksTG performance improvement on CUDA PR 468 - Faster
IQ3_KTandIQ4_KTPR 453 - Zen4: Faster PP for
IQ2_KS, IQ4_KS, IQ5_KSPR 428 - Fast GEMM/GEMV for
IQ1_SPR 212 - AVX-VNNI optimizations PR 1446 PR 1455 PR 1467 PR 1474 PR 1482
Features
- New split mode "graph" for multi GPU setups PR 1022
- Fused delta-net for Qwen3-Next and Qwen3.5-MoE PR 1315 PR 1333 PR 1362 PR 1373
- Hadamard transforms for K-cache and V-cache PR 1033 PR 1034 PR 1527
- Auto-fit offloaded tensors to available VRAM (MoE and dense models) PR 1501 PR 1504, allows per GPU fit margin PR 1872
- Checkpoints for recurrent models PR 1310 PR 1398
- MTP decoding support for popular models like GLM-4.x MoE 1270, Qwen 3.5/3.6 1698 1745, Gemma 4 1744, GLM 5 1890
- Self speculative decoding, ngram PR 1261, suffix PR 1646
- DFlash initial support PR 1970
- GLM-DSA architecture indexer cache
- String ban function for all completions PR 1185 PR 1243
- Expiring Logit Bias PR 1731
- OpenAI
/v1/responsesAPI endpoint PR 1184 - Function call support PR 628
- jinja template support PR 677
- Webui: New Features for Conversations, Settings, and Chat Messages PR 618, MCP PR 1904
- On-demand tensor reload PR 1989
- Dynamic control vector management endpoints PR 1223
- Legacy quants conversion schemes in
convert_hf_to_gguf.pyPR 449,Q6_0in PR 483 - Adaptive-P Sampler PR 1100 implemented as designed by it's author; supported on Webui
- Multi-modal Vision support in
llama-mtmd-cliPR 798 and inllama-serverPR 901 - mikupad as an alternative WebUI PR 558
- June 8 2025: Webui updated (legacy still available when
--path ./examples/server/public_legacyis passed) PR 481 - June 8 2025: RPC improvements PR 480
- June 7 2025: Add an endpoint that lists all the saved prompt caches to server PR 502
- June 6 2025: Make prompt cache saving and restoring MLA aware PR 497
- June 3 2025: Added samplers, XTC PR 486, top-n σ PR 489.
- May 22 2025: Refactor
iqk_mul_mat.cppwhich speeds up compilation time significantly. PR 435 - May 17 2025: Option to enable or disable the CPU FA kernels PR 429.
- May 12 2025: User can now control if/which operations with tensors held in RAM are offloaded to the GPU. See PR 405
- May 12 2025: Compatibility issues with mainline
llama.cppGGUFs for DeepSeek models with MLA enabled were resolved in PR 394. The lower prompt processing performance resulting from usingllama.cpp-style MLA GGUFs was recovered in PR 409. - April 21 2025: ik_llama.cpp builds and runs successfully on Android (using termux), see PR 336
- March 1 2025: Smart Expert Reduction for faster DeepSeek inference PR 239
- Feb 25 2025: Tensor overrides for better control where model weights are stored (GPU or CPU) PR 232
- Feb 23 2025:
sweep-bench- better performance benchmarking PR 225 - Feb 19 2025:
Q8_KV- new type for 8-bit KV-cache quantization PR 208 - March 7 2025: Custom quantization mixes using regular expressions PR 244
Performance improvements
- Better GPU offload strategy for MoE models when using hybrid HPU/CPU inference, see PR 520
- Much faster rng sampling PR 1187
- May 13 2025: Better CPU FA performance for DeepSeek-Lite. PR 410
- May 11 2025: Slightly faster flash attention for DeepSeek models on CUDA, along with extending compatibility to Touring or newer GPUs. PR 408
- May 4 2025: Significant token generation performance improvement on CUDA with Flash Attention for GQA models. For details and benchmarks. PR 370
- April 17 2025: Better CPU Flash Attention token generation performance. PR 332
- April 3 2025: Much faster MoE implementation on Metal. PR 307
- March 25 2025: Better MoE performance on CUDA PR 283
- March 23 2025: Better batched processing speed for DeepSeek models PR 282
- March 18 2025: Reduce compute buffer size PR 237
- March 10 2025: Better TG performance for MoE models on CUDA PR 248
- Feb 23 2025: Fused FFN ops for faster MoE inference PR 229
Flash-MLA
- May 7 2025: 🚀 FlashMLA-3 for DeepSeek models on CUDA. PR 386. Caveat: Ampere or newer Nvidia GPU required
- March 21 2025: 🚀 FlashMLA-3: fastest CPU-only inference for DeepSeek models PR 273
- March 17 2025: 🚀 FlashMLA-2 performance improvements PR 253
- March 12 2025: Allow
Q8_0KV cache with FlashMLA-2 on CUDA PR 265 - March 9 2025: 🚀 FlashMLA on CUDA PR 247
- March 8 2025: 🚀 Faster FlashMLA CPU implementation PR 243
- March 3 2025: 🚀 Introducing FlashMLA - MLA with Flash Attention PR 240
- Feb 27 2025: MLA without transposed cache PR 235
- Feb 13 2025: Allow
Q8_0quantized cache with MLA PR 206 - Feb 11 2025: 🚀 Flash Attention support for DeepSeek models PR 200
- Feb 9 2025: 🚀 MLA for DeepSeek models PR 188
Fixes
- Fix bug in MMVQ kernel PR 446
- Fix AVX2 implementation of
IQ4_K, IQ4_KS, IQ5_K, IQ6_KPR 427 - Fix standard attention on the CPU PR 421
- Fix imatrix calculation for MLA models PR 411
- Fix new CUDA FA on Touring PR 413
- Fix SER. CPU: PR 415 CUDA: PR 416
Resources
There is no single point of reference describing all new ik_llama.cpp features. Pull requests often contain detailed information, so browsing the PRs is often the best way to learn about new features and how to use them. In addition
- The Wiki page has performance comparisons to mainline
llama.cpp - This guide is a good place to start if you came here because of DeepSeek models
- This discussion is about running DeepSeek-V3/R1 on a 16 x 3090 setup
- This discussion describes the new quantization types available in
ik_llama.cpp
Testing
Function Calls Tests
To run the function calls test suite:
cd build
cmake --build . --target test-function-calls
./bin/test-function-calls
The test suite covers parser functionality, streaming, error handling, content cleaning, and server integration. All tests should pass to ensure production readiness.
Contributing
Contributions in form of pull requests, issue submissions (bug reports, feature requests), or general discussions, are welcome.
License
- subprocess.h - Single-header process launching solution for C and C++ - Public domain
- server
- GBNF grammars
Development documentation
Seminal papers and background on the models
If your issue is with model generation quality, then please at least scan the following links and papers to understand the limitations of LLaMA models. This is especially important when choosing an appropriate model size and appreciating both the significant and subtle differences between LLaMA models and ChatGPT:
- LLaMA:
- GPT-3
- GPT-3.5 / InstructGPT / ChatGPT:
Completions
Command-line completion is available for some environments.
Bash Completion
$ build/bin/llama-cli --completion-bash > ~/.llama-completion.bash
$ source ~/.llama-completion.bash
Optionally this can be added to your .bashrc or .bash_profile to load it
automatically. For example:
$ echo "source ~/.llama-completion.bash" >> ~/.bashrc
Dependencies
- yhirose/cpp-httplib - Single-header HTTP server, used by
llama-server- MIT license - stb-image - Single-header image format decoder, used by multimodal subsystem - Public domain
- nlohmann/json - Single-header JSON library, used by various tools/examples - MIT License
- miniaudio.h - Single-header audio format decoder, used by multimodal subsystem - Public domain
- subprocess.h - Single-header process launching solution for C and C++ - Public domain