571d0d540d
model: rotate injected K/V cache for DFlash ( #25823 )
...
* dflash: rotate injected K/V cache when using K/V quantization
* Update src/models/dflash.cpp
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
* clearer format
* remove trailing whitespace
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2026-07-18 15:02:18 +02:00
Ruixiang Wang and GitHub
a55a8c5266
convert : fix dflash target tokenizer mismatch during conversion ( #25733 )
...
* spec: fix dflash target tokenizer mismatch during conversion
* fix ci ty check
2026-07-16 15:19:47 +03:00
Ruixiang Wang and GitHub
152d337fad
spec: support spec-draft-p-min in DFlash ( #25246 )
...
* spec: support spec-draft-p-min in DFlash
* dflash: add n_min guard
* dflash: guard both n_min and n_max
2026-07-03 15:40:06 +02:00
Ruixiang Wang and GitHub
fa72bc6826
dflash: refactor draft model conversion ( #25110 )
...
* dflash: refactor draft model conversion
* apply fix for eagle3 convert
2026-06-28 20:31:48 +02:00
d1b34251bc
spec : add DFlash support ( #22105 )
...
* spec: add DFlash v2 support
* dflash: support sliding window attention per layer_types
* docs: add dflash section
---------
Co-authored-by: Kashif Rasul <kashif.rasul@gmail.com >
2026-06-28 16:01:34 +03:00
b14e3fb90c
spec: support eagle3 for qwen3.5 & 3.6 ( #24593 )
...
* spec: support qwen3.5 & 3.6 eagle3 draft
* eagle3: Add deferred boundary checkpoints restore support for hybrid models
* apply suggestions
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
* spec: adapt to API change
* spec: fix naming
* cont : add TODO
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2026-06-19 13:08:50 +03:00
Ruixiang Wang and GitHub
1a2dea29b9
spec: fix segfault error on long prompts for eagle3 ( #24707 )
2026-06-17 17:29:49 +03:00
Ruixiang Wang and GitHub
a1824902b5
spec: add backend sampling support for eagle3 ( #24655 )
2026-06-16 12:05:52 +03:00
635b65ad7a
spec: add spec metrics mean acceptance length and acceptance rate per position ( #24536 )
...
* spec: add spec metrics mean acceptance length and acceptance per pos
* fix as suggestion
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
* fix as suggestion
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
* fix as suggestion
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
* fix as suggestions
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2026-06-16 10:23:09 +03:00
88a39274ec
spec: add EAGLE3 speculative decoding support ( #18039 )
...
* llama : enable layer input extraction
* spec: support eagle3
* eagle3: fix params bug
* eagle3: support Gemma4 eagle3 from RedHatAI
* eagle3: set sync when get features from target
Co-authored-by: tnhnyzc <115956684+tnhnyzc@users.noreply.github.com >
* eagle3 : fix ubatch handling in embd_layer_inp extraction and encoder
Co-authored-by: Doğaç Eldenk <dogacel@gmail.com >
* eagle3: adapt to upstream changes
* eagle3: fix rebase issues and adapt to upstream changes
* eagle3:exclude the eagle3 arch from test-llama-archs
* eagle3: fix editorconfig check failures
* eagle3: fix multi-seq issue in d2t vocab mapping
* cont : minor style / clean-up
* spec : remove `common_speculative_setup_draft_model()`
* llama : clean-up unused API
* eagle3: set d2t vocab mapping in decode graph
* cont : assert layer inputs are configured
* hparams : use n_embd_inp instead of n_embd_target_features
* eagle3: make output.weight optional and inherit from target model when needed
* haparams : generic norm-before-residual param
* llama-ext : consistent names
* cont : fix
* hparams : remove target_hidden_size
* cparams : rename output_layer_inp -> embeddings_layer_inp
* arch : reuse ATTN_NORM_2 instead of adding new hidden norm
* llama : clean-up names
* cont : add assert + comment
* Update conversion/llama.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
Co-authored-by: tnhnyzc <115956684+tnhnyzc@users.noreply.github.com >
Co-authored-by: Doğaç Eldenk <dogacel@gmail.com >
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
2026-06-12 10:21:06 +03:00
Ruixiang Wang and GitHub
689a9a470e
server-bench : add speed-bench for speculative decoding benchmarking ( #23869 )
...
* spec: add speed-bench support for benchmarking
* speed-bench : add trailing newline to requirements.txt
* speed-bench : bump datasets to 4.8.0 to fix ty check
* server-bench : remove now-unused type: ignore after datasets bump
2026-05-29 23:09:47 +02:00
Ruixiang Wang and GitHub
2fc8d1851e
doc: fix spec mtp typo ( #23435 )
2026-05-21 09:30:55 +03:00
