furyhawk/ik_llama.cpp
1
0
Fork 0
mirror of https://github.com/ikawrakow/ik_llama.cpp.git synced 2026-07-21 02:05:35 +00:00
Code Issues Packages Projects Releases Wiki Activity
4,735 Commits 952 Branches 1 Tag
main
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Johannes GäßlerGitHubslaren
 		5b4e0a2a38 common: fix warning (#8036) 
* common: fix warning

* Update common/common.cpp

Co-authored-by: slaren <slarengh@gmail.com>

---------

Co-authored-by: slaren <slarengh@gmail.com>
 2024-06-20 16:40:13 +02:00
Johannes GäßlerandGitHub 24dfdbb1a3 CUDA: stream-k decomposition for MMQ (#8018) 
* CUDA: stream-k decomposition for MMQ

* fix undefined memory reads for small matrices
 2024-06-20 14:39:21 +02:00
Johannes GäßlerandGitHub 8c1fed631e CUDA: faster q2_K, q3_K MMQ + int8 tensor cores (#7921) 
* CUDA: faster q2_K, q3_K MMQ + int8 tensor cores

* try CI fix

* try CI fix

* try CI fix

* fix data race

* rever q2_K precision related changes
 2024-06-14 18:41:49 +02:00
Johannes GäßlerandGitHub c9ff309cdb CUDA: fix broken oob check for FA vec f32 kernel (#7904) 2024-06-12 17:41:51 +02:00
Johannes GäßlerandGitHub a9e26d8c45 llama-bench: more compact markdown tables (#7879) 2024-06-11 14:45:40 +02:00
Johannes GäßlerandGitHub cb7240ad05 CUDA: int8 tensor cores for MMQ (q4_K, q5_K, q6_K) (#7860) 2024-06-11 08:26:07 +02:00
Johannes GäßlerandGitHub fc5f0f1647 CUDA: use tensor cores for MMQ (#7676) 
* CUDA: int8 tensor cores for MMQ (legacy quants)

* fix out-of-bounds writes

* __builtin_assume -> GGML_CUDA_ASSUME

* fix writeback returning too early
 2024-06-10 11:45:13 +02:00
Johannes GäßlerandGitHub d4dc8e168b CUDA: revise q8_1 data layout for mul_mat_q (#7824) 2024-06-09 09:42:25 +02:00
Johannes GäßlerandGitHub e8ac0b3518 server: update cache_prompt documentation [no ci] (#7745) 2024-06-07 11:15:49 +02:00
Johannes GäßlerandGitHub 4ac499892a CUDA: refactor mmq, dmmv, mmvq (#7716) 
* CUDA: refactor mmq, dmmv, mmvq

* fix out-of-bounds write

* struct for qk, qr, qi

* fix cmake build

* mmq_type_traits
 2024-06-05 16:53:00 +02:00
Johannes GäßlerandGitHub 57c1334819 make: fix debug options not being applied to NVCC (#7714) 2024-06-03 16:28:58 +02:00
Johannes GäßlerandGitHub 0f8cdc3051 Fix FlashAttention debug test, FP32 assert (#7684) 2024-06-01 23:26:10 +02:00
Johannes GäßlerandGitHub 474d8c7bac CUDA: fix Pascal FA, deq. KV to FP16 for batch > 8 (#7681) 2024-06-01 15:47:04 +02:00
Johannes GäßlerandGitHub 0ef2e997e2 CUDA: quantized KV support for FA vec (#7527) 
* CUDA: quantized KV support for FA vec

* try CI fix

* fix commented-out kernel variants

* add q8_0 q4_0 tests

* fix nwarps > batch size

* split fattn compile via extern templates

* fix flake8

* fix metal tests

* fix cmake

* make generate_cu_files.py executable

* add autogenerated .cu files

* fix AMD

* error if type_v != FP16 and not flash_attn

* remove obsolete code
 2024-06-01 08:44:14 +02:00
Johannes GäßlerandGitHub 4f660b95b5 scripts: update compare_llama_bench.py [no ci] (#7673) 2024-05-31 16:26:21 +02:00
Johannes GäßlerandGitHub 01a30eac03 README: explain parallel build [no ci] (#7618) 2024-05-30 09:52:39 +02:00
Johannes GäßlerandGitHub 22109fa246 make: add --device-debug to NVCC debug flags (#7542) 2024-05-27 19:34:40 +02:00
Johannes GäßlerandGitHub 6e74ae0a64 CUDA: fix FA out-of-bounds reads (#7479) 2024-05-23 00:31:20 +02:00
Johannes GäßlerandGitHub b491ac4240 CUDA: fix FA out-of-bounds writes (#7465) 2024-05-22 17:58:25 +02:00
Johannes GäßlerandGitHub 71bf04b8bd CUDA: remove incorrect precision check (#7454) 2024-05-22 10:24:29 +02:00
Johannes GäßlerandGitHub 1f2bce9bc2 CUDA: fix unused warning in mmq.cu (#7442) 2024-05-21 20:27:12 +03:00
Johannes GäßlerandGitHub 260949cad5 CUDA: deduplicate mmq code (#7397) 2024-05-21 16:02:12 +02:00
Johannes GäßlerandGitHub a2a24aec6f perplexity: update README FP16 results [no ci] (#7413) 2024-05-20 18:15:38 +02:00
Johannes GäßlerandGitHub 59a38d4847 ggml: implement quantized KV cache for FA (#7372) 2024-05-19 16:46:13 +02:00
Johannes GäßlerandGitHub a742a54fd0 server: add test for token probs (#7347) 2024-05-19 16:26:02 +02:00
Johannes GäßlerandGitHub 9ae757d0b5 server: fix seed being reported back (#7382) 2024-05-19 17:06:33 +03:00
Johannes GäßlerandGitHub 6fe8769d65 CUDA: deduplicate FlashAttention code (#7352) 2024-05-18 12:36:25 +02:00
Johannes GäßlerandGitHub 97cd158809 server: correct --threads documentation [no ci] (#7362) 2024-05-18 11:10:47 +02:00
Johannes GäßlerandGitHub 9eecf291b4 CUDA: faster large batch FA without tensor cores (#7314) 2024-05-17 18:54:52 +02:00
Johannes GäßlerandGitHub 228b8cd135 tokenization: add warning for double BOS (#7332) 2024-05-17 09:59:57 +02:00
Johannes GäßlerandGitHub cbd77f7974 server bench: fix bench not waiting for model load (#7284) 2024-05-15 08:44:16 +02:00
Johannes GäßlerandGitHub 2d2147923e perplexity: add BF16 vs. FP16 results (#7150) 2024-05-13 13:03:27 +02:00
Johannes GäßlerandGitHub 871641d19e CUDA: add FP32 FlashAttention vector kernel (#7188) 
* CUDA: add FP32 FlashAttention vector kernel

* fixup! CUDA: add FP32 FlashAttention vector kernel

* fixup! fixup! CUDA: add FP32 FlashAttention vector kernel

* fixup! fixup! fixup! CUDA: add FP32 FlashAttention vector kernel
 2024-05-12 19:40:45 +02:00
Johannes GäßlerandGitHub 70a18260b2 server: fix reported top tokens for temperature 0 (#7203) 2024-05-11 10:11:28 +02:00
Johannes GäßlerandGitHub b02a859891 CUDA: generalize FP16 fattn vec kernel (#7061) 
* CUDA: generalize FP16 fattn vec kernel

* disable unsupported head sizes for AMD in test

* try AMD fix

* fix batch size 2-8

* partially revert changes
 2024-05-09 14:32:02 +02:00
Johannes GäßlerandGitHub 4e708b6d16 JSON: [key] -> .at(key), assert() -> GGML_ASSERT (#7143) 2024-05-08 21:53:08 +02:00
Johannes GäßlerandGitHub e56a09c3dd server: fix incorrectly reported token probabilities (#7125) 
* server: normalize token probabilities

* fix temperature == 0.0f
 2024-05-07 23:07:58 +02:00
Johannes GäßlerandGitHub a350b6a103 Server: add tests for batch size, different seeds (#6950) 2024-05-01 17:52:55 +02:00
Johannes GäßlerandGitHub 2dd44593ef CUDA: CUDART < 11.7 workaround for __hmax, __hmax2 (#7019) 2024-05-01 14:46:37 +02:00
Johannes GäßlerandGitHub 5ab303493c perplexity: more statistics, added documentation (#6936) 
* perplexity: more statistics, added documentation

* add LLaMA 3 8b scoreboard
 2024-04-30 23:36:27 +02:00
Johannes GäßlerandGitHub ca998d7836 llama : fix typo LAMMAFILE -> LLAMAFILE (#6974) 2024-04-29 15:36:22 +03:00
Johannes GäßlerandGitHub 4adf11a516 README: add graphic for matrix multiplication (#6881) 2024-04-24 21:29:13 +02:00
Johannes GäßlerandGitHub 95878a0936 Server: fix seed for multiple slots (#6835) 
* Server: add tests for consistent results

* sampling: separate rng per sampling context
 2024-04-24 11:08:36 +02:00
Johannes GäßlerandGitHub c419c245fb CUDA: fix matrix multiplication logic for tests (#6667) 2024-04-14 00:21:55 +02:00
Johannes GäßlerandGitHub 5a5d9cbbe2 compare-llama-bench.py: fix long hexsha args (#6424) 2024-04-01 13:30:43 +02:00
Johannes GäßlerandGitHub df091faf85 Fixed lookup compilation issues on Windows (#6273) 2024-03-24 14:21:17 +01:00
Johannes GäßlerandGitHub 56d74c8210 lookup: complement data from context with general text statistics (#5479) 
* lookup: evaluation tools, use corpus/previous gens

* fixup! lookup: evaluation tools, use corpus/previous gens

* fixup! lookup: evaluation tools, use corpus/previous gens

* fixup! lookup: evaluation tools, use corpus/previous gens

* fixup! lookup: evaluation tools, use corpus/previous gens
 2024-03-23 01:24:36 +01:00
Johannes GäßlerandGitHub 470cef7162 CUDA: fix DEBUG_CUDA_MALLOC (#5729) 2024-02-26 15:36:38 +01:00
Johannes GäßlerGitHubJared Van Bortel
 		990dfea10c make: add error message for bad CUDA version (#5444) 
* make: add error message for bad CUDA version

* Update Makefile

Co-authored-by: Jared Van Bortel <cebtenzzre@gmail.com>

---------

Co-authored-by: Jared Van Bortel <cebtenzzre@gmail.com>
 2024-02-13 12:38:37 +01:00
Johannes GäßlerGitHubslaren
 		8377d606b8 CUDA: mul_mat_vec_q tiling, refactor mul mat logic (#5434) 
* CUDA: mul_mat_vec_q tiling, refactor mul mat logic

Co-authored-by: slaren <slarengh@gmail.com>

---------

Co-authored-by: slaren <slarengh@gmail.com>
 2024-02-11 19:08:39 +01:00
Johannes GäßlerandGitHub 6834564e30 lookup: add print for drafting performance (#5450) 2024-02-11 12:44:51 +01:00
Johannes GäßlerandGitHub 788f40265d CUDA: more warps for mmvq on NVIDIA (#5394) 2024-02-08 21:56:40 +01:00
Johannes GäßlerandGitHub 5d5a6cfa7b fix trailing whitespace (#5407) 2024-02-08 11:36:54 +01:00
Johannes GäßlerGitHubGeorgi Gerganov
 		5a19feb61f sampling: fix top_k <= 0 (#5388) 
* sampling: fix top_k <= 0

* Update llama.cpp

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2024-02-08 09:46:30 +01:00
Johannes GäßlerandGitHub f3a0ec2947 CUDA: fixed mmvq kernel for bs 2,3,4 and -sm row (#5386) 2024-02-07 12:40:26 +01:00
Johannes GäßlerandGitHub e3ee2b0879 CUDA: mul_mat_vec_q max. batch size 8 -> 4 (#5370) 2024-02-06 19:43:06 +02:00
Johannes GäßlerandGitHub d293e063ce CUDA: mul_mat_vec_q for batch sizes > 1 (#5351) 2024-02-06 14:44:06 +01:00
Johannes GäßlerandGitHub d9e98a5b79 make: Use ccache for faster compilation (#5318) 
* make: Use ccache for faster compilation
 2024-02-05 19:33:00 +01:00
Johannes GäßlerGitHubGeorgi Gerganov
 		468c6071d7 README: updated introduction (#5343) 
* README: updated introduction

* readme : update

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2024-02-05 15:55:10 +01:00
Johannes GäßlerandGitHub 3f5c2d9c0d make: add nvcc info print (#5310) 2024-02-03 20:15:13 +01:00
Johannes GäßlerandGitHub dacfbb8ada make: fix nvcc optimization flags for host code (#5309) 2024-02-03 20:14:59 +01:00
Johannes GäßlerandGitHub 371396c599 Apply min_p to unsorted tokens (#5115) 2024-01-28 09:59:49 +01:00
Johannes GäßlerandGitHub 93efcb8e31 Tests for min_p, sampling queue (#5147) 2024-01-28 09:35:14 +01:00
Johannes GäßlerandGitHub fa690025e6 CUDA: more info when no device code (#5088) 2024-01-23 13:31:56 +01:00
Johannes GäßlerandGitHub f2426af97b ggml: cache sin/cos for RoPE (#4908) 2024-01-13 21:41:37 +01:00
Johannes GäßlerandGitHub 7079775d2b compare-llama-bench: tweak output format (#4910) 2024-01-13 15:52:53 +01:00
Johannes GäßlerandGitHub 50f828eab9 CUDA: faster q8_0 -> f16 dequantization (#4895) 2024-01-12 20:38:54 +01:00
Johannes GäßlerandGitHub d80c163451 CUDA: fix softmax compile for old CUDA versions (#4862) 2024-01-12 12:30:41 +01:00
Johannes GäßlerandGitHub 3091bc1ab0 Python script to compare commits with llama-bench (#4844) 2024-01-10 01:04:33 +01:00
Johannes GäßlerandGitHub 635bddbf8b CUDA: faster softmax via shared memory + fp16 math (#4742) 2024-01-09 08:58:55 +01:00
Johannes GäßlerandGitHub 770ec541f9 CUDA: fixed redundant value dequantization (#4809) 2024-01-07 17:24:08 +01:00
Johannes GäßlerandGitHub b0a9bb90f9 Print backend name on test-backend-ops failure (#4751) 2024-01-04 09:43:23 +01:00
Johannes GäßlerandGitHub 15800d1753 CUDA: fixed tensor cores not being used on RDNA3 (#4697) 2023-12-30 13:52:01 +01:00
Johannes GäßlerandGitHub 77eb20b5ce CUDA: fix tensor core logic for Pascal and HIP (#4682) 2023-12-29 23:12:53 +01:00
Johannes GäßlerandGitHub 075c1781f6 CUDA: fixed row rounding for 0 tensor splits (#4594) 2023-12-23 09:16:33 +01:00
Johannes GäßlerandGitHub 0b657fce5a CUDA: mul_mat_id always on GPU for batches >= 32 (#4553) 2023-12-21 18:42:59 +01:00
Johannes GäßlerandGitHub 825bb4313f llama : disable per-tensor info prints on model load (#4562) 2023-12-21 18:34:17 +02:00
Johannes GäßlerGitHubslaren
 		164d9c4ddf CUDA: Faster Mixtral prompt processing (#4538) 
* CUDA: make MoE tensors contiguous for batch size>1

* Update ggml-cuda.cu

Co-authored-by: slaren <slarengh@gmail.com>

---------

Co-authored-by: slaren <slarengh@gmail.com>
 2023-12-20 15:41:22 +01:00
Johannes GäßlerandGitHub 2c58f6bb1b CUDA: use only 1 thread if fully offloaded (#2915) 2023-09-21 11:43:53 +03:00
Johannes GäßlerandGitHub 8821f4ae78 CUDA: fix peer access logic (#3231) 2023-09-17 23:35:20 +02:00
Johannes GäßlerandGitHub 94a0ea6e76 CUDA: enable peer access between devices (#2470) 2023-09-17 16:37:53 +02:00
Johannes GäßlerandGitHub 64dbee2c5e CUDA: fix scratch malloced on non-main device (#3220) 2023-09-17 14:16:22 +02:00
Johannes GäßlerGitHubHenri Vasserman
 		a1999004f0 CUDA: mul_mat_q RDNA2 tunings (#2910) 
* CUDA: mul_mat_q RDNA2 tunings

* Update ggml-cuda.cu

Co-authored-by: Henri Vasserman <henv@hot.ee>

---------

Co-authored-by: Henri Vasserman <henv@hot.ee>
 2023-09-13 11:20:24 +02:00
Johannes GäßlerandGitHub 99793f90fe CUDA: fix LoRAs (#3130) 2023-09-13 00:15:33 +02:00
Johannes GäßlerandGitHub 54ef27dff3 CUDA: fix mul_mat_q not used for output tensor (#3127) 2023-09-11 22:58:41 +02:00
Johannes GäßlerandGitHub f4cfe0248a CUDA: lower GPU latency + fix Windows performance (#3110) 2023-09-11 19:55:51 +02:00
Johannes GäßlerandGitHub 36e69da6ba CUDA: add device number to error messages (#3112) 2023-09-11 13:00:24 +02:00
Johannes GäßlerandGitHub d638c0a4ed CUDA: mul_mat_q=true llama_context_params default (#2912) 2023-08-30 21:46:19 +02:00
Johannes GäßlerandGitHub 76c5d8c5a4 YAML result logging + preset script (#2657) 2023-08-28 17:59:39 +02:00
Johannes GäßlerandGitHub adddb70f08 CUDA: fix RoPE asserts, block sizes (#2833) 2023-08-28 14:23:55 +03:00
Johannes GäßlerandGitHub 466a79f7b4 CUDA: use mul_mat_q kernels by default (#2683) 2023-08-22 22:47:05 +02:00
Johannes GäßlerandGitHub 0fc9b45a9a README: fix LLAMA_CUDA_MMV_Y documentation (#2647) 2023-08-17 23:57:59 +02:00
Johannes GäßlerandGitHub 677c32921e CUDA: launch_bounds, small q4_K, q5_K mmq refactor (#2596) 2023-08-14 10:41:22 +02:00
Johannes GäßlerandGitHub e1305c82c3 CUDA: Fixed OpenLLaMA 3b mmq, reduced compile time (#2590) 2023-08-13 00:24:45 +02:00
Johannes GäßlerandGitHub ade4f55531 CUDA: tuned mul_mat_q kernels (#2546) 2023-08-09 09:42:34 +02:00
Johannes GäßlerandGitHub fe11463cef CUDA: tighter VRAM scratch size for 65b/70b (#2551) 2023-08-08 14:38:16 +02:00
Johannes GäßlerandGitHub 19cf204509 Fixed mmap prefetch for GPU offloading (#2529) 2023-08-07 10:09:40 +02:00
Johannes GäßlerandGitHub d5060f4590 CUDA: faster k-quant mul_mat_q kernels (#2525) 2023-08-05 18:20:44 +02:00
Johannes GäßlerandGitHub 5dc74e92ed CUDA: faster non k-quant mul_mat_q kernels (#2483) 2023-08-02 18:04:04 +02:00
Johannes GäßlerandGitHub c06707dd78 CUDA: Fix models with output size != 32000 (#2480) 2023-08-02 16:48:10 +02:00