furyhawk/ik_llama.cpp
1
0
Fork 0
mirror of https://github.com/ikawrakow/ik_llama.cpp.git synced 2026-07-21 10:15:35 +00:00
Code Issues Packages Projects Releases Wiki Activity

Add Unicode allowlist (#1597)

Browse Source 
* initial commit

* cleanup

* fix whitelist arg parsing and simplify keyword search state

* rename white* to allow*

* add vocab_pieces init function, rename update functions, delete accidentally added file

* delete temporary bias code

* auto-generate fill function with script data inside

* deduplicate allowlist unicode rule parsing

* minor cleanup

* delete unnecessary header

* refactor allowlist to support sequential rule sets via keywords

* add early exit for zero-rules case

* delete accidentally added file
This commit is contained in:
dungquixote42
2026-04-10 18:22:57 +02:00
committed by GitHub
parent 5720a4131a
commit 869b83bc49
13 changed files with 2393 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
include/llama.h
+2
View File
@@ -1559,4 +1559,6 @@ llama_token llama_sample_token_with_rng(struct llama_context * ctx, llama_token_
#endif // LLAMA_API_INTERNAL
size_t llama_fill_from_utf8(void* utf8, void* cpts, void* scripts);
#endif // LLAMA_H