6 Commits
Author SHA1 Message Date
Adrien GallouëtandGitHub aff6eb6e75 tokenize : drop --stdin mutual-exclusion check (#25672)
match cli and completion, which don't enforce it
2026-07-15 18:41:51 +02:00
Adrien GallouëtandGitHub 8ff8c4299d tokenize : align usage by using common args (#25516)
Migrate the tokenize tool to common_params_parse, replacing its
hand-rolled argv parsing, Windows UTF-8 handling and file reading
with the shared common helpers.

Expose the model-sourcing flags (-m, -mu, -dr, -hf, -hff, --offline,
HF_TOKEN) to LLAMA_EXAMPLE_TOKENIZE, and register --ids, --stdin,
--no-bos, --no-parse-special and --show-count as common args.
parse_special defaults to true for TOKENIZE to preserve the old
behavior. Errors now go through LOG_ERR instead of fprintf(stderr).

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-07-14 15:20:53 +02:00
Georgi GerganovandGitHub 6990e2f1f7 libs : rename libcommon -> libllama-common (#21936)
* cmake : allow libcommon to be shared

* cmake : rename libcommon to libllama-common

* cont : set -fPIC for httplib

* cont : export all symbols

* cont : fix build_info exports

* libs : add libllama-common-base

* log : add common_log_get_verbosity_thold()
2026-04-17 11:11:46 +03:00
cb8f4fa3f8 Fix locale-dependent float printing in GGUF metadata (#17331)
* Set C locale for consistent float formatting across all binaries.

* Add C locale setting to all tools binaries

Add std::setlocale(LC_NUMERIC, "C") to all 16 binaries in the tools/
directory to ensure consistent floating-point formatting.

* Apply suggestion from @JohannesGaessler

---------

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
2026-03-04 09:30:40 +01:00
Yuri KhrustalevandGitHub 07808ebb07 cmake : Do not install tools on iOS targets (#15903) 2025-09-16 09:54:44 +07:00
1d36b3670b llama : move end-user examples to tools directory (#13249)
* llama : move end-user examples to tools directory

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2025-05-02 20:27:13 +02:00