Adrien GallouëtandGitHub 8ff8c4299d tokenize : align usage by using common args (#25516) ...

Migrate the tokenize tool to common_params_parse, replacing its
hand-rolled argv parsing, Windows UTF-8 handling and file reading
with the shared common helpers.

Expose the model-sourcing flags (-m, -mu, -dr, -hf, -hff, --offline,
HF_TOKEN) to LLAMA_EXAMPLE_TOKENIZE, and register --ids, --stdin,
--no-bos, --no-parse-special and --show-count as common args.
parse_special defaults to true for TOKENIZE to preserve the old
behavior. Errors now go through LOG_ERR instead of fprintf(stderr).

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

2026-07-14 15:20:53 +02:00

..

CMakeLists.txt

libs : rename libcommon -> libllama-common (#21936)

2026-04-17 11:11:46 +03:00

tokenize.cpp

tokenize : align usage by using common args (#25516)

2026-07-14 15:20:53 +02:00