mirror of
https://github.com/ggml-org/llama.cpp.git
synced 2026-07-21 18:25:57 +00:00
Adrien GallouëtandGitHub
8ff8c4299d
Migrate the tokenize tool to common_params_parse, replacing its hand-rolled argv parsing, Windows UTF-8 handling and file reading with the shared common helpers. Expose the model-sourcing flags (-m, -mu, -dr, -hf, -hff, --offline, HF_TOKEN) to LLAMA_EXAMPLE_TOKENIZE, and register --ids, --stdin, --no-bos, --no-parse-special and --show-count as common args. parse_special defaults to true for TOKENIZE to preserve the old behavior. Errors now go through LOG_ERR instead of fprintf(stderr). Signed-off-by: Adrien Gallouët <angt@huggingface.co>