mirror of
https://github.com/ggml-org/llama.cpp.git
synced 2026-07-21 10:15:53 +00:00
server: (router) rework -hf preset repo (#24739)
* server: temporary remove HF remote preset * rework remove preset.ini support * rm unused get_remote_preset_whitelist() * print warning * add docs * rm stray file
This commit is contained in:
+36
-38
@@ -8,55 +8,53 @@ The INI preset feature, introduced in [PR#17859](https://github.com/ggml-org/lla
|
||||
|
||||
When running multiple models on the server (router mode), INI preset files can be used to configure model-specific parameters. Please refer to the [server documentation](../tools/server/README.md) for more details.
|
||||
|
||||
### Using a Remote Preset
|
||||
### Using a Hugging Face Preset
|
||||
|
||||
> [!NOTE]
|
||||
> [!IMPORTANT]
|
||||
>
|
||||
> This feature is currently only supported via the `-hf` option.
|
||||
> Please only use presets that you can trust! Unknown presets may be unsafe
|
||||
|
||||
For GGUF models hosted on Hugging Face, you can include a `preset.ini` file in the root directory of the repository to define specific configurations for that model.
|
||||
You can push your preset to Hugging Face Hub and share with other users by:
|
||||
1. Creating an empty model repository on Hugging Face
|
||||
2. Creating a `preset.ini` file in the root directory of the repository
|
||||
|
||||
Example:
|
||||
Example of a `preset.ini`:
|
||||
|
||||
```ini
|
||||
hf-repo-draft = username/my-draft-model-GGUF
|
||||
temp = 0.5
|
||||
top-k = 20
|
||||
top-p = 0.95
|
||||
[*]
|
||||
ctx-size = 0
|
||||
mmap = 1
|
||||
kv-unified = 1
|
||||
parallel = 4
|
||||
spec-default = 1
|
||||
|
||||
[Qwen3.5-4B]
|
||||
hf = unsloth/Qwen3.5-4B-GGUF:Q4_K_M
|
||||
ctx-size = 262144
|
||||
batch-size = 2048
|
||||
ubatch-size = 2048
|
||||
top-p = 1.0
|
||||
top-k = 0
|
||||
min-p = 0.01
|
||||
temp = 1.0
|
||||
|
||||
[gpt-oss-120b-hf]
|
||||
hf = ggml-org/gpt-oss-120b-GGUF
|
||||
ctx-size = 262144
|
||||
batch-size = 2048
|
||||
ubatch-size = 2048
|
||||
top-p = 1.0
|
||||
top-k = 0
|
||||
min-p = 0.01
|
||||
temp = 1.0
|
||||
chat-template-kwargs = {"reasoning_effort": "high"}
|
||||
```
|
||||
|
||||
For security reasons, only certain options are allowed. Please refer to [preset.cpp](../common/preset.cpp) for the complete list of permitted options.
|
||||
|
||||
Example usage:
|
||||
|
||||
Assuming your repository `username/my-model-with-preset` contains a `preset.ini` with the configuration above:
|
||||
|
||||
```sh
|
||||
llama-cli -hf username/my-model-with-preset
|
||||
|
||||
# This is equivalent to:
|
||||
llama-cli -hf username/my-model-with-preset \
|
||||
--hf-repo-draft username/my-draft-model-GGUF \
|
||||
--temp 0.5 \
|
||||
--top-k 20 \
|
||||
--top-p 0.95
|
||||
```
|
||||
|
||||
You can also override preset arguments by specifying them on the command line:
|
||||
The preset will be loaded similarly to the `--models-preset` option. Therefore, you can also override certain params via CLI arguments:
|
||||
|
||||
```sh
|
||||
# Force temp = 0.1, overriding the preset value
|
||||
llama-cli -hf username/my-model-with-preset --temp 0.1
|
||||
```
|
||||
|
||||
If you want to define multiple preset configurations for one or more GGUF models, you can create a blank HF repo for each preset. Each HF repo should contain a `preset.ini` file that references the actual model(s):
|
||||
|
||||
```ini
|
||||
hf-repo = user/my-model-main
|
||||
hf-repo-draft = user/my-model-draft
|
||||
temp = 0.8
|
||||
ctx-size = 1024
|
||||
; (and other configurations)
|
||||
llama-cli -hf username/my-preset --temp 0.1
|
||||
```
|
||||
|
||||
### Named presets
|
||||
|
||||
Reference in New Issue
Block a user