2 Commits
Author SHA1 Message Date
Thiago PadilhaandGitHub cb489bc0fb convert_hf_to_gguf: support split MTP export for HY V3 (#25641)
- Add a supports_mtp_export capability to ModelBase so architectures can opt
  into --mtp and --no-mtp without extending a central class allowlist.
- Enable the capability for the existing Qwen3.5/3.6 and Step3.5/3.7
  implementations, and for HY V3, whose converter already supports
  filtering the appended MTP layers.
2026-07-14 11:43:15 +02:00
Thiago PadilhaandGitHub 6c487e2f79 server: enforce prompt cache RAM limit (#25070)
Before this commit, --cache-ram was not a hard limit:

- The cache always kept at least one entry, even if that entry exceeded the
  RAM/token limits.
- Old entries were only evicted for the RAM/token limits after saving the new
  one, which could cause the cache to temporarily exceed the RAM/token limits
  even if individual entries were below the limit.

Now, ensure that the RAM limit is strict with these changes:

- Skip saving state to cache if by itself it exceeds the RAM limit.
- Evict old entries as necessary to make the new entry fit.

Additionally, token-limit cleanup may now evict the last remaining cache entry
instead of always preserving one.
2026-07-07 15:24:35 +02:00