server : prevent LRU eviction of models still loading

In router mode with models_max, unload_lru() could select a model in
LOADING state as the LRU victim. Because the child's stdin monitoring
thread starts only after model loading completes, the exit command
sits unread in the buffer and the stop_timeout expires before the
child ever receives it, causing a force-kill.

LOADING models still count toward capacity to prevent resource
exhaustion from concurrent loads, but are excluded from eviction.
This commit is contained in:
Georgi Gerganov
2026-07-10 16:49:57 +03:00
parent 0badc06ab5
commit 3203ab47a6
+3
View File
@@ -751,6 +751,9 @@ void server_models::unload_lru() {
for (const auto & m : mapping) {
if (m.second.meta.is_running()) {
count_active++;
if (m.second.meta.status == SERVER_MODEL_STATUS_LOADING) {
continue;
}
if (m.second.meta.last_used < lru_last_used) {
lru_model_name = m.first;
lru_last_used = m.second.meta.last_used;