perplexity: signal-driven hot-swap mode for persistent per-tensor PPL/KLD benchmarking (extends #1989) (#2131)

* perplexity: add signal-driven hot-swap mode for persistent KLD/PPL benchmarking

llama-perplexity can now stay resident and be driven through control/status files (reload/compute/exit): it reloads only the tensors that changed on disk and recomputes PPL/KLD without ever reloading the full model. File-based signalling works on Windows, macOS and Linux. The reload returning-to-original path now refreshes tensor data from disk instead of
reattaching stale weights.

* Not Cygwin specific

docs/development/on-demand-tensor-reload.md

@@ -135,21 +135,57 @@ Returns `true` if at least one tensor was reloaded. When this happens, the funct
 | Variable | Purpose |
 |----------|---------|
 | `LLAMA_HOTSWAP_ENABLED` | Enables the hot-swap loop in `perplexity` and the health-check hook in `server`. |
-| `LLAMA_PERPLEXITY_PRE_RELOAD_SCRIPT` | Path to an executable script run between perplexity iterations (e.g., to regenerate/re-quantize a tensor file). |
+| `LLAMA_PERPLEXITY_PRE_RELOAD_SCRIPT` | Path to an executable script run between perplexity iterations (e.g., to regenerate/re-quantize a tensor file). Legacy loop mode only. |
+| `LLAMA_HOTSWAP_CONTROL_FILE` | Path to a command file polled by `perplexity`. Setting it (together with `LLAMA_HOTSWAP_ENABLED`) switches the tool into **signal mode** (see below). |
+| `LLAMA_HOTSWAP_STATUS_FILE` | Path to a status file written by `perplexity` in signal mode. Required when `LLAMA_HOTSWAP_CONTROL_FILE` is set. |
 
 ---
 
 ## Integration Points
 
-### `examples/perplexity/perplexity.cpp`
-When `LLAMA_HOTSWAP_ENABLED` is set, the tool runs in a loop:
+### `examples/perplexity/perplexity.cpp` (legacy loop mode)
+When `LLAMA_HOTSWAP_ENABLED` is set (and `LLAMA_HOTSWAP_CONTROL_FILE` is not), the tool runs in a loop:
 
 1. Perform an initial `llama_reload_changed_tensors()` to apply any pending changes before the first evaluation.
-2. Compute perplexity (or Hellaswag, etc.).
+2. Compute perplexity (or Hellaswag, KL divergence, etc.).
 3. Print timings and write logs.
 4. Execute the optional pre-reload script.
 5. Call `llama_reload_changed_tensors(ctx)`. If no tensors changed, exit; otherwise repeat from step 2.
 
+### `examples/perplexity/perplexity.cpp` (signal mode)
+
+When both `LLAMA_HOTSWAP_ENABLED` and `LLAMA_HOTSWAP_CONTROL_FILE` are set (the latter requires `LLAMA_HOTSWAP_STATUS_FILE` too), the tool **never terminates on its own**. Instead, an external driver (e.g. `benchmark_each_tensor.sh --hotswap` from Thireus' GGUF Tool Suite) swaps GGUF shards on disk and steers the process through two plain files. Files are used instead of POSIX signals so the exact same protocol works on Windows, macOS and Linux.
+
+**Status file** (written atomically by `perplexity` via tmp-file + rename, single line):
+
+```
+<state> <iter> <seq>
+```
+
+* `<state>` — `computing` (an evaluation is running), `done` (evaluation finished, waiting for a command), `reload_failed` (a `reload` command found no changed tensor; still waiting), `exiting` (an `exit` command was accepted).
+* `<iter>` — 1-based index of the evaluation. It only advances when a `reload` or `compute` command is accepted.
+* `<seq>` — sequence number of the last accepted command (`0` before any command).
+
+**Control file** (written by the driver — ideally atomically via tmp-file + rename —, consumed and deleted by `perplexity`, single line):
+
+```
+<command> <seq>
+```
+
+* `reload <seq>` — call `llama_reload_changed_tensors()`; on success re-evaluate, on "nothing changed" publish `reload_failed` and keep waiting.
+* `compute <seq>` — re-evaluate without reloading (useful when the on-disk state already matches the desired state).
+* `exit <seq>` — publish `exiting`, free the model and terminate (aliases: `quit`, `stop`).
+* `<seq>` must increase monotonically; a command whose `<seq>` is not greater than the last accepted one is deleted and ignored, which makes duplicate delivery harmless. A command without `<seq>` is always executed (convenient for manual testing: `echo reload > ctrl`).
+
+Driver loop in a nutshell:
+
+1. Swap the shard(s) of the tensor(s) to benchmark, start `llama-perplexity` with the three environment variables, redirecting stdout+stderr to a session log; wait for `done 1`.
+2. Capture the per-benchmark log by remembering the session log's byte size before each command and slicing from that offset once `done <iter>` appears.
+3. Restore the previous shard(s), swap the next one(s), write `reload <seq>` and wait for `done <iter+1>` — both the restore and the new swap are applied by the same reload.
+4. When finished, write `exit <seq>`.
+
+This mode works with any of the evaluation flavours (`perplexity`, `--kl-divergence`, HellaSwag, ...); KL divergence is fully re-entrant since the base-logits file is re-read on every iteration.
+
 ### `examples/server/server.cpp`
 On every health-check (`/health`) request, if `LLAMA_HOTSWAP_ENABLED` is set, the server calls `llama_reload_changed_tensors()`. This provides a convenient, external trigger: simply `touch` or overwrite a tensor’s source GGUF file and poll `/health` to apply the change.
 

examples/perplexity/perplexity.cpp

@@ -21,6 +21,9 @@
 #include <fstream>
 #include <sstream>
 #include <cstdlib>
+#include <cerrno>
+#include <chrono>
+#include <string>
 
 #if defined(_MSC_VER)
 #pragma warning(disable: 4244 4267) // possible loss of data
@@ -29,6 +32,89 @@
 // Public C API for hot-swap (defined in src/llama.cpp)
 extern "C" bool llama_reload_changed_tensors(struct llama_context * ctx);
 
+// ------------------------------------------------------------------
+// Hot-swap signal mode: file-based signalling so an external driver can
+// swap GGUF shards on disk and ask this process to reload + recompute
+// without ever restarting it. Plain files are used instead of POSIX
+// signals so the exact same protocol works on Windows, macOS and Linux.
+//
+// Protocol:
+//   - status file (written by us, read by the driver), single line:
+//         <state> <iter> <seq>
+//     where <state> is one of computing/done/reload_failed/exiting,
+//     <iter> is the 1-based index of the current computation and <seq>
+//     is the sequence number of the last accepted command (0 = none).
+//   - control file (written by the driver, read + deleted by us):
+//         <command> <seq>
+//     where <command> is one of:
+//         reload  - reload tensors that changed on disk, then recompute
+//         compute - recompute without reloading anything
+//         exit    - terminate the process (quit/stop are synonyms)
+//     <seq> must increase monotonically; commands whose <seq> is not
+//     greater than the last accepted one are ignored (this makes
+//     duplicate delivery harmless). A command without <seq> is always
+//     executed (convenient for manual use: `echo reload > ctrl`).
+// ------------------------------------------------------------------
+
+static void hotswap_write_status(const char * path, const char * state, int iter, int seq) {
+    const std::string tmp = std::string(path) + ".tmp";
+    // The status file is the driver's only view of our progress: losing a
+    // single update would deadlock the whole protocol (we would wait for a
+    // command the driver will never send). So never give up - retry until the
+    // update is published. Transient failures are expected on Windows while
+    // another process (the driver polling, antivirus, indexer) briefly holds
+    // one of the files open; publish via tmp + rename so a reader can never
+    // observe a partially written status.
+    for (long attempt = 0; ; ++attempt) {
+        FILE * f = fopen(tmp.c_str(), "wb"); // binary: keep LF line ending on Windows
+        if (f) {
+            fprintf(f, "%s %d %d\n", state, iter, seq);
+            fclose(f);
+            std::remove(path); // Windows: rename fails when the destination exists
+            if (std::rename(tmp.c_str(), path) == 0) {
+                return;
+            }
+        }
+        if (attempt > 0 && attempt % 500 == 0) { // roughly every 10 seconds
+            fprintf(stderr, "hotswap: still trying to publish status '%s' to file '%s': %s\n", state, path, strerror(errno));
+            fflush(stderr);
+        }
+        std::this_thread::sleep_for(std::chrono::milliseconds(20));
+    }
+}
+
+// Returns true when a complete command was read (and the control file deleted).
+// A file without a terminating newline is considered partially written and is
+// left in place to be retried on the next poll.
+static bool hotswap_read_command(const char * path, std::string & cmd, int & seq) {
+    std::ifstream in(path, std::ios::binary);
+    if (!in) {
+        return false;
+    }
+    std::stringstream ss;
+    ss << in.rdbuf();
+    in.close();
+    const std::string content = ss.str();
+    if (content.find('\n') == std::string::npos) {
+        return false;
+    }
+    cmd.clear();
+    std::istringstream parser(content);
+    parser >> cmd;
+    if (!(parser >> seq)) {
+        seq = -1; // no sequence number provided
+    }
+    if (cmd.empty()) {
+        return false;
+    }
+    // Delete so the command is not picked up twice; retry in case the writer
+    // still briefly holds the file open (Windows sharing semantics).
+    for (int attempt = 0; attempt < 50 && std::remove(path) != 0; ++attempt) {
+        std::this_thread::sleep_for(std::chrono::milliseconds(20));
+    }
+    return true;
+}
+
 struct results_perplexity {
     std::vector<llama_token> tokens;
     double                   ppl_value;
@@ -2060,13 +2146,37 @@ int main(int argc, char ** argv) {
         fprintf(stderr, "%s\n", gpt_params_get_system_info(params).c_str());
     }
 
-    const char * hotswap_env = std::getenv("LLAMA_HOTSWAP_ENABLED");
-    const char * pre_script  = std::getenv("LLAMA_PERPLEXITY_PRE_RELOAD_SCRIPT");
+    const char * hotswap_env  = std::getenv("LLAMA_HOTSWAP_ENABLED");
+    const char * pre_script   = std::getenv("LLAMA_PERPLEXITY_PRE_RELOAD_SCRIPT");
+    const char * hotswap_ctrl = std::getenv("LLAMA_HOTSWAP_CONTROL_FILE");
+    const char * hotswap_stat = std::getenv("LLAMA_HOTSWAP_STATUS_FILE");
+
+    // Signal-driven hot-swap mode: never exit on our own; after every
+    // computation wait for a command in the control file (see the protocol
+    // description at the top of this file).
+    const bool hotswap_signal_mode = hotswap_env && hotswap_ctrl && *hotswap_ctrl;
+
+    if (hotswap_signal_mode && !(hotswap_stat && *hotswap_stat)) {
+        fprintf(stderr, "%s: error: LLAMA_HOTSWAP_CONTROL_FILE requires LLAMA_HOTSWAP_STATUS_FILE to be set as well\n", __func__);
+        llama_free(ctx);
+        llama_free_model(model);
+        llama_backend_free();
+        return 1;
+    }
 
     if (hotswap_env) {
         llama_reload_changed_tensors(ctx);
     }
 
+    int hotswap_iter = 1; // 1-based index of the current computation
+    int hotswap_seq  = 0; // sequence number of the last accepted command
+    if (hotswap_signal_mode) {
+        std::remove(hotswap_ctrl); // discard stale commands from a previous run
+        fprintf(stderr, "%s: hot-swap signal mode enabled (control='%s', status='%s')\n",
+                __func__, hotswap_ctrl, hotswap_stat);
+        hotswap_write_status(hotswap_stat, "computing", hotswap_iter, hotswap_seq);
+    }
+
     while (true) {
         struct results_perplexity results;
         if (params.hellaswag) {
@@ -2084,6 +2194,62 @@ int main(int argc, char ** argv) {
         llama_print_timings(ctx);
         write_logfile(ctx, params, model, results);
 
+        if (hotswap_signal_mode) {
+            fprintf(stderr, "%s: hot-swap iteration %d finished, waiting for the next command\n", __func__, hotswap_iter);
+            fflush(stdout);
+            fflush(stderr);
+            hotswap_write_status(hotswap_stat, "done", hotswap_iter, hotswap_seq);
+
+            bool terminate = false;
+            while (true) {
+                std::string cmd;
+                int seq = -1;
+                if (!hotswap_read_command(hotswap_ctrl, cmd, seq)) {
+                    std::this_thread::sleep_for(std::chrono::milliseconds(200));
+                    continue;
+                }
+                if (seq >= 0 && seq <= hotswap_seq) {
+                    fprintf(stderr, "%s: ignoring stale hot-swap command '%s' (seq %d <= %d)\n",
+                            __func__, cmd.c_str(), seq, hotswap_seq);
+                    continue;
+                }
+                if (seq >= 0) {
+                    hotswap_seq = seq;
+                }
+                fprintf(stderr, "%s: hot-swap command received: %s\n", __func__, cmd.c_str());
+                if (cmd == "exit" || cmd == "quit" || cmd == "stop") {
+                    terminate = true;
+                    break;
+                }
+                if (cmd == "reload") {
+                    if (llama_reload_changed_tensors(ctx)) {
+                        break;
+                    }
+                    fprintf(stderr, "%s: hot-swap reload requested, but no tensor has changed on disk\n", __func__);
+                    fflush(stderr);
+                    hotswap_write_status(hotswap_stat, "reload_failed", hotswap_iter, hotswap_seq);
+                    continue;
+                }
+                if (cmd == "compute") {
+                    break; // recompute without reloading
+                }
+                fprintf(stderr, "%s: unknown hot-swap command '%s' ignored\n", __func__, cmd.c_str());
+                fflush(stderr);
+            }
+            if (terminate) {
+                fprintf(stderr, "%s: hot-swap exit requested, terminating\n", __func__);
+                fflush(stdout);
+                fflush(stderr);
+                hotswap_write_status(hotswap_stat, "exiting", hotswap_iter, hotswap_seq);
+                break;
+            }
+
+            hotswap_iter++;
+            llama_reset_timings(ctx); // per-iteration timings, like separate runs would report
+            hotswap_write_status(hotswap_stat, "computing", hotswap_iter, hotswap_seq);
+            continue;
+        }
+
         if (pre_script) {
             fprintf(stderr, "%s: executing pre-reload script: %s\n", __func__, pre_script);
 #ifdef _WIN32

src/llama-reload.cpp

@@ -144,6 +144,22 @@ static bool is_original_snapshot_buffer(llama_model & model, ggml_backend_buffer
     return false;
 }
 
+// ------------------------------------------------------------------
+// mmap guard: tensors whose data lives inside a (read-only) file mapping
+// must not be written to in place
+// ------------------------------------------------------------------
+static bool tensor_data_in_mmap(const llama_model & model, const void * data) {
+    if (!data) return false;
+    for (const auto & mapping : model.mappings) {
+        if (!mapping) continue;
+        const char * base = (const char *) mapping->addr();
+        if ((const char *) data >= base && (const char *) data < base + mapping->size()) {
+            return true;
+        }
+    }
+    return false;
+}
+
 // ------------------------------------------------------------------
 // Final size estimator
 // ------------------------------------------------------------------
@@ -633,6 +649,17 @@ static bool reload_tensor_split_path(
         for (const auto & sib : src.sibling_names) {
             reattach_split_tensor_to_shared(model, sib.c_str());
         }
+        // Refresh the data from the current file bytes: the file changed on
+        // disk even though its dtype matches the original snapshot. For a
+        // restored original file this copies identical bytes; for a same-dtype
+        // swap it loads the real new data instead of keeping stale weights.
+        if (!host_buf.empty()) {
+            if (tensor_data_in_mmap(model, tensor->data)) {
+                LLAMA_LOG_WARN("%s: tensor '%s' is mmap-backed; skipping in-place data refresh\n", __func__, name);
+            } else {
+                ggml_backend_tensor_set(tensor, host_buf.data(), 0, host_buf.size());
+            }
+        }
         return true;
     }
 
@@ -673,7 +700,6 @@ static bool reload_tensor_non_split_path(
         bool returning_to_original,
         ggml_backend_buffer_t old_buf)
 {
-		(void)model;
 		(void)curr_type;
 #ifndef NDEBUG
     const char * name = ggml_get_name(tensor);
@@ -690,6 +716,17 @@ static bool reload_tensor_non_split_path(
             tensor->ne[i] = src.original_ne[i];
             tensor->nb[i] = src.original_nb[i];
         }
+        // Refresh the data from the current file bytes: the file changed on
+        // disk even though its dtype matches the original snapshot. For a
+        // restored original file this copies identical bytes; for a same-dtype
+        // swap it loads the real new data instead of keeping stale weights.
+        if (!host_buf.empty()) {
+            if (tensor_data_in_mmap(model, tensor->data)) {
+                LLAMA_LOG_WARN("%s: tensor '%s' is mmap-backed; skipping in-place data refresh\n", __func__, ggml_get_name(tensor));
+            } else {
+                ggml_backend_tensor_set(tensor, host_buf.data(), 0, host_buf.size());
+            }
+        }
         src.state = tensor_reload_source::reload_state::ON_ORIGINAL;
         return true;
     }
@@ -861,6 +898,16 @@ bool reload_info::reload_tensor(const char * name, llama_model & model) {
         host_buf.resize(need);
         file.read(host_buf.data(), (std::streamsize)need);
         if (!file || (size_t)file.gcount() != need) return false;
+    } else {
+        // The on-disk dtype matches the original snapshot, but the bytes may
+        // still differ (the file did change on disk, e.g. a same-dtype tensor
+        // quantized with a different imatrix). Read them so the returning path
+        // can refresh the tensor data after reattaching the original buffers.
+        size_t need = src.original_nbytes;
+        if (file_nbytes < need) return false;
+        host_buf.resize(need);
+        file.read(host_buf.data(), (std::streamsize)need);
+        if (!file || (size_t)file.gcount() != need) return false;
     }
 
     bool ok = false;