* ci: implement build matrix for CUDA/CPU containers with dynamic tagging
* fix: Updated Docker images/build-container.yml
* fix: Updated the documentation about Docker
* fix: Set Arch for 3090s
* fix: Updated build step name.
* fix: Set target ARCH as a variable
* feat: Added cleanup step
* feat: Added docker-bake and updated workflow
* fix: Issue with REPO_OWNER variable
* fix: Updated workflow to solve errors
* fix: Updated branch format
* fix: Wrong naming
* Update docker-bake.hcl
* Update build-container.yml
* Update ik_llama-cuda.Containerfile
* Update ik_llama-cpu.Containerfile
* Update docker-bake.hcl
* Update build-container.yml
* Removed action/cache
* added -sSL for reliability and fixed the URL path
* added -sSL for reliability and fixed the URL path CUDA containerfile
* fix: correct Dockerfile RUN command syntax errors
- Combine split apt-get install commands in both Containerfiles
- Fix broken cmake command continuation in ik_llama-cuda.Containerfile
* fix: correct llama-swap download URL in Containerfiles
- Fix broken line continuation in curl download URL for llama-swap
* perf: improve ccache configuration in Containerfiles
- Add CCACHE_UMASK=000 for cache accessibility across stages
- Add CCACHE_MAXSIZE=1G to prevent unbounded growth
- Initialize ccache with ccache -i during build stage
* fix: remove problematic ccache initialization from Containerfiles
- ccache -i fails because CCACHE_DIR mount doesn't exist yet at build time
* fix: add git to CPU Containerfile build dependencies
- Resolves CMake warning about missing Git for build info
* chore: optimize Containerfile with smaller images and better healthchecks
- Add --no-install-recommends to all apt-get commands for smaller image size
- Add ca-certificates to base stage for HTTPS support
- Merge redundant build copy commands from 3 layers to 1
- Fix llama-swap version from 198 to v199 (latest release)
- Add HEALTHCHECK configuration with interval/timeout/retries to server and swap stages
- Copy /app/lib in server stage to fix container startup
* chore: fix CUDA Containerfile healthchecks and swap version
- Add /app/lib copy in server stage to fix container startup
- Fix llama-swap version from 198 to v199 (latest release)
- Add HEALTHCHECK configuration with interval/timeout/retries
* chore: fix indentation in Containerfiles and add LD_LIBRARY_PATH for server target
* fix: add --break-system-packages flag for pip in CPU Containerfile
* feat: add git bind mount for build info and NCCL support for CUDA
* fix: remove libnccl-dev from CUDA build (already included in base image)
* fix: added Markdown files to ignore files
* feat: use BUILD_NUMBER-COMMIT pattern for docker image tags
- Add BUILD_NUMBER and LLAMA_COMMIT to build workflow
- Update docker-bake.hcl to use version tag format matching llama-server --version output
- Format: VARIANT-BUILD_NUMBER-COMMIT (e.g., cu12-full-4406-3bc90dfd)
* fix: fetch full git history for accurate BUILD_NUMBER
- Add fetch-depth: 0 to actions/checkout to get all commits
- This ensures git rev-list --count HEAD returns correct total commit count
* fix: fetch full git history in Dockerfile for accurate BUILD_NUMBER
- Add git fetch --unshallow to get complete commit history during build
- This ensures build-info.cpp is generated with correct LLAMA_BUILD_NUMBER
* chore: update GitHub Actions to latest versions for Node.js 24 compatibility
- docker/setup-buildx-action@v3 -> v4
- docker/login-action@v3 -> v4
* chore: update all GitHub Actions to Node.js 24 compatible versions
- actions/checkout@v4 -> v6
- docker/setup-buildx-action@v3 -> v5
- docker/login-action@v3 -> v6
- docker/bake-action@v5 -> v7
* fix: use CI-passed BUILD_NUMBER and LLAMA_COMMIT in Dockerfile
- Add BUILD_NUMBER and LLAMA_COMMIT as build args
- Fall back to git commands if not provided
- Pass values explicitly to cmake for accurate build info
* fix: pass BUILD_NUMBER and LLAMA_COMMIT as Docker build args
- Add BUILD_NUMBER and LLAMA_COMMIT to docker bake args
- These will be used by the Containerfile for accurate build info
* fix: revert docker actions to v4 (latest available versions)
* fix: calculate BUILD_NUMBER and LLAMA_COMMIT directly in Containerfile
- Removed ARG defaults since we calculate from git during build
- Use git rev-list --count HEAD and git rev-parse for accurate version info
- Falls back to 0/unknown if git commands fail
* feat: calculate BUILD_NUMBER and LLAMA_COMMIT in Containerfiles
- Add git-based version calculation in both CPU and CUDA Containerfiles
- Remove .git bind mount (git is copied with COPY .)
- Pass build info to CMake for accurate llama-server --version output
* feat: calculate BUILD_NUMBER and LLAMA_COMMIT in Containerfiles
- Add git-based version calculation using git rev-list and git rev-parse
- Copy .git directory separately to ensure git commands work during build
- Pass build info to CMake for accurate llama-server --version output
* fix: cache improvements for CUDA and CPU builds
* fix: "/.git": not found
* fix: Unnecessary mv llama-swap
* fix: Remove BUILD_NUMBER and LLAMA_COMMIT from docker file, calculated by cmake proc
* fix: remove .git from dockerignore for local and CI builds
- Enables cmake to access .git directory during Docker build
- Required for version calculation in llama-server binary
- GitHub Actions uses explicit mount via bake action set parameter
* fix: Remove mounts key from Build and Push step in gh workflow
* ci: add .git verification step before build
* refactor: standardize Containerfile structure and remove .git mount dependency
- Remove --mount=type=bind,source=.git,target=.git from both Containerfiles
- Replace COPY . . with git clone for cleaner build context
- Add CUSTOM_COMMIT ARG for optional custom commit switching
- Standardize ARG/ENV ordering and comment formatting across CPU/CUDA variants
- Install ca-certificates before git clone to fix SSL verification issues
- Rename 'Structured artifact collection' to 'Collect build artifacts'
* ci: remove broken cache pruning step
* ci: remove broken prune-cache job
- Remove prune-cache job that was failing due to missing .git directory
- The job required a checkout step and the cache pruning logic was non-critical
* chore: Removed step for Verifying .git existance in GH workflow
* fix: ensure build always proceeds even if git switch fails
- Add '|| true' to git switch command so build continues on failure
- This prevents the entire RUN step from failing when CUSTOM_COMMIT is invalid
* fix: resolve Docker build pipeline issues
- Remove external git clone from Containerfiles, use build context directly
- Add BUILD_NUMBER and BUILD_COMMIT as CMake cache variables in build-info.cmake
- Fix .devops/tools.sh inclusion by using explicit COPY for hidden directories
- Set USE_CCACHE=true for CI builds
- Clean up unused SHA_SHORT variable from docker-bake.hcl
Fixes: Build steps were cached incorrectly due to external git clone ignoring the actual build context source.
* fix: include .git in Docker build context and add verification
* ci: add .git directory verification step after checkout
* build: fix .git mount path for Docker build context compatibility
* build: fix .git mount path for Docker build context compatibility
* docker: include .git in build context for version calculation
* ci: add .git directory verification step after checkout
* chore: Removed unecessary Verify .git step (It was a test)
* docs: update README with docker-bake and build-local.sh instructions
* docs: remove build-local.sh reference (not in repo)
* ci: optimize disk usage by limiting fetch depth and cleaning workspace
---------
Co-authored-by: HP Prodesk <sourceupdev@gmail.com>
* wip: separate llama_context for MTP with graph reuse
* wip: fix KV cache desync with separate MTP context
* refactor: remove dead mtp logic code, encapsulate KV mirroring
* mtp-context: derive args directly from the main model's context
* mtp: fix kv cache positions
* clean small comments
* minor refactor for context shift
* Use build_std_attention for Gemma4 when possible
It is possible for the 26b MoE and 31b dense models.
It is not possible for the E4B/E2B vaiants because they
don't have KV cache in each layer.
* Standardize Gemma4 dense ffn
* WIP: Gemma4 split mode graph
Runs but produces NaNs
* WIP: Gemma4 split mode graph
Runs but very high PPL. At least it is no longer NaN.
* WIP
* This works!
* Put attn_norm, attn_post_norm, ffn_norm, ffn_post_norm on all GPUs
* Fix crash when saving/loading KV cache
* WIP: split mode graph for Gemma4-MoE - crashes
* Split mode graph for Gemma4-MoE - this works
* Disable SWA optimization
Something goes wrong there
* Consolidate MoE and dense graph parallel
* Use build_std_attention for Gemma4 when possible
It is possible for the 26b MoE and 31b dense models.
It is not possible for the E4B/E2B vaiants because they
don't have KV cache in each layer.
* Standardize Gemma4 dense ffn
* WIP: Gemma4 split mode graph
Runs but produces NaNs
* WIP: Gemma4 split mode graph
Runs but very high PPL. At least it is no longer NaN.
* WIP
* This works!
* Put attn_norm, attn_post_norm, ffn_norm, ffn_post_norm on all GPUs
* Fix crash when saving/loading KV cache
* wip: build spec tuner for spefic args
* wip: test different reward system
* spec-tune: fix the reward to find best params given a good TPS
* spec-tune: refactor logic for its own file
* minor clean for comments and modules
* Little maintenance
* llama-quantize : Add the missing items in the help
* Add GGML_MAX_CONTEXTS define in the general cmakelist.txt
* Make the KV cache (CPU) based warnings clearer
* Correct placement of GGML_MAX_CONTEXTS definition
* Revert wrong indents
This reverts commit d0728cbb6c.
* Moving the GGML_MAX_CONTEXTS definition to src/CMakeLists.txt
* Update warning message for unsupported KV cache types
* forgotten antislash
* Gemma4: WIP
* Gemma4: WIP - runs with totally wrong results
* Gemma4: WIP - add CPU 512, 512 FA
* Gemma4: WIP
It gives a meaningful response in llama-cli, but PPL is still much too
high. Is this due to tokenizer issues?
* Gemma4: this works
I had forgotten the softcap on the final output.
* Remove log
* Gemma4: WIP E4B/E2B
* Gemma4: Q4B/E2B appear to work now
* gemma4: tokenizer fixes
Add HAVE_FANCY_SIMD path that processes 16 rows at a time using 512-bit
operations, combining two R8 groups via _mm512_inserti32x8. Reuses the
existing qx_r8_q8_dot_product 512-bit overload for the inner dot product.
Also updates num_rows for Q8_1 to 16 under HAVE_FANCY_SIMD.
Co-authored-by: Adam Caldwell <accaldwell@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* wip: port MTP architecture
Ports the Multi-Token Prediction (MTP) architecture to the older `llama.cpp` codebase used by `ikllama`.
Changes include:
- Updating `llama_batch` to support `mtp_params`.
- Modifying `llama_decode_internal` (and `encode`) to handle MTP operations (Warmup, Update, Draft).
- Adding public APIs for MTP state management (`llama_set_draft_input_hidden_state`).
- Adapting the embedding extraction logic to skip MTP update passes.
* Refactors `server_slot` to support generic speculative decoding (MTP or Draft Model).
* core: enable hybrid outputs (logits + embeddings) for MTP support
* fix(mtp): correct KV-cache slot finding for updates
* fix(mtp): persist hidden states to prevent context corruption during drafting
* refactor(mtp): clean unused code
* fix(mtp): update server to new functions name
* fix(mtp): fix graph and save hidden state
* mtp: refactor integration, context params and kv cache search
* mtp: fix hidden state extraction and speculative acceptance flow
* server: fix MTP warmup for long prompts and reset token buffer
* llama: refactor MTP operation state to context parameters
* server: fix n_past calculation in MTP acceptance
* llama: fix mtp enable flags
* speculative: refactor MTP to use common_speculative interface
* context: remove unused signatures
* clip: fix deprecated enum-enum conversion warning
* common: fix format string crash in help message
* context: fix mtp activation logic
* llamat: always use the extracted embedding
* llama: get all embeddings to kv cache
* llama: revert logit to not run mtp for not supported arch
* llama: allocate all the n_outputs for MTP
* wip
* server-context: get only the last embedding for hidden state
* ggml-backend: fix array of bounds in debug build
* server-context: run mt kv update to each prompt batch
* revert segmentation fault fixes
* glm-mtp(feat): optimize graph embedding and recursive drafting
