* fix: Updating cleanup step
* fix: Updated trigger for build-container
* fix: Unset token for cleanup step
* fix: Set build-container cleanup step without run-dry
* fix: Removed 100 commits from checkout actions
* fix: Enable the whole history
* test: Suggestion from mcm007 on build-container
* fix: add token to package cleanup step
Explicitly pass GITHUB_TOKEN to the delete-package-versions action to ensure it has sufficient authorization for package deletion.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: - Error: delete version API failed. Package not found.
* fix: Deleted LLAMA_COMMIT from build
* fix: Removed id-token permission
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
get_mmq_x_max_host() returned 128 for NVIDIA Volta when built with
GGML_CUDA_FORCE_MMQ, but get_mmq_x_max_device() caps at
MMQ_DP4A_MAX_BATCH_SIZE (= 64) for the same configuration because the
wider mmq_x device kernels are not instantiated on sm_70. The host
dispatcher selects an mmq_x in [8, mmq_x_max] and instantiates the
matching template; any pick > 64 hits NO_DEVICE_CODE at runtime and
the kernel returns without writing output, producing zero-valued
activations and incoherent generation.
Repro on Tesla V100 (sm_70) with a Q6_K Qwen3.5-MoE model:
cmake -B build -DGGML_CUDA=ON -DGGML_CUDA_FORCE_MMQ=ON \
-DGGML_CUDA_NO_PEER_COPY=ON -DCMAKE_CUDA_ARCHITECTURES=70
./llama-cli -m model.gguf -ngl 99 -p "<~200-token prompt>"
# stderr: mmq.cuh:3941: ERROR: CUDA kernel mul_mat_q has no
# device code compatible with CUDA arch 700.
Cap the host bound at MMQ_DP4A_MAX_BATCH_SIZE on Volta regardless of
GGML_CUDA_FORCE_MMQ; this matches the device template availability and
yields correct output. Measured on a single V100-SXM2:
before patch: PP unusable (NO_DEVICE_CODE), TG ~100 t/s
after patch: PP ~630 t/s, TG ~102 t/s
The change is a no-op on architectures that have int8 MMA (Turing+),
on AMD/HIP, and on builds without GGML_CUDA_FORCE_MMQ.
In llama_kv_cache_init() call, params.type_v_first was incorrectly passed
twice instead of params.type_v_last.
This caused V cache in the last N layers to use type_v_first instead of type_v_last.
Fix: Replace second params.type_v_first with params.type_v_last.
* ci: implement build matrix for CUDA/CPU containers with dynamic tagging
* fix: Updated Docker images/build-container.yml
* fix: Updated the documentation about Docker
* fix: Set Arch for 3090s
* fix: Updated build step name.
* fix: Set target ARCH as a variable
* feat: Added cleanup step
* feat: Added docker-bake and updated workflow
* fix: Issue with REPO_OWNER variable
* fix: Updated workflow to solve errors
* fix: Updated branch format
* fix: Wrong naming
* Update docker-bake.hcl
* Update build-container.yml
* Update ik_llama-cuda.Containerfile
* Update ik_llama-cpu.Containerfile
* Update docker-bake.hcl
* Update build-container.yml
* Removed action/cache
* added -sSL for reliability and fixed the URL path
* added -sSL for reliability and fixed the URL path CUDA containerfile
* fix: correct Dockerfile RUN command syntax errors
- Combine split apt-get install commands in both Containerfiles
- Fix broken cmake command continuation in ik_llama-cuda.Containerfile
* fix: correct llama-swap download URL in Containerfiles
- Fix broken line continuation in curl download URL for llama-swap
* perf: improve ccache configuration in Containerfiles
- Add CCACHE_UMASK=000 for cache accessibility across stages
- Add CCACHE_MAXSIZE=1G to prevent unbounded growth
- Initialize ccache with ccache -i during build stage
* fix: remove problematic ccache initialization from Containerfiles
- ccache -i fails because CCACHE_DIR mount doesn't exist yet at build time
* fix: add git to CPU Containerfile build dependencies
- Resolves CMake warning about missing Git for build info
* chore: optimize Containerfile with smaller images and better healthchecks
- Add --no-install-recommends to all apt-get commands for smaller image size
- Add ca-certificates to base stage for HTTPS support
- Merge redundant build copy commands from 3 layers to 1
- Fix llama-swap version from 198 to v199 (latest release)
- Add HEALTHCHECK configuration with interval/timeout/retries to server and swap stages
- Copy /app/lib in server stage to fix container startup
* chore: fix CUDA Containerfile healthchecks and swap version
- Add /app/lib copy in server stage to fix container startup
- Fix llama-swap version from 198 to v199 (latest release)
- Add HEALTHCHECK configuration with interval/timeout/retries
* chore: fix indentation in Containerfiles and add LD_LIBRARY_PATH for server target
* fix: add --break-system-packages flag for pip in CPU Containerfile
* feat: add git bind mount for build info and NCCL support for CUDA
* fix: remove libnccl-dev from CUDA build (already included in base image)
* fix: added Markdown files to ignore files
* feat: use BUILD_NUMBER-COMMIT pattern for docker image tags
- Add BUILD_NUMBER and LLAMA_COMMIT to build workflow
- Update docker-bake.hcl to use version tag format matching llama-server --version output
- Format: VARIANT-BUILD_NUMBER-COMMIT (e.g., cu12-full-4406-3bc90dfd)
* fix: fetch full git history for accurate BUILD_NUMBER
- Add fetch-depth: 0 to actions/checkout to get all commits
- This ensures git rev-list --count HEAD returns correct total commit count
* fix: fetch full git history in Dockerfile for accurate BUILD_NUMBER
- Add git fetch --unshallow to get complete commit history during build
- This ensures build-info.cpp is generated with correct LLAMA_BUILD_NUMBER
* chore: update GitHub Actions to latest versions for Node.js 24 compatibility
- docker/setup-buildx-action@v3 -> v4
- docker/login-action@v3 -> v4
* chore: update all GitHub Actions to Node.js 24 compatible versions
- actions/checkout@v4 -> v6
- docker/setup-buildx-action@v3 -> v5
- docker/login-action@v3 -> v6
- docker/bake-action@v5 -> v7
* fix: use CI-passed BUILD_NUMBER and LLAMA_COMMIT in Dockerfile
- Add BUILD_NUMBER and LLAMA_COMMIT as build args
- Fall back to git commands if not provided
- Pass values explicitly to cmake for accurate build info
* fix: pass BUILD_NUMBER and LLAMA_COMMIT as Docker build args
- Add BUILD_NUMBER and LLAMA_COMMIT to docker bake args
- These will be used by the Containerfile for accurate build info
* fix: revert docker actions to v4 (latest available versions)
* fix: calculate BUILD_NUMBER and LLAMA_COMMIT directly in Containerfile
- Removed ARG defaults since we calculate from git during build
- Use git rev-list --count HEAD and git rev-parse for accurate version info
- Falls back to 0/unknown if git commands fail
* feat: calculate BUILD_NUMBER and LLAMA_COMMIT in Containerfiles
- Add git-based version calculation in both CPU and CUDA Containerfiles
- Remove .git bind mount (git is copied with COPY .)
- Pass build info to CMake for accurate llama-server --version output
* feat: calculate BUILD_NUMBER and LLAMA_COMMIT in Containerfiles
- Add git-based version calculation using git rev-list and git rev-parse
- Copy .git directory separately to ensure git commands work during build
- Pass build info to CMake for accurate llama-server --version output
* fix: cache improvements for CUDA and CPU builds
* fix: "/.git": not found
* fix: Unnecessary mv llama-swap
* fix: Remove BUILD_NUMBER and LLAMA_COMMIT from docker file, calculated by cmake proc
* fix: remove .git from dockerignore for local and CI builds
- Enables cmake to access .git directory during Docker build
- Required for version calculation in llama-server binary
- GitHub Actions uses explicit mount via bake action set parameter
* fix: Remove mounts key from Build and Push step in gh workflow
* ci: add .git verification step before build
* refactor: standardize Containerfile structure and remove .git mount dependency
- Remove --mount=type=bind,source=.git,target=.git from both Containerfiles
- Replace COPY . . with git clone for cleaner build context
- Add CUSTOM_COMMIT ARG for optional custom commit switching
- Standardize ARG/ENV ordering and comment formatting across CPU/CUDA variants
- Install ca-certificates before git clone to fix SSL verification issues
- Rename 'Structured artifact collection' to 'Collect build artifacts'
* ci: remove broken cache pruning step
* ci: remove broken prune-cache job
- Remove prune-cache job that was failing due to missing .git directory
- The job required a checkout step and the cache pruning logic was non-critical
* chore: Removed step for Verifying .git existance in GH workflow
* fix: ensure build always proceeds even if git switch fails
- Add '|| true' to git switch command so build continues on failure
- This prevents the entire RUN step from failing when CUSTOM_COMMIT is invalid
* fix: resolve Docker build pipeline issues
- Remove external git clone from Containerfiles, use build context directly
- Add BUILD_NUMBER and BUILD_COMMIT as CMake cache variables in build-info.cmake
- Fix .devops/tools.sh inclusion by using explicit COPY for hidden directories
- Set USE_CCACHE=true for CI builds
- Clean up unused SHA_SHORT variable from docker-bake.hcl
Fixes: Build steps were cached incorrectly due to external git clone ignoring the actual build context source.
* fix: include .git in Docker build context and add verification
* ci: add .git directory verification step after checkout
* build: fix .git mount path for Docker build context compatibility
* build: fix .git mount path for Docker build context compatibility
* docker: include .git in build context for version calculation
* ci: add .git directory verification step after checkout
* chore: Removed unecessary Verify .git step (It was a test)
* docs: update README with docker-bake and build-local.sh instructions
* docs: remove build-local.sh reference (not in repo)
* ci: optimize disk usage by limiting fetch depth and cleaning workspace
* fix: cleanup step in workflow
---------
Co-authored-by: HP Prodesk <sourceupdev@gmail.com>
* ci: implement build matrix for CUDA/CPU containers with dynamic tagging
* fix: Updated Docker images/build-container.yml
* fix: Updated the documentation about Docker
* fix: Set Arch for 3090s
* fix: Updated build step name.
* fix: Set target ARCH as a variable
* feat: Added cleanup step
* feat: Added docker-bake and updated workflow
* fix: Issue with REPO_OWNER variable
* fix: Updated workflow to solve errors
* fix: Updated branch format
* fix: Wrong naming
* Update docker-bake.hcl
* Update build-container.yml
* Update ik_llama-cuda.Containerfile
* Update ik_llama-cpu.Containerfile
* Update docker-bake.hcl
* Update build-container.yml
* Removed action/cache
* added -sSL for reliability and fixed the URL path
* added -sSL for reliability and fixed the URL path CUDA containerfile
* fix: correct Dockerfile RUN command syntax errors
- Combine split apt-get install commands in both Containerfiles
- Fix broken cmake command continuation in ik_llama-cuda.Containerfile
* fix: correct llama-swap download URL in Containerfiles
- Fix broken line continuation in curl download URL for llama-swap
* perf: improve ccache configuration in Containerfiles
- Add CCACHE_UMASK=000 for cache accessibility across stages
- Add CCACHE_MAXSIZE=1G to prevent unbounded growth
- Initialize ccache with ccache -i during build stage
* fix: remove problematic ccache initialization from Containerfiles
- ccache -i fails because CCACHE_DIR mount doesn't exist yet at build time
* fix: add git to CPU Containerfile build dependencies
- Resolves CMake warning about missing Git for build info
* chore: optimize Containerfile with smaller images and better healthchecks
- Add --no-install-recommends to all apt-get commands for smaller image size
- Add ca-certificates to base stage for HTTPS support
- Merge redundant build copy commands from 3 layers to 1
- Fix llama-swap version from 198 to v199 (latest release)
- Add HEALTHCHECK configuration with interval/timeout/retries to server and swap stages
- Copy /app/lib in server stage to fix container startup
* chore: fix CUDA Containerfile healthchecks and swap version
- Add /app/lib copy in server stage to fix container startup
- Fix llama-swap version from 198 to v199 (latest release)
- Add HEALTHCHECK configuration with interval/timeout/retries
* chore: fix indentation in Containerfiles and add LD_LIBRARY_PATH for server target
* fix: add --break-system-packages flag for pip in CPU Containerfile
* feat: add git bind mount for build info and NCCL support for CUDA
* fix: remove libnccl-dev from CUDA build (already included in base image)
* fix: added Markdown files to ignore files
* feat: use BUILD_NUMBER-COMMIT pattern for docker image tags
- Add BUILD_NUMBER and LLAMA_COMMIT to build workflow
- Update docker-bake.hcl to use version tag format matching llama-server --version output
- Format: VARIANT-BUILD_NUMBER-COMMIT (e.g., cu12-full-4406-3bc90dfd)
* fix: fetch full git history for accurate BUILD_NUMBER
- Add fetch-depth: 0 to actions/checkout to get all commits
- This ensures git rev-list --count HEAD returns correct total commit count
* fix: fetch full git history in Dockerfile for accurate BUILD_NUMBER
- Add git fetch --unshallow to get complete commit history during build
- This ensures build-info.cpp is generated with correct LLAMA_BUILD_NUMBER
* chore: update GitHub Actions to latest versions for Node.js 24 compatibility
- docker/setup-buildx-action@v3 -> v4
- docker/login-action@v3 -> v4
* chore: update all GitHub Actions to Node.js 24 compatible versions
- actions/checkout@v4 -> v6
- docker/setup-buildx-action@v3 -> v5
- docker/login-action@v3 -> v6
- docker/bake-action@v5 -> v7
* fix: use CI-passed BUILD_NUMBER and LLAMA_COMMIT in Dockerfile
- Add BUILD_NUMBER and LLAMA_COMMIT as build args
- Fall back to git commands if not provided
- Pass values explicitly to cmake for accurate build info
* fix: pass BUILD_NUMBER and LLAMA_COMMIT as Docker build args
- Add BUILD_NUMBER and LLAMA_COMMIT to docker bake args
- These will be used by the Containerfile for accurate build info
* fix: revert docker actions to v4 (latest available versions)
* fix: calculate BUILD_NUMBER and LLAMA_COMMIT directly in Containerfile
- Removed ARG defaults since we calculate from git during build
- Use git rev-list --count HEAD and git rev-parse for accurate version info
- Falls back to 0/unknown if git commands fail
* feat: calculate BUILD_NUMBER and LLAMA_COMMIT in Containerfiles
- Add git-based version calculation in both CPU and CUDA Containerfiles
- Remove .git bind mount (git is copied with COPY .)
- Pass build info to CMake for accurate llama-server --version output
* feat: calculate BUILD_NUMBER and LLAMA_COMMIT in Containerfiles
- Add git-based version calculation using git rev-list and git rev-parse
- Copy .git directory separately to ensure git commands work during build
- Pass build info to CMake for accurate llama-server --version output
* fix: cache improvements for CUDA and CPU builds
* fix: "/.git": not found
* fix: Unnecessary mv llama-swap
* fix: Remove BUILD_NUMBER and LLAMA_COMMIT from docker file, calculated by cmake proc
* fix: remove .git from dockerignore for local and CI builds
- Enables cmake to access .git directory during Docker build
- Required for version calculation in llama-server binary
- GitHub Actions uses explicit mount via bake action set parameter
* fix: Remove mounts key from Build and Push step in gh workflow
* ci: add .git verification step before build
* refactor: standardize Containerfile structure and remove .git mount dependency
- Remove --mount=type=bind,source=.git,target=.git from both Containerfiles
- Replace COPY . . with git clone for cleaner build context
- Add CUSTOM_COMMIT ARG for optional custom commit switching
- Standardize ARG/ENV ordering and comment formatting across CPU/CUDA variants
- Install ca-certificates before git clone to fix SSL verification issues
- Rename 'Structured artifact collection' to 'Collect build artifacts'
* ci: remove broken cache pruning step
* ci: remove broken prune-cache job
- Remove prune-cache job that was failing due to missing .git directory
- The job required a checkout step and the cache pruning logic was non-critical
* chore: Removed step for Verifying .git existance in GH workflow
* fix: ensure build always proceeds even if git switch fails
- Add '|| true' to git switch command so build continues on failure
- This prevents the entire RUN step from failing when CUSTOM_COMMIT is invalid
* fix: resolve Docker build pipeline issues
- Remove external git clone from Containerfiles, use build context directly
- Add BUILD_NUMBER and BUILD_COMMIT as CMake cache variables in build-info.cmake
- Fix .devops/tools.sh inclusion by using explicit COPY for hidden directories
- Set USE_CCACHE=true for CI builds
- Clean up unused SHA_SHORT variable from docker-bake.hcl
Fixes: Build steps were cached incorrectly due to external git clone ignoring the actual build context source.
* fix: include .git in Docker build context and add verification
* ci: add .git directory verification step after checkout
* build: fix .git mount path for Docker build context compatibility
* build: fix .git mount path for Docker build context compatibility
* docker: include .git in build context for version calculation
* ci: add .git directory verification step after checkout
* chore: Removed unecessary Verify .git step (It was a test)
* docs: update README with docker-bake and build-local.sh instructions
* docs: remove build-local.sh reference (not in repo)
* ci: optimize disk usage by limiting fetch depth and cleaning workspace
---------
Co-authored-by: HP Prodesk <sourceupdev@gmail.com>
* wip: separate llama_context for MTP with graph reuse
* wip: fix KV cache desync with separate MTP context
* refactor: remove dead mtp logic code, encapsulate KV mirroring
* mtp-context: derive args directly from the main model's context
* mtp: fix kv cache positions
* clean small comments
* minor refactor for context shift
* Use build_std_attention for Gemma4 when possible
It is possible for the 26b MoE and 31b dense models.
It is not possible for the E4B/E2B vaiants because they
don't have KV cache in each layer.
* Standardize Gemma4 dense ffn
* WIP: Gemma4 split mode graph
Runs but produces NaNs
* WIP: Gemma4 split mode graph
Runs but very high PPL. At least it is no longer NaN.
* WIP
* This works!
* Put attn_norm, attn_post_norm, ffn_norm, ffn_post_norm on all GPUs
* Fix crash when saving/loading KV cache
* WIP: split mode graph for Gemma4-MoE - crashes
* Split mode graph for Gemma4-MoE - this works
* Disable SWA optimization
Something goes wrong there
* Consolidate MoE and dense graph parallel
* Use build_std_attention for Gemma4 when possible
It is possible for the 26b MoE and 31b dense models.
It is not possible for the E4B/E2B vaiants because they
don't have KV cache in each layer.
* Standardize Gemma4 dense ffn
* WIP: Gemma4 split mode graph
Runs but produces NaNs
* WIP: Gemma4 split mode graph
Runs but very high PPL. At least it is no longer NaN.
* WIP
* This works!
* Put attn_norm, attn_post_norm, ffn_norm, ffn_post_norm on all GPUs
* Fix crash when saving/loading KV cache
* wip: build spec tuner for spefic args
* wip: test different reward system
* spec-tune: fix the reward to find best params given a good TPS
* spec-tune: refactor logic for its own file
* minor clean for comments and modules
* Little maintenance
* llama-quantize : Add the missing items in the help
* Add GGML_MAX_CONTEXTS define in the general cmakelist.txt
* Make the KV cache (CPU) based warnings clearer
* Correct placement of GGML_MAX_CONTEXTS definition
* Revert wrong indents
This reverts commit d0728cbb6c.
* Moving the GGML_MAX_CONTEXTS definition to src/CMakeLists.txt
* Update warning message for unsupported KV cache types
* forgotten antislash
* Gemma4: WIP
* Gemma4: WIP - runs with totally wrong results
* Gemma4: WIP - add CPU 512, 512 FA
* Gemma4: WIP
It gives a meaningful response in llama-cli, but PPL is still much too
high. Is this due to tokenizer issues?
* Gemma4: this works
I had forgotten the softcap on the final output.
* Remove log
* Gemma4: WIP E4B/E2B
* Gemma4: Q4B/E2B appear to work now
* gemma4: tokenizer fixes
Add HAVE_FANCY_SIMD path that processes 16 rows at a time using 512-bit
operations, combining two R8 groups via _mm512_inserti32x8. Reuses the
existing qx_r8_q8_dot_product 512-bit overload for the inner dot product.
Also updates num_rows for Q8_1 to 16 under HAVE_FANCY_SIMD.
Co-authored-by: Adam Caldwell <accaldwell@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
