What it does

llama-swap can keep several models resident at once, but it only trusts the combinations declared in its matrix: block and does not check free memory. Declare one that does not fit and a model runs out of memory; declare too few and headroom is wasted. llama-matrix is a companion to llama-swap, a separate project: it measures each model on your machine and writes that block for you. It does nothing to your model weights.

Measures real footprints

measure loads each model alone, waits for its allocation to settle and records the GPU memory it occupies, and host RAM the same way. A footprint is cached by its memory-affecting flags, so an unrelated config edit never forces a re-measure.

Declares everything that fits

build finds every maximal combination whose measured footprints sum under the budget minus a safety margin, and checks each one against host RAM as well. --budget keeps room for other apps.

Applies, then checks

build --apply backs up config.yaml, splices in the block, waits for the hot-reload and rolls back on any anomaly. validate loads the tightest declared combination and compares what it occupies with the prediction.

Made for scripts and agents

Every command takes --json, and llama-matrix --llm prints the full guide, every command plus the design, for a person or an AI agent.

A first look

The example from the README: a roster where a couple of coders, a general model and small aux services fit together, but a 122B model does not. Small services ride along in every set, each pack is a group of models that may be resident together, and a heavy model runs alone.

matrix:
  evict_costs:                               # what to keep when something has to go
    gemma: 10                                #   chat models outrank the image pool…
    z-image-turbo: 1                         #   …which reloads in seconds
  sets:
    aux:    "embed & rerank & whisper"       # small services; ride along with every set
    pack1:  "gemma & glm-flash & +aux"       # three models co-resident, under budget
    pack2:  "coder-30b & gemma & +aux"
    heavy_qwen122: "qwen122 & +aux"          # too big to share - runs alone (+ aux)

Install

Homebrew, on macOS and Linux

Installs a prebuilt binary, so no compiler is needed. The tap is trusted once first, as Homebrew 6 and later ask.

brew trust --tap jakobhviid/tapbrew install jakobhviid/tap/llama-matrix

Or one line, without Homebrew

No compiler and no root: downloads the prebuilt binary to ~/.local/bin.

curl -fsSL https://raw.githubusercontent.com/jakobhviid/llama-matrix/main/install.sh | sh

Or from source

Needs a Rust toolchain. The binary lands in target/release/llama-matrix.

cargo build --release

Then point it at your llama-swap, once

Finds your config.yaml and endpoint, detects the GPU budget and writes llama-matrix.toml. It needs a running llama-swap with the matrix engine.

llama-matrix setup

Documentation

README: requirements, every command and the known limitations
WORKFLOWS.md: the operating loops, from setup to measure, build and apply
ARCHITECTURE.md: the memory model and how it is built
SPEC.md: the config, the measurement store and the matrix format
PRINCIPLES.md: the design rules
ROADMAP.md: what it does not do yet
Releases