Guide: Q4_0 4-bit KV cache with per-channel mean-centering (setup + calibration)

[khosravipasha]

What it does

Stores the KV cache in Q4_0 (4-bit) to cut KV-cache memory, and applies a precomputed per-(kv-head, channel) bias to the K vector before it is quantized, reducing the symmetric-quantization error for channels with a nonzero activation mean. Zero decode-time cost (one subtract at cache write). Works on hybrid (attention + recurrent/SSM) models as of #53.

The three PRs

• kv-cache: optional per-channel K-cache mean-centering for Q4_0 #51 — core per-channel K-cache mean-centering for Q4_0.
• kv-mean-center: add make-calib-corpus.sh self-generated corpus helper #52 — make-calib-corpus.sh: generate calibration text from the model's own output (no external corpus needed).
• kv-cache: mean-centering on hybrid models + calibration-basis guard (composes with K rotation) #53 — hybrid-memory support + a calibration-basis guard that rejects a K-rotation mismatch at load.

Prerequisites

• Build from source (all three PRs are on the branch); this produces llama-cli / llama-server and llama-kv-mean-center.
• curl, jq, gzip for the corpus generator.

Steps

1. Generate calibration data (example helper; bring a larger, representative corpus for real use):

tools/kv-mean-center/make-calib-corpus.sh -s ./llama-server -m model.gguf -o corpus.txt -g 999

2. Calibrate the bias (produces a model-specific bias GGUF; calibrated with K-rotation inactive):

llama-kv-mean-center -m model.gguf -f corpus.txt -o kv-bias.gguf -ngl 999

3. Run inference — pass the Q4_0 KV flags and the bias, and match the K-rotation state used at calibration:

LLAMA_ATTN_ROT_DISABLE=1 llama-cli -m model.gguf -ngl 999 -fa on \
  --cache-type-k q4_0 --cache-type-v q4_0 --kv-mean-center kv-bias.gguf -p "..."

Notes

• The bias GGUF must be passed on every inference run (--kv-mean-center); it is not stored in the model.
• Calibration and inference must agree on the K-rotation state. The tools calibrate with rotation off, so inference needs LLAMA_ATTN_ROT_DISABLE=1; the kv-cache: mean-centering on hybrid models + calibration-basis guard (composes with K rotation) #53 basis guard refuses to load on a mismatch (this is intentional, to avoid silently-wrong centering).
• The bias is model-specific (recalibrate per model).
• The corpus/calibration scripts are examples — for production quality, use a larger and more representative calibration corpus.
• Requires flash attention (-fa on) for the quantized KV cache.