← Lab
kvbits
confirmedPython · llama.cppInference & servingJuly 5, 2026
the question
Quantizing the KV cache to 4-bit saves memory -- but does the key cache tolerate it as well as the value cache?
what came out
No. On real llama.cpp the key cache needs >=8 bits while the value cache stays lossless at 4-bit. Asymmetric Kq8/Vq4 buys a 59% KV reduction at <1% perplexity; symmetric 4-bit is catastrophic.
method & receipts
- Result: confirmed
- Reproducible — one script re-runs the whole thing from scratch.
- Machine-readable claims — every reported number, in a checkable file.
- Tested — a correctness/benchmark suite ships alongside the code.
→ read the code and re-run it
github.com/v-code01/kvbits