Skip to main content
Vansh Verma
← Lab

kvquant

null resultPython · llama.cppInference & servingJuly 6, 2026

the question

Decode is memory-bandwidth-bound, so shouldn't a smaller (quantized) KV cache be faster?

what came out

No -- both the speedup and the drift-reduction predictions were falsified. Quantizing the KV cache makes decode *slower*: the dequant cost outweighs the fewer bytes read. A counter-intuitive, honest null.

method & receipts

  • Result: null result — an honest negative, reported as it came out.
  • Pre-registered — the prediction was written down and committed before the run.
  • Reproducible — one script re-runs the whole thing from scratch.
  • Independently reviewed — an adversarial review pass, kept in the repo.
  • Machine-readable claims — every reported number, in a checkable file.
  • Tested — a correctness/benchmark suite ships alongside the code.

→ read the code and re-run it

github.com/v-code01/kvquant