Skip to main content
Vansh Verma
← Lab

decodedrift

confirmedPython · llama.cppInference & servingJuly 6, 2026

the question

Does per-token decode latency drift up as the KV cache grows?

what came out

Yes -- inter-token latency rises linearly with position (slope positive, CI excludes zero), it's material over a long generation, and the bigger model drifts faster in absolute terms. The decode-time complement to the prefill O(n^2) wall. All three predictions held.

method & receipts

  • Result: confirmed
  • Pre-registered — the prediction was written down and committed before the run.
  • Reproducible — one script re-runs the whole thing from scratch.
  • Independently reviewed — an adversarial review pass, kept in the repo.
  • Machine-readable claims — every reported number, in a checkable file.
  • Tested — a correctness/benchmark suite ships alongside the code.

→ read the code and re-run it

github.com/v-code01/decodedrift