Skip to main content
Vansh Verma
← Lab

ropelowrank

mixedshipPython · CPUInference & servingJuly 12, 2026

the question

Does RoPE inflate the effective rank of the key cache and block low-rank SVD compression, while values stay at most as low-rank as pre-RoPE keys?

what came out

On Qwen2.5, pre-RoPE keys are strikingly low-rank (12 to 17 of 64 to 128 dims at 99% energy); RoPE inflates that 2.5x to 3.8x at 99% (5x to 8x at 90%), surviving mean-centering. Since RoPE is a per-row isometry, compressing pre-RoPE then rotating is free. The values-low-rank prediction is false: V exceeds pre-RoPE K.

method & receipts

  • Result: mixed
  • Pre-registered — the prediction was written down and committed before the run.
  • Reproducible — one script re-runs the whole thing from scratch.
  • Tested — a correctness/benchmark suite ships alongside the code.

→ read the code and re-run it

github.com/v-code01/ropelowrank