attnrank

null resultPythonInference & servingJuly 11, 2026

the question

Is causal-decoder attention low-rank, as the Linformer premise assumes?

what came out

No. On an exact SVD oracle over Qwen2.5, the 99%-energy rank grows LINEARLY with sequence length (alpha ~1.0), near-universally across heads -- softmax destroys the head-dim-bounded low rank of the QK^T scores. The fixed-k constant-rank premise is broken for causal decoders.

method & receipts

Result: null result — an honest negative, reported as it came out.
Pre-registered — the prediction was written down and committed before the run.

→ read the code and re-run it

github.com/v-code01/attnrank