causalrank

the question

Does the rank<=d bottleneck cited for linear attention still hold once the causal mask is applied?

what came out

No. With a strictly positive feature map the causal mixing matrix tril(phi(Q)phi(K)^T) is full rank N for every d, including d=1 (measured rank 512), because its diagonal is strictly positive. The empirical low-rank behavior is energy concentration (stable rank ~1.24, condition number up to 2.6e10), not the feature-dimension bottleneck.

→ read the code and re-run it