qkceiling

the question

Is there an exact, tight, dimension-independent ceiling on the largest single softmax weight under QK-normalized attention?

what came out

Yes: w_max = 1/(1+(n-1)e^{-2g}), attained at the collinear config and never crossed (0/100000, matched to 2e-13), exactly dimension-independent. The critical length n* = 1+e^{2g} splits the variants: RMSNorm-QK (g ~ sqrt(d)) never binds, while unit-L2-QK (g=1) caps any key below 0.023% at 32k tokens.

→ read the code and re-run it