sinktemp

the question

Does softmax attention conserve per-key mass, and what sets the cost of restoring doubly-stochastic key conservation as attention sharpens?

what came out

Column sums have mean exactly 1 but per-key mass diverges, ranging [0.12, 2.96] at beta=4. Sinkhorn restores every column to 1; its Birkhoff contraction kappa=tanh(beta*Delta_0/4) is exactly linear in inverse temperature, so iterations to 1e-6 blow up from 57 to 258638 as beta goes 0.5 to 2. Key conservation is exponentially expensive for sharp attention.

→ read the code and re-run it