mupblind

the question

Under muP's 1/d attention-logit scaling, does attention start maximally diffuse at initialization with an entropy gap that collapses as Theta(1/d) with width, unlike standard 1/sqrt(d)?

what came out

Measured across head dim 16 to 512, muP's init entropy gap collapses with log-log slope -1.001 (Theta(1/d)) while standard 1/sqrt(d) stays flat at -0.002. muP starts attention maximally diffuse and sharpens only as training aligns queries and keys, the gap rising from 0.004 to 1.9.

→ read the code and re-run it