rmsgauge

the question

Is the scale of a weight feeding an RMSNorm a gauge freedom that the loss and gradient cannot change, leaving weight decay as its only control?

what came out

Scaling the pre-RMSNorm weight leaves the output invariant (3e-15), so by Euler's theorem the gradient's radial component is exactly zero (<grad,W> ~1e-15). Only decoupled weight decay moves the norm, settling at 2*wd*||W*||^2=lr^2*||g||^2, and the effective angular learning rate grows like 1/||W|| (0.0025, 0.010, 0.040 at norm scales 1, 1/2, 1/4).

→ read the code and re-run it