normnull

confirmedcertificatePython · PyTorch · Apple M4Model behavior & reasoningJuly 22, 2026

the question

What is the exact linear-algebraic form of a normalization layer's backward pass, and how do RMSNorm and LayerNorm differ?

what came out

Each backward pass is an exact scaled orthogonal projector: RMSNorm rank d-1 (kills scale), LayerNorm rank d-2 (kills scale and the mean/DC direction), verified to machine precision in fp64. The difference is exactly one dimension, the mean. Under LayerNorm every gradient is mean-free (sum(dx) at the fp32 floor); RMSNorm's is about six orders larger.

method & receipts

Result: confirmed
Tested — a correctness/benchmark suite ships alongside the code.

→ read the code and re-run it

github.com/v-code01/normnull