varsplit

the question

In a Pre-LN transformer at initialization, do the MLP and attention contributions to residual-stream variance grow on the same depth schedule, and can a single depth-uniform residual scale keep per-layer variance balanced?

what came out

The MLP variance increment is depth-flat (slope +0.0003, 95% CI straddling zero) while the attention increment rises (slope +0.010, CI strictly positive) with token rank collapse. The two invert (MLP about 5x attention at layer 1, attention about 3x MLP by layer 48), so a single uniform residual scale cannot balance both.

→ read the code and re-run it