Skip to main content
Vansh Verma
← Lab

zlossgauge

confirmedshipPythonModel behavior & reasoningJuly 22, 2026

the question

What is the relationship between softmax cross-entropy's shift symmetry and the z-loss added for large-model training?

what came out

Cross-entropy is exactly flat along the all-ones logit direction, so its gradient sums to zero and the total logit is conserved (drift 6e-14 over 300 steps). z-loss's gradient sums to 2*logsumexp(z), exactly the missing gauge component, breaking the conservation on purpose (drift 421).

method & receipts

  • Result: confirmed
  • Reproducible — one script re-runs the whole thing from scratch.
  • Tested — a correctness/benchmark suite ships alongside the code.

→ read the code and re-run it

github.com/v-code01/zlossgauge