adamfreeze

the question

Does storing Adam's second moment in bf16 bias it away from the true second moment, and can loss scaling correct that bias?

what came out

A bf16 second moment freezes once beta2 > 1 - 2^-(t+1) = 0.99609, so standard beta2=0.999 stops tracking. The bias flips sign with gradient skew: 0.24x low on steady gradients, 2.4x high on spiky ones. Loss scaling is scale-invariant and cannot fix it; stochastic rounding restores an unbiased 1.0x.

→ read the code and re-run it