cedeadzone

the question

Does a fused bf16 cross-entropy kernel that rounds the softmax probability before subtracting one zero out the gradient of high-confidence tokens?

what came out

R(p_c)-1 rounds to exactly zero for every p_c >= 1 - 2^-9 = 0.998047, so a fused bf16 head gives no gradient on tokens the model is over 99.8% confident about, while the fp32 subtract-then-round path stays provably immune. On a 32k-vocab softmax at logit gap 17 the fused head returns 0 versus -9.9e-4.

→ read the code and re-run it