attnscale

the question

What does the 1/sqrt(d) scaling in attention do at inference, beyond the gradient argument?

what came out

It keeps logits O(1) so exp doesn't overflow fp16 (unscaled logits cross the 11.09 threshold from d=64 up) and softmax doesn't collapse to one-hot -- the same fact the gradient argument names, seen from a 16-bit datapath. One predicted effect (a bf16 accuracy gain) was honestly falsified.

→ read the code and re-run it