maskoverflow

the question

For a fully masked attention row, is the safe mask value determined by how much attention it leaks, or by whether the masked logit overflows?

what came out

Leakage flushes to a bit-identical zero below the per-dtype exp floors (-104 fp32, -93 bf16, -18 fp16), making the leakage ranking vacuous. The real discriminator is overflow: -1e9 casts to -inf and NaNs a fully masked row in float16 while finfo.min is safe, inverting the float32 intuition. Contamination reaches a full NaN sequence in exactly two attention layers.

→ read the code and re-run it