dropskew

the question

Does storing the inverted-dropout keep-scale in bfloat16 introduce a deterministic per-layer bias that survives averaging over masks?

what came out

Rounding the scale gives an exact bias (1-p)*round_dtype(1/(1-p))-1 that does not average out and compounds as (1+bias)^L. At p=0.1 in bf16 the scale is 1.109375 versus 1.111..., a -0.16% per-layer bias reaching -4.9% train/eval drift over 32 layers. It vanishes only at bf16-dyadic rates like p=0.2.

→ read the code and re-run it