ropealias

the question

Does computing rotary position embeddings in bf16 cause a long-context model to lose the ability to distinguish nearby token positions?

what came out

Confirmed. bf16 resolves position only to ULP(p) ~ p/128, so adjacent positions collapse to one rotary angle: 87% of adjacent pairs aliased at position 1024, 99% at 16384, 100% at 131072. Aliased positions give bit-identical rotary vectors and identical attention scores. The phase must be fp32; the cos/sin cache tolerates bf16.

→ read the code and re-run it