packreset

the question

When packing documents for training with RoPE, does resetting position ids at document boundaries change the model's output or gradients?

what came out

No. RoPE logits depend only on relative offset, so per-document reset under the block mask is an exact no-op: forward differs ~1e-15, gradients ~2e-15, non-accumulating out to offset 100000. Under a full causal mask the second document leaks ~70% of its mass across the boundary and reset does not fix it (0.70 vs 0.72). The mask is load-bearing; the dual holds for absolute PEs.

→ read the code and re-run it