Skip to main content
Vansh Verma
← Lab

seedzero

confirmedupstream bugPython · transformersCluster & infrastructureJuly 25, 2026

the question

Does transformers' DataCollatorForLanguageModeling honor seed=0 for reproducible MLM masking?

what came out

transformers data_collator.py DataCollatorForLanguageModeling guards its seeded generator with if self.seed (truthiness), so seed=0 is dropped and MLM masking plus -100 labels fall back to the global RNG. seed=0 labels differ across global states while every nonzero seed is reproducible. Fix is if self.seed is not None.

method & receipts

  • Result: confirmed
  • Reproducible — one script re-runs the whole thing from scratch.
  • Tested — a correctness/benchmark suite ships alongside the code.

→ read the code and re-run it

github.com/v-code01/seedzero