Skip to main content
Vansh Verma
← Lab

doubleshift

confirmedupstream bugPython · transformersModels, training & reasoningJuly 25, 2026

the question

Does transformers MoshiForCausalLM shift its training labels exactly once when computing the text loss?

what came out

transformers MoshiForCausalLM.forward pre-shifts labels then passes them positionally into ForCausalLMLoss, which shifts them again; every logit is scored against the token two positions ahead and row boundaries bleed at the flatten seam. The sibling CSM model passes shift_labels by keyword with labels=None correctly. Since fixed upstream in transformers#48493.

method & receipts

  • Result: confirmed
  • Reproducible: one script re-runs the whole thing from scratch.
  • Tested: a correctness/benchmark suite ships alongside the code.

→ read the code and re-run it

github.com/v-code01/doubleshift