shufdup

the question

Does litdata's FullShuffle preserve disjoint per-worker sample coverage across epochs on multiple nodes?

what came out

From epoch two on multiple nodes, the intra-node reshuffle re-indexes the original full chunk intervals with a stream that already lists split chunks twice, so each split chunk is handed whole to two workers. In the smallest multi-node case one third of samples (10 of 30) are trained twice, silently.

→ read the code and re-run it