spantail

confirmedupstream bugPython · datatroveCluster & infrastructureJuly 24, 2026

the question

Does datatrove's sentence-dedup restore the full short duplicate span it means to keep?

what came out

datatrove's sentence-dedup fills removed_span under an 'elif not removed_span' guard, so it holds only the run's first sentence. When min_words_to_remove_span keeps a short duplicate span, exactly n-1 of n sentences are silently dropped and the word gate undercounts. The fix is an else branch accumulating the whole run.

method & receipts

Result: confirmed
Reproducible — one script re-runs the whole thing from scratch.
Tested — a correctness/benchmark suite ships alongside the code.

→ read the code and re-run it

github.com/v-code01/spantail