Skip to main content
Vansh Verma
← Lab

decodeclean

confirmedupstream bugPython · transformersInference & servingJuly 25, 2026

the question

Does transformers' batched decode() honour the tokenizer's configured clean_up_tokenization_spaces?

what came out

transformers' batched decode() branch pops clean_up_tokenization_spaces with a hardcoded False, while single decode() and batch_decode() resolve it from the tokenizer config. A bert-base-uncased tokenizer with the flag True decodes one sequence to "i don't think so." alone but "i don ' t think so." inside a batch.

method & receipts

  • Result: confirmed
  • Reproducible — one script re-runs the whole thing from scratch.
  • Tested — a correctness/benchmark suite ships alongside the code.

→ read the code and re-run it

github.com/v-code01/decodeclean