detok

confirmedshipPython · CPUInference & servingJuly 12, 2026

the question

Does streaming byte-level BPE detokenization token-by-token corrupt a codepoint exactly when the tokenizer emits it as more than one token, and what is the minimal correct lookahead?

what came out

Yes, checked over 127,096 codepoints with no exceptions: naive per-token decode emits U+FFFD iff the vocab splits the codepoint (0% of ASCII, 40% of 2-byte, 76% of 3-byte, 99% of 4-byte). The minimal streaming fix holds back at most 3 bytes and 3 tokens, both bounds tight.

method & receipts

Result: confirmed
Pre-registered — the prediction was written down and committed before the run.
Reproducible — one script re-runs the whole thing from scratch.
Tested — a correctness/benchmark suite ships alongside the code.

→ read the code and re-run it

github.com/v-code01/detok