Skip to main content
Vansh Verma
← Lab

chrfshort

confirmedupstream bugPython · nltkInference & servingJuly 25, 2026

the question

Does nltk's chrF average over the effective n-gram order for short strings?

what came out

nltk's corpus_chrf divides the summed per-order F-scores by max_len (6) rather than the effective order count, so a perfect match shorter than six characters scores length/6: 'c' scores 0.1667 not 1.0 while sacrebleu returns 1.0. The fix is to divide by the orders that actually have n-grams.

method & receipts

  • Result: confirmed
  • Reproducible — one script re-runs the whole thing from scratch.
  • Tested — a correctness/benchmark suite ships alongside the code.

→ read the code and re-run it

github.com/v-code01/chrfshort