Skip to main content
Vansh Verma
← Lab

terflip

confirmedupstream bugPython · lm-evaluation-harnessInference & servingJuly 24, 2026

the question

Is lm-evaluation-harness's TER metric registered with the correct `higher_is_better` direction?

what came out

lm-evaluation-harness registers the TER metric `higher_is_better=True`, contradicting its own "Lower is better" docstring, while bleu and chrf are correct. Since TER is returned un-negated, a worse-translating system is picked as best and an up arrow is printed next to an error rate on WMT/FLORES/IWSLT. Fix: `higher_is_better=False`.

method & receipts

  • Result: confirmed
  • Reproducible — one script re-runs the whole thing from scratch.
  • Tested — a correctness/benchmark suite ships alongside the code.

→ read the code and re-run it

github.com/v-code01/terflip