Skip to main content
Vansh Verma
← Lab

terflip

confirmedupstream bugPython · lm-evaluation-harnessInference, serving & evaluationJuly 24, 2026

the question

Is lm-evaluation-harness's TER metric registered with the correct `higher_is_better` direction?

what came out

lm-evaluation-harness registers the TER metric `higher_is_better=True`, contradicting its own "Lower is better" docstring, while bleu and chrf are correct. TER is returned un-negated, so on the eight WMT14/16 and IWSLT2017 tasks that inherit the default (the FLORES configs override it) a worse-translating system is picked as best and an up arrow is printed next to an error rate. Fixed upstream in lm-evaluation-harness#3993.

method & receipts

  • Result: confirmed
  • Reproducible: one script re-runs the whole thing from scratch.
  • Tested: a correctness/benchmark suite ships alongside the code.

→ read the code and re-run it

github.com/v-code01/terflip