Skip to main content
Vansh Verma
← Lab

merdenom

confirmedupstream bugPython · torchmetricsInference & servingJuly 25, 2026

the question

Does torchmetrics use the correct number of aligned positions for Match Error Rate and Word Information metrics?

what came out

torchmetrics MER, WIL, and WIP use `max(len(ref), len(hyp))` for aligned positions and hit count, which is short by `min(D, I)` whenever an alignment has both a deletion and an insertion. On ref "b c d" / hyp "a b c" MER reports 0.667 vs the defined 0.5; WER is unaffected. Fix: use `edit_distance + hits`.

method & receipts

  • Result: confirmed
  • Reproducible — one script re-runs the whole thing from scratch.
  • Tested — a correctness/benchmark suite ships alongside the code.

→ read the code and re-run it

github.com/v-code01/merdenom