Skip to main content
Vansh Verma
← Lab

maxf1

confirmedupstream bugPython · lightevalInference & servingJuly 24, 2026

the question

Does lighteval report the positive-class F1 for GLUE MRPC and QQP?

what came out

lighteval metrics_corpus.py CorpusLevelF1Score.compute_corpus returns np.max over the per-class F1 array from average=None instead of fscore[1], the positive-class F1 GLUE MRPC/QQP define. On majority-negative tasks a null model scores 0.889 and two models can rank in the wrong order.

method & receipts

  • Result: confirmed
  • Reproducible — one script re-runs the whole thing from scratch.
  • Tested — a correctness/benchmark suite ships alongside the code.

→ read the code and re-run it

github.com/v-code01/maxf1