← Lab
maxf1
confirmedupstream bugPython · lightevalInference, serving & evaluationJuly 24, 2026
the question
Does lighteval report the positive-class F1 for GLUE MRPC and QQP?
what came out
lighteval metrics_corpus.py CorpusLevelF1Score.compute_corpus returns np.max over the per-class F1 array from average=None instead of fscore[1], the positive-class F1 GLUE MRPC/QQP define. On majority-negative tasks a null model scores 0.889 and two models can rank in the wrong order. Also reported upstream in lighteval#1275 (opened 2026-06-25, still open).
method & receipts
- Result: confirmed
- Reproducible: one script re-runs the whole thing from scratch.
- Tested: a correctness/benchmark suite ships alongside the code.
→ read the code and re-run it
github.com/v-code01/maxf1