Skip to main content
Vansh Verma
← Lab

simflip

confirmedupstream bugPython · sentence-transformersInference, serving & evaluationJuly 24, 2026

the question

Does sentence-transformers' BinaryClassificationEvaluator flag the Manhattan/Euclidean similarity direction correctly?

what came out

sentence-transformers' BinaryClassificationEvaluator flags Manhattan and Euclidean greater_is_better=False, but their score_fns return negative distance (higher = more similar). The flag runs the threshold search, average precision, and labelling backwards: reported Manhattan accuracy is 49.5% versus a true 100%. The fix is greater_is_better=True, since merged upstream in sentence-transformers#3886.

method & receipts

  • Result: confirmed
  • Reproducible: one script re-runs the whole thing from scratch.
  • Tested: a correctness/benchmark suite ships alongside the code.

→ read the code and re-run it

github.com/v-code01/simflip