← Lab
calibration
null resultPython · CPUModel behavior & reasoningJuly 6, 2026
the question
Does a small model's multiple-choice confidence match its accuracy?
what came out
No -- systematically overconfident, measured from exact top-logprobs. A 2x2 (size x ARC difficulty) study: miscalibration compounds with both smaller size and harder task.
method & receipts
- Result: null result — an honest negative, reported as it came out.
- Reproducible — one script re-runs the whole thing from scratch.
- Independently reviewed — an adversarial review pass, kept in the repo.
- Machine-readable claims — every reported number, in a checkable file.
- Tested — a correctness/benchmark suite ships alongside the code.
→ read the code and re-run it
github.com/v-code01/calibration