kappaweight

confirmedupstream bugPython · riverSystems & data structuresJuly 25, 2026

the question

Does river's CohenKappa use the weighted total as its denominator under non-unit sample weights?

what came out

river/metrics/kappa.py CohenKappa.get divides weighted confusion-matrix totals by the unweighted cm.n_samples instead of cm.total_weight, which sibling Accuracy uses. Under non-unit weights kappa is wrong (0.822 vs sklearn's 0.153) and observed agreement p0 can exceed one (10.0).

method & receipts

Result: confirmed
Reproducible — one script re-runs the whole thing from scratch.
Tested — a correctness/benchmark suite ships alongside the code.

→ read the code and re-run it

github.com/v-code01/kappaweight