← Lab
goodput
confirmedPython · llama.cppInference & servingJuly 6, 2026
the question
How much concurrency can a server absorb before it's pure latency cost?
what came out
There's a goodput knee at the slot count: throughput maxes at C=4 (211 tok/s on 1.5B, 572 on 0.5B), and C=8/12/16 add no throughput -- only latency, which grows roughly linearly (1.52s->4.54s on 1.5B). The smaller model has a higher ceiling but the same knee. All three predictions held.
method & receipts
- Result: confirmed
- Pre-registered — the prediction was written down and committed before the run.
- Reproducible — one script re-runs the whole thing from scratch.
- Independently reviewed — an adversarial review pass, kept in the repo.
- Machine-readable claims — every reported number, in a checkable file.
- Tested — a correctness/benchmark suite ships alongside the code.
→ read the code and re-run it
github.com/v-code01/goodput