pyramidskew

confirmedupstream bugPython · CPythonInference & servingJuly 23, 2026

the question

Does PyramidKV's per-layer KV-cache allocator keep the total budget exactly constant at L*C as its same-budget comparisons assume?

what came out

No. The integer floor in the step size makes the retained total exactly L*C + (L/2)*r with r = (max_num-min_num) mod (L-1), always at or above L*C, never under. The gap reaches 11.33% at C=128, L=32 and shrinks with budget, so PyramidKV's same-budget comparisons run at a larger cache than the baselines.

method & receipts

Result: confirmed
Reproducible — one script re-runs the whole thing from scratch.
Tested — a correctness/benchmark suite ships alongside the code.

→ read the code and re-run it

github.com/v-code01/pyramidskew