adafactorlift

confirmedcertificatePython · PyTorchModel behavior & reasoningJuly 22, 2026

the question

What exactly does Adafactor's factored second-moment estimate represent, and how much information does it lose per step?

what came out

V_hat = r c^T/s is exactly the contingency-table independence model of the squared-gradient matrix, and its per-step information loss equals half the G-test statistic exactly (D=19.364982=G^2/2). Against Adam each coordinate's step is mis-scaled by exactly 1/sqrt(lift), lossless only when V is rank-1 separable.

method & receipts

Result: confirmed
Tested — a correctness/benchmark suite ships alongside the code.

→ read the code and re-run it

github.com/v-code01/adafactorlift