negvar

confirmedupstream bugPython · pytorch_optimizerModel behavior & reasoningJuly 25, 2026

the question

Does pytorch_optimizer's QHAdam keep a non-negative second-moment accumulator on default hyperparameters?

what came out

pytorch_optimizer's QHAdam second-moment line drops parentheses, adding 1.0 - beta2_adj*g^2 instead of (1-beta2_adj)*g^2. For |g|>1 the variance accumulator goes negative (-1.998 after three steps) and its sqrt yields NaN on default hyperparameters; for |g|<1 it inflates the denominator ~40x. The first-moment line is correct.

method & receipts

Result: confirmed
Reproducible — one script re-runs the whole thing from scratch.
Tested — a correctness/benchmark suite ships alongside the code.

→ read the code and re-run it

github.com/v-code01/negvar