clonemom

confirmedupstream bugPython · pytorch_optimizerModel behavior & reasoningJuly 25, 2026

the question

Does pytorch_optimizer's CAME optimizer preserve first-moment momentum for one-dimensional parameters?

what came out

pytorch_optimizer CAME's non-factored branch aliases update=exp_avg then mul_(lr) in place, overwriting the persistent beta1 momentum for every 1D parameter (norm gains, biases) each step. Momentum stays pinned near lr magnitude (3e-05 vs 0.68) and the trajectory diverges 68% within six steps. Fix: exp_avg.clone().

method & receipts

Result: confirmed
Reproducible — one script re-runs the whole thing from scratch.
Tested — a correctness/benchmark suite ships alongside the code.

→ read the code and re-run it

github.com/v-code01/clonemom