constdecay

confirmedupstream bugPython · torch-optimizerModel behavior & reasoningJuly 25, 2026

the question

Does torch_optimizer's SGDW apply decoupled weight decay in proportion to each parameter?

what came out

torch_optimizer/sgdw.py SGDW.step writes p.data.add_(weight_decay, alpha=-lr), subtracting a constant -lr*wd from every element instead of the proportional p*(1-lr*wd). A zero parameter drifts to -0.05 and a magnitude-100 weight is under-decayed 100x. The momentum line above passes a tensor operand.

method & receipts

Result: confirmed
Reproducible — one script re-runs the whole thing from scratch.
Tested — a correctness/benchmark suite ships alongside the code.

→ read the code and re-run it

github.com/v-code01/constdecay