← Lab
dfcdrop
confirmedupstream bugPython · pytorch_optimizerModel behavior & reasoningJuly 25, 2026
the question
Does pytorch_optimizer's default DiffGrad apply its diffGrad friction coefficient?
what came out
pytorch_optimizer's DiffGrad computes dfc = sigmoid(|g_prev-g|)*exp_avg but the default rectify=False path updates with bare exp_avg and returns before weight decay, so default DiffGrad is plain Adam. A dfc-applying fix diverges 51.7% after five steps, and weight_decay is silently ignored.
method & receipts
- Result: confirmed
- Reproducible — one script re-runs the whole thing from scratch.
- Tested — a correctness/benchmark suite ships alongside the code.
→ read the code and re-run it
github.com/v-code01/dfcdrop