Skip to main content
Vansh Verma
← Lab

clipflip

confirmedupstream bugPython · timmModel behavior & reasoningJuly 25, 2026

the question

Does timm's AdafactorBigVision RMS update clip contract over-sized updates to the threshold?

what came out

timm's AdafactorBigVision divides the update by min(1, RMS*threshold) instead of max(1, RMS/threshold), flipping both the reciprocal and the clamp direction. The clip never shrinks a large update (RMS 5 passes as 5) and amplifies small ones. A gradient spike passes through unchanged. The sibling adafactor.py is correct.

method & receipts

  • Result: confirmed
  • Reproducible — one script re-runs the whole thing from scratch.
  • Tested — a correctness/benchmark suite ships alongside the code.

→ read the code and re-run it

github.com/v-code01/clipflip