softcapgrad

the question

What is the exact backward law of tanh logit soft-capping, and does bf16 reintroduce the hard-clip zero-gradient dead-zone it was meant to avoid?

what came out

The gradient factor is exactly 1-(z/c)^2, confidence-proportional attenuation, verified to 3.3e-16 over 2M points. In bf16 tanh saturates to 1.0 at a = 3.47c (a = 104 and 173 at Gemma2's caps 30 and 50), giving a bit-exact zero gradient reached by the most confident logits. End-to-end slowdown is only ~1.1x, not the pointwise several-x.

→ read the code and re-run it