foxwindow

the question

Does the Forgetting Transformer's effective attention window have an exact distribution, and how does it relate to ALiBi?

what came out

The effective window is the first-passage time of the forget-gate walk, with exact mean L/kappa + E[X^2]/(2kappa^2) (matched to 2e-3 against 200k-trial simulation) and variance sigma^2 L/kappa^3 (within about one percent). ALiBi is exactly the zero-variance limit, so data-dependence shows up as window variance.

→ read the code and re-run it