Skip to main content
Vansh Verma
← Lab

momentwo

confirmedshipPythonModel behavior & reasoningJuly 22, 2026

the question

How does the momentum term in Titans neural memory change the stability of its delta-rule forgetting?

what came out

It reduces the update to an exact second-order recurrence (closure to 5.7e-16), raising the stable learning-rate ceiling to theta*c < (1+eta)(2-alpha)/2, roughly double the delta rule's. The retention horizon becomes sqrt(eta(1-alpha)) and adds damped ringing, staying below divergence for eta<1.

method & receipts

  • Result: confirmed
  • Reproducible — one script re-runs the whole thing from scratch.
  • Tested — a correctness/benchmark suite ships alongside the code.

→ read the code and re-run it

github.com/v-code01/momentwo