massiveact

mixedshipPython · PyTorchModel behavior & reasoningJuly 12, 2026

the question

Are Qwen2.5 residual-stream massive activations a size-invariant law across model sizes?

what came out

Every size (0.5B, 1.5B, 3B) has one fixed channel at 600-950x the median RMS, with 100% of its mass on the first token. But the pre-registered relative-depth invariance is false: onset is pinned to absolute layer 2-3, not a constant fraction of depth.

method & receipts

Result: mixed
Pre-registered — the prediction was written down and committed before the run.
Reproducible — one script re-runs the whole thing from scratch.

→ read the code and re-run it

github.com/v-code01/massiveact