← Lab
dropmean
confirmedcertificatePythonModel behavior & reasoningJuly 22, 2026
the question
In cut-cross-entropy, what exactly sets the error introduced by dropping sub-threshold vocabulary entries from the gradient?
what came out
Above V*=32/eps (4096 in bf16) the filter drops the entire non-target gradient, and the error equals (1-p_t) times the probability-weighted unembedding row-mean, bit-exact and independent of tau. It stays 10.91 from tau=2^-9 to 2^-15, refuting the per-term-truncation justification by four orders of magnitude.
method & receipts
- Result: confirmed
- Reproducible — one script re-runs the whole thing from scratch.
- Tested — a correctness/benchmark suite ships alongside the code.
→ read the code and re-run it
github.com/v-code01/dropmean