primegap

the question

Is MDM-Prime's variational objective a valid upper bound on the negative log-likelihood?

what came out

MDM-Prime applies the per-unit MDLM weight to a joint sub-token reconstruction term while masking sub-tokens independently, so at the optimal decoder L_vb = H - I/2, half the within-token multi-information below the entropy. It is not an upper bound on NLL; Zipf and skewed token laws violate it, independent digits stay tight.

→ read the code and re-run it