Skip to main content
Vansh Verma
← Lab

dorafanin

confirmedupstream bugPython · PEFTModel behavior & reasoningJuly 25, 2026

the question

Is PEFT DoraLinearVariant.unmerge the exact inverse of merge on transformers Conv1D layers?

what came out

PEFT's DoraLinearVariant.unmerge divides by the bare dora_factor.view(-1,1) and omits the fan_in_fan_out transpose that both merge paths apply. On the Conv1D layers PEFT auto-flags for GPT-2 family, a trained square layer round-trips off by 0.14 per element (silent base corruption); non-square raises. nn.Linear round-trips clean.

method & receipts

  • Result: confirmed
  • Reproducible — one script re-runs the whole thing from scratch.
  • Tested — a correctness/benchmark suite ships alongside the code.

→ read the code and re-run it

github.com/v-code01/dorafanin