← Lab
svdscale
confirmedupstream bugPython · PEFTModels, training & reasoningJuly 24, 2026
the question
Does PEFT add_weighted_adapter's svd combination apply each LoRA adapter's scaling factor exactly once?
what came out
PEFT add_weighted_adapter's svd family (the default) multiplies each adapter weight by target.scaling and then sums get_delta_weight tensors, which already carry scaling, applying it twice. At the standard lora_alpha=2r the merged adapter is exactly twice too strong; the cat path applies scaling once. Already reported by the PEFT maintainers in peft#3111 (opened 2026-03-18) and fixed there (merged 2026-09-10).
method & receipts
- Result: confirmed
- Reproducible: one script re-runs the whole thing from scratch.
- Tested: a correctness/benchmark suite ships alongside the code.
→ read the code and re-run it
github.com/v-code01/svdscale