DoRA, rsLoRA, and Modern PEFT

DoRA (Weight-Decomposed Low-Rank Adaptation, NVIDIA, arXiv:2402.09353, ICML 2024 Oral) fixes a structural deficit of LoRA: LoRA couples magnitude and direction updates proportionally (when the low-rank product B·A shifts a column, both its magnitude and its direction move together in a fixed ratio), whereas full fine-tuning updates them independently. LoRA cannot rotate a direction without rescaling its magnitude, and vice versa — roughly half of what full FT does, and the half LoRA cannot express at any rank. DoRA decomposes the weight into a per-column magnitude vector m and a unit-normalized direction, applies a LoRA-style adapter ONLY to the direction, and adapts the magnitude via the tiny vector m. It closes ~half the gap between LoRA and full FT at ~5–10% more VRAM, and after merge the magnitude and direction fold back into a single weight tensor that is shape-identical to the base — ZERO inference overhead, no serving-stack changes.

rsLoRA (rank-stabilized LoRA, arXiv:2312.03732) fixes the high-rank paradox: vanilla LoRA's α/r scaling collapses toward zero as rank grows (at r=64, α=16 the scale is 0.25; at r=128 it is 0.125), making the adapter numerically present but functionally inert and training unstable. rsLoRA changes one character — α/r becomes α/√r — keeping forward-pass magnitudes stable as rank climbs (at r=64, α=16 the scale becomes 2.0). Rule of thumb: above r=32, use rsLoRA; below that it does no harm but offers little. rsLoRA STACKS with DoRA, it does not compete: DoRA edits the magnitude/direction decomposition on top of the frozen base; rsLoRA edits the scaling factor inside the direction-side LoRA adapter. They are orthogonal knobs — the combination beats either alone, especially at high rank. Enabling both is two flags: use_dora=True and use_rslora=True in LoraConfig.

PiSSA (Principal Singular values and Singular Vectors Adaptation, arXiv:2404.02948, NeurIPS 2024) is a drop-in INITIALIZATION alternative: instead of LoRA's random-A/zero-B start, it runs a fast SVD on the frozen W₀, splits into principal and residual components, and seeds the adapter from the principal part while freezing the residual as the new base. Faster convergence and stronger NLU quality at the same parameter count; initialization takes seconds. Tradeoff: it reshuffles frozen vs trained, so merge-back must go into the residual base. GaLore (Gradient Low-Rank Projection, arXiv:2403.03507, ICML 2024) is categorically different — NOT an adapter. It trains ALL parameters (full-parameter FT) but projects the gradient into a low-rank subspace before the optimizer sees it, so the dominant memory cost (the optimizer state — Adam's ~56 GB first/second-moment buffers for a 7B) is compressed. GaLore fit a 7B full-param run on a single 24 GB RTX 4090. Use case: 'I want full-FT quality on a memory-limited node' — the bridge when the FT10 decision says full FT but the GPU says no.

The 2026 practical sweet spot is DoRA + rsLoRA at appropriate rank (r=32–64, α≈√r) — the default for new work, configured with two flags. On memory-constrained consumer hardware, QDoRA (DoRA on a 4-bit quantized base, the DoRA analogue of QLoRA) is the leading quality/cost point: full-FT-adjacent quality from a setup that fits a 7B-class model on a single consumer card, with negligible memory overhead on top of QLoRA and the same DoRA-over-LoRA quality lift. When true full-parameter FT is required (large domain shift adapters cannot capture, per FT10), GaLore replaces vanilla full FT. VeRA (~10× fewer params via shared frozen random matrices; niche — storage-bottleneck multi-adapter only), AdaLoRA (adaptive rank allocation; largely superseded by DoRA), and MiSS (claims SOTA; not widely reproduced) are placed on the niche/research axis — do not adopt in production without a controlled A/B on your own held-out eval. The cardinal anti-pattern is chasing novelty without measuring: adopt a method only after a controlled same-base/same-data/same-eval A/B; DoRA earned default status through consistent gains reproduced by independent teams, a single-paper claim does not.