# Teaching Script — Module FT09: DoRA, rsLoRA, and Modern PEFT

**Course**: Course 3 — LLM Fine-Tuning Masterclass
**Module**: FT09 — DoRA, rsLoRA, and Modern PEFT
**Duration**: ~40 minutes (spoken at ~140 wpm)
**Format**: Verbatim transcript with `[SLIDE N]` cues. Read aloud or use as speaker notes.

---

[SLIDE 1 — Title]

Welcome to module FT nine, DoRA, rsLoRA, and Modern PEFT. This is the second module of Pillar Two, Parameter-Efficient Fine-Tuning. In FT zero-eight you learned LoRA and QLoRA — the trunk of the PEFT tree. In FT nine I am going to show you what grew on that trunk in twenty twenty-four, and which of those branches you should actually be using in twenty twenty-six.

Here is the one-sentence version. Modern PEFT, specifically DoRA plus rsLoRA, closes roughly half the gap between vanilla LoRA and full fine-tuning, at about five to ten percent more VRAM, and zero inference overhead after you merge. That is not a marginal improvement. It is the new default. By the end of this module you will know exactly what changed, why, and which two flags to set.

[SLIDE 2 — The problem DoRA solves]

Let us start with the structural question the DoRA authors asked. Liu and colleagues at NVIDIA, paper at arXiv two-four-zero-two-point-nine-three-five-five-three, ICML twenty-twenty-four oral. The question was: how does a LoRA update actually differ, structurally, from a full fine-tuning update? Not in size — we know LoRA is smaller. In geometry.

They did a weight-decomposition analysis. Every weight matrix can be split into two components: a magnitude component, which is the norm of each column or filter, and a direction component, which is the unit-normalized matrix. Track how each component moves during training, for full FT, for LoRA, and for an ideal. The finding is the foundation of this module.

Full fine-tuning updates magnitude and direction INDEPENDENTLY. Sometimes a filter's direction rotates a lot while its magnitude barely changes. Sometimes the magnitude scales up while the direction holds. The two are decoupled. LoRA, by contrast, updates them PROPORTIONALLY. When the low-rank product B times A shifts a column, both its magnitude and its direction move together, locked in a fixed ratio determined by the rank and the scaling factor. LoRA cannot rotate a direction without also rescaling its magnitude, and vice versa.

That is the structural deficit DoRA names. It is roughly half of what full FT does — the half LoRA cannot express at any rank. No amount of turning the rank up fixes it, because the coupling is structural, not capacity-limited.

[SLIDE 3 — The DoRA fix: decompose]

The fix. DoRA, Weight-Decomposed Low-Rank Adaptation, rewrites the weight update. Take the frozen base weight W-zero. Normalize it column-wise to get the pure direction. Scale it back up by a learned per-column magnitude vector m — that is a tiny vector, one scalar per column. Apply a LoRA-style low-rank update ONLY to the direction. Magnitude is handled by the vector m. Direction is where the LoRA adapter lives.

Now the optimizer can do the two things full FT does and LoRA cannot. It can change the direction of a filter while holding its magnitude fixed. It can change the magnitude of a filter without rotating its direction. The expressive gap closes by roughly half. And it costs almost nothing — the magnitude vector is tiny, the adapter is the same size, and the overhead is on the order of five to ten percent more VRAM.

The load-bearing production property: after training, magnitude and direction fold back into a single merged weight tensor that is shape-identical to the base. A merged DoRA adapter is indistinguishable from the base model at inference. Same forward pass, same memory, same serving path, no special kernel. Zero inference overhead. This is why DoRA can be the default — adopting it requires no changes downstream. The export, the quantizer, the harness all see a normal model.

[SLIDE 4 — Where DoRA wins most]

Two regimes where DoRA's advantage is largest. First, low ranks, r at or below sixteen. At low rank vanilla LoRA is most constrained by the coupling — the proportional lock dominates. DoRA's decoupling buys the most expressive headroom exactly where cheap adapters need it most. This matters enormously for QLoRA-class work on consumer cards, where you cannot afford a high rank.

Second, domain shifts. When the target distribution is far from the pretraining distribution — a new language family, a new modality, a heavy style transfer — full FT updates magnitude and direction aggressively. DoRA can approximate that. LoRA cannot. Empirically DoRA's lead over LoRA grows with the size of the distribution shift.

The corollary: for a vanilla instruction-tuning SFT on a model already close to the target distribution, the DoRA advantage is real but smaller. DoRA is not a uniform ten-percent win on every task. It is a large win where you need it most — low rank and domain shift — and a small win where LoRA was already adequate. That nuance matters when you read the benchmarks.

[SLIDE 5 — rsLoRA: fixing high-rank instability]

A different problem, and a different fix. Counterintuitive fact about LoRA: turning the rank up does not always help. You would expect rank two-fifty-six to dominate rank sixteen — more parameters, more capacity. In practice vanilla LoRA often gets WORSE or stalls at high rank. The rsLoRA paper, arXiv two-three-one-two-point-zero-three-seven-three-two, named and fixed this.

The cause is the scaling factor. Vanilla LoRA scales the update by alpha over r. At rank eight with alpha sixteen, the scale is two-point-zero. At rank sixty-four with the same alpha, the scale collapses to zero-point-two-five. As rank grows the effective update shrinks toward zero — the adapter is numerically present but functionally inert. The optimizer is trying to learn a meaningful update through a zero-point-two-five attenuator, and the gradients flowing back are correspondingly tiny. Higher rank buys capacity in theory and takes it away in practice.

The rsLoRA fix is one character. The scaling becomes alpha over the square root of r. At rank sixty-four with alpha sixteen, the scale is now sixteen over root sixty-four — that is two-point-zero, the same as rank eight under vanilla scaling. The forward pass stays stable as rank climbs. That is why rsLoRA's full name is rank-STABILIZED LoRA. Rule of thumb: if you are going above r equals thirty-two, use rsLoRA. Below that, it does no harm but offers little.

[SLIDE 6 — rsLoRA stacks with DoRA]

The key combinatorial fact of this module. rsLoRA and DoRA operate on DIFFERENT parts of the adapter update. DoRA adds the magnitude-direction decomposition on top of the frozen base. rsLoRA fixes the scaling factor inside the direction-side LoRA adapter. They are orthogonal. You use BOTH. A DoRA decomposition whose direction-side adapter is scaled by alpha over square-root of r instead of alpha over r. Empirically the combination beats either alone, especially at high rank.

These are not competing methods you choose between. They are complementary knobs you turn together. If you have been treating the PEFT menu as a single-choice question — pick DoRA OR rsLoRA OR PiSSA — that framing is wrong. DoRA and rsLoRA stack. PiSSA is an initialization layer you can add on top. The 2026 default is the stack.

[SLIDE 7 — PiSSA: smarter initialization]

PiSSA, arXiv two-four-zero-four-point-zero-two-nine-four-eight, NeurIPS twenty-twenty-four. It attacks a different part of the LoRA recipe: the initialization. Vanilla LoRA initializes A with random Gaussian noise and B with zeros, so the adapter starts as the identity — no update — and the optimizer builds the update from scratch.

PiSSA asks: what if we initialized the adapter from the most important directions already present in the weight matrix? It runs a fast SVD on the frozen W-zero, splits it into principal components — the top singular values and vectors — and residual components, and initializes the adapter from the principal part while freezing the residual as the new base. The LoRA B and A matrices start seeded with the top-r singular vectors of W-zero, scaled by the singular values, instead of zero and random.

The payoff: faster convergence and stronger final quality on NLU benchmarks, at the same parameter count and per-step cost as LoRA. The initialization takes only seconds. The tradeoff is that PiSSA modifies the frozen base — it reshuffles which part is frozen and which is trained — so the merge-back workflow is slightly different. You must merge into the residual base, not the original. That is a one-time accounting cost, not a quality cost.

PiSSA is a drop-in initialization alternative, not a replacement for DoRA or rsLoRA. You can combine them. In practice PiSSA is most often chosen when convergence speed matters — short training budgets — and the task is NLU-heavy. Classification, extraction, natural language inference, where the principal components carry the task signal.

[SLIDE 8 — GaLore: the full-FT bridge]

GaLore sits in a different category. DoRA, rsLoRA, PiSSA are all ADAPTER methods — they freeze the base and train a small set of extra parameters. GaLore does the opposite in one sense and the same in another. arXiv two-four-zero-three-point-zero-three-five-zero-seven, MLSys twenty-twenty-four.

The observation: the dominant memory cost during training is not the weights and not the gradients. It is the optimizer state. Adam keeps two fp32 buffers — first and second moment — per parameter. For a seven-billion-parameter model that is about fifty-six gigabytes of optimizer state alone — the thing that makes full fine-tuning impossible on a twenty-four-gigabyte card. LoRA sidesteps it by training only the adapter.

GaLore's move. Train ALL the parameters — full fine-tuning — but project the gradient into a low-rank subspace before the optimizer sees it. The weights are full-rank and get fully updated. But the optimizer state lives in the low-rank subspace. Periodically the subspace is recomputed via SVD of recent gradient history and rotated. Result: full-parameter learning at a fraction of the optimizer memory. GaLore fit a seven-billion-parameter full-param run on a single twenty-four-gigabyte RTX 4090.

The use case in one sentence: I want full-FT quality on a memory-limited node. If the FT ten decision says you need full-parameter FT — a large domain shift adapters cannot capture — but you cannot afford vanilla full FT's optimizer memory, GaLore is the bridge. It is slower than LoRA per step, but it trains the whole model, which no adapter does. GaLore is NOT competing with DoRA for SFT steering — DoRA wins there. GaLore is what you reach for when the decision says full FT and the GPU says no.

[SLIDE 9 — The standard-vs-niche axis]

The PEFT literature produces a new method every few weeks. Most do not graduate to default status. Place the also-rans honestly.

Standard, the methods you actually use in twenty twenty-six. DoRA — magnitude-direction decomposition, the default. rsLoRA — alpha over square-root of r, the default at high rank. PiSSA — SVD initialization, a strong alternative init. GaLore — the full-FT bridge.

Niche or research. VeRA — shared frozen random matrices plus per-layer scaling vectors, about ten times fewer params than LoRA, but quality is not consistently there. Useful only when adapter STORAGE, not training memory, is the bottleneck. AdaLoRA — adaptive rank allocation, largely superseded by DoRA; the problem it solved is better solved by DoRA's decomposition. MiSS — claims SOTA, not widely reproduced. Treat as research awareness, not a production default.

The axis is not about recency. It is about whether a method earned default status through consistent gains across many tasks and models, reproduced by independent teams, with ecosystem support. DoRA earned it. VeRA, AdaLoRA, MiSS did not. This distinction matters because of the anti-patterns coming up.

[SLIDE 10 — The 2026 sweet spot]

After all the methods, here is what you actually run in twenty twenty-six.

On a real GPU — sixteen to twenty-four gigabytes, bf sixteen, no quantization — DoRA plus rsLoRA at rank thirty-two to sixty-four, alpha set to roughly the square root of r. This is the default. It closes about half the gap to full FT, costs five to ten percent more VRAM than vanilla LoRA, and merges to zero inference overhead.

On a consumer card — eight to twelve gigabytes, quantized — QDoRA. DoRA on top of a four-bit quantized base. The DoRA analogue of QLoRA, and the leading quality-slash-cost point in PEFT. Full-FT-adjacent quality from a setup that fits a seven-billion-class model on a single consumer card.

When you genuinely need full FT per the FT ten decision — GaLore, not vanilla full FT. Same quality target, fraction of the optimizer memory.

The shape of the decision. Start with DoRA plus rsLoRA. If the GPU is too small, drop to QDoRA. If DoRA is measured-insufficient on a held-out eval — not vibes — the next step up is GaLore full FT, NOT vanilla LoRA at higher rank.

The configuration is two flags. use-dora equals true, use-rslora equals true, inside LoraConfig. That is the entire upgrade from the FT zero-eight vanilla LoRA config to the twenty twenty-six default. No new training loop, no new optimizer, no new export pipeline. Two flags.

[SLIDE 11 — The three anti-patterns]

Three things to avoid.

First, chasing novelty without measuring. The PEFT field publishes a new method monthly. Each claims a win on some benchmark. Switching your production stack to every new method is how you spend a quarter and ship nothing. The discipline: only adopt a method after you have measured it against your current default on YOUR held-out eval, on YOUR data, with a controlled A-B. DoRA earned its default status through consistent gains across many tasks and models, reproduced by independent teams. A single-paper claim is not that.

Second, ignoring the rank-stability issue at high rank. This is the most common silent failure of "I just turned the rank up to one-twenty-eight and it didn't help." Vanilla LoRA attenuates the update by one over r. At rank one-twenty-eight with alpha sixteen the scale is zero-point-one-two-five. The adapter is numerically present but functionally inert. The fix is one flag — use-rslora equals true. If you are going above r equals thirty-two and you are not using rsLoRA, you are almost certainly wasting your rank budget and blaming the data.

Third, full fine-tuning where DoRA would suffice. The most expensive anti-pattern. Full FT costs ten to twenty times the memory and compute of DoRA, requires a much bigger GPU or GaLore, produces a non-swappable artifact — no hot-swapping adapters — and on the majority of steering tasks delivers quality within a couple of points of DoRA. The FT ten decision tree exists to prevent this. The default is DoRA plus rsLoRA. Full FT is the exception you justify with a measured eval gap, not the starting point.

[SLIDE 12 — What you can now do]

Let me close on what you can now do.

One. Explain what DoRA fixes about LoRA — the coupling of magnitude and direction — and defend the half-the-gap-at-five-to-ten-percent claim.

Two. State why rsLoRA swaps alpha over r for alpha over square-root of r, why it matters only at high rank, and why it STACKS with DoRA.

Three. Distinguish the PEFT family on a standard-versus-niche axis. DoRA plus rsLoRA as the default. PiSSA as a drop-in init. GaLore as the full-FT bridge. VeRA, AdaLoRA, MiSS as niche.

Four. Configure DoRA plus rsLoRA — two flags — and position QDoRA as the leading quality-slash-cost point.

Five. Avoid the three anti-patterns: chasing novelty without measuring, ignoring rank-stability, and full FT where DoRA suffices.

The lab is a controlled LoRA-versus-DoRA A-B on your own data. Run it. The only basis on which to adopt a new PEFT method is to measure it on your distribution. Next is FT ten, Full FT versus PEFT, the decision — where DoRA sits on the decision tree, and when you escalate to GaLore.