Full FT vs PEFT: The Decision
Module FT10 · Course 3 — LLM Fine-Tuning Masterclass
45 minutes · the judgment module — default to PEFT for ~95% of adaptation; full FT only for genuinely higher-rank tasks
Prereqs: FT08 (LoRA/QLoRA), FT09 (DoRA, rsLoRA). This is where the PEFT pillar becomes a decision.
Pillar 2 — Parameter-Efficient Fine-Tuning
The decision rule
Default to PEFT. Full fine-tuning must justify a ~30× cost premium and demonstrate that PEFT provably cannot reach the target.
In most teams the instinct runs the opposite way: "full FT is the real thing; LoRA is the compromise."
That instinct is wrong twice. Wrong on cost — and wrong on what the two methods actually do.
The cost asymmetry
PEFT (QLoRA) — the default
$1,500
Single RTX 4090. 7B trains in an evening.
~1.5% of params. Steering tasks: equivalent quality.
Full FT — the exception
~$50K
Multi-GPU node (H100-class). 7B: tens of $thousands of compute.
100% of params. Justified only for higher-rank tasks.
~30× cost premium. The default-to-full-FT instinct must defend this spend. It usually cannot.
Why PEFT wins the default (the steering thesis, applied)
Fine-tuning steers behavior. The useful change lives in a low-rank subspace (intrinsic dimension, FT00).
When the task is steering, full FT is over-parameterized, not more powerful. More degrees of freedom than the task needs = more drift, more forgetting, more noise-fitting.
For format, style, instruction, preference, refusal removal — a low-rank update is the correct representation, not a compromise.
So PEFT is not "almost as good for less money." It is the method whose inductive bias matches the problem.
The three conditions for full FT
The only honest exits from the PEFT default. Each is rare. Each must be checked, not assumed.
1 · New reasoning pathways — the model needs a new way to think, not to redirect existing patterns (most "reasoning" FT is activation, not installation)
2 · Extreme domain shift — new knowledge the base lacks, where RAG is structurally insufficient (the rare CPT case; check RAG first)
3 · Large batch sizes — LoRA degrades faster than full FT (Biechler et al., arXiv:2410.21228)
None of these is "I want better quality." All three require evidence that PEFT's low-rank solution is the wrong geometry.
Structural non-equivalence — why the choice matters
Biechler et al. (arXiv:2410.21228, "An Illusion of Equivalence") examined LoRA vs full FT weights via SVD:
LoRA and full FT produce structurally different weight matrices. They are not approximations of each other.
| Path | Geometry | Best for |
|---|
| LoRA | Low-rank solution | Steering — correct representation; more robust, less forgetting |
| Full FT | Higher-rank solution | Higher-rank tasks; can capture changes LoRA cannot — at cost of more drift |
This turns the choice from a cost decision into a design decision: which solution's geometry fits the task?
Intruder dimensions (the LoRA-degradation mechanism)
Biechler et al. also found the mechanism behind LoRA's degradation in some regimes:
"Intruder dimensions" — spurious directions introduced during LoRA training (especially at low rank / suboptimal hyperparameters) that correlate with catastrophic forgetting.
This is actionable, not a verdict against PEFT:
- Avoid very small ranks (FT09)
- Use
alpha = 2 × rank
- Prefer DoRA — closes ~half the gap to full FT
The finding sharpens how you configure PEFT. It does not invalidate it.
GaLore — the bridge
The tension
Full FT: higher-rank updates (some tasks need them) at ~30× memory.
PEFT: low-rank updates at low memory, but geometrically constrained.
GaLore (arXiv:2403.03507)
Project the optimizer state (not the weights) to low rank via SVD.
Weights update in full → higher-rank solution at near-LoRA memory.
7B on a single 24GB GPU (8-bit Adam). Same class of hardware QLoRA uses. Full-FT geometry on a PEFT-class node.
GaLore collapses the cost-quality dichotomy: "I need full-FT quality" and "I have a 4090" are no longer exclusive.
The escalation ladder
Start cheap. Measure. Escalate one rung at a time, with evidence.
PEFT (LoRA / QLoRA) · THE DEFAULT · ~95% of adaptation
↑ escalate if quality bar unmet
DoRA / higher-rank PEFT · first escalation rung; closes ~half the gap
↑ if still underperforming
GaLore · THE BRIDGE · full-FT quality, PEFT memory
↑ only when memory is no constraint
Full FT · 100% params, full optimizer state, ~30× memory
Never start at full FT on speculation. "We'll probably need it eventually" burns ~30× cost upfront and, for steering, produces a worse solution.
The decision matrix
| Scenario | Recommendation | Defense (one sentence) |
|---|
| Medical chatbot formatting | PEFT (DoRA/QLoRA) | Steering — model knows medicine; you shape presentation |
| New programming language | Full FT / CPT | Knowledge gap; low-rank can't install semantics (check RAG first) |
| Brand voice / style | PEFT (LoRA/DoRA) | Archetypal steering — redirect surface generation |
| Tool-use JSON formatting | PEFT (QLoRA) | Steering — model knows tools; make format deterministic |
| Uncensoring (authorized agent) | PEFT / abliteration | Low-rank by definition; full FT risks more forgetting |
| Domain QA over a KB | RAG first; PEFT for behavior | Knowledge — retrieve, never memorize |
The shape that recurs: if the base can already do it with a perfect prompt, it's steering → PEFT.
Anti-patterns
Defaulting to full FT "because it's better." It often isn't, and it's 30× the cost. For steering it's over-parameterized and more prone to forgetting.
Ignoring structural non-equivalence. Treating LoRA as "full FT minus some quality." The two are structurally distinct solutions — reasoning about "quality per dollar" inverts the correct decisions.
Full FT for format and style tasks. The specific, common instance of the first anti-pattern. Format and style are always PEFT. If you take one rule from this module, take that one.
What you can now do
- State the decision rule and defend it with the cost asymmetry and the structural non-equivalence finding.
- Name the three conditions that genuinely push toward full FT.
- Place GaLore on the escalation ladder as the bridge.
- Take any adaptation scenario and produce a defensible recommendation in one sentence.
The lab has no code. Six scenarios, one decision card each. The reasoning is the deliverable.
Next: FT11 — The Training Loop with TRL