# Teaching Script — Module FT10: Full FT vs PEFT: The Decision

**Course**: Course 3 — LLM Fine-Tuning Masterclass
**Module**: FT10 — Full FT vs PEFT: The Decision
**Duration**: ~35 minutes (spoken at ~140 wpm)
**Format**: Verbatim transcript with `[SLIDE N]` cues. Read aloud or use as speaker notes.

---

[SLIDE 1 — Title]

Welcome back. This is module FT ten — Full FT versus PEFT, the Decision. It sits at the top of pillar two, the PEFT pillar, right after eight and nine taught you how LoRA, QLoRA, DoRA, and rsLoRA actually work. This module is different. There is almost no code. This is the judgment module. By the end of it you will know, for any adaptation task, whether to reach for an adapter or for full fine-tuning — and you will be able to defend the call in one sentence.

Here is the headline, and I want you to sit with it because it inverts what most teams do. Default to PEFT for about ninety-five percent of production adaptation. Full fine-tuning is the exception, and the exception has to justify itself.

[SLIDE 2 — The decision rule]

The decision rule, stated cleanly: default to PEFT. Full fine-tuning must justify a roughly thirty-times cost premium and must demonstrate that PEFT provably cannot reach the target.

Read that twice. In most organizations the instinct runs the opposite direction. People say: full fine-tuning is the real thing; LoRA is the cheap compromise. That instinct is wrong, and it is wrong in two ways at once. It is wrong on cost, and — more subtly — it is wrong on what the two methods actually do.

[SLIDE 3 — The cost asymmetry]

The cost half is the easy half. Full fine-tuning a seven-billion-parameter model from a usable state typically needs a multi-GPU node — H-one-hundred-class hardware. A conservative budget for a serious full-FT run on seven-billion is in the tens of thousands of dollars of compute. QLoRA on the same seven-billion runs on a single RTX forty-ninety — a fifteen-hundred-dollar consumer card — and trains in an evening. That is on the order of a thirty-fold difference in hardware cost for the majority of adaptation tasks where the two methods reach equivalent quality.

So any default that starts from the more expensive option has to defend the spend. That alone should flip the default. But the cost is not the interesting part.

[SLIDE 4 — Why PEFT wins the default (the steering thesis applied)]

The interesting part is the structural half, and it comes straight from the course thesis. Recall FT zero-zero. Fine-tuning steers behavior; it does not teach knowledge. If fine-tuning is steering, then the useful change lives in a low-rank subspace. That is the intrinsic-dimension result, and it is the entire reason adapters work at under one percent of parameters.

Here is the reframe that makes this module click. When the task is steering, full fine-tuning is not more powerful. It is over-parameterized. It has more degrees of freedom than the task requires, which means it has more room to drift, more room to forget, more room to fit noise. For a steering task — format, style, instruction-following, preference, refusal removal — a low-rank update is not a compromise. It is the correct representation of the change you are trying to make.

So PEFT is not "almost as good for less money" on these tasks. It is the method whose inductive bias matches the problem. That is why it wins the default for about ninety-five percent of the work you will ever do.

[SLIDE 5 — The three conditions for full FT]

So when is full FT the right call? Only when the task is not pure steering — when the change you need is genuinely higher-rank. That narrows the field to three conditions, and I want you to treat each as rare. Each must be checked, not assumed.

Condition one: genuinely new reasoning pathways. The model needs to learn a new way to think, not merely redirect an existing pattern. Be suspicious here. Most "reasoning" fine-tuning is not this. When you SFT a model to produce chain-of-thought, or GRPO it on math rewards, you are activating reasoning the base already has. That is steering. The genuine case is rarer: the model cannot produce the kind of reasoning required even with a perfect prompt and a long context. Even then the evidence is mixed, but this is the most defensible of the three exits.

[SLIDE 6 — Conditions two and three]

Condition two: extreme domain shift requiring new knowledge. Be very suspicious of this one. The cardinal error from FT zero-zero is treating fine-tuning as knowledge injection. For the overwhelming majority of "the model doesn't know X" problems, the answer is RAG — retrieve X and put it in the context. RAG is cheaper, more reliable, auditable, and updateable without retraining. Reach for it first. Continued pretraining, or full FT on a large domain corpus, is the rare exception. It is defensible when the domain is so specialized and so voluminous that retrieval becomes intractable — a new programming language, a deep professional vocabulary — and when you have the corpus to make it work, which is usually the binding constraint.

Condition three is empirical, not conceptual. Shuttleworth and colleagues, in the twenty-twenty-four paper "An Illusion of Equivalence," found that under certain training regimes — notably larger batch sizes — LoRA degrades faster than full fine-tuning. The mechanism is tied to the structural difference between the two methods, which we will get to. The practical implication: if you are training at large batch sizes and LoRA underperforms, full FT — or a higher-rank PEFT variant like DoRA — is the honest response. This is the narrowest of the three exits, but it is the one most likely to surprise a team that has internalized "LoRA is always enough."

[SLIDE 7 — Structural non-equivalence]

Now the finding that turns this from a cost decision into a design decision. Shuttleworth and colleagues examined the weight matrices produced by LoRA and by full fine-tuning using singular value decomposition. The result: the two methods produce structurally different weight matrices. They are not approximations of each other. LoRA finds a low-rank solution; full FT finds a higher-rank solution. The two can reach similar behavior on a benchmark, but they get there via different geometry.

This matters for two reasons. First, it confirms the steering thesis. If fine-tuning were a single well-defined knowledge-injection operation, LoRA and full FT would converge to similar weights. They do not. Multiple solutions exist — exactly what you expect for steering, and the opposite of what you expect for knowledge injection.

[SLIDE 8 — Why the choice matters]

Second — and this is the decision-relevant part — it means the choice between PEFT and full FT is not "cheap approximation versus expensive ground truth." It is a choice between two structurally distinct solutions with different generalization profiles. LoRA's low-rank solution can be more robust on pure steering tasks: less room to overfit, less catastrophic forgetting. Full FT's higher-rank solution can capture changes LoRA cannot, at the cost of more drift and more forgetting.

The paper also identified a concrete mechanism behind LoRA's degradation in some regimes: "intruder dimensions" — spurious directions introduced during LoRA training that correlate with forgetting. These appear at low ranks and suboptimal hyperparameters. This is why FT nine's guidance exists: avoid very small ranks, use alpha equals two times rank, prefer DoRA. The intruder-dimension finding does not invalidate PEFT. It sharpens how you configure it.

[SLIDE 9 — The decision implication]

So here is the operational form of the decision. Which solution's inductive bias matches your task? Pure steering — format, style, instruction, preference, refusal: the low-rank solution is correct. PEFT. Higher-rank change — new reasoning, genuine domain knowledge: the low-rank solution may be too constrained. Full FT or CPT. Uncertain: start with PEFT, DoRA, reasonable rank. Measure. Only escalate to full FT if PEFT provably fails and you have a hypothesis for why the task is higher-rank. Start cheap, measure, escalate with evidence.

[SLIDE 10 — GaLore, the bridge]

Now the method that collapses the cost-quality dichotomy. GaLore — Gradient Low-Rank Projection, from twenty-twenty-four. The honest tension in the decision is that full FT gives you higher-rank updates, which a minority of tasks need, but at roughly thirty times the memory cost. GaLore is the bridge.

The idea: instead of constraining the weights to low rank — that is LoRA — constrain the optimizer's gradient projection to low rank. The weights themselves are full-parameter and update in full. Only the optimizer state, which is the bulk of the memory cost in full FT, is projected into a low-rank form via periodic SVD. The result: full-parameter learning — you get the higher-rank solution — at a memory footprint approaching LoRA's. The paper demonstrates training a seven-billion model on a single twenty-four-gigabyte consumer GPU with eight-bit Adam. The same class of hardware QLoRA uses.

Where GaLore sits: when you want full-FT quality because you have evidence the task is higher-rank, but you are on a memory-limited node, GaLore is the answer. The escalation ladder becomes PEFT, then DoRA or higher-rank PEFT, then GaLore, then full FT when memory is no constraint. GaLore is the rung that makes "I need full-FT quality" and "I have a forty-ninety" compatible.

[SLIDE 11 — The decision matrix]

Let me put six realistic scenarios on the table so you can feel the decision. Medical chatbot formatting — structure outputs as a differential, cite sources, flag uncertainty. PEFT, DoRA or QLoRA. Pure steering; the model already knows medicine. Adding a new programming language the base has never seen. Full FT or CPT, after confirming RAG is insufficient. Genuine knowledge gap; this is the rare CPT case. Style transfer — match a brand voice. PEFT, LoRA or DoRA. Archetypal steering. Tool-use formatting — reliable JSON function-calling. PEFT, QLoRA. Steering; the model already understands tools. Uncensoring an authorized-use agent. PEFT, LoRA, or abliteration. Steering away from a direction; full FT is overkill and risks more forgetting. Domain QA over a proprietary knowledge base. RAG first; PEFT only for the behavior of citing. Knowledge gap; retrieve the answers, never memorize them.

Notice the shape that recurs. If the base can already do it with a perfect prompt and the right context, it is steering, and PEFT is correct. The two rows that break the pattern are knowledge problems — and note that even there, the first move is RAG, not full FT.

[SLIDE 12 — Anti-patterns]

Three anti-patterns to leave with. First, defaulting to full FT "because it's better." It often is not, and it is thirty times the cost. For pure steering it is over-parameterized and more prone to forgetting. Second, ignoring the structural non-equivalence finding — treating LoRA as "full FT minus some quality." The two are structurally distinct solutions, not points on a quality continuum. A team that ignores this will over-spend on full FT for steering and under-spend on it for the rare tasks that need it. Third, full FT for format and style tasks. The specific, common instance of the first anti-pattern. Format and style are always PEFT. If you take one rule from this module, take that one.

[SLIDE 13 — What you can now do]

You can now state the decision rule and defend it with the cost asymmetry and the structural non-equivalence finding. You can name the three conditions that genuinely push toward full FT. You can place GaLore on the escalation ladder as the bridge. And you can take any adaptation scenario and produce a defensible recommendation in one sentence.

The lab for this module has no code. It is six scenarios and a decision card for each. Do it. The reasoning is the deliverable. Next, module FT eleven: the training loop with TRL. Now that you know which rung of the ladder to start on, we will show you the loop that climbs it.

---

*End of module FT10. Duration: approximately thirty-five minutes at one-hundred-forty words per minute.*