Module FT10 — Full FT vs PEFT: The Decision

Full FT vs PEFT: The Decision

The judgment module at the adapter/steer boundary. Default to PEFT for ~95% of production adaptation; full fine-tuning only for genuinely higher-rank tasks — and the structural reason why the choice is a design decision, not a cost decision.

minutes

artifacts

sub-sections

Most teams default to full fine-tuning 'because it's better.' It usually isn't, and it's ~30× the cost. The decision rule: default to PEFT; full FT must justify its premium and demonstrate PEFT provably cannot reach the target. The structural non-equivalence finding (Shuttleworth et al., arXiv:2410.21228) — LoRA and full FT are not approximations of each other — turns this from a cost trade-off into a design decision: which solution's geometry fits your task?

Key Claims

Load-Bearing Claims

Default to PEFT for ~95% of production adaptation. Full FT must justify a ~30× cost premium and demonstrate PEFT provably cannot reach the target. The exception defends itself; the default does not.

Full FT is over-parameterized for steering, not more powerful. If fine-tuning steers behavior (FT00), the useful change is low-rank. For format, style, instruction, preference, refusal — a low-rank update is the correct representation. Full FT's extra degrees of freedom buy drift and forgetting, not quality.

Three conditions genuinely push toward full FT: (1) new reasoning pathways the base cannot produce, (2) extreme domain shift where RAG is structurally insufficient (the rare CPT case), (3) large batch sizes where LoRA degrades faster (Shuttleworth et al.). Each is rare; each must be checked, not assumed.

LoRA and full FT are structurally non-equivalent (arXiv:2410.21228) — they produce different weight matrices, not approximations of each other. LoRA finds a low-rank solution; full FT a higher-rank one. The choice is which geometry fits the task, not how much quality you can afford.

GaLore is the bridge: full-parameter learning (higher-rank solution) at near-LoRA memory, by projecting the optimizer state — not the weights — to low rank. The escalation ladder is PEFT → DoRA → GaLore → full FT, one rung at a time, with evidence.

After This Module

State the decision rule — default to PEFT for ~95% of production adaptation — and justify it with the cost asymmetry (~30× premium) and the structural non-equivalence finding.

Identify the three conditions that genuinely push toward full FT: new reasoning pathways, extreme domain shift (RAG-insufficient), and the large-batch LoRA-degradation regime.

Explain structural non-equivalence (Shuttleworth et al., arXiv:2410.21228): LoRA and full FT are structurally distinct solutions, which is why the choice is a design decision, not a cost decision.

Place GaLore on the escalation ladder as the bridge — full-FT quality at PEFT-class memory — and apply the ladder (PEFT → DoRA → GaLore → full FT) one rung at a time with evidence.

Apply the decision matrix to a realistic adaptation scenario and produce a defensible FT-vs-PEFT-and-which-PEFT recommendation in one sentence.

Artifacts

Teaching Document

~3,200 words; 6 sub-sections — the decision rule & why PEFT wins the default, the three conditions for full FT, structural non-equivalence, GaLore as the bridge, the decision matrix, anti-patterns. The judgment module: less code, more decision architecture.

READ

Diagrams

5 Mermaid diagrams (dark #14141f / #5eead4) — the decision tree, the three full-FT conditions, structural non-equivalence (LoRA low-rank vs full FT higher-rank), the ~30× cost asymmetry, and GaLore as the bridge in the escalation ladder.

READ

Slide Deck

12-slide reveal.js deck (exact head/style template) — title 'Module FT10 — Full FT vs PEFT: The Decision', footer 'Course 3 — LLM Fine-Tuning Masterclass · FT10 · Pillar 2'.

READ

Teaching Script