SFT: The Baseline
Module FT12 · Course 3 — LLM Fine-Tuning Masterclass
60 minutes · The opener of Pillar 3 — the foundation every alignment method builds on.
Prereq: FT11 (The Training Loop with TRL).
Pillar 3 — Alignment & Preferences
SFT — what it is, what it is not
Supervised fine-tuning steers an already-capable base toward a target behavior — format, style, instruction-following, tool-call formatting, refusal calibration — by training on (input, target) pairs. It does not inject knowledge.
What SFT does well (steering)
- Format — strict JSON, citations, schema
- Style / tone — concise, formal, cautious
- Instruction following — multi-step, edge cases
- Tool-call formatting — exact fields the harness parses
- Refusal calibration — refuse the right things
What SFT does NOT do
The cardinal error: "fine-tune on our medical records and it will learn medicine." It won't — it memorizes surface patterns and fails out-of-distribution. For knowledge, use RAG.
The FT00 thesis, restated for Layer 3: if the base could already produce the behavior unreliably, that's SFT. If it couldn't at all, no SFT will fix it.
SFT is stage 1 of the post-training stack
Everything else (DPO, GRPO) builds on top of it.
1. SFT · format · style · instruction-following · the baseline (this module)
↓ builds on top
2. DPO · preferences: better vs worse · Module FT13
↓ builds on top
3. GRPO · reasoning on verifiable rewards · Module FT14 (optional)
Why SFT is first: DPO assumes the model already produces plausible responses in the right format (it just ranks them). GRPO assumes it already produces candidate solutions (it just rewards correct ones). Skip SFT and the later stages have no substrate to refine.
The SFT mix — the steering wheel
A production SFT dataset is a blend. The mix encodes what you want the model to be.
45% general
35% domain
10% tool
10% safety
| Source | Share | Why it's there |
|---|
| General instruction-following | 40–50% | The substrate. Magpie (FT05) / teacher distillation. Keeps it a general assistant. |
| Domain examples | 30–40% | Your use cases. Where the domain lift comes from. |
| Tool-use formatting | 5–10% | Exact tool-call schema (FT07 hygiene). |
| Safety calibration | 5–10% | Refusals + compliance. Balance matters. |
Cardinal data rule: curation beats volume. A 2,000-example set you read and curated beats 50,000 raw outputs you didn't. (LIMA, Magpie, all of Pillar 1.)
Mixture imbalance has a signature
Too much of one source skews the model — predictably.
| Imbalance | What happens |
|---|
| Too much general | Domain lift evaporates — you retrained it to be the generic assistant it already was. |
| Too much domain | Catastrophic forgetting — a specialist that lost general capability. |
| Too much tool-use | Format leaks into everything — tool calls on plain questions. |
| Too much safety | Refusal-happy — declines reasonable requests. |
The right ratio is found empirically, by eval. But the starting ratio — ~half general, a third domain, a sixth tool/safety — consistently produces a model that is both capable and domain-lifted, without forgetting.
The three SFT failure modes
Each with a signature (what you see) and a mitigation (what you do).
1. Catastrophic forgetting
cause: too-narrow data
signature: domain up, general down
fix: mix general data · prefer LoRA
2. Mode collapse
cause: low-diversity data
signature: every output looks the same
fix: diversity filtering (FT06) · mix sources
3. Format leakage
cause: template bugs (FT07) — training schema ≠ harness schema
signature: fluent output the harness can't parse (wrong JSON/tool fields)
fix: use the tokenizer chat template · test the round-trip · hand-craft exact schema
All three are steering failures, not knowledge failures. The loss curve looks fine on all three — only eval catches them.
When to stop at SFT, when to escalate
The rule: SFT is enough when there is a single correct response. Escalate to DPO when you must express better versus worse between multiple plausible responses.
| Your task | Tool | Why |
|---|
| One right answer (correct JSON, right tool call, desired summary) | SFT | (input, target) — no preference to express. |
| Several acceptable, with preference (concise vs verbose, cite vs not) | SFT → DPO | (input, preferred, rejected) — express a direction. |
| Reasoning with checkable answers (math, code) | SFT → GRPO | RL rewards the correct solution. |
Most pipelines: SFT first, then escalate only if a quality gap remains that a later stage could close. Don't build a DPO pipeline you don't need. Don't skip one you do.
The three-axis eval
If you didn't eval, you didn't fine-tune. You hoped.
| Axis | What it detects | If it fails |
|---|
General benchmarks
MMLU, MT-Bench, AlpacaEval | Did the model forget? | Forgetting — add general data, re-run. |
Domain eval
held-out (input, ideal-output) | The domain lift — the point of the fine-tune | No lift — SFT didn't work for its purpose. |
Format compliance
parse-success rate on a sample | Does it emit the right format? | Format leakage (FT07) — fix the template. |
The deliverable of an SFT project: "domain lift was X, general capability changed by Y, format compliance is Z%." Without that triangle, you are shipping on faith.
SIDEBAR — Continued Pretraining (CPT)
The one exception to "steering not teaching." Actually shifts knowledge. De-emphasized.
What it is
Keep pretraining the base (next-token on a large corpus) to shift its knowledge distribution toward a domain. The one Layer 3 technique that teaches, not just steers.
The dilemma
Plasticity-stability: learn the new domain (plasticity) without destroying existing knowledge (stability). Catastrophic forgetting is the failure.
Mitigations (in order)
- Mixture-of-source / replay — domain:general 1:1 to 2:1. Replay anchors existing knowledge. (Same mechanism as the SFT general mix.)
- Low-perplexity token filtering — train on the informative middle. NeurIPS 2025: mitigates forgetting.
- Small LR + warmup/cooldown — large LRs forget.
When CPT vs SFT: CPT for substantial new knowledge (rare). SFT for behavior (almost always). RAG is usually better for knowledge. Most practitioners need domain SFT/LoRA, not CPT.
Anti-patterns
SFT-only on narrow data (forgetting). Pure domain/tool/safety. The model becomes a specialist that forgot general capability. Fix: mix general data; prefer LoRA.
Ignoring mixture ratios. Dumping all data in unbalanced. The dominant source skews the model. Fix: track mix %; eval general alongside domain.
Skipping eval. Shipping without held-out eval, general benchmarks, format checks. You can't detect forgetting, mode collapse, or format leakage. Fix: the three-axis triangle, every time.
Treating SFT as knowledge injection. Fine-tuning on documents to "teach" content. Surface-pattern memorization, OOD failure. Fix: SFT for behavior, RAG for knowledge. The cardinal error.
The lab — build an SFT mix
The opener of Pillar 3. Build, train, and eval a real alignment dataset.
- Construct a 2,000-sample SFT mix — 45% Magpie general + 35% domain + tool/safety
- Domain: medical, legal, or security (your choice)
- Train with TRL SFTTrainer (reusing the FT11 loop), LoRA, held-out eval
- Evaluate on the three axes: general, domain lift, format compliance
- Report the domain lift vs the base — the point of the fine-tune
Stretch goal — provoke forgetting: re-build with 70% domain and watch general capability degrade while domain lift rises. The gap between them is the forgetting, made visible. That imbalance is the lesson.
Consumer GPU (RTX 4090 / 24GB) or Colab. Runnable Python in 07-lab-spec.md.
What you can now do
- State what SFT can and cannot do — steers behavior, does not inject knowledge — and defend it with the thesis.
- Build a high-quality SFT mixture at defensible ratios (general + domain + tool + safety).
- Diagnose the three failure modes — forgetting, mode collapse, format leakage — and name each mitigation.
- Decide when to stop at SFT vs escalate to DPO (single correct response vs better/worse preferences).
- Explain the modern post-training stack (SFT → DPO → GRPO) and place SFT as stage 1, the foundation.
Next: FT13 — The DPO Family and Preferences