# Module FT12 — SFT: The Baseline **Course**: Course 3 — LLM Fine-Tuning Masterclass **Module**: FT12 — SFT: The Baseline **Duration**: 60 minutes **Level**: Senior Engineer and above **Prerequisites**: FT11 (The Training Loop with TRL) --- ## Learning Objectives After completing this module, you will be able to: 1. State what SFT can and cannot do — *steers* format, style, behavior, and instruction-following; does *not* inject knowledge — and defend the distinction with the FT00 thesis and the LoRA-vs-full-FT evidence. 2. Build a high-quality SFT mixture — general instruction-following (Magpie / teacher distillation from FT05), domain examples, tool-use formatting, and safety calibration — with defensible ratios (40–50% general, 30–40% domain, 10–20% tool/format/safety) and explain how too much of one source skews the model. 3. Diagnose the three SFT failure modes — catastrophic forgetting (too-narrow data), mode collapse (low-diversity data), and format leakage (template bugs from FT07) — and name the mitigation for each (mix general data, diversity filtering, template hygiene). 4. Decide when to stop at SFT versus escalate to preference methods (FT13): SFT suffices when there is a single correct response; escalate to DPO when you must express "better versus worse." 5. Explain the modern post-training stack — SFT → DPO → (GRPO for reasoning) — and place SFT as stage 1, the foundation every subsequent alignment method builds on. --- # 12.1 — What SFT Is, What It Is Not *The baseline of alignment. Everything else (DPO, GRPO) builds on top. Get the baseline right or the rest is built on sand.* ## The sentence > **Supervised fine-tuning steers an already-capable base model toward a target behavior — format, style, instruction-following, tool-call formatting, refusal calibration — by training on (input, target-output) pairs. It does not inject knowledge.** This is the FT00 thesis restated for the steering layer. The base model already saw trillions of tokens during pretraining; it already knows how to answer questions, write code, and structure prose. What it does not reliably do is answer *in your format*, *in your tone*, *following your instructions*, *calling your tools the way you want them called*. SFT closes that gap. It redirects probability mass from "a plausible continuation" to "the continuation you want." That redirection is steering, and it is the foundation of alignment. The corollary that protects every decision downstream: **if the base model, given a perfect prompt, could already produce the target behavior unreliably or in the wrong format — that is an SFT problem. If it could not produce the behavior at all, no amount of SFT will reliably get you there.** You need a different base, continued pretraining (sidebar, 12.6), or — usually the right answer — RAG. RAG is not this course, but for knowledge it is the correct tool. SFT is for behavior. ## What SFT does well | Steering target | What "steered" looks like | | --- | --- | | **Format** | "Respond in strict JSON." "Always cite sources." "Use this schema." | | **Style / tone** | Concise, formal, safety-cautious, persona-consistent | | **Instruction following** | Obey multi-step instructions, handle edge cases, stop at the right point | | **Tool-call formatting** | Emit `` blocks with the exact fields your harness parses | | **Refusal calibration** | Refuse the requests you want refused; comply with the rest (Pillar 5) | These are all behaviors the base is *capable* of. SFT makes them *reliable*. That reliability is what "alignment" means at this layer — the model does the right thing on demand, in your format, under your conditions. ## What SFT does not do It does not pour facts into the weights. The cardinal error — restated because it is the most expensive mistake in the field — is "we'll fine-tune the model on our medical records and it will learn medicine." It will not. It will *appear* to, by memorizing surface patterns, and then fail catastrophically on the first case that differs from the training distribution. If you want the model to *use* your medical knowledge base in a particular way (always cite, always flag uncertainty, always structure as a differential), that is steering — SFT is the right tool. If you want the model to *know* something it does not, retrieve. The single exception — continued pretraining — is the sidebar at the end of this module. ## Why SFT is the foundation of the stack Everything in Layer 3 builds on top of SFT. The modern post-training pipeline is three stages: 1. **SFT** — teach the model the format, the style, and the basic behavior. (This module.) 2. **DPO** (or KTO, or another preference method) — refine the model's *preferences* now that it produces the right *format*. (Module FT13.) 3. **GRPO** (or RL with verifiable rewards) — sharpen *reasoning* on problems with checkable answers. (Module FT14.) SFT is stage 1 because it gives the preference and reasoning stages something to work with. DPO assumes the model already produces plausible responses in the right format — it just ranks them. GRPO assumes the model already produces candidate solutions — it just rewards the correct ones. Skip SFT and the later stages have no substrate to refine. This is why every real alignment pipeline starts here, and why "SFT the baseline" is the most accurate one-line description of this module. --- # 12.2 — Building a High-Quality SFT Mix *The steering wheel (Pillar 1) for the steering layer (Pillar 3). The dataset is what points the model; the algorithm is just how hard you push.* ## The mix A production SFT dataset is not a single source. It is a *blend* of complementary sources, each covering a behavior the model must reliably exhibit. A typical, defensible mix: | Source | Share | Why it is there | | --- | --- | --- | | **General instruction-following** | 40–50% | The substrate. Magpie-synthesized instructions (FT05) or teacher-distilled pairs (GPT-4, Claude, a 70B you control). Keeps the model a general assistant. | | **Domain examples** | 30–40% | Your actual use cases — medical Q&A, legal summaries, code review, security advisories. This is where the *domain lift* comes from. | | **Tool-use formatting** | 5–10% | The exact tool-call format your harness parses. Teaches the model to *emit* calls correctly (FT07 template hygiene). | | **Safety calibration** | 5–10% | Refusals and clarifications for the requests you want handled that way. (Pillar 5 covers this in depth.) | The numbers are a starting point, not a law. The principle behind them is the part to internalize: **the mix encodes what you want the model to be.** Too much of one source and the model skews toward it. ## Why mixture ratios matter Mixtures are a balancing act, and imbalance has a visible signature: - **Too much general data** → the model is a fine general assistant but your domain lift evaporates. You trained the model to be Qwen2.5-Instruct, which it already was. - **Too much domain data** → the model becomes a domain specialist that has forgotten how to be a general assistant. It answers medical questions well and refuses to write a Python function. This is **catastrophic forgetting from too-narrow data** (12.3) — the failure SFT mixture ratios exist to prevent. - **Too much tool-use data** → every response tries to emit a tool call, even when the user asked a plain question. The format leaks into everything. - **Too much safety data** → the model becomes refusal-happy, declining reasonable requests. (Pillar 5.) Balance matters as much for safety as for capability. The right ratio is found empirically, by eval (12.5). But the starting ratio — roughly half general, a third domain, a sixth tool/safety — is the one that consistently produces a model that is both capable *and* domain-lifted, without forgetting. ## Sourcing each component - **General instruction-following:** your FT05 output is the first place to look. Magpie-Air (from an 8B) gives broad diversity cheaply; Magpie-Pro (from a 70B) gives higher response quality. Teacher distillation (GPT-4/Claude/R1 on your prompts) gives you control over the prompt distribution but risks style leakage (FT05) — mix teachers to avoid it. Whatever you pick, run it through the FT06 pipeline (dedup, filter, decontaminate) before it touches the trainer. - **Domain examples:** real, curated `(input, output)` pairs from your domain. If you have logs of (user query, ideal response), those are gold. If you do not, synthesize them with a teacher, but curate aggressively — domain data quality matters more than volume (the LIMA lesson: ~1k clean examples can outperform 50k noisy ones). - **Tool-use formatting:** hand-craft a few hundred examples in the *exact* tool-call schema your harness parses. This is where FT07 template hygiene bites hardest — if your training examples use a slightly different format than your inference harness expects, the model learns the wrong format and your tools break at runtime. - **Safety calibration:** a small, balanced set of (harmful-ish query, refusal/redirect) pairs alongside (reasonable query, compliance). The balance matters — too many refusals and the model becomes useless. ## The cardinal data rule > **Curation beats volume. A 2,000-example set you actually read and curated will steer better than 50,000 raw outputs you did not look at.** This is the lesson of LIMA, Magpie, and the entire Pillar 1 arc, restated for SFT because it is still the most common mistake. People believe scale-of-data is the lever. It is not — scale-of-*curation* is. Read your data, dedup it, filter it, decontaminate it, balance the mix. Then train. --- # 12.3 — SFT Failure Modes *Three ways an SFT job goes wrong, each with a signature and a mitigation. Learn to spot them before you ship.* ## Failure 1 — Catastrophic forgetting (too-narrow data) The model trains on a narrow distribution — pure domain, pure tool-use, pure safety — and loses the general capability the base had. You fine-tuned on medical Q&A and now the model writes a worse Python function than the base did. The knowledge did not leave the weights (FT00: steering, not teaching), but the *behavioral disposition* shifted — the model's probability mass concentrated on the narrow distribution and away from general assistance. **Signature:** domain metrics go up, general benchmarks go down. The gap between them is the forgetting. **Mitigation:** mix general data into the SFT set (the 40–50% general in 12.2). This is the *same* mechanism as the continued-pretraining sidebar's mixture-of-source training (12.6) — replay general data alongside domain data so the model does not collapse onto the narrow distribution. LoRA (FT08) also mitigates forgetting implicitly: the low-rank adapter cannot move enough parameters to forget as much as full FT can. This is one reason LoRA is the default for SFT. ## Failure 2 — Mode collapse (low-diversity data) The training data is low-diversity — every response looks the same (same opening, same structure, same length) — and the model collapses onto that mode. Every output starts with "Great question!" Every summary is three bullets. Every code answer is wrapped in the same preamble. The model is not broken; it is *boring*, and boring is a quality failure. **Signature:** every generation looks the same, regardless of input. Diversity metrics (n-gram diversity, embedding spread) collapse. **Mitigation:** diversity filtering (FT06). Before training, measure the diversity of your set and either drop near-duplicates (semantic dedup) or actively sample to maximize embedding-space spread (diversity sampling). Mix sources — a blend of Magpie + teacher + domain naturally has more diversity than any single source. And do not over-train: mode collapse worsens with epochs past convergence, because the model keeps sharpening on the dominant mode. ## Failure 3 — Format leakage (template bugs) The chat template applied during training does not match the template the inference harness applies, *or* the training examples use a different format than the harness expects. The model learns a format that works in training and breaks in production. This is the FT07 failure, now manifest at the alignment layer. **Signature:** the model produces fluent, confident output that the harness cannot parse — tool calls with the wrong field names, JSON that does not validate, responses that cut off because the EOS token is wrong. The loss curve looks fine (the model learned *a* format); the production behavior is broken. **Mitigation:** template hygiene (FT07). Use the model's own tokenizer chat template (TRL does this for you — `dataset_text_field="messages"`). Test the round-trip: apply the template, tokenize, decode, and confirm the output matches what your inference path produces. For tool calls, hand-craft examples in the *exact* schema the harness parses and verify a few against the harness's parser before training. One template bug poisons the whole run. --- # 12.4 — When to Stop at SFT, When to Escalate *The decision that saves you from building a preference pipeline you did not need — or skipping one you did.* ## The rule > **SFT is enough when there is a single correct response. Escalate to DPO (FT13) when you need to express "better versus worse" between multiple plausible responses.** SFT trains on `(input, target)` pairs — one right answer per input. It is the right tool when the target is unambiguous: the correct tool call, the right JSON, the desired summary. There is no "preference" to express; there is the answer and there is not-the-answer. SFT nails the answer. DPO trains on `(input, preferred, rejected)` triples — two answers per input, with a label saying which is better. It is the right tool when multiple responses are *plausible* but you have a preference: more helpful, less verbose, more cautious, better cited. SFT cannot express that gradient — it can only show the model one target. DPO shows it a *direction*. ## How to tell which side you are on Ask, for your task: **is there one right answer, or several acceptable answers with a quality ordering?** - **One right answer:** "Return the JSON `{"status": "ok"}` for this input." → SFT. - **Several acceptable, with preference:** "Be concise, not verbose. Cite sources when you make a claim. Be helpful, not guarded." → SFT to establish the format, then DPO to refine the preference. - **Reasoning with checkable answers:** "Solve this math problem; the correct solution gets rewarded." → SFT then GRPO (FT14). Most real pipelines do SFT first, then escalate *only if* the SFT model still has a quality gap that preferences could close. Do not build a DPO pipeline you do not need. Do not skip one you do. ## The escalation sequence ``` SFT (format + behavior) → enough? ship. → quality gap preferences could close? → DPO (FT13) → reasoning gap verifiable rewards could close? → GRPO (FT14) ``` SFT is always step 1. Whether you stop there depends on the gap that remains and whether a later stage is the tool for that gap. --- # 12.5 — Evaluating an SFT Run *If you did not eval, you did not fine-tune. You hoped.* The eval obligation from FT11 applies in full: hold out a split, eval every N steps, watch the train-vs-eval gap, ship the lowest-eval-loss checkpoint. For SFT specifically, three evals matter: 1. **General benchmarks** (MMLU, MT-Bench, AlpacaEval) — did the model forget general capability? Compare to the base. If these dropped, you have catastrophic forgetting (12.3); add general data and re-run. 2. **Domain eval** — a held-out set of domain `(input, ideal-output)` pairs, scored by exact match, BLEU/ROUGE, or an LLM judge. This is the *domain lift* — the whole point of the fine-tune. If it did not go up, the SFT did not work for its purpose. 3. **Format compliance** — does the model emit the right format (valid JSON, parseable tool calls, correct citations) at the rate you need? Measure the parse-success rate on a sample. A model that produces great prose in the wrong format is a production failure. The deliverable of an SFT project is the report: "domain lift was X, general capability changed by Y, format compliance is Z%." Without that triangle, you are shipping on faith. ## Harness Connection — Where the SFT'd model lands An SFT'd model that passes the eval triangle is not yet a product. It is a checkpoint. The checkpoint becomes useful the moment it is dropped into **a Pi-derived harness** — the Layer 5 boundary from FT00. Recall the contract: Pi (Course 1, DD-01) was designed so that the underlying model is swappable without harness changes — the *future-proof test*. Your SFT'd model satisfies that contract by construction. The four-tool ReAct loop, the max-iteration guard, the system prompt do not change because you steered the model. What changes is the model's *behavior inside the loop*: the tool calls land in the format FT07 promised, the responses follow the SFT'd distribution, the refusal calibration matches your steering data. Two practical consequences for an SFT project: 1. **Steer toward the harness's format, not the chat template's.** The single most common SPT deployment bug is a model steered on a chat-template format that the harness never sends. FT07 (chat templates) and FT20 (serving) cover this; the rule is that the SFT inference format must equal the harness request format byte-for-byte. If you SFT on `<|im_start|>...<|im_end|>` and the Pi loop sends OpenAI-style `{"role":...}` JSON, format leakage (Failure 3) shows up as malformed tool calls in production. 2. **The harness is where you observe SFT regressions.** A general-benchmark drop (MMLU down 3 points) is interesting; a harness audit log showing the SFT'd model now hallucinates tool arguments 12% of the time is the signal that matters. The Pi/Tau loop is the production eval rig — run the SFT'd model in it before you ship, and read the audit log. FT23 makes this the synthesis. The unsteered base in the unbounded Pi loop is a curiosity. The SFT'd model in the bounded Pi loop is a product. Fine-tuning without the harness is academic; the harness is what makes the steered model useful. --- # 12.6 — SIDEBAR: Continued Pretraining (CPT) *The one exception to "steering not teaching." Actually shifts the model's knowledge distribution. De-emphasized because almost nobody reading this needs it, and RAG is usually better.* ## What CPT is Continued pretraining takes the base model and keeps pretraining it — next-token prediction on a large corpus — to shift its *knowledge distribution* toward a domain. This is the one technique in Layer 3 that genuinely teaches (adapts) rather than steers. Train a general base on a few billion tokens of medical literature and the model's internal representation of medicine shifts. It now "knows" the domain differently than the base did. This is why CPT is the exception: it is the only Layer 3 technique that moves the model on the *knowledge* axis rather than the *behavior* axis. Everything else in this pillar — SFT, DPO, GRPO, abliteration — is behavior. ## The plasticity-stability dilemma CPT has a fundamental tension: **to learn the new domain (plasticity), the model must change its weights; but changing weights risks destroying what it already knew (stability).** This is the plasticity-stability dilemma, and its most visible symptom is **catastrophic forgetting** — the model learns the new domain and forgets the old one. Train too hard on medical text and the model forgets how to code. The mitigations, in order of effectiveness: 1. **Mixture-of-source / replay training.** Do not train on pure domain data. Mix domain tokens with general tokens at roughly 1:1 to 2:1 (domain : general). The general tokens *replay* what the model knew, anchoring it while the domain tokens shift it. This is the single most effective anti-forgetting technique — and it is the same mechanism as the SFT general-data mix (12.3), just at pretraining scale. 2. **Low-perplexity token filtering.** Drop tokens the model already finds high-perplexity (likely garbage) and tokens it finds trivially low-perplexity (already mastered, no signal). Train on the *informative middle* of the perplexity distribution. The NeurIPS 2025 finding formalized this: token-perplexity reduction — focusing CPT on tokens the model can actually learn from — mitigates catastrophic forgetting while improving domain adaptation. This is FT06 perplexity filtering, scaled to pretraining. 3. **Small learning rate with warmup and cooldown.** CPT uses pretraining-scale LRs (much smaller than SFT LRs), with a warmup to stabilize the start and a cooldown to settle the weights at the end. Large LRs move weights too fast and forget; small LRs with careful scheduling adapt without destroying. ## When CPT versus SFT | Question | If yes | | --- | --- | | Do you need the model to *know* a substantial new domain it currently does poorly on? | CPT (then SFT on top for behavior). | | Do you need the model to *behave* differently (format, style, tools)? | SFT. (Almost always the right answer.) | | Could RAG give the model the knowledge at inference time instead? | Use RAG. For most practitioners, RAG is better than CPT for knowledge. | ## Why this is de-emphasized CPT is expensive (billions of tokens, pretraining-scale compute), finicky (forgetting, LR schedules, mixture ratios), and — for most real use cases — unnecessary. The situations where you genuinely need to shift the model's knowledge distribution, rather than steer its behavior or retrieve knowledge at inference, are rare. When in doubt: **domain SFT or LoRA is enough; RAG is the knowledge tool.** Reach for CPT only when you have a large domain corpus, a clear knowledge gap SFT cannot close, and the budget to do it right. --- ## Anti-Patterns ### SFT-only on narrow data (forgetting) Training on pure domain or pure tool-use data. The model becomes a specialist that forgot how to be a general assistant. Mitigation: mix general data (the 40–50% in 12.2); prefer LoRA, which forgets less than full FT. ### Ignoring mixture ratios Dumping all available data in without balancing sources. The dominant source skews the model. Mitigation: track the mix percentages explicitly; eval general capability alongside domain lift. ### Skipping eval Shipping an SFT model without held-out eval, general benchmarks, or format-compliance checks. You cannot detect forgetting, mode collapse, or format leakage without eval. Mitigation: the eval triangle from 12.5, every time. ### Treating SFT as knowledge injection Fine-tuning on documents expecting the model to *learn* the content. It memorizes surface patterns and fails out-of-distribution. Mitigation: SFT for behavior, RAG for knowledge, CPT (rarely) for genuine adaptation. The cardinal error of the field. --- ## Key Terms | Term | Definition | | --- | --- | | **SFT (Supervised Fine-Tuning)** | Training an instruct model on (input, target-output) pairs to steer format, style, instruction-following, tool formatting, and refusal calibration. The baseline of alignment. | | **SFT mix** | The blended dataset of general instruction-following + domain + tool-use + safety sources. Typical ratios: 40–50% general, 30–40% domain, 10–20% tool/safety. | | **Catastrophic forgetting** | Loss of general capability from training on too-narrow data. Mitigated by mixing general data and by LoRA (low-rank updates forget less). | | **Mode collapse** | All outputs converging to a single style/structure from low-diversity training data. Mitigated by diversity filtering (FT06) and source mixing. | | **Format leakage** | The model learning a chat/tool-call format that does not match the inference harness. The FT07 failure at the alignment layer. | | **The post-training stack** | SFT → DPO → (GRPO). SFT is stage 1; preference and reasoning methods build on top of the SFT'd model. | | **Domain lift** | The improvement on domain-specific eval after SFT — the point of the fine-tune. Measured against a held-out domain set. | | **Continued pretraining (CPT)** | The one Layer 3 technique that shifts knowledge distribution, not just behavior. The exception to "steering not teaching." De-emphasized. | | **Plasticity-stability dilemma** | The CPT tension: learn the new domain (plasticity) without destroying existing knowledge (stability). Catastrophic forgetting is the failure mode. | | **Mixture-of-source / replay training** | Mixing domain tokens with general tokens (1:1 to 2:1) during CPT to anchor existing knowledge while adapting. The primary anti-forgetting technique. | --- ## Lab Exercise See `07-lab-spec.md`. The "Build an SFT Mix" lab: construct a 2,000-sample SFT dataset blending Magpie-synthesized general instructions (from FT05) with a domain subset (medical, legal, or security — your choice), train with TRL (building on FT11), eval, and report the domain-lift versus the base. Consumer-GPU. --- ## References 1. **Module FT00** — *The Steering Stack*. The thesis this module extends: fine-tuning steers behavior; it does not teach knowledge. 2. **Module FT05** — *Synthetic Data*. Magpie self-synthesis and teacher distillation — the sources of the general instruction-following data in the SFT mix. 3. **Module FT06** — *Dedup, Filter, Decontaminate*. The curation pipeline every SFT data source must pass through; diversity filtering against mode collapse; perplexity filtering (the CPT connection). 4. **Module FT07** — *Tokenizers & Chat Templates*. Template hygiene — the root of format leakage. 5. **Module FT11** — *The Training Loop with TRL*. The mechanics (SFTTrainer, loss curves, eval) this module's lab builds on. 6. **Zhou et al. (2023)** — *LIMA: Less Is More for Alignment*. arXiv:2305.11206. ~1k curated examples can align a model. The curation-beats-volume evidence for SFT data. 7. **Xu et al. (2024)** — *Magpie: Alignment Data from Nothing*. arXiv:2406.08464, ICLR 2025. Self-synthesized instruction-following data — the modern default for the general portion of the SFT mix. 8. **NeurIPS 2025** — *Token-perplexity reduction for mitigating catastrophic forgetting in continued pretraining*. The CPT perplexity-filtering finding cited in the sidebar. 9. **Module FT13** — *The DPO Family and Preferences*. The escalation target when SFT is not enough.