{ "module": "FT12 — SFT: The Baseline", "course": "3 — LLM Fine-Tuning Masterclass", "version": "1.0.0", "duration_minutes": 40, "total_questions": 15, "bloom_distribution": { "target": "20% recall / 40% application / 40% analysis-design", "actual": { "recall": 3, "application": 6, "analysis": 6 } }, "passing_score_percent": 70, "questions": [ { "id": "Q01", "bloom": "recall", "type": "multiple_choice", "prompt": "What is SFT, and what is its role in the post-training stack?", "options": [ "A technique that injects new knowledge into a base model's weights by training on domain documents.", "Supervised fine-tuning on (input, target-output) pairs to steer format, style, instruction-following, tool formatting, and refusal calibration. It is stage 1 of the stack (SFT → DPO → GRPO) and steers behavior without injecting knowledge.", "A preference-learning method that trains on (input, preferred, rejected) triples to refine which of two responses is better.", "A reinforcement-learning method that rewards correct reasoning on problems with checkable answers." ], "answer_index": 1, "rationale": "SFT trains an instruct model on (input, target-output) pairs to steer behavior — format, style, instruction-following, tool-call formatting, refusal calibration. It is stage 1 of the modern post-training stack; DPO (option C) and GRPO (option D) are the later stages that build on top of the SFT'd model. SFT does NOT inject knowledge (option A is the cardinal error). This is the FT00 thesis restated for Layer 3." }, { "id": "Q02", "bloom": "recall", "type": "multiple_choice", "prompt": "What are the typical mixture ratios for a production SFT dataset, and what does each source cover?", "options": [ "100% domain examples; general and tool data are not needed for SFT.", "General instruction-following 40-50%, domain examples 30-40%, tool-use formatting 5-10%, safety calibration 5-10%.", "Tool-use formatting 60%, domain 30%, general 10% — tools are the dominant behavior to steer.", "Safety calibration 50%, general 30%, domain 20% — safety must always be the largest share." ], "answer_index": 1, "rationale": "The defensible starting mix: general 40-50% (Magpie/teacher distillation — the substrate that prevents forgetting), domain 30-40% (your use cases — where domain lift comes from), tool-use 5-10% (exact tool-call schema), safety 5-10% (refusals + compliance, balanced). The mix encodes what you want the model to be. 100% domain (A) causes catastrophic forgetting; tool-dominant (C) causes format leakage into everything; safety-dominant (D) causes refusal-happiness." }, { "id": "Q03", "bloom": "recall", "type": "multiple_choice", "prompt": "What is the modern post-training stack, and why is SFT always stage 1?", "options": [ "GRPO → DPO → SFT; SFT is last because it finalizes the format after reasoning is sharpened.", "SFT → DPO → (GRPO for reasoning). SFT is stage 1 because it gives the later stages a substrate: DPO assumes the model already produces plausible responses in the right format (it just ranks them); GRPO assumes it already produces candidate solutions (it just rewards correct ones).", "DPO → SFT → GRPO; DPO must come first to establish preferences before format.", "Pretraining → SFT → RAG; RAG is stage 3 of the post-training stack." ], "answer_index": 1, "rationale": "The modern stack is SFT → DPO → (GRPO for reasoning). SFT is stage 1 because DPO and GRPO build on top of an SFT'd model: DPO refines preferences assuming the model already produces the right format; GRPO sharpens reasoning assuming the model already produces candidate solutions. Skip SFT and the later stages have nothing to refine. RAG is NOT part of the post-training stack (it is an inference-time retrieval technique)." }, { "id": "Q04", "bloom": "application", "type": "multiple_choice", "prompt": "You are building an SFT dataset for a customer-support assistant. Your goal is reliable JSON output for a ticket-routing tool. Which mixture is most defensible?", "options": [ "100% tool-call examples — maximize tool-use exposure.", "Roughly 45% general instruction-following, 35% support-domain examples, 10% tool-call formatting, 10% safety/clarification — balanced so the model routes correctly without forgetting general capability or becoming refusal-happy.", "80% safety calibration, 20% general — safety is paramount for customer-facing models.", "50% domain, 50% tool-use, 0% general — the model only needs to do support and tools." ], "answer_index": 1, "rationale": "The defensible mix balances all sources: general (substrate, prevents forgetting), domain (the support use case), tool-use (the routing schema), and safety/clarification (balanced refusals). 100% tool-use (A) causes format leakage into everything (tool calls on plain questions). Safety-dominant (C) makes the model refusal-happy. 0% general (D) causes catastrophic forgetting — the model forgets how to be a general assistant. The 45/35/10/10 split keeps the model capable, domain-lifted, and format-compliant." }, { "id": "Q05", "bloom": "application", "type": "multiple_choice", "prompt": "Your task: make the model always cite its sources and structure medical answers as a differential diagnosis. There is one correct way to format each answer. SFT or DPO, and why?", "options": [ "DPO — you need to express preferences between multiple acceptable response styles.", "SFT — there is a single correct response format (cite sources, structure as a differential). SFT trains on (input, target) pairs, which is exactly how you express 'the one right format.' DPO is for when multiple responses are plausible and you have a better/worse ordering.", "GRPO — citation correctness is a verifiable reward.", "Continued pretraining — the model must learn medical knowledge to cite sources." ], "answer_index": 1, "rationale": "The rule: SFT when there is a single correct response; DPO when you must express 'better vs worse' between multiple plausible responses. 'Always cite sources and structure as a differential' is ONE correct format per input — an SFT problem. DPO (A) is overkill and cannot express a single target as cleanly. GRPO (C) is for verifiable-reward reasoning, not format. CPT (D) is the cardinal error — citation is a behavior, not knowledge; the model does not need to LEARN medicine to cite sources in the right format." }, { "id": "Q06", "bloom": "application", "type": "multiple_choice", "prompt": "You want to improve the model's multi-step math reasoning by rewarding correct solutions. After SFT, the model produces candidate solutions but they are often wrong. What is the correct escalation?", "options": [ "More SFT — train on more math examples until the model is always correct.", "GRPO (FT14) — RL with verifiable rewards. The model already produces candidate solutions (SFT gave it the substrate); reward the mathematically correct ones to sharpen reasoning. This is exactly what GRPO is for.", "DPO — label which of two solutions is preferred.", "Continued pretraining on math textbooks to inject the knowledge." ], "answer_index": 1, "rationale": "The escalation path: SFT (format + behavior) → if a reasoning gap remains with checkable answers → GRPO. The model already produces candidate solutions (SFT established the substrate); GRPO rewards the correct ones to sharpen reasoning on verifiable-reward problems like math. More SFT (A) hits diminishing returns on reasoning quality. DPO (C) is for preference, not verifiable correctness. CPT (D) is overkill — the issue is reasoning behavior, not missing knowledge (RAG or the base already has the math)." }, { "id": "Q07", "bloom": "application", "type": "multiple_choice", "prompt": "After SFT, you measure: domain accuracy +15%, MMLU -12%, and the model writes worse Python than the base. What happened, and what is the fix?", "options": [ "The SFT worked — domain lift is the only metric that matters.", "Catastrophic forgetting from too-narrow data. Domain metrics rose but general capability dropped because the training set was too domain-heavy. Fix: raise the general-instruction-following share toward 40-50%, prefer LoRA (low-rank forgets less than full FT), and re-eval to confirm general benchmarks recover while domain lift holds.", "The learning rate was too high; lower it and retrain.", "The model is overfitting; stop training earlier." ], "answer_index": 1, "rationale": "This is catastrophic forgetting (failure mode 1): the model trained on too-narrow data and lost general capability — domain up, general down. The fix is mixture: raise the general source to 40-50% (the substrate that anchors general capability), and prefer LoRA over full FT because low-rank adapters cannot move enough parameters to forget as aggressively. Domain lift alone (A) is not success — the three-axis report must show general capability preserved. LR (C) and early stopping (D) do not fix a mixture problem." }, { "id": "Q08", "bloom": "application", "type": "multiple_choice", "prompt": "Your fine-tuned model produces fluent, confident output, but your harness fails to parse its tool calls — the field names are wrong. The base model produced the correct format. The loss curve was healthy throughout training. What is the cause and fix?", "options": [ "The learning rate was too low; the model did not learn the format. Raise the LR.", "Format leakage (the FT07 failure at the alignment layer). The training examples used a tool-call schema that differs from what the harness parses. Fix: hand-craft tool examples in the EXACT schema the harness expects, use the model's tokenizer chat template, and test the round-trip (apply template, tokenize, decode, compare to inference) before training.", "The model is undertrained; add more epochs.", "Switch from LoRA to full FT for more capacity." ], "answer_index": 1, "rationale": "Format leakage: the chat/tool template in training differs from the inference harness, so the model learned a format that breaks in production. The loss curve looked healthy because the model learned A format, just not THE format. The fix is template hygiene (FT07): hand-craft examples in the exact harness schema, use the tokenizer's chat template (TRL does this via dataset_text_field='messages'), and verify the round-trip before training. LR (A), more epochs (C), and full FT (D) all compound the error — the issue is the data/template, not the optimizer." }, { "id": "Q09", "bloom": "application", "type": "multiple_choice", "prompt": "You need to decide between SFT and continued pretraining (CPT) for a project. Your goal: the model should adopt a concise, formal tone and always emit a specific JSON schema. Which do you choose and why?", "options": [ "CPT — tone and format require shifting the model's knowledge distribution.", "SFT — tone (style) and JSON schema (format/tool) are BEHAVIORS the base can already produce; they are steering targets, not knowledge. CPT is for shifting the knowledge distribution, which is not needed here. SFT on (input, target) pairs is the correct, cheaper tool.", "Both equally — run CPT then SFT in parallel and pick the better one.", "RAG — retrieve the JSON schema at inference instead of training." ], "answer_index": 1, "rationale": "Tone and format are behaviors (steering), not knowledge. The base can already produce concise/formal prose and JSON; SFT makes these reliable. CPT (A) is the wrong tool — it shifts knowledge distribution and is far more expensive; using it for format/tone is massive overkill. RAG (D) is for knowledge retrieval, not behavior; retrieving a schema does not make the model emit it reliably. The decision tree from 12.6: behavior → SFT; knowledge → (RAG usually, CPT rarely)." }, { "id": "Q10", "bloom": "analysis", "type": "multiple_choice", "prompt": "A team argues: 'We'll fine-tune the model on our 50,000 medical records and it will learn medicine.' Why is this wrong, and what should they do instead?", "options": [ "They are right — 50,000 records is enough to teach the model medicine.", "This is the cardinal error (treating SFT as knowledge injection). SFT will appear to work by memorizing surface patterns, then fail out-of-distribution on the first case unlike the training set. For the model to USE the medical knowledge base in a particular way (always cite, flag uncertainty, structure as a differential), use SFT on behavior. For the model to KNOW the records, use RAG (retrieve at inference). CPT only if a genuine, large-scale knowledge gap exists.", "They should use DPO instead of SFT to learn the medicine.", "They should use CPT, which reliably teaches knowledge without any forgetting risk." ], "answer_index": 1, "rationale": "The cardinal error of the field: treating fine-tuning as knowledge injection. SFT does not pour facts into weights — it memorizes surface patterns and fails OOD. The correct split: SFT for BEHAVIOR (how to use the knowledge — cite, structure, flag); RAG for KNOWLEDGE (retrieve the records at inference). CPT (D) is the one exception that shifts knowledge, but it has forgetting risk and is rarely the right answer — and 50k records is not the scale CPT is for. DPO (C) is for preferences, not knowledge." }, { "id": "Q11", "bloom": "analysis", "type": "multiple_choice", "prompt": "Why does LoRA mitigate catastrophic forgetting more than full-parameter fine-tuning on the same narrow data? What does this illustrate about the FT00 thesis?", "options": [ "LoRA uses a lower learning rate, so it updates weights more slowly and forgets less.", "LoRA injects low-rank adapters (under 1% of params) while the base is frozen. Because the adapter is low-rank, it can only move the model within a low-dimensional subspace — it physically cannot shift enough parameters to forget as aggressively as full FT. This illustrates the FT00 thesis: fine-tuning is steering (a low-rank operation), not knowledge injection (which would need full-rank updates).", "LoRA does not update any weights, so it cannot forget anything.", "LoRA and full FT forget equally; the difference is only in training speed." ], "answer_index": 1, "rationale": "LoRA's adapters are low-rank — they can only move the model within a low-dimensional subspace, so they cannot shift enough parameters to forget as aggressively as full FT (which updates every weight). This is a direct consequence of the FT00 thesis: if fine-tuning is steering (and steering is low-rank), then a low-rank adapter captures the useful changes while being structurally constrained from large destructive moves. It is NOT because of LR (A — LoRA uses a HIGHER LR than full FT). LoRA does update adapter weights (C is wrong), and it forgets measurably less than full FT (D is wrong)." }, { "id": "Q12", "bloom": "analysis", "type": "multiple_choice", "prompt": "In continued pretraining (CPT), what is the plasticity-stability dilemma, and why does mixture-of-source/replay training address it?", "options": [ "The dilemma is choosing between a plastic (small) and stable (large) model. Replay training uses both.", "The dilemma: to learn the new domain (plasticity) the model must change weights, but changing weights risks destroying existing knowledge (stability) — catastrophic forgetting. Mixture-of-source/replay training mixes domain tokens with general tokens (1:1 to 2:1) so the general tokens REPLAY what the model knew, anchoring it while the domain tokens shift it. It is the single most effective anti-forgetting technique.", "The dilemma is that plastic models cannot be stable. Replay training picks one.", "The dilemma is about learning rate; replay training uses a constant LR to stay stable." ], "answer_index": 1, "rationale": "The plasticity-stability dilemma: plasticity (learning the new domain) requires weight changes, but changes risk stability (destroying existing knowledge) — forgetting is the failure. Mixture-of-source/replay training mixes domain with general tokens (1:1 to 2:1) so the general tokens replay existing knowledge, anchoring it during adaptation. This is the same mechanism as the SFT general-data mix (the 40-50% general that prevents forgetting in SFT), scaled to pretraining. It is not about model size (A), choosing one property (C), or LR (D)." }, { "id": "Q13", "bloom": "analysis", "type": "multiple_choice", "prompt": "A team trains an SFT model with a healthy loss curve but ships it without eval. In production, users report the model gives overly verbose answers and occasionally refuses reasonable requests. Which failure(s) did skipping eval prevent them from catching, and what should they have measured?", "options": [ "No failure — the loss curve was healthy, so the model is fine.", "Skipping eval prevented them from detecting: (1) style/verbosity drift (mode collapse toward verbose outputs from low-diversity or teacher-style data) and (2) refusal calibration imbalance (too much safety data). They should have measured the three-axis triangle: general benchmarks (forgetting), domain eval (lift), and format/behavior compliance (verbosity, refusal rate on reasonable requests). The loss curve cannot detect any of these — only held-out eval can.", "Only the verbosity — refusal calibration cannot be measured.", "Only catastrophic forgetting — the other issues are subjective." ], "answer_index": 1, "rationale": "The loss curve measures optimization, not behavior. A healthy loss tells you the model fit the data — it does NOT tell you whether the data produced the right behavior (verbosity, refusal rate, format). The three-axis eval (general benchmarks, domain lift, format/behavior compliance including refusal rate on reasonable requests) is the only way to catch these. Skipping eval is the 'flying blind' anti-pattern (FT11/FT12): you cannot detect forgetting, mode collapse, format leakage, or calibration imbalance without held-out eval and behavioral probes. Options C and D wrongly narrow what eval can measure." }, { "id": "Q14", "bloom": "analysis", "type": "multiple_choice", "prompt": "Two teams build SFT models on the same domain. Team A uses 50,000 raw, unfiltered teacher outputs. Team B uses 2,000 hand-curated, deduplicated, decontaminated examples. Team B's model scores higher on domain eval. Why, and what principle does this illustrate?", "options": [ "Team A's model is undertrained; 50,000 examples need more epochs.", "Curation beats volume. Team B's 2,000 curated examples (read, deduped, filtered, decontaminated — the FT06 pipeline) steered better than 50,000 raw outputs because scale-of-CURATION is the lever, not scale-of-data. Noisy data fits noise; clean data steers precisely. This is the LIMA/Magpie/Pillar 1 lesson: a small, curated set outperforms a large, uncurated one for steering.", "Team B got lucky; with enough data Team A would always win.", "Team A's teacher was weaker; switching teachers would fix it." ], "answer_index": 1, "rationale": "The cardinal data rule: curation beats volume. Team B's disciplined 2,000 (dedup, filter, decontaminate from FT06) outperformed 50,000 raw outputs because the model fits whatever it is shown — clean data steers precisely, noisy data fits noise. This is the LIMA finding (~1k curated examples aligned a model), the Magpie finding (filtered 300K beat unfiltered 1M), and the Pillar 1 thesis. It is not undertraining (A), luck (C), or the teacher (D) — it is the data quality, period." }, { "id": "Q15", "bloom": "analysis", "type": "multiple_choice", "prompt": "Why is SFT called 'the baseline' of alignment, and what goes wrong if a team skips SFT and goes straight to DPO on a base model?", "options": [ "SFT is called 'baseline' because it is the simplest method and rarely used in practice.", "SFT is the baseline because every subsequent alignment method builds on the SFT'd model. DPO assumes the model already produces plausible responses in the right format and just ranks them. On a raw base (no SFT), DPO has no substrate to refine — the base's outputs are unformatted and inconsistent, so preference rankings have nothing coherent to operate on. SFT establishes the format/style substrate that DPO refines.", "SFT is the baseline because it must always be the only method used.", "Skipping SFT is fine; DPO works equally well on raw base models." ], "answer_index": 1, "rationale": "SFT is 'the baseline' because it is the foundation: DPO and GRPO build on top of an SFT'd model. DPO refines preferences assuming the model already produces plausible, correctly-formatted responses — it ranks them. On a raw base, outputs are unformatted and inconsistent, so DPO's preferred/rejected rankings have no coherent substrate to operate on; the result is poor. SFT establishes the format, style, and basic behavior that preference and reasoning methods then refine. 'Baseline' = foundation (not optional, not the only method). Skipping SFT for DPO directly (D) is a known failure mode." } ] }