{ "module": "FT13 — The DPO Family and Preferences", "course": "3 — LLM Fine-Tuning Masterclass", "version": "1.0.0", "duration_minutes": 45, "total_questions": 15, "bloom_distribution": { "target": "20% recall / 40% application / 40% analysis-design", "actual": { "recall": 3, "application": 6, "analysis": 6 } }, "passing_score_percent": 70, "questions": [ { "id": "Q01", "bloom": "recall", "type": "multiple_choice", "prompt": "What is DPO, and what did it replace in the alignment pipeline?", "options": [ "A new reward-model architecture that scores preference pairs more accurately than the RLHF reward model.", "A reparameterization of the RLHF objective into a logistic classification loss on preference pairs — no reward model, no RL loop. The model's own log-probabilities (relative to a reference) serve as the implicit reward.", "A reinforcement-learning algorithm that improves on PPO by using a clipped objective with preference pairs instead of scalar rewards.", "A data-augmentation technique that generates synthetic preference pairs from SFT demonstration data." ], "answer_index": 1, "rationale": "DPO (Direct Preference Optimization, Rafailov et al. 2023, arXiv:2305.18290) reparameterizes the RLHF objective: the reward function is expressible in terms of the policy itself, so you can cut out the reward model and the RL loop. What remains is a contrastive logistic loss on preference pairs. The model IS its own reward model — its log-probabilities relative to the frozen reference serve as the implicit reward. It is not a reward model, not an RL algorithm, and not a data technique." }, { "id": "Q02", "bloom": "recall", "type": "multiple_choice", "prompt": "What is the reference model (π_ref) in DPO, and why is it load-bearing?", "options": [ "A separately trained reward model that scores chosen and rejected responses.", "The frozen SFT model you are improving. DPO computes the policy's log-probabilities MINUS the reference's for both responses, anchoring the policy to prevent arbitrary drift. DPO needs a coherent reference distribution — it is not a base-model technique.", "The base pretrained model, used as a fallback when the SFT model overfits.", "A copy of the policy used only for KL-divergence regularization during the RL loop." ], "answer_index": 1, "rationale": "The reference model is the frozen SFT model — the one DPO is improving. The loss contrasts the policy's log-probs against the reference's (log π(chosen) − log π_ref(chosen), etc.), which anchors the policy and bounds drift. This is load-bearing: without a coherent reference, 'preference' is meaningless. DPO on a base (non-SFT) model produces garbage because the base distribution isn't coherent instruct behavior. There is no separate reward model in DPO, and no RL loop." }, { "id": "Q03", "bloom": "recall", "type": "multiple_choice", "prompt": "What is the format of a preference dataset for DPO-family methods (excluding KTO)?", "options": [ "A list of prompts with a single target response (same as SFT).", "Triples of {prompt, chosen, rejected} — two responses to the same prompt, one better and one worse by your definition of better.", "Unpaired binary labels (good/bad) per response, with no shared prompt.", "A reward score (float) per response, learned from human ratings." ], "answer_index": 1, "rationale": "DPO-family methods (except KTO) consume pairs: {prompt, chosen, rejected}. The prompt is shared; chosen and rejected are two responses, one better, one worse. DPO trains the model to assign relatively more probability to the chosen kind. Option A is SFT format. Option C is KTO's data shape (unpaired binary). Option D is reward-model training data, not DPO. KTO is the one method built for unpaired binary feedback." }, { "id": "Q04", "bloom": "application", "type": "multiple_choice", "prompt": "You have an SFT'd model and 5,000 preference pairs. β is set to 0.1. Train reward rises steadily, but after step 400 the eval win rate stalls and the model starts producing oddly sycophantic responses. What is happening and what should you do?", "options": [ "The model is converging; let it run longer to reach the win-rate plateau.", "Over-optimization — β is too low, the policy is drifting too far from the reference and fitting the noise/bias in the data. Fix: raise β (e.g., to 0.3), switch to IPO (which caps the implicit reward growth), or improve the data quality.", "The reference model is corrupted; reload it.", "Switch from DPO to GRPO because the reward is verifiable." ], "answer_index": 1, "rationale": "Rising train reward with stalling/falling eval win rate plus sycophancy is the textbook over-optimization signal — the policy has drifted too far from the reference (β too low) and is fitting the quirks of the preference data. The fixes are: (1) raise β to strengthen the anchor, (2) switch to IPO which adds regularization that caps implicit reward growth, or (3) most often, improve the data (overfitting on bad/noisy data is fundamentally a data problem). Option D is wrong — the reward is subjective preference (win rate), not verifiable, so GRPO does not apply." }, { "id": "Q05", "bloom": "application", "type": "multiple_choice", "prompt": "Your team has collected 10,000 user interactions labeled thumbs-up/thumbs-down, with no pairing between responses. Which method do you use, and why NOT DPO?", "options": [ "DPO — relabel the thumbs-up/thumbs-down as chosen/rejected pairs.", "KTO (Kahneman-Tversky Optimization) — it is the one method built for unpaired binary feedback. DPO requires {prompt, chosen, rejected} pairs where two responses to the same prompt are compared; your data has no pairing.", "SimPO — it is reference-free and handles any feedback type.", "GRPO — verifiable rewards work for binary labels." ], "answer_index": 1, "rationale": "KTO (Ethayarajh et al. 2024) is specifically designed for unpaired binary (good/bad) feedback — it uses a prospect-theoretic loss that does not require pairing. DPO (and SimPO, ORPO, IPO) require {prompt, chosen, rejected} triples where two responses to the same prompt are compared. Your thumbs-up/down data has no pairing, so artificially constructing pairs would inject noise. SimPO is reference-free but still needs pairs. GRPO is for verifiable rewards (checkable answers), not binary preference labels." }, { "id": "Q06", "bloom": "application", "type": "multiple_choice", "prompt": "You want to align a base model to be helpful and harmless, but you want to skip the separate SFT stage to save compute. You have preference pairs. Which method do you choose?", "options": [ "DPO — it can run directly on a base model.", "ORPO — it combines SFT and preference optimization into one training stage with no reference model, saving a full training pass.", "SimPO — it is reference-free so it skips the SFT requirement.", "IPO — the regularization makes base-model training stable." ], "answer_index": 1, "rationale": "ORPO (Hong et al. 2024) is the method that combines SFT and preference into a single stage with no reference model — you skip the separate SFT pass. The tradeoff is less control over SFT quality since it's entangled with the preference loss. DPO (A) and IPO (D) both REQUIRE an SFT'd reference model — running them on a base model is a cardinal anti-pattern. SimPO (C) is reference-free but still assumes an SFT'd starting point; it does not include the SFT loss term." }, { "id": "Q07", "bloom": "application", "type": "multiple_choice", "prompt": "You are starting a new preference-alignment pipeline today. You have an SFT'd model, paired preference data, and a 24GB consumer GPU where VRAM is tight. The teaching doc suggests SimPO as a 'strong modern default.' What concrete advantage does SimPO give you here?", "options": [ "It trains faster because it uses a smaller loss function.", "It removes the reference-model memory overhead (no frozen second copy of the model in VRAM) and adds length normalization so the model can't win by producing shorter/longer responses. Roughly half the memory of DPO.", "It produces higher-quality outputs because it uses a stronger optimizer.", "It does not require a preference dataset, only prompts." ], "answer_index": 1, "rationale": "SimPO (Meng et al. 2024, arXiv:2405.14734) is reference-free — no frozen second copy of the model in VRAM — which roughly halves memory compared to DPO. This matters on a 24GB consumer GPU. It also adds length normalization, preventing the model from gaming the loss by producing shorter/longer responses. In benchmarks it consistently beats DPO. It does NOT skip the preference dataset (option D is wrong — it still needs pairs), and its advantage is memory/normalization, not a different optimizer (C) or smaller loss (A)." }, { "id": "Q08", "bloom": "application", "type": "multiple_choice", "prompt": "Your task is to make a model better at solving competitive programming problems, where solutions can be checked by running them against test cases. Do you use the DPO family or GRPO, and why?", "options": [ "DPO — preference pairs of good/bad solutions will teach the model to write better code.", "GRPO (FT14) — the reward is verifiable (tests pass/fail), so on-policy RL with a verifier can explore and discover correct solutions that offline preference pairs cannot. The DPO family is for subjective preference, not checkable answers.", "SimPO — it is the modern default and beats DPO on all benchmarks including code.", "KTO — binary pass/fail labels are unpaired feedback." ], "answer_index": 1, "rationale": "This is the central line of Pillar 3. Code correctness is a VERIFIABLE reward — a test runner can check it. That means GRPO (on-policy RL with verifiable rewards, FT14) is the right tool: the model generates multiple candidate solutions, the verifier scores them, and the policy explores and discovers correct reasoning paths. The DPO family (including SimPO) is offline and cannot explore — it only learns from fixed pairs. Even with perfect preference pairs, DPO cannot discover solutions outside the dataset. KTO (D) misreads the data shape — pass/fail is a verifier output, not binary preference feedback." }, { "id": "Q09", "bloom": "application", "type": "multiple_choice", "prompt": "You sample 20 pairs from your preference dataset and find you cannot articulate why 'chosen' is better than 'rejected' in most of them — they look nearly identical. What will happen if you run DPO on this data, and what is the fix?", "options": [ "DPO will smooth out the noise and produce a slightly better model.", "DPO will fit the noise — the model degrades because there is no learnable signal. Fix: improve the data so each pair has a clear, articulable difference. The quality bar for DPO is HIGHER than for SFT.", "DPO will ignore the pairs and leave the model unchanged.", "Raise β to make the model learn faster from weak signal." ], "answer_index": 1, "rationale": "Preference data with no real signal is the 'steering without a steering wheel' anti-pattern. DPO will fit the noise (the random labels), which degrades the model — it steers precisely in a meaningless direction. The fix is to improve the data: each pair must have a clear, articulable difference a reasonable person could identify. The quality bar for DPO is HIGHER than for SFT because the signal is subtler (a preference, not a target). Raising β (D) would make it drift LESS, but on noiseless data it still fits noise — the problem is the data, not the hyperparameter." }, { "id": "Q10", "bloom": "analysis", "type": "multiple_choice", "prompt": "A colleague argues: 'DPO is just SFT with extra steps — you train on chosen responses like SFT, and the rejected ones don't matter much.' Why is this wrong, and what does the rejected response actually do in the loss?", "options": [ "They're right — DPO is approximately SFT on chosen responses; rejected is just regularization.", "Wrong. DPO is a CONTRASTIVE loss — it pushes the policy to assign more relative probability to chosen OVER rejected (both relative to the reference). The rejected response is load-bearing: without it, there is no contrast and no preference signal. SFT has no notion of 'better than'; DPO's entire mechanism is the relative ordering.", "Wrong — DPO trains only on rejected responses to teach the model what NOT to say.", "Wrong — the rejected response is only used for the KL penalty, not the loss." ], "answer_index": 1, "rationale": "DPO is fundamentally contrastive. The loss is -log σ(β · [(log π(chosen) − log π_ref(chosen)) − (log π(rejected) − log π_ref(rejected))]). The rejected term is subtracted from the chosen term — the loss is about the DIFFERENCE. Without rejected, there is no contrast and no preference learning; you'd just have SFT on chosen. The rejected response defines what the model should assign LESS probability to, relative to the reference. This relative ordering — 'chosen is better than rejected' — is the entire preference signal. SFT has no such notion." }, { "id": "Q11", "bloom": "analysis", "type": "multiple_choice", "prompt": "Why can pure offline methods (the DPO family) NOT do on-policy exploration, and what does this mean for tasks with verifiable rewards like math?", "options": [ "They can — DPO generates responses during training and learns from them.", "DPO learns only from the fixed pairs you gave it; it cannot try a response, see it fail, and try a different one. For verifiable rewards (math, code), this is a fundamental limitation — on-policy RL (GRPO, FT14) can generate candidates, have a verifier score them, and discover correct reasoning paths the model didn't know it could produce. Offline methods are capped by what's in the dataset.", "Offline methods are actually better for math because preference pairs are cleaner than verifier scores.", "DPO can explore if you set β low enough." ], "answer_index": 1, "rationale": "This is the core limitation of the DPO family. DPO is offline — it trains only on pre-existing pairs, with no generation during training. It cannot explore the solution space. For subjective preference (tone, helpfulness) this is fine — there's no exploration to do. But for verifiable rewards (math correct, tests pass), on-policy RL (GRPO) generates multiple candidates, has a verifier score them, and updates toward the winners — discovering correct reasoning paths that weren't in any dataset. β (D) controls drift from the reference, not exploration capability. Option C inverts the truth: for verifiable rewards, exploration is strictly better." }, { "id": "Q12", "bloom": "analysis", "type": "multiple_choice", "prompt": "A team tries DPO directly on a base model (not SFT'd). The loss decreases but the resulting model produces incoherent, low-quality text. Diagnose the failure and explain why SFT-first is required.", "options": [ "The β was too high; lower it.", "The preference data was low quality; clean it.", "DPO on a base model is a cardinal anti-pattern. A base model's outputs are continuations, not instruct responses — the baseline distribution is incoherent, so 'preference' over it is meaningless. DPO needs the SFT starting point to establish a coherent response distribution to refine. The reference must be an SFT'd model. Fix: SFT first, then DPO.", "The model is too small; use a larger base." ], "answer_index": 2, "rationale": "DPO on a base model is a cardinal anti-pattern because the reference distribution must be coherent instruct behavior. A base model produces text continuations — its output distribution is not the 'response' distribution DPO's contrastive loss assumes. 'Preference' over base-model continuations is meaningless, so the loss decreases but the model degrades into incoherent text. The fix is to SFT first (establishing format and instruction-following), making the SFT model the reference, THEN run DPO. β, data quality, and model size are secondary — the fundamental error is skipping SFT. (ORPO is the exception that bakes SFT into the same loss.)" }, { "id": "Q13", "bloom": "analysis", "type": "multiple_choice", "prompt": "DPO and GRPO are both 'alignment' methods. A student asks: 'If DPO is simpler and more stable, why not use it for everything, including math reasoning?' Give the precise reason GRPO wins for verifiable-reward tasks.", "options": [ "GRPO is always more stable than DPO, so it's preferred for hard tasks.", "For verifiable-reward tasks (math, code), GRPO can explore on-policy: generate multiple candidates, have a verifier score them, and discover correct solutions NOT present in any fixed dataset. DPO is capped by what's in its preference pairs — it cannot discover reasoning paths the dataset doesn't contain. Exploration is strictly better when rewards are checkable.", "GRPO uses a larger model, so it produces better math.", "DPO cannot reduce loss on math problems, only on text." ], "answer_index": 1, "rationale": "The reason GRPO wins for verifiable-reward tasks is on-policy EXPLORATION. The model generates multiple candidate solutions, a verifier (math checker, test runner) scores them, and the policy is updated toward the winners. This lets the model DISCOVER correct reasoning paths it didn't know it could produce — paths that may not be in any offline dataset. DPO is capped by its preference pairs; it can only learn relative preferences between responses someone already generated. For subjective preference (tone, style) there's no exploration benefit and DPO's stability wins. For checkable answers, exploration is strictly better. GRPO is not inherently more stable (A — RL is less stable), doesn't use a larger model (C), and DPO CAN reduce loss on math text (D) — it just can't discover new solutions." }, { "id": "Q14", "bloom": "analysis", "type": "multiple_choice", "prompt": "Your DPO run shows train reward rising and eval win rate rising — both improving, no sign of overfitting. But you notice the model's responses have gotten noticeably longer, and the win rate gains disappear when you control for length. What happened, and which variant would you switch to?", "options": [ "The data is noisy; clean it and re-run DPO.", "Length hacking — the model learned to produce longer responses because they correlate with 'chosen' in your data, not because they're actually better. Switch to SimPO, which adds length normalization so the model can't win by producing longer/shorter responses.", "β is too low; raise it to stop the drift.", "Switch to KTO because the feedback is effectively binary." ], "answer_index": 1, "rationale": "This is length hacking — a known DPO failure mode where the model exploits length correlations in the preference data rather than learning genuine quality. The win-rate gains vanish when you control for length, confirming the model is gaming a surface feature. SimPO (Meng et al. 2024) specifically addresses this with length normalization: the reward is computed per-token (averaged), so the model cannot inflate its score by producing more tokens. This is one of SimPO's two key advantages over DPO (the other being reference-free operation). β (C) wouldn't fix length gaming specifically. KTO (D) is for unpaired data, which this isn't." }, { "id": "Q15", "bloom": "analysis", "type": "multiple_choice", "prompt": "The module says 'DPO steers preference; it does not teach facts.' Connect this to the FT00 thesis ('fine-tuning steers behavior, not knowledge'). If a model gives factually wrong medical answers, will DPO fix it — and what is the correct intervention?", "options": [ "Yes — DPO on pairs of correct/incorrect medical answers will teach the model the facts.", "No — DPO will make the model more confidently wrong or steer it to refuse more eagerly, but it will NOT inject the missing knowledge. The correct intervention for factual gaps is RAG (retrieval) or a different/knowledge-richer base model. DPO steers preference (how the model responds); it does not teach facts (what the model knows). This is the FT00 thesis restated for Layer 3.", "No — DPO only works for code and math, not medical text.", "Yes, but only if β is set very low." ], "answer_index": 1, "rationale": "This is the FT00 thesis ('fine-tuning steers behavior; it does not teach knowledge') applied to the DPO layer. DPO shifts the model's preferences — which response style it favors, whether it refuses, how it phrases things. It does NOT inject factual knowledge. DPO on correct/incorrect medical pairs would make the model prefer the 'correct' phrasing pattern, but it won't reliably fix factual errors because the underlying knowledge distribution is unchanged — the model will appear to improve then fail on out-of-distribution cases. The correct intervention for factual gaps is RAG (retrieve the facts) or a base model with stronger domain coverage. This is exactly why RAG exists: for knowledge, you retrieve; for behavior, you fine-tune." } ] }