# Teaching Script — Module FT13: The DPO Family and Preferences

**Course**: Course 3 — LLM Fine-Tuning Masterclass
**Module**: FT13 — The DPO Family and Preferences
**Duration**: ~45 minutes (spoken at ~140 wpm)
**Format**: Verbatim transcript with `[SLIDE N]` cues. Read aloud or use as speaker notes.

---

[SLIDE 1 — Title]

Welcome to module FT thirteen, The DPO Family and Preferences. This is the module that opens pillar three, alignment. Everything in FT twelve — SFT, the baseline — set up the behavior distribution. Now we refine it. We teach the model not just what to say, but which version of what to say is better. That is preference alignment, and the DPO family is how almost everyone does it today.

If you have not done FT twelve, go back. This module assumes you have an SFT'd model. DPO builds on top of SFT — it does not replace it. The reference model in DPO is your SFT model, and we'll come back to why that matters.

[SLIDE 2 — The trajectory]

First, the history, because it explains why the field looks the way it does. In twenty twenty-two, the alignment pipeline was RLHF with PPO. Three stages. SFT a base model. Train a separate reward model on human preference pairs. Then run PPO — proximal policy optimization — where the language model generates, the reward model scores, and an RL loop updates the policy toward higher reward, with a KL penalty keeping it close to the SFT model. That's how InstructGPT and the original ChatGPT were aligned.

It worked. It was also expensive and painful. Three models in memory at once — the policy, the frozen reference, and the reward model. PPO is notoriously finicky. The reward model has its own failure modes — reward hacking, misgeneralization. A hyperparameter mistake produces silent degradation, not a clean crash.

Then, in May twenty twenty-three, Rafailov and colleagues published DPO — Direct Preference Optimization. The insight was mathematical. The optimal RLHF policy and the reward function are linked. You can reparameterize the whole thing — express the reward in terms of the policy itself — and that lets you cut out the reward model and the RL loop entirely. What remains is a simple classification loss on preference pairs, trained with the same optimizer stack as SFT.

DPO collapsed a three-model RL pipeline into a single supervised loss you can run on one GPU. Overnight, RLHF in most teams came to mean DPO or one of its variants. The variants came fast — IPO, KTO, ORPO, SimPO — each fixing a specific DPO failure mode.

The one place PPO survived: verifiable rewards. When a checker can compute the reward — math is correct, tests pass — on-policy RL explores and discovers solutions that offline methods cannot. That is module FT fourteen, GRPO. It is the central judgment of this pillar.

[SLIDE 3 — RLHF/PPO, what DPO replaced]

Let me be concrete about what DPO replaced, because "three models in memory" is abstract until you feel it. Stage one: SFT, same as FT twelve. Stage two: train a separate reward model — a whole other transformer — to predict which response a human would prefer. Stage three: run PPO. The policy generates, the reward model scores, and you do a policy-gradient update with a clipped objective and a KL penalty to the reference. That KL penalty is what keeps the model from drifting into gibberish to chase reward.

Each stage has its own training loop, its own hyperparameters, its own failure modes. The reward model can be hacked — the policy finds inputs that score high but are nonsense. PPO can be unstable — the clip ratio, the value function, the GAE lambda. It works, and the people who made it work deserve enormous credit, but it is not something most teams want to maintain. DPO replaced almost all of it for general alignment.

[SLIDE 4 — The DPO mechanism]

Now the mechanism. Here's the loss, simplified. For each preference pair — prompt, chosen, rejected — DPO computes the model's log probability for both responses. It subtracts what the reference model would have assigned. Then it takes the difference of those two, multiplies by beta, applies a sigmoid, and takes the negative log. That's the loss. Backprop. Optimizer step. Done.

Three things to notice. First, no reward model. The model is the reward model — its own log-probabilities, relative to the reference, serve as the implicit reward. Second, no RL. This is a standard supervised loss. Same training mechanics as SFT. You already know how to run it from FT eleven. Third — and this is load-bearing — the reference model. The contrast is always against pi-ref. That's what anchors the policy. Without it, the model could drift arbitrarily.

[SLIDE 5 — beta and the reference model]

Two things you must understand to use DPO well. Beta, and the reference model.

Beta is the drift temperature. It controls how strongly the loss anchors the policy to the reference. Low beta — point-zero-one — weak anchor. The policy can drift far. Faster alignment, and faster over-optimization, reward hacking, capability degradation. The model enthusiastically matches your preference data, including its flaws. High beta — point-five and up — strong anchor. The policy barely moves. Safe but weak. Typical values: point-one to point-five, with point-one as a common starting point. The right beta is empirical. Too low, you over-optimize. Too high, nothing changes. Raschka's work and Phil Schmid's twenty twenty-five guide both treat beta tuning as the central practical question of DPO.

The reference model is, by construction, your SFT model. DPO is not a base-model technique. The whole pipeline assumes you already have an SFT'd model that produces coherent, formatted responses, and you want to shift its preferences. The reference is that SFT model, frozen at the start of DPO. The policy is a copy of it, trainable. Every gradient step pushes the policy's preferences away from the reference, in the direction your data points.

This is why DPO on a base model is a cardinal anti-pattern. A base model has not been taught to produce coherent instruct responses. Preference over its outputs is meaningless because the baseline distribution is wrong. DPO needs the SFT starting point. SFT then DPO is the standard pipeline. Always.

[SLIDE 6 — The variant decision tree]

Now the family. DPO spawned variants. Each fixes a specific failure mode. You don't need to learn all of them deeply. You need the decision tree.

First question: what shape is your data? If you have preference pairs — prompt, chosen, rejected — you can use DPO, IPO, SimPO, ORPO, R-DPO. If you only have unpaired binary feedback — thumbs up, thumbs down, no pairing — you use KTO. That's the one method built for unpaired data.

Second: do you want a reference model? Reference-free methods — SimPO and ORPO — cut memory and compute roughly in half. No frozen second copy in VRAM. For consumer-GPU work, that matters.

Third: do you already have an SFT model? If yes, use DPO, SimPO, or IPO to add preferences. If you want to skip the SFT stage entirely, use ORPO — it does SFT and preference in one pass, no reference. The tradeoff is less control over the SFT quality.

Fourth: is DPO overfitting? You'll see it as train reward rising while eval degrades, or the model getting sycophantic. Switch to IPO, which caps how much the implicit reward can grow. Or raise beta. Or fix the data.

And if you're starting a new preference pipeline today with no specific reason to pick DPO, SimPO is a defensible first choice. It removes the reference model and adds length normalization, so the model can't win by just producing shorter or longer responses. In the authors' benchmarks, it consistently beat DPO. Many production teams have moved to it.

The honest summary: DPO is the baseline you understand first, because it makes the mechanism clear. SimPO and ORPO are the modern compute-efficient choices. IPO and KTO are specialists. You'll use DPO or SimPO about ninety percent of the time.

[SLIDE 7 — Building a preference dataset]

The dataset is the steering wheel. DPO on bad data steers you into a wall. The format is simple: prompt, chosen, rejected. The prompt is shared. Chosen and rejected are two responses — one better, one worse, by your definition of better.

Three sources. Human annotation — present a prompt and two responses to a human, ask which is better. Expensive, slow, high quality. That's how the original datasets were built. AI feedback — use a strong model as a judge with a rubric. Cheap, fast, scalable. UltraFeedback, RLAIF, Constitutional AI all use this. Quality depends on the judge model. Synthetic — build pairs without a judge. Chosen is a good response, rejected is a degraded version — the base model's lower-quality output, a truncated response, or a response with a specific failure injected. That's what our lab does. Cheap, controllable, excellent for targeting a specific behavior.

The signal test. Sample twenty pairs and ask yourself: could a reasonable person tell which is better, and why? If you can't articulate the difference, your model can't learn from it. Preference data with no real signal is the most expensive version of the "steering without a steering wheel" anti-pattern. The quality bar for DPO is higher than for SFT, because the signal is subtler — a preference, not a target.

[SLIDE 8 — DPO family vs GRPO]

The central judgment of pillar three. Where does the DPO family end, and where does GRPO begin? The line is the reward.

The DPO family optimizes offline preferences — fixed pairs from a dataset. It cannot explore. It cannot try a response, see it fail, and try a different one. It learns only from the pairs you gave it.

GRPO — group relative policy optimization, module FT fourteen — optimizes on-policy with verifiable rewards. The model generates multiple candidates. A verifier scores them — a math checker, a test runner, a tool-call validator. The policy is updated to favor the winners. The model explores.

The rule is simple. If the reward can be computed by a verifier — the answer is checkable — use GRPO. If the reward is a human or aesthetic judgment — this response is better — use the DPO family. For a math problem where you can check the answer, exploring is strictly better. The model discovers correct reasoning paths it didn't know it could produce. For "be more concise and less sycophantic," there's no verifier. You need preferences. DPO is the tool.

Pure offline methods cannot do on-policy exploration. That is their fundamental limitation. Everything in FT thirteen and FT fourteen follows from this line.

[SLIDE 9 — The standard pipeline and anti-patterns]

The canonical pipeline. Start with a base model. SFT it on demonstration data — that's your reference. Then DPO, or SimPO, or ORPO, on top, using a preference dataset, to shift the model's preferences. This is the pipeline almost every aligned instruct model you've used went through. SFT establishes the behavior distribution. DPO refines it.

The anti-patterns. First, DPO on a base model, not SFT'd. No coherent reference distribution. Produces garbage. Always SFT first. Second, over-optimization — beta too low. The policy drifts, fits the noise and bias in your data. You see it as rising train reward but degrading eval, increasing sycophancy, or the model breaking on out-of-distribution inputs. Raise beta, switch to IPO, or fix the data. Third, preference data with no real signal. If chosen and rejected are nearly indistinguishable, DPO fits noise and the model degrades. Fourth, treating DPO as a knowledge tool. DPO steers preference. It does not teach facts. If your model doesn't know something, DPO will not fix it — it'll make the model more confidently wrong, or steer it to refuse more eagerly. Knowledge is RAG or continued pretraining. Preference is DPO. That's the FT zero-zero thesis, restated for this layer.

[SLIDE 10 — The lab: SFT then DPO]

The lab. You take your SFT'd model from FT twelve — or a provided one if you skipped that. You build a five-hundred-pair synthetic preference dataset. Chosen is a good response. Rejected is a degraded version — a base-model refusal, a low-quality output, or a response with a specific failure. You run DPO via TRL's DPOTrainer on top of the SFT model. And you measure the win-rate improvement on a held-out set, using a model judge.

You will see the win-rate move. That is alignment, felt. The model's preferences shifted measurably because of the data you gave it. Consumer-GPU runnable. About forty-five to sixty minutes.

[SLIDE 11 — What you can now do]

You can now trace the trajectory from RLHF to DPO to SimPO. You can explain the DPO mechanism — a reparameterized logistic loss on preference pairs, no reward model, no RL. You can apply the variant decision tree by data shape and reference budget. You can build a preference dataset from human, AI-judge, or synthetic sources. And you can run the SFT-then-DPO pipeline and recognize the anti-patterns.

That is the DPO family. Next, module FT fourteen: GRPO and verifiable rewards. Now we cross the line — from subjective preference to checkable answers, from offline pairs to on-policy exploration. That is where alignment meets reasoning.

---

*End of module FT thirteen. Duration: approximately forty-five minutes at one-hundred-forty words per minute.*