Module FT13 — The DPO Family and Preferences

The DPO Family and Preferences

The module that opens Pillar 3. Trace preference alignment from RLHF/PPO to DPO and its variants (IPO, KTO, ORPO, SimPO), learn the reparameterization that made the reward model disappear, and run DPO on top of your SFT'd model — then measure the win-rate improvement.

minutes

artifacts

sub-sections

In 2023, Rafailov et al. showed that the RLHF reward function can be reparameterized as the policy itself — collapsing a three-model RL pipeline into a single supervised loss on preference pairs. DPO and its family (IPO, KTO, ORPO, SimPO) became the default for general preference alignment, and PPO retired to its specialty: verifiable rewards (FT14). In this module you learn the mechanism (a contrastive logistic loss anchored to a reference — your SFT model), the variant decision tree (when to use DPO vs SimPO vs KTO vs ORPO), and the line between offline preference methods and on-policy RL. The lab runs DPO via TRL's DPOTrainer on top of your FT12 model and measures the win-rate gain.

Key Claims

Load-Bearing Claims

DPO reparameterizes RLHF as a logistic classification loss on preference pairs — no reward model, no RL. Rafailov et al. (arXiv:2305.18290) showed the optimal policy's reward is expressible in terms of the policy itself. The model IS its own reward model: its log-probabilities relative to a frozen reference serve as the implicit reward. A contrastive loss pushes the policy to assign more relative probability to chosen than rejected. Same optimizer stack as SFT; runnable on one GPU.

The reference model is load-bearing and must be your SFT'd model. DPO is not a base-model technique. The contrast is always against the frozen reference (π_ref = your SFT model). Without a coherent reference distribution, 'preference' is meaningless — DPO on a base model produces garbage. SFT-then-DPO is the standard pipeline. β (the drift temperature, typically 0.1–0.5) controls how far the policy can drift from the reference.

The variant decision tree is driven by data shape and the reference-model budget. DPO (paired, baseline) · IPO (when DPO overfits) · KTO (unpaired binary feedback — the one method for thumbs-up/down) · ORPO (combines SFT + preference in one stage, no reference) · SimPO (reference-free, length-normalized; beats DPO in benchmarks — a strong modern default) · R-DPO (regularized, niche). DPO or SimPO covers ~90% of real work.

The DPO family is for subjective preference; GRPO (FT14) wins for verifiable rewards. The central judgment of Pillar 3: if a checker can compute the reward (math correct, tests pass), use on-policy RL (GRPO) which can explore and discover correct solutions. If the reward is a human/aesthetic judgment ('which response is better'), use the DPO family. Pure offline methods cannot do on-policy exploration — that is their fundamental limitation.

After This Module

Trace the trajectory of preference alignment — RLHF/PPO → DPO → DPO variants (IPO, KTO) → reference-free methods (SimPO, ORPO) — and state why PPO largely retired for general alignment while surviving for verifiable rewards (FT14).

Explain the DPO mechanism (Rafailov et al. 2023): a reparameterization of the RLHF objective into a logistic classification loss on preference pairs, with no reward model and no RL, optimized against a reference policy (your SFT'd model).

Apply the DPO variant decision tree: when to use DPO, IPO (overfitting), KTO (unpaired binary feedback), ORPO (one-step SFT+preference), SimPO (reference-free, length-normalized modern default), and R-DPO.

Build a preference dataset {prompt, chosen, rejected} from human annotation, AI feedback (model-as-judge), or synthetic construction, and recognize the anti-pattern of preference data with no real signal.

Run the standard SFT-then-DPO pipeline, and explain why DPO on top of a base (non-SFT'd) model is a cardinal error.

Artifacts

Teaching Document

~3,600 words; 6 sub-sections — the RLHF→DPO→variants trajectory, the DPO mechanism (reparameterization + β + reference model), building a preference dataset, the variant decision tree, DPO vs GRPO (the line between preferences and verifiable rewards), the standard pipeline and anti-patterns

READ

Diagrams

5 Mermaid diagrams — the RLHF→DPO→variants trajectory (with the PPO-survives branch to GRPO), the DPO mechanism (pairs → contrastive loss), the variant decision tree, the SFT→DPO pipeline, DPO family vs GRPO (offline preference vs on-policy verifiable rewards)

READ

Slide Deck

11 reveal.js slides — title, the trajectory, RLHF/PPO replaced, the DPO mechanism, β and the reference, the variant decision tree, building a preference dataset, DPO vs GRPO, the standard pipeline and anti-patterns, the lab, what you can now do

READ

Teaching Script