# Teaching Script — Module FT14: GRPO and Verifiable Rewards

**Course**: Course 3 — LLM Fine-Tuning Masterclass
**Module**: FT14 — GRPO and Verifiable Rewards
**Duration**: ~55 minutes (spoken at ~140 wpm; this is the flagship module, longer than most)
**Format**: Verbatim transcript with `[SLIDE N]` cues. Read aloud or use as speaker notes.

---

[SLIDE 1 — Title]

Welcome to module FT fourteen, GRPO and Verifiable Rewards. This is Pillar Four — Reasoning Models — and it is the flagship module of the 2025–2026 course. Everything before this was preparation for the question this module answers: how do you train a reasoning model? The o-one, R-one, Qwen-three-style models that defined the frontier. The answer is RL on verifiable rewards, and the algorithm is GRPO. The prerequisite is FT thirteen. DPO taught you preference optimization — the offline method. This module is the on-policy counterpart: the model generates its own attempts, a verifier checks them, and the policy learns from the distribution of its own errors.

[SLIDE 2 — The decision: DPO vs GRPO]

The most important judgment in this module comes down to one question: can you write a verifier? A deterministic function that returns correct or incorrect for a given response. If you can — math correctness, code execution, tool-use success — you use GRPO, RL on that verifiable reward, and you get on-policy exploration. If you cannot, because the reward is only expressible as a preference — style, helpfulness, tone — you use DPO, the offline family from FT thirteen.

Why does this matter? Because of what on-policy exploration buys you. DPO learns from a fixed dataset of pairs that someone already wrote. The model re-scores them; it never generates a new response during the loss. The curriculum is frozen before training. That is fine for subjective preference. But reasoning rewards are verifiable. A math answer is right or wrong — you can run the check. The moment you have a verifier, the right move is to let the model generate its own attempts, verify them, and learn from its own current behavior. That is structurally more informative than learning from a frozen snapshot of someone else's attempts.

[SLIDE 3 — Why DPO is wrong for reasoning]

The intuition. Imagine teaching a student long division two ways. The DPO way: show them a thousand pre-labeled good and bad worked examples. They learn to imitate the good ones. But you never see their mistakes — only the mistakes in your fixed set. The RL way: give them a problem, let them attempt it N times, check each against the answer key, tell them which were right. They learn from the distribution of their own errors — including errors your fixed dataset never contained. For verifiable skill, the RL way is strictly more informative. The verifier generates an infinite curriculum tailored to this model's current weak spots.

And a precise reading of the emergence claim. DeepSeek R-one-Zero showed reasoning can emerge from RL alone on a base, with no SFT cold start. What emerged was not new knowledge. The base already had the capability from pretraining. What emerged was reliable use of it — self-verification, backtracking, longer chains of thought — because RL with a correctness reward selects for the behaviors that produce correct answers. This is the FT zero-zero thesis — steering, not teaching — in its purest form.

[SLIDE 4 — GRPO: drop the critic]

Now the algorithm. GRPO — Group Relative Policy Optimization, from DeepSeek, introduced in DeepSeekMath, arXiv two-four-zero-two-point-zero-three-three-zero-zero. The key move: drop the learned value function, the critic. Standard PPO has two networks: the policy, and a value function — the critic — that estimates expected reward, used to compute the advantage. The critic is typically the same size as the policy. So PPO for an LLM roughly doubles the memory: policy, critic, plus a reference for the KL. Three models.

GRPO eliminates the critic by computing the advantage relative to a group of sampled responses. The recipe, per prompt: sample N responses from the current policy — say eight or sixteen. Score each with the verifier. Compute the group mean. The advantage of each response is its reward minus the group mean, normalized by the standard deviation. Then a clipped PPO-style update, with a KL anchor. No critic. The group is the baseline. Memory drops from three models to two — roughly halves the memory of PPO.

[SLIDE 5 — The clarification: GRPO ≈ RLOO]

Now the clarification that saves you from reading the literature wrong. GRPO is essentially RLOO — REINFORCE Leave-One-Out — with a slightly different advantage computation. Both sample N responses per prompt and use a group or leave-one-out baseline. They are not fundamentally different algorithms. RLOO, Kool et al twenty nineteen, computes the baseline for response i as the mean of the other N-minus-one. GRPO uses the mean of all N, including i. A small bias-variance tradeoff, not a different family. This is why TRL ships GRPOTrainer and RLOOTrainer side by side, and they behave nearly identically. The mental model: PPO is policy plus a learned critic that estimates the baseline. GRPO and RLOO are policy plus a group-mean baseline computed from the N samples, with no learning. The critic is replaced by Monte Carlo estimation over the group. When you can cheaply sample N responses, the empirical mean is a bias-free, lower-variance estimate of expected reward — and it requires no extra parameters, training, or memory. The tradeoff: N forward passes per prompt instead of one. For reasoning, where the reward is cheap and sample diversity is the point, this trade is overwhelmingly favorable.

[SLIDE 6 — The GRPO++ landscape]

GRPO works, but production recipes patch specific failure modes. Cameron Wolfe's survey is the canonical map. DAPO, from ByteDance, addresses two issues. Clip-higher decouples the PPO clip range for positive versus negative advantages — standard symmetric clipping suppresses exploration once the policy is doing well. Clip-higher lets positive advantages push further. Dynamic sampling filters out prompts where all N samples get the same reward — all-correct or all-wrong. Those contribute zero gradient signal and waste compute. Dr. GRPO removes a length bias. The standard GRPO objective, averaged per-token, biases toward shorter responses — shorter sequences have fewer tokens to dilute the per-token advantage. Dr. GRPO re-weights to remove this, preventing "shorter equals higher reward" as a spurious shortcut. GSPO does sequence-level importance weighting rather than token-level — token-level ratios are noisy on long chain-of-thought, so one weight per full sequence stabilizes credit assignment.

How to read the landscape: the base is GRPO or RLOO. The variants are patches. In production, recipes mix and match — "GRPO with clip-higher and dynamic sampling" means GRPO plus the DAPO patches. OLMo three, twenty twenty-five, from the Allen Institute, is the open-data production anchor — a better citation than the closed frontier labs when someone asks for a reproducible recipe.

[SLIDE 7 — The DeepSeek-R1 pipeline]

Now the canonical recipe. DeepSeek R-one, arXiv two-five-zero-one-point-one-two-nine-four-eight, and in Nature. Every reasoning-RL paper since cites it. R-one-Zero is the experiment that changed the field's mind. DeepSeek took DeepSeek-V-three-Base — a base model, no instruction tuning — and applied RL directly with verifiable rewards. No cold-start SFT. The result: reasoning behavior emerged. Self-verify, backtrack, long chains of thought — selected for by the correctness reward. R-one-Zero's limitation: readability. The emergent chains were messy, mixed-language, hard to parse. Pure RL optimizes for correctness, not format.

The published R-one pipeline is four stages. One: cold-start SFT — high-quality reasoning data gives a readable chain-of-thought scaffold, fixing R-one-Zero's readability. You are not teaching reasoning; you are giving a clean format. Two: reasoning RL — GRPO on verifiable rewards, sharpening the capability. Three: rejection-sampling SFT — generate candidates with the RL model, filter by reward, keep the correct ones, SFT a fresh model on the survivors. This bakes quality in without the instability of continued RL — it is the subject of FT fifteen. Four: final RL for alignment — a broader reward mix, reasoning plus safety, preserving the reasoning while aligning for general use.

And the distillation surprise. DeepSeek distilled R-one into six smaller dense models — one-point-five-B to seventy-B — via SFT only, no RL, on roughly eight hundred thousand curated reasoning samples from R-one. R-one-Distill-Qwen-thirty-two-B beats OpenAI o-one-mini. This made chain-of-thought distillation a standard technique: pay the RL cost once to produce a teacher, then stamp out cheaper students via SFT alone. It works because the students already had the capability; the CoT data steers them to use it — FT zero-zero again.

[SLIDE 8 — The Qwen3 pipeline]

The other canonical reference. Qwen three, arXiv two-five-zero-five-point-zero-nine-three-eight-eight, May twenty twenty-five. It industrializes the R-one recipe. Four stages. One: long-CoT cold start — SFT on long chain-of-thought, setting a prior toward thorough reasoning. Two: reasoning RL — GRPO-style on verifiable rewards, large-scale. Three: thinking mode fusion — merge thinking mode, long CoT, with non-thinking mode, fast direct answers, into one model, controlled by a mode flag. This solves "sometimes you want thirty seconds of reasoning, sometimes a one-line answer" without two models. Four: general RL — broad capability and alignment, preserving reasoning.

Qwen three reports thirty-six trillion-plus pretraining tokens — among the largest openly disclosed. Scale matters: RL amplifies what the base can do. The thinking-budget mechanism is the runtime complement: instruct the model, via a token budget, to spend bounded thinking compute before answering — short for easy, long for hard. Adaptive latency and accuracy. R-one versus Qwen three in one line: R-one is the science — emergence, the pipeline, distillation. Qwen three is the engineering — fusion, budgets, scale. Both canonical; cite both.

[SLIDE 9 — The reward-verification loop]

Now the heart of the method. The loop, per training step. One: generate N responses per prompt from the current policy. Two: verify each — parse the answer, execute the check. Run the code, compare the math answer, validate the tool output. Return a scalar reward. Three: compute advantages relative to the group — reward minus mean, over standard deviation. No learned critic. Four: update the policy toward high-advantage responses, away from low-advantage, with a KL anchor to the reference.

The verifier is the load-bearing component. A weak verifier — substring matching, leniency, a judge that rewards confidence — produces a signal the model will game. That is reward hacking. A strong verifier — execute the code, compare the parsed answer exactly, validate structure — produces a signal that tracks correctness. The quality of your reasoning RL is bounded by the quality of your verifier. This is the analog of FT zero-zero's "your data matters more than your algorithm": here, your reward matters more than your optimizer. A great GRPO config on a hackable reward steers you into a wall. Learn to write verifiers as carefully as you write training data.

[SLIDE 10 — Tooling: TRL, verl, OpenRLHF]

Which trainer do you reach for? Determined by scale. TRL GRPOTrainer, from HuggingFace — integrates with the transformers, peft, accelerate stack. You supply a reward function, plain Python, a dataset of prompts, and a config. The trainer handles group sampling, advantage, the clipped objective, the KL. This is the right starting point for research, small models — one-B to seven-B on a single GPU — and the lab. Its limit: not built for the multi-node, high-throughput generation that frontier-scale RL requires.

verl, also HybridFlow, from ByteDance — the standard for large-scale GRPO and PPO. Built on Ray, it separates generation — rollout — from training — gradient. That architecture makes RL at scale tractable, because generation is bandwidth-bound and parallel, while training is compute-bound and orchestrated. If you are training thirty-B-plus on a cluster, this is the tool.

OpenRLHF — faster than verl on some micro-benchmarks. But a masked-mean bug was found in its advantage and loss computation that silently miscalculated the per-token mean under certain padding conditions — subtly wrong gradients on long sequences with heavy padding, exactly the reasoning-CoT setting. The bug has been addressed, but the consensus crystallized as: TRL or verl and you are fine. Use TRL for prototyping, verl for scale. OpenRLHF only with a specific reason and a check that the masked-mean issue does not affect your padding.

[SLIDE 11 — The verifiable reward function]

Here is the reward function you will write in the lab, for a math task. It takes prompts, completions, and ground-truth answers. For each completion, it parses the last boxed answer — the model is prompted to put its final answer in a boxed block — and compares it exactly to the gold answer. One point for an exact match, zero otherwise. This is the load-bearing component. It parses the answer and checks it exactly — not a substring match, not a judge. For code rewards you would exec the completion and check the output. The strength of your verifier is the ceiling on your RL run. Memorize that.

[SLIDE 12 — Anti-patterns]

The anti-patterns. Using DPO for reasoning — you forfeit on-policy exploration on a verifiable task. Use GRPO with a verifier. No reward verification — a hackable reward, substring or leniency, and the model games it, scoring high, outputting wrong. Use a verifier, not a judge. Skipping the cold-start SFT — the R-one-Zero path can work but produces readability problems. Give the model a CoT scaffold first. Ignoring the readability problem of pure-RL — RL optimizes correctness, not format. Plan the readability layer. Treating GRPO variants as interchangeable — "use GRPO" is underspecified. Mid-training stalls often need a specific patch — clip-higher, length-debias, sequence-level weighting. Know which failure each patch addresses.

[SLIDE 13 — The lab]

The lab is GRPO on a Math Task. Run TRL GRPOTrainer on Qwen two-point-five one-point-five-B Instruct with a verifiable math reward — parse the boxed answer, compare to ground truth exactly. You write the reward function, configure GRPO — the number of samples per prompt, the KL penalty — train, and eval on held-out math. The moment of truth is the reward curve: it climbs, step over step, because on-policy exploration is working. The model generates attempts, the verifier checks them, and the policy moves toward the behaviors that produce correct answers. That climb is the whole method, visible. It is a heavy lab — a single consumer GPU works for the small model; a cloud GPU is the stretch goal. The stretch goals: swap in verl for scale, or write a code-execution reward for a coding task.

[SLIDE 14 — What you can now do]

You can now explain why RL, not DPO, for reasoning — verifiable rewards enable on-policy exploration offline DPO cannot do. You can state the GRPO mechanism — N samples, group-relative advantage, no critic, roughly half the memory — and you know GRPO is essentially RLOO. You can map the GRPO-plus-plus landscape to the failure each patches, and name OLMo three as a production recipe. You can recite the R-one pipeline and the Qwen three pipeline plus the thinking budget. You can choose between TRL and verl, and you know the OpenRLHF caveat. And you can write a verifiable reward function and explain why verifier quality bounds the whole run. Pillar Four continues with FT fifteen — CoT Distillation and Rejection-Sampling FT — where the reasoning capability, once trained via the expensive RL pipeline you just learned, gets propagated to cheaper students via SFT alone. DPO taught you preference. Now you have exploration — the tool that builds reasoning models.

---

*End of module FT fourteen. Duration: approximately fifty-five minutes at one-hundred-forty words per minute.*