Module FT14 — GRPO and Verifiable Rewards

GRPO and Verifiable Rewards

The flagship 2025–2026 module. Reasoning models (o1 / R1 / Qwen3-style) are trained via RL on verifiable rewards. GRPO (Group Relative Policy Optimization, DeepSeek) samples N responses per prompt and computes advantages relative to the group mean — eliminating the PPO critic and roughly halving memory. Why RL not DPO for reasoning (verifiable rewards enable on-policy exploration), the GRPO≈RLOO clarification, the GRPO++ landscape (DAPO, Dr. GRPO, GSPO), the DeepSeek-R1 pipeline (R1-Zero → 4-stage R1 → distillation), the Qwen3 4-stage pipeline + thinking budget, and the tooling decision (TRL vs verl vs OpenRLHF).

minutes

artifacts

sub-sections

FT13's DPO learns from fixed preference pairs — offline, never generating. For reasoning, the reward is verifiable (math correctness, code execution, tool-use success), and that enables on-policy exploration DPO structurally cannot do: the model generates its own attempts, a verifier checks them, and the policy learns from the distribution of its own errors. GRPO (DeepSeek, DeepSeekMath arXiv:2402.03300) is the algorithm that became the default — sample N responses per prompt, compute advantages relative to the group mean (not a learned value function), drop the PPO critic, roughly halve the memory. DeepSeek-R1 proved reasoning can EMERGE from RL alone (R1-Zero), fixed the readability with a 4-stage pipeline (cold-start SFT → reasoning RL → rejection-sampling SFT → final RL), and showed SFT-only distillation beats o1-mini. Qwen3 industrialized the recipe (thinking-mode fusion, thinking budgets, 36T pretraining). The lab runs GRPO on a verifiable math reward and watches the reward curve climb.

Key Claims

Load-Bearing Claims

RL — not DPO — is the correct tool for reasoning because reasoning rewards are VERIFIABLE (math correctness, code execution, tool-use success). A deterministic verifier enables on-policy exploration: the model generates N attempts per prompt, the verifier scores each, and the policy learns from the distribution of ITS OWN current errors — including errors no fixed dataset contained. DPO learns from frozen (chosen, rejected) pairs and never generates; it is right for subjective preference, structurally wrong for verifiable skill. Decision rule: verifiable reward → GRPO; preference-only → DPO.

GRPO (Group Relative Policy Optimization, DeepSeek, DeepSeekMath arXiv:2402.03300) eliminates the PPO learned value function (critic) by computing the advantage relative to the GROUP MEAN of N sampled responses: A_i = (r_i − mean(r)) / std(r). The group is a bias-free, lower-variance-than-REINFORCE Monte Carlo estimate of expected reward under the current policy, requiring no extra params/training/memory. Memory drops from ~3 models (policy + critic + reference) to ~2 (policy + reference) → roughly halves memory vs PPO. Used in DeepSeek-R1.

GRPO is essentially RLOO (REINFORCE Leave-One-Out, Kool et al. 2019) with a slightly different advantage computation. Both sample N responses per prompt and use a group/leave-one-out baseline with NO learned critic. RLOO's baseline for response i = mean of the other N−1; GRPO's = mean of all N. A small bias/variance tradeoff, NOT a different family. This is why TRL ships GRPOTrainer and RLOOTrainer side by side and they behave nearly identically — siblings, not strangers.

The GRPO++ landscape (per Cameron Wolfe's survey) patches specific failure modes: DAPO (ByteDance, arXiv:2503.14476) — clip-higher (decouple ± clip for exploration) + dynamic sampling (skip zero-variance groups); Dr. GRPO — removes the per-token length bias; GSPO — sequence-level importance weighting for stable long-CoT credit assignment. These are PATCHES, not different algorithms — production recipes mix and match ('GRPO with clip-higher and dynamic sampling' = GRPO + DAPO patches). OLMo 3 (2025) is the open-data production reference.

DeepSeek-R1 (arXiv:2501.12948 + Nature s41586-025-09422-z): R1-Zero proved reasoning EMERGES from pure RL on a base (no SFT cold start) — self-verify, backtrack, long CoT are SELECTED FOR by the correctness reward (FT00: steering capability the base already has, not teaching). R1-Zero's limitation: poor readability. R1 fixes it with a 4-stage pipeline: cold-start SFT (readable CoT scaffold) → reasoning RL (GRPO) → rejection-sampling SFT (generate, filter by reward, retrain — → FT15) → final RL (alignment). Distillation: 6 dense students via SFT ONLY (no RL) on ~800K curated CoT samples; R1-Distill-Qwen-32B beats OpenAI o1-mini — made CoT distillation standard.

Qwen3 (arXiv:2505.09388, May 2025) industrializes the recipe: 4-stage post-training (long-CoT cold start → reasoning RL → thinking-mode fusion merging thinking + non-thinking into one model → general RL), 36T+ pretraining tokens, and a thinking-budget runtime mechanism for adaptive compute (short budget for easy, long for hard). R1 = the science (emergence, pipeline, distillation); Qwen3 = the engineering (fusion, budgets, scale). Both canonical; cite both.

Tooling: TRL GRPOTrainer for moderate scale (research, 1B–7B, learning, integrates with transformers/peft/accelerate — supply a Python reward function); verl (HybridFlow, ByteDance) for production-scale distributed RL on Ray (separates generation/rollout from training/gradient — the architecture that makes 30B+ tractable); OpenRLHF is faster on some micro-benchmarks BUT a masked_mean bug was found (silently wrong on padded long CoT) — community consensus: 'TRL or verl and you're fine.'

The verifier is the load-bearing component of any reasoning-RL run. A weak verifier (substring match, leniency, a judge rewarding confidence) → reward hacking (model games the shortcut, scores high, outputs wrong). A strong verifier (execute code and check output; parse and compare the math answer exactly; validate tool-output structure) → signal tracks true correctness. Reasoning-RL quality is bounded by verifier quality — the analog of FT00's 'your data matters more than your algorithm.'

After This Module

Explain why RL — not DPO — is the correct tool for reasoning: verifiable rewards (math correctness, code execution, tool-use success) enable on-policy exploration that offline preference datasets structurally cannot provide.

State the GRPO mechanism — sample N responses per prompt, compute advantages relative to the group mean (A_i = (r_i − mean(r))/std(r)) rather than a learned value function — and explain how eliminating the critic from PPO roughly halves the memory footprint.

Articulate the clarification that GRPO is essentially RLOO (REINFORCE Leave-One-Out) with a slightly different advantage computation, and why this matters for reading the literature.

Map the GRPO++ landscape (DAPO, Dr. GRPO, GSPO) to the specific failure modes each variant fixes, and name OLMo 3 as a concrete production recipe.

Recite the DeepSeek-R1 pipeline — R1-Zero (pure RL, no cold start) → R1 (cold-start SFT → reasoning RL → rejection-sampling SFT → final RL) → distillation — and explain why each stage exists.

Recite the Qwen3 4-stage post-training pipeline and the thinking-budget mechanism.

Choose between TRL GRPOTrainer (moderate scale) and verl (production-grade distributed RL on Ray) based on scale, and name OpenRLHF's masked_mean caveat.

Write a verifiable reward function (parse answer, execute/check, return reward) and explain why verifier quality bounds the entire RL run.

Artifacts

Teaching Document

~3,900 words; 7 sub-sections — why RL not DPO for reasoning (verifiable rewards → on-policy exploration), GRPO mechanism (group-relative advantage, no critic, ~half memory), the GRPO≈RLOO clarification, the GRPO++ landscape (DAPO/Dr.GRPO/GSPO + OLMo 3), the DeepSeek-R1 pipeline (R1-Zero → 4-stage R1 → distillation), the Qwen3 4-stage pipeline + thinking budget, tooling (TRL/verl/OpenRLHF) + the reward-verification loop. Anti-patterns, key terms, references.

READ

Diagrams

6 Mermaid diagrams (dark #14141f/#5eead4) — GRPO vs PPO (drop the critic, group baseline); the R1 pipeline (R1-Zero → 4-stage R1 → distillation); the Qwen3 4-stage pipeline + thinking budget; DPO vs GRPO decision (offline preference vs on-policy verifiable); the GRPO++ variant landscape (failure → patch map); the reward-verification loop (generate → verify → advantage, with reward-hacking warning).

READ

Slide Deck

14 reveal.js slides using the exact course dark template — title, DPO-vs-GRPO decision, why DPO is wrong for reasoning, GRPO drop-the-critic, GRPO≈RLOO, GRPO++ landscape, R1 pipeline, Qwen3 pipeline, reward-verification loop, tooling, the verifiable reward function (code), anti-patterns, lab, objectives.

READ

Teaching Script