# Module FT14 — GRPO and Verifiable Rewards
**Course**: Course 3 — LLM Fine-Tuning Masterclass
**Module**: FT14 — GRPO and Verifiable Rewards
**Duration**: 90 minutes
**Level**: Senior Engineer and above
**Prerequisites**: FT13 — The DPO Family and Preferences
---
## Learning Objectives
After completing this module, you will be able to:
1. Explain **why RL — not DPO — is the correct tool for reasoning**: verifiable rewards (math correctness, code execution, tool-use success) enable on-policy exploration that offline preference datasets structurally cannot provide. DPO learns from fixed pairs; GRPO generates, verifies, and learns from its own attempts.
2. State the GRPO (Group Relative Policy Optimization, DeepSeek) mechanism — sample N responses per prompt, compute advantages relative to the *group mean* rather than a learned value function — and explain how eliminating the critic from PPO roughly halves the memory footprint.
3. Articulate the clarification that GRPO is essentially RLOO (REINFORCE Leave-One-Out) with a slightly different advantage computation, and why this matters for how you read the literature.
4. Map the GRPO++ landscape (DAPO, Dr. GRPO, GSPO) to the specific failure modes each variant fixes, and name OLMo 3 as a concrete production recipe.
5. Recite the DeepSeek-R1 pipeline — R1-Zero (pure RL, no cold start) → R1 (cold-start SFT → reasoning RL → rejection-sampling SFT → final RL) → distillation — and explain why each stage exists.
6. Recite the Qwen3 4-stage post-training pipeline and the thinking-budget mechanism.
7. Choose between TRL `GRPOTrainer` (moderate scale) and verl (production-grade distributed RL on Ray) based on your scale, and name OpenRLHF's caveat.
---
# 14.1 — Why RL, Not DPO, for Reasoning
*The single most important judgment in this module. Get it wrong and you reach for an offline method on a problem that demands exploration.*
## The structural difference
FT13 covered the DPO family. DPO takes a *preference dataset* — pairs of (chosen, rejected) responses — and trains the model to assign higher likelihood to the chosen one. The dataset is fixed before training begins. The model never generates a new response during the DPO loss; it only re-scores the ones a human or a judge already wrote.
That is exactly right for preference alignment ("this answer is more helpful than that one"), because preferences are subjective and a fixed dataset of judgments is the signal you have.
It is exactly wrong for reasoning, for one reason: **reasoning rewards are verifiable.** A math answer is correct or it is not — you can execute the check. A program compiles and passes tests or it does not — you can run it. A tool call retrieved the right document or it did not. You do not need a human to judge these; you need a *verifier*. And the moment you have a verifier, the right move is to let the model *generate its own attempts, verify them, and learn from the distribution of its own successes and failures*.
That is on-policy exploration. DPO cannot do it. DPO learns from attempts a human (or a stronger model) already made. GRPO generates N fresh attempts per prompt, scores them with the verifier, and shapes the policy toward the ones that passed. The model is learning from *its own* current behavior, not from a frozen snapshot of someone else's.
## The intuition that makes it click
Imagine teaching a student long division two ways:
- **DPO way:** show them 1,000 worked examples where someone else divided correctly and incorrectly, labeled "good" and "bad." They learn to imitate the good ones. But you never see *their* mistakes — only the mistakes in your fixed set.
- **RL way:** give them a problem, let them attempt it N times, check each attempt against the answer key, and tell them which attempts were right. They learn from the *distribution of their own errors* — including errors your fixed dataset never contained.
For subjective taste (writing style), the DPO way is fine. For verifiable skill (math, code, tool use), the RL way is strictly more informative, because the verifier lets you cheaply generate an infinite curriculum of labeled attempts tailored to *this model's* current weak spots. The frontier of reasoning quality in 2025–2026 is built on this fact.
> **The decision rule:** if your reward is verifiable (you can write a function that returns correct/incorrect), use RL on that reward. If your reward is only expressible as a preference (a human or a judge must compare two outputs), use DPO. Reasoning is the canonical verifiable domain; hence RL.
## The "emergence" claim, stated precisely
DeepSeek-R1-Zero showed that reasoning behavior can *emerge* from RL alone on a base model — no SFT cold start. This is the headline finding that made the field pivot to RL in early 2025. But read it precisely: what emerged was not new *knowledge*. The base model (DeepSeek-V3-Base) already had the capability from pretraining. What emerged was reliable *use* of that capability — self-verification, backtracking, longer chains of thought — because RL with a correctness reward *selects for* the behaviors that produce correct answers. This is the FT00 thesis ("steering, not teaching") in its purest form: RL is steering the model to use reasoning pathways it already has.
---
# 14.2 — GRPO: Group Relative Policy Optimization
*The algorithm that became the default for reasoning RL. Introduced in DeepSeekMath, used in DeepSeek-R1.*
## The mechanism
GRPO (Shao et al., DeepSeekMath, arXiv:2402.03300) is a simplification of PPO for the LLM setting. The key move: **drop the learned value function (the critic).**
Standard PPO has two networks: the policy (the model you are training) and a value function (the critic) that estimates the expected reward from each state, used to compute the advantage (how much better an action was than expected). The critic is typically the same size as the policy. So PPO for an LLM roughly *doubles* the memory: one copy for the policy, one for the critic, plus a reference policy for the KL penalty. Three models in memory.
GRPO eliminates the critic by computing the advantage **relative to a group of sampled responses**, not relative to a learned expectation. The recipe, per prompt:
1. Sample **N** responses from the current policy (e.g., N=8 or 16).
2. Score each with the reward function (verifier).
3. Compute the group mean reward.
4. The advantage of each response is its reward minus the group mean: `A_i = (r_i − mean(r)) / std(r)` (normalized).
5. Update the policy via a clipped PPO-style objective, with a KL penalty to the reference policy to stay anchored.
No critic. The group *is* the baseline. Memory drops from three models (policy + critic + reference) to two (policy + reference), often further reducible because the reference can be the frozen pre-RL weights and the KL can be computed cheaply.
## The clarification: GRPO ≈ RLOO
Here is the clarification that saves you from reading the literature wrong: **GRPO is essentially RLOO (REINFORCE Leave-One-Out) with a slightly different advantage computation.** Both sample N responses per prompt and use a leave-one-out / group-mean baseline. They are not fundamentally different algorithms. The REINFORCE-Leave-One-Out estimator (Kool et al., 2019) computes the baseline as the mean of the *other* N−1 responses for each response i; GRPO uses the mean of all N (including i). The difference is a small bias/variance tradeoff, not a different family. When you see "GRPO" and "RLOO" discussed as alternatives, understand they are siblings, not strangers — both are variance-reduced REINFORCE with a group baseline and no critic. This is why TRL ships `GRPOTrainer` and `RLOOTrainer` side by side and they behave nearly identically in practice.
> **Mental model:** PPO = policy + learned critic (estimates baseline). GRPO/RLOO = policy + group-mean baseline (computed from the N samples, no learning). The critic is replaced by Monte Carlo estimation over the group.
## Why the group baseline works
The advantage's job is to tell the policy "this response was better or worse than expected." A learned critic estimates "expected" from state features. But when you can cheaply sample N responses, the *empirical mean of those N* is a lower-variance, bias-free estimate of the expected reward under the current policy — and it requires no extra parameters, no extra training, no extra memory. The tradeoff: you pay N forward passes per prompt instead of one. For reasoning, where the reward is cheap to compute (run the verifier) and the sample diversity is the whole point, this trade is overwhelmingly favorable.
## The on-policy / off-policy spectrum (and why production GRPO is neither pure)
> **Harness recall — RI-002 nuance:** this module so far describes GRPO as cleanly "on-policy" — the model generates rollouts, scores them, and updates from its own current behavior. That is the textbook picture, and it is how R1-Zero is usually summarized. **Production GRPO is not purely on-policy. It is 1–2 steps off-policy in practice, and that fact is load-bearing for how you reason about the training system.** The pure on-policy ideal would mean: sample N responses from policy θ, compute advantages, take a gradient step to θ′, then *throw away those samples and resample from θ′*. Real systems do not do that. The moment you reuse rollouts across multiple gradient steps (to amortize the cost of generation), or run multiple gradient micro-batches per generation batch, or interleave generation and training across decoupled workers (14.7), the samples that inform the θ′ update were drawn from θ — one or two policy versions stale. That staleness is the definition of *off-policy*.
**The spectrum:**
| Regime | Rollouts are from | Used for how many updates | On/off-policy |
| --- | --- | --- | --- |
| Pure on-policy | The current policy θ | 1 — discarded after | Strictly on-policy |
| **Standard production GRPO** | **Policy θ, held for the step** | **1 step, but micro-batched** | **~1 step off-policy (negligible)** |
| Rollout-reuse GRPO | Policy θ | 2–4 gradient steps | Mildly off-policy — needs importance correction |
| Async / decoupled (14.7) | A generation worker pinned at θ_g | Until the trainer at θ_t pulls them | Off-policy by (θ_t − θ_g) versions |
The reason the nuance matters is not pedantic. The whole training-system architecture in 14.7 (async RL, disaggregated prefill/decode, weight-drift measurement) exists *because* production GRPO tolerates being 1–2 steps off-policy. If it did not, none of those systems would work — you could never decouple generation from training, never reuse a rollout, never pipeline the two phases. The field's move to "1–2 steps off-policy is fine" is precisely what unlocked the throughput gains of 2025–2026. Two operational consequences:
1. **Importance sampling correction.** The further off-policy you go, the more the PPO-style importance ratio `π(θ_t)/π(θ_g)` matters. Standard GRPO clips it (that is part of what the clip in the objective does). At 1–2 steps the ratio is near 1 and the clip rarely fires; at 4+ steps (rollout reuse, async) it starts to bite and you may need to recompute or reweight. GSPO's sequence-level importance weighting (14.3) is, in part, a response to this.
2. **The phrase "on-policy" in the literature is a spectrum endpoint, not a description.** When a paper says "GRPO is on-policy," it means "on-policy relative to PPO with a replay buffer," not "every gradient step uses fresh samples from the exact current weights." Read it as the ideal the algorithm is *approximating*, and then look at the trainer config (rollout batch, mini-batches per generation, number of reuse steps) to see where the run actually sits on the spectrum.
The earlier framing in this module — "GRPO generates, verifies, and learns from its own attempts" — remains correct and is the right mental model. Just do not mistake it for a claim that the rollouts and the gradient are computed against byte-identical weights. They are not, in any production system, and the small gap is what the async training system in 14.7 is designed to manage.
---
# 14.3 — The GRPO++ Landscape
*GRPO works, but production recipes patch specific failure modes. The variant names matter because the 2025–2026 literature uses them constantly. Cameron Wolfe's survey is the canonical map.*
## The failure modes and their fixes
**DAPO (ByteDance, arXiv:2503.14476)** addresses two issues. *Clip-higher* decouples the PPO clip range for positive vs. negative advantages — standard symmetric clipping suppresses exploration once the policy is doing well, because good responses can only be upweighted up to the clip. Clip-higher lets positive advantages push further. *Dynamic sampling* filters out prompts where all N samples get the same reward (all-correct or all-wrong) — those contribute zero gradient signal (the advantage is zero for every sample) and waste compute. Skip them, spend the budget on prompts with reward variance.
**Dr. GRPO** removes a length bias. The standard GRPO objective, when averaged per-token, introduces a bias toward shorter responses (shorter sequences have fewer tokens to dilute the per-token advantage). Dr. GRPO re-weights to remove this, preventing the model from learning "shorter = higher reward" as a spurious shortcut.
**GSPO (Group Sequence Policy Optimization)** does sequence-level importance weighting rather than token-level. The argument: token-level importance ratios in GRPO/PPO are noisy and can mis-credit long sequences. GSPO computes one importance weight per full sequence, which stabilizes training on long chain-of-thought outputs where the reasoning spans hundreds of tokens.
## OLMo 3 as a concrete recipe
The Allen Institute's OLMo 3 (2025) is a useful production anchor: it publishes a recipe that combines GRPO-family RL with a verifiable-reward curriculum, demonstrating the pattern at open-data scale. When someone asks "what does a real, reproducible reasoning-RL pipeline look like end to end?", OLMo 3 is a better citation than the closed frontier labs. Read it as the open counterpoint to DeepSeek-R1 and Qwen3.
> **How to read the landscape:** the base algorithm is GRPO/RLOO. The variants are *patches*. DAPO patches exploration (clip-higher) and sample efficiency (dynamic sampling). Dr. GRPO patches length bias. GSPO patches credit assignment. In practice, production recipes mix and match — "GRPO with clip-higher and dynamic sampling" is a common description, and that means "GRPO + the DAPO patches." The name GRPO++ is the umbrella for this patched family.
---
# 14.4 — The DeepSeek-R1 Pipeline
*The canonical recipe. Published as R1 (arXiv:2501.12948) and in Nature (s41586-025-09422-z). Every reasoning-RL paper since cites it.*
## R1-Zero: the proof that RL alone surfaces reasoning
R1-Zero is the experiment that changed the field's mind. DeepSeek took DeepSeek-V3-Base — a base model, no instruction tuning, no SFT — and applied RL directly with verifiable rewards (math, code, plus a formatting reward to encourage a readable chain-of-thought structure). No cold-start SFT.
The result: reasoning behavior *emerged*. The model began to self-verify, backtrack, and produce longer, more careful chains of thought — not because it was taught to, but because those behaviors were *selected for* by the correctness reward. Behaviors that lead to correct answers get upweighted; the policy drifts toward them. This is pure steering of capability the base already had (FT00), but the steering is strong enough that the *behavioral phenotype* looks like new capability.
R1-Zero's limitation: readability and format. The emergent chains were often messy, mixed languages, and hard to parse. Pure RL optimizes for correctness, not for the human-facing format. This is why R1 (the full pipeline) adds the SFT stages.
## R1: the multi-stage recipe
The published DeepSeek-R1 pipeline is four stages:
1. **Cold-start SFT.** A small amount of high-quality reasoning data (long CoT examples, curated) is used to SFT the base. This gives the model a readable chain-of-thought *format* to build on — solving R1-Zero's readability problem before RL starts. You are not teaching reasoning here; you are giving the model a clean scaffold.
2. **Reasoning RL.** GRPO on verifiable reasoning rewards. This is where the reasoning capability is sharpened. The model explores, the verifier rewards correct answers, the policy moves toward behaviors that produce them.
3. **Rejection-sampling SFT.** Take the RL-trained model, generate many candidate responses across a diverse prompt set, *filter by the reward* (keep the correct ones), and SFT a fresh model on this curated set. This is the critical bridge: RL produced high-quality reasoning traces; rejection sampling distills them into a clean SFT dataset; SFT on that set bakes the quality in *without* the instability of continued RL. (This is the technique FT15 covers in depth — CoT distillation and rejection-sampling FT.)
4. **Final RL for alignment.** A second RL stage, this time with a broader reward mix (reasoning correctness + safety/helpfulness) to align the model for general use. The reasoning capability from stage 2/3 is preserved; alignment is layered on top.
The published Nature paper frames this as the recipe that produced a frontier-class reasoning model from an open-weights base.
## Distillation: the surprise
The R1 paper's most practically influential finding: **distillation works better than expected.** DeepSeek took the R1 model and distilled it into 6 smaller dense models (1.5B to 70B, Qwen and Llama bases) via **SFT only** — no RL — on roughly 800K curated reasoning samples generated by R1. The resulting DeepSeek-R1-Distill-Qwen-32B *beats OpenAI o1-mini* on reasoning benchmarks.
This single result made chain-of-thought distillation a standard technique. The implication: once you have one strong reasoning teacher (trained via the expensive RL pipeline), you can stamp out cheaper reasoning students via SFT alone. That is the entire subject of FT15. The reason it works, again, is the FT00 thesis: the student bases already had the capability; the distilled CoT data steers them to use it. No new knowledge is transferred — only the behavioral pattern of careful, verified reasoning.
---
# 14.5 — The Qwen3 Pipeline
*The other canonical 2025 reference. Qwen3 technical report (arXiv:2505.09388, May 2025). Reads as a refinement and industrialization of the R1 recipe.*
## The 4-stage post-training
Qwen3's post-training is four stages:
1. **Long-CoT cold start.** SFT on long chain-of-thought data (curated, high-quality). Like R1's cold-start SFT, this establishes a readable reasoning scaffold. Qwen3 emphasizes the *long* aspect — the cold-start data features extended reasoning traces, setting the model's prior toward thorough CoT.
2. **Reasoning RL.** GRPO-style RL on verifiable rewards. This sharpens reasoning, exactly as in R1 stage 2. Scale here is large — Qwen3 reports substantial RL compute.
3. **Thinking mode fusion.** A Qwen3-specific step: merge the "thinking" mode (long CoT) with a "non-thinking" mode (fast, direct answers) into a single model. The model learns to do both, controlled by a mode flag. This is the practical answer to "sometimes you want the model to reason for 30 seconds, sometimes you want a one-line answer" — you should not need two models.
4. **General RL.** A broad RL stage for capability and alignment across the full task distribution, preserving the reasoning from stages 2–3 while improving general helpfulness.
## Pretraining scale and the thinking budget
Qwen3 reports 36T+ pretraining tokens — among the largest openly-disclosed pretraining budgets. The scale matters because reasoning RL *amplifies* what the base can do; a richer base gives RL more to work with. The thinking-budget mechanism is the deployment-side complement: the model can be instructed (via a token budget) to spend a bounded amount of "thinking" compute before answering, trading latency for accuracy adaptively. Easy questions get short budgets; hard questions get long budgets. This is the runtime analog of the training-time exploration: spend compute where it pays off.
> **R1 vs Qwen3, in one line:** R1 established that RL-on-verifiable-rewards surfaces reasoning and that distillation propagates it; Qwen3 industrialized the recipe (thinking/non-thinking fusion, thinking budgets, massive pretraining) into a single controllable model. Both are canonical; cite both.
---
# 14.6 — Tooling: TRL, verl, OpenRLHF
*Which trainer do you actually reach for? The answer is determined by scale.*
## TRL GRPOTrainer — moderate scale, the default starting point
HuggingFace TRL ships `GRPOTrainer` (and the sibling `RLOOTrainer`). It integrates with the standard transformers/peft/accelerate stack you already use for SFT and DPO. You supply a reward function (plain Python: takes a list of generated strings, returns a list of floats), a dataset of prompts, and a config. The trainer handles the group sampling, advantage computation, clipped objective, and KL penalty.
This is the right starting point for: research prototypes, small models (1B–7B on a single GPU or small cluster), and the lab in this module. It is what you use to *learn* GRPO and to validate a reward function before scaling. Its limitation: it is not built for the multi-node, high-throughput generation that frontier-scale RL requires. When your generation cost dominates and you need thousands of parallel rollouts, you outgrow TRL.
## verl (HybridFlow, ByteDance) — production-grade distributed RL
verl is the standard for large-scale GRPO/PPO. Built on Ray for distributed orchestration, it separates the *generation* (rollout) phase from the *training* (gradient) phase — the architecture that makes RL at scale tractable, because generation and training have very different compute profiles (generation is memory-bandwidth-bound and embarrassingly parallel; training is compute-bound and needs careful orchestration). verl handles the placement, the actor/rollout/reference decomposition, and the large-batch generation that reasoning RL needs. If you are training a 30B+ model on a cluster, this is the tool. ByteDance and others use it for production reasoning RL.
## OpenRLHF — the caveat
OpenRLHF is faster than verl on some micro-benchmarks (single-node, specific configs). But a `masked_mean` bug was found in its advantage/loss computation that silently miscalculated the per-token mean under certain padding conditions — leading to subtly wrong gradients on long sequences with heavy padding (exactly the reasoning-CoT setting). The bug has been addressed, but the community consensus crystallized as: **"TRL or verl and you're fine."** Use TRL for prototyping, verl for scale. Treat OpenRLHF as opt-in only if you have verified your specific config against the patched version.
> **The decision tree:** single GPU, learning, small model → TRL `GRPOTrainer`. Cluster, large model, production → verl. OpenRLHF only with a specific reason and a check that the masked_mean issue does not affect your padding pattern.
---
# 14.7 — The Async RL Training System (RI-301)
*The most significant gap in most RL-for-LLM curricula. GRPO is taught as an algorithm — sample N, score, take a gradient step. But GRPO runs on a **training system**, and that system is where the production engineering lives. verl's one-line description in 14.6 ("separates generation from training") is the entire tip of this iceberg. This section is the rest of it.*
## The decoupled trainer/generator architecture
The first thing to internalize is that a production RL-for-LLM run is *two workloads stapled together*, and they have nothing in common computationally:
- **Generation (rollout).** The model autoregressively produces N responses per prompt. This is **memory-bandwidth-bound** — you are streaming weights through the matrix-vector multiplies of decode, one token at a time, doing very little math per byte loaded. It is also **embarrassingly parallel** across the N samples and across prompts. Batch size barely matters past a point; you are latency-bound on each sequence.
- **Training (gradient).** You take the scored rollouts and run forward+backward over the policy. This is **compute-bound** — dense matrix-matrix multiplies, high arithmetic intensity, benefits enormously from large batches and tensor parallelism.
A single coupled loop ("generate a batch, immediately train on it, repeat") forces both workloads onto the same hardware topology and serializes them. That works at TRL scale (one GPU, small model) and collapses at verl scale (a cluster, 30B+ model). The production answer — verl, and the broader 2025–2026 async-RL literature (Skywork-Reasoner, SeedRoll, the NeoX/OpenRLHF async modes) — is to **decouple**: run generation and training as separate worker pools that communicate through a shared rollout buffer, each sized and scheduled for its own compute profile.
```text
Decoupled trainer/generator (the verl / HybridFlow pattern)
+-------------------+ +------------------------+ +------------------+
| Generation pool | rollouts | Rollout buffer | batches | Training pool |
| (memory-BW bound, | --------> | (scored, | -------> | (compute-bound, |
| many workers, | | optionally | | TP/PP sharded, |
| weight shards) | <-------- | importance-tagged) | <------- | large batches) |
+-------------------+ weight +------------------------+ policy +------------------+
sync update
```
The two pools can be sized independently. You over-provision generation (where the N-sample cost lives) and right-size training. The rollout buffer absorbs the rate mismatch. This is the architecture that makes thousand-rollout GRPO affordable.
## The tail-latency problem (stragglers)
Decoupling surfaces a new failure mode that does not exist in the coupled loop: **the straggler problem.** In a synchronous regime, a gradient step cannot complete until every rollout in the batch has been generated *and* verified. Reasoning rollouts have enormous length variance — one sample produces a 200-token answer, another produces a 4,000-token chain-of-thought and explores three dead ends before concluding. The long tail dominates wall-clock time. If you wait for all N×P rollouts to finish, your step time is set by the *slowest* sequence in the batch, and GPU utilization on the workers that finished early collapses to zero while they wait.
This is the classic distributed-systems tail-latency amplification, applied to RL. The mitigations, in order of how aggressive they are:
1. **Dynamic batching / continuous generation.** Workers do not wait for the batch to fill; finished rollouts stream into the buffer as they complete. The trainer pulls a batch when it is ready, not when the slowest worker finishes.
2. **Length-based rollout capping.** Enforce a hard max-generation length and a thinking budget (the Qwen3 mechanism from 14.5). The tail is bounded by construction; you trade a little coverage of very-long-reasoning for predictable step time.
3. **Straggler cancellation / speculative discard.** Once enough rollouts for a prompt have completed (e.g., N of the requested M), cancel the rest. The advantage estimate from N samples is fine; the cancelled samples were going to be the long-tail ones anyway.
4. **Async regimes (below).** Stop waiting at all. The trainer updates whenever it has a batch; the generators keep producing. Tail latency stops blocking the gradient — at the cost of staleness.
## Disaggregated prefill / decode
The same disaggregation that serving stacks (FT20) use for inference applies to the generation half of RL, and for the same reason: **prefill and decode have different compute profiles.** Prefill (processing the prompt) is compute-bound and batch-friendly; decode (generating tokens one at a time) is memory-bandwidth-bound and batch-hostile. In a coupled engine, a long prompt inflates the KV cache for every subsequent decode step on that worker.
Disaggregated generation routes prefill to one set of workers (optimized for throughput, large batches of prompts) and decode to another (optimized for bandwidth, many independent sequences). The KV cache is handed off between them. This is the architecture behind SGLang, TensorRT-LLM's disaggregated mode, and the generation side of verl's HybridFlow. For RL it has a second benefit: the decode pool can be independently scaled to absorb the straggler tail (point 2 above), while the prefill pool runs flat-out.
## Weight-drift measurement in off-policy regimes
The moment you decouple generation from training, you are off-policy (recall the spectrum from 14.2's callout). The generation workers hold weights at version θ_g; the trainer has moved on to θ_t. The quantity that governs whether this is safe is **weight drift** — some norm of (θ_t − θ_g). Drift translates directly into the importance-sampling ratio `π(θ_t)/π(θ_g)` blowing up, which is what makes off-policy gradients wrong.
Production systems measure drift continuously and use it as a control signal:
- **KL between generator and trainer policy.** Compute the per-token KL of the generation policy against the current training policy on a held-out probe set. This is a direct, sampleable measure of how far apart the two distributions have grown. When it crosses a threshold, force a weight sync.
- **Effective sample size (ESS).** The standard importance-sampling diagnostic (`(Σw)² / Σw²` for importance weights w). When ESS drops below a fraction of the raw batch size, the off-policy correction is consuming most of your samples and you are effectively training on a much smaller batch — sync and regenerate.
- **Gradient-norm surprise.** If the gradient on a stale batch is wildly different in magnitude from recent on-policy gradients, the importance ratios have blown up. Treat it as a sync trigger.
The rule of thumb that emerged in 2025–2026: **weight drift of 1–2 policy versions is free; beyond ~4 versions the importance correction starts eating the signal and you should resync.** This is why "1–2 steps off-policy is fine" (14.2 callout) is the practical operating point — it is the drift at which the importance ratio is still near 1 and no correction is meaningfully needed.
## Zero-advantage handling
Back at the algorithm layer, GRPO has a silent failure mode that the training system has to cope with at scale: **the all-same-reward group.** When all N samples for a prompt receive identical reward (all correct, or all wrong), every advantage `A_i = (r_i − mean)/std` is zero, the gradient contribution is zero, and compute was spent for nothing. On easy prompts (model gets them all right) and hard prompts (model gets them all wrong) this is the *common* case, not the exception — and at cluster scale it means a large fraction of your generation budget produces zero learning signal.
This is what DAPO's **dynamic sampling** (14.3) addresses: detect zero-variance groups and skip them, reallocating the generation budget to prompts where the model has reward variance (the ones where it gets some right and some wrong — the actual learning frontier). The training-system implication: you need an online filter that discards zero-advantage groups *before* they consume a training batch slot, and you need the generation pool to over-sample so enough variance-positive groups survive the filter. This couples back to the straggler work — the cheap-to-generate all-correct groups are also the ones you want to throw away, so continuous generation + dynamic filtering is the combined answer.
A second subtlety: **numerical zero-advantage.** When std is tiny but nonzero (N−1 identical rewards, one slightly different), the normalization by std amplifies noise into a large spurious advantage. Production trainers clamp std with an epsilon floor (`std + 1e-4` or similar) and/or mask groups where std is below threshold. Get this wrong and the model trains on numerically-amplified noise — a subtle, silent instability.
## FP8 and mixed-precision implications
The 2025 move to FP8 training (Hopper, Blackwell) changes the RL system in two specific ways:
1. **Generation side: FP8 KV cache and FP8 matmuls roughly double decode throughput.** This is where FP8 pays off most cleanly for RL — decode is bandwidth-bound, FP8 halves the bytes moved per weight and per KV-cache element, and the quality impact on rollout fidelity is small (the rollouts are scored by a deterministic verifier anyway, so small numerical drift in generation is absorbed by the reward signal rather than corrupting a label).
2. **Training side: FP8 forward/backward is real but fragile.** The gradient paths in GRPO are longer than in SFT (advantage computation, importance ratios, KL penalty, clipping) and each is a place a denormal FP8 underflow can silently zero out a signal. Production FP8 RL trainers keep the advantage, importance-ratio, and KL computations in BF16/FP32 and apply FP8 only to the dense matmuls. The ESS and gradient-norm diagnostics from the weight-drift section double as FP8 health monitors — a sudden ESS collapse on a mixed-precision run is often a precision artifact, not a policy drift.
The Hopper/Blackwell NVFP4/MXFP4 frontier (FT19) is, at the time of writing, still on the training side; for RL generation it is increasingly production-ready. Treat FP8 for generation as default-on and FP8 for training as opt-in with monitoring.
## Sync vs async vs fully-async regimes
These choices form a three-point design space, and the right pick is determined by your tolerance for staleness versus your need for throughput. The diagram maps them.
```mermaid
flowchart TB
subgraph SYNC["Sync regime — fully on-policy, lowest throughput"]
direction TB
S1["Gen pool: sample batch from theta_t"]
S2["Wait for ALL rollouts + verify"]
S3["Trainer: 1 step to theta_t+1"]
S4["Push theta_t+1 to gen pool"]
S1 --> S2 --> S3 --> S4 --> S1
end
subgraph ASYNC["Async regime — decoupled, bounded staleness"]
direction TB
A1["Gen pool: continuous rollout
into buffer at theta_g"]
A2["Trainer: pulls batch,
steps to theta_t"]
A3["Weight sync when KL > eps
typ. every 1-4 steps"]
A1 -. rollouts .-> A2
A2 --> A3
A3 -. theta_t .-> A1
end
subgraph FULL["Fully-async regime — max throughput, unbounded staleness"]
direction TB
F1["Gen pool: runs independently,
never blocks on trainer"]
F2["Trainer: updates on whatever is in buffer,
whenever it is ready"]
F3["Drift monitor: ESS / KL / grad-norm
triggers resync or rollback"]
F1 -. stale rollouts .-> F2
F2 --> F3
F3 -. resync .-> F1
end
SYNC -->|"throughput wall
at cluster scale"| ASYNC
ASYNC -->|"push for max GPU util
accept drift mgmt"| FULL
style S1 fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
style S2 fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
style S3 fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
style S4 fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
style A1 fill:#14141f,stroke:#f0a868,stroke-width:1.5px,color:#e4e4e8
style A2 fill:#14141f,stroke:#f0a868,stroke-width:1.5px,color:#e4e4e8
style A3 fill:#14141f,stroke:#f0a868,stroke-width:1.5px,color:#e4e4e8
style F1 fill:#14141f,stroke:#f08080,stroke-width:1.5px,color:#e4e4e8
style F2 fill:#14141f,stroke:#f08080,stroke-width:1.5px,color:#e4e4e8
style F3 fill:#08080c,stroke:#3a3a48,stroke-width:1px,color:#9494a0,stroke-dasharray:3 3
```
**Reading the diagram:** moving left-to-right trades on-policy purity for GPU utilization. Sync is the textbook picture — every gradient is on-policy, but the straggler tail sets step time and the cluster idles while it waits. Async (the production default in verl/Skywork/SeedRoll) decouples the two pools and resyncs on a measured-drift trigger; this is where "1–2 steps off-policy" lives. Fully-async stops syncing on a schedule at all and relies on drift monitors to catch when the importance correction has gone bad — max throughput, but you are now running a distributed-systems consensus problem alongside your RL loop.
**Type:** Architecture comparison. **Purpose:** Map the three operating regimes for production RL and the trade each one makes. **Reading the diagram:** the arrows are the migration path a team follows as it scales. Most teams stop at async; fully-async is a research/frontier setting. The drift-monitoring box in fully-async is non-optional — without it, the run silently diverges.
> **The takeaway:** GRPO-the-algorithm is half a page of math. GRPO-the-production-system is a decoupled, async, mixed-precision, drift-monitored distributed training rig in which the algorithm is the inner loop. The 14.6 tooling decision (TRL vs verl) is, in part, a decision about how much of this system you are willing to build yourself. TRL gives you the coupled/sync regime out of the box — correct, slow, fine for 7B. verl gives you the decoupled/async regime — the only thing that scales to 30B+ reasoning RL. There is no version of frontier reasoning RL that runs in the coupled loop.
---
# 14.8 — The Reward-Verification Loop
*The heart of the method. Get this right and everything else is plumbing.*
The loop, per training step:
1. **Generate** N responses per prompt from the current policy.
2. **Verify** each response with the reward function: parse the answer, execute the check (run the code, compare the math answer, validate the tool output), return a scalar reward.
3. **Compute advantages** relative to the group: `A_i = (r_i − mean) / std`.
4. **Update** the policy toward high-advantage responses, away from low-advantage ones, with a KL anchor to the reference.
The verifier is the load-bearing component. A weak verifier (e.g., substring matching that is fooled by formatting) produces a reward signal the model will *game* — reward hacking. A strong verifier (execute the code and check the output; compare the parsed math answer exactly) produces a signal that genuinely tracks correctness. **The quality of your reasoning RL is bounded by the quality of your verifier.** This is the analog of FT00's "your data matters more than your algorithm": here, your *reward* matters more than your optimizer. A great GRPO config on a hackable reward steers you into a wall.
This is why the lab uses a *code-execution* reward function for math — it parses the model's answer and checks it against the ground truth, which is far more robust than substring or LLM-judge matching for correctness. Learn to write verifiers as carefully as you write training data.
---
## Anti-Patterns
### Using DPO for reasoning
DPO on a fixed preference dataset for a verifiable task. You forfeit on-policy exploration — the model never learns from *its own* mistakes, only from the mistakes in your fixed set. For math/code/tool-use, where the reward is checkable, this is leaving the main signal on the table. Use GRPO with a verifier.
### No reward verification (reward hacking)
A reward function that can be gamed (substring match, leniency, a judge that rewards confidence over correctness). The model will find the shortcut and optimize for it — producing outputs that score high on the reward but are wrong. The fix is a *verifier*, not a *judge*: execute the code, compare the parsed answer exactly, validate the tool output structurally. If you must use a judge (for non-verifiable rewards), you are back in DPO territory, not GRPO.
### Skipping the cold-start SFT
Going straight to RL on a raw base (the R1-Zero path) *can* work — it is the proof-of-emergence experiment — but it produces the R1-Zero problems: poor readability, mixed-language chains, hard-to-parse formats. The R1 full pipeline adds cold-start SFT *for a reason*. Unless you are specifically studying emergence, give the model a readable CoT scaffold before RL. It is one SFT stage and it saves you the readability cleanup later.
### Ignoring the readability problem of pure-RL
The complement of the above: even if pure-RL reasoning emerges, the output is often not in a deployable format. Production reasoning models need readable, parseable chains. Plan for the readability layer (cold-start SFT, and/or rejection-sampling SFT to clean up RL outputs) — do not assume RL alone yields a shippable artifact.
### Treating GRPO variants as interchangeable without reading the patches
"Use GRPO" is underspecified. DAPO's clip-higher changes exploration; Dr. GRPO's length-debias changes the length distribution of outputs; GSPO changes credit assignment. If your model stops improving mid-training, the fix is often a *specific* GRPO++ patch, not "more compute." Know which failure mode each patch addresses.
---
## Key Terms
| Term | Definition |
| --- | --- |
| **GRPO** | Group Relative Policy Optimization (DeepSeek). Samples N responses per prompt, computes advantages relative to the group mean. Eliminates the PPO critic → ~halves memory. |
| **RLOO** | REINFORCE Leave-One-Out. The sibling algorithm GRPO is essentially equivalent to — both use a group/leave-one-out baseline, no learned critic. |
| **Verifiable reward** | A reward computable by a deterministic checker (math correctness, code execution, tool-use success) — enables on-policy RL exploration. Contrast with preference rewards (DPO). |
| **On-policy exploration** | The model generates its own training attempts (rollouts) and learns from their verified outcomes, rather than from a fixed offline dataset. |
| **Advantage (group-relative)** | `A_i = (r_i − mean(r)) / std(r)` — how much better response i was than the group average. Replaces the PPO learned-critic advantage. |
| **R1-Zero** | DeepSeek's pure-RL-on-base experiment — proved reasoning can emerge from RL alone, no SFT cold start. Has readability/format issues. |
| **R1 pipeline** | Cold-start SFT → reasoning RL → rejection-sampling SFT → final alignment RL. The canonical multi-stage recipe. |
| **Rejection-sampling SFT** | Generate candidates with the RL model, filter by reward (keep correct), SFT a fresh model on the curated set. The bridge from RL to a stable SFT model (FT15). |
| **Distillation (CoT)** | SFT a smaller student on a strong reasoning teacher's CoT outputs. R1-Distill-Qwen-32B (SFT-only) beats o1-mini. (FT15.) |
| **DAPO** | ByteDance GRPO variant: clip-higher (decoupled clip for exploration) + dynamic sampling (skip zero-variance groups). |
| **Dr. GRPO** | Variant that removes the length bias in per-token advantage averaging. |
| **GSPO** | Group Sequence Policy Optimization — sequence-level (not token-level) importance weighting. |
| **Thinking budget** | Qwen3's runtime mechanism: bound the "thinking" compute per query, trading latency for accuracy adaptively. |
| **verl** | ByteDance's HybridFlow — production-grade distributed RL on Ray. The standard for large-scale GRPO. |
| **Reward hacking** | The model exploits a weak verifier's shortcut, scoring high on the reward while being wrong. Bounded by verifier quality. |
| **Decoupled trainer/generator** | The production RL architecture: separate worker pools for generation (bandwidth-bound, parallel) and training (compute-bound, sharded), communicating through a rollout buffer. The verl pattern. |
| **Straggler problem (tail latency)** | In sync RL, step time is set by the slowest (longest) rollout in the batch. Mitigated by dynamic batching, length capping, straggler cancellation, or going async. |
| **Disaggregated prefill/decode** | Routing prompt-processing (prefill, compute-bound) and token-generation (decode, bandwidth-bound) to separate worker pools with a KV-cache handoff. Same pattern as FT20 serving, applied to RL generation. |
| **Weight drift** | The norm of (θ_trainer − θ_generator). The quantity that governs how off-policy a decoupled run is; monitored via KL, ESS, gradient-norm surprise. 1–2 versions is free; 4+ needs resync. |
| **Zero-advantage group** | A prompt where all N rollouts receive the same reward → all advantages are zero → no gradient signal. DAPO's dynamic sampling filters these out. Numerical zero-advantage (tiny std) amplifies noise; clamp with an epsilon floor. |
| **Sync / async / fully-async regimes** | The three RL training-system operating points: sync (fully on-policy, straggler-bound), async (decoupled, bounded-staleness resync on drift, production default), fully-async (max throughput, unbounded staleness, drift-monitored). |
| **Effective sample size (ESS)** | `(Σw)² / Σw²` for importance weights w — the standard off-policy diagnostic. ESS collapse signals the importance correction is eating the batch and a weight resync is needed. |
---
## Lab Exercise
See `07-lab-spec.md`. The "GRPO on a Math Task" lab: run TRL `GRPOTrainer` on a small base with a verifiable math reward (GSM8K correctness checked by a Python reward function that parses the answer and verifies it). Watch the reward curve climb. Heavy lab — single consumer GPU for the small model, cloud GPU stretch goal. Full runnable Python including the reward function, the GRPO config, training, and eval.
---
## References
1. **Shao et al. (2024)** — *DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models*. arXiv:2402.03300. Introduces GRPO (group-relative advantage, no critic).
2. **DeepSeek-AI (2025)** — *DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning*. arXiv:2501.12948. The R1-Zero and R1 pipeline.
3. **DeepSeek-AI (2025)** — *DeepSeek-R1* (Nature). s41586-025-09422-z. The peer-reviewed R1 pipeline.
4. **Qwen Team (2025)** — *Qwen3 Technical Report*. arXiv:2505.09388. The 4-stage post-training, thinking-budget, 36T pretraining.
5. **Kool et al. (2019)** — *Buy 4 REINFORCE Samples, Get a Baseline for Free*. The RLOO estimator — the sibling GRPO is essentially equivalent to.
6. **Cameron Wolfe** — *GRPO++ survey* (Distill, Substack). The canonical map of DAPO, Dr. GRPO, GSPO and the failure modes they patch.
7. **Yu et al. (2025)** — *DAPO: An Open-Source LLM Reinforcement Learning System*. arXiv:2503.14476. Clip-higher + dynamic sampling.
8. **Allen AI (2025)** — *OLMo 3*. Open-data production reasoning-RL recipe.
9. **HuggingFace TRL** — *GRPOTrainer documentation*. The moderate-scale entry point.
10. **HybridFlow / verl** — *volcengine/verl* (GitHub). ByteDance's production distributed RL framework on Ray.
11. **Interconnects (Nathan Lambert)** — Coverage of the RL-for-reasoning landscape, GRPO variants, and the OpenRLHF masked_mean discussion.
12. **FT13 — The DPO Family and Preferences** — The prerequisite. The offline-preference baseline this module contrasts against.
13. **FT15 — CoT Distillation and Rejection-Sampling FT** — The follow-on. Covers the distillation and RFT stages that R1/Qwen3 use after RL.
14. **Sheng et al. (2024)** — *HybridFlow: A Flexible and Efficient RLHF Framework*. The verl architecture paper — the canonical reference for the decoupled trainer/generator pattern, actor/rollout/reference decomposition, and large-batch generation on Ray.
15. **Skywork-Reasoner Team (2025)** — *Skywork-Reasoner: Scaling RL for Math*. Production reference for async-RL regime choices, weight-drift management, and the straggler mitigations summarized in 14.7.
16. **Zhong et al. (2025)** — *Disaggregated Serving (prefill/decode split)* family of papers (e.g., DistServe). The architecture 14.7 borrows from serving (FT20) for the generation half of RL.
17. **Dao et al. / Fu et al. (2024–2025)** — *Async RL training* line of work (Async-RLHF, Skywork-async modes). The case that production GRPO is 1–2 steps off-policy and the drift-monitoring regimes that make that safe.