{ "module": "FT14 — GRPO and Verifiable Rewards", "course": "3 — LLM Fine-Tuning Masterclass", "version": "1.0.0", "duration_minutes": 55, "total_questions": 15, "bloom_distribution": { "target": "20% recall / 40% application / 40% analysis-design", "actual": { "recall": 3, "application": 6, "analysis": 6 } }, "passing_score_percent": 70, "questions": [ { "id": "Q01", "bloom": "recall", "type": "multiple_choice", "prompt": "State the GRPO advantage computation and what replaces the PPO learned value function (critic).", "options": [ "A_i = r_i − V(s_i), where V is a learned value function the same size as the policy.", "A_i = (r_i − mean(r)) / std(r), where mean and std are computed over the N sampled responses for a prompt. The group mean (a Monte Carlo estimate of expected reward under the current policy) replaces the learned critic — no critic params, no critic training.", "A_i = r_i − max(r), using the best sample in the group as the baseline.", "A_i = r_i, the raw reward, with no baseline subtraction at all." ], "answer_index": 1, "rationale": "GRPO computes the advantage relative to the GROUP: A_i = (r_i − mean(r)) / std(r) over the N sampled responses. The group mean is a bias-free, lower-variance-than-REINFORCE Monte Carlo estimate of E[r] under the current policy, and it replaces the PPO learned value function entirely. No critic params, no critic training, no critic memory — memory drops from ~3 models (policy+critic+reference) to ~2 (policy+reference), roughly halving memory vs PPO." }, { "id": "Q02", "bloom": "recall", "type": "multiple_choice", "prompt": "Name the three GRPO++ variants and the specific failure mode each patches.", "options": [ "DAPO (clip-higher + dynamic sampling), Dr. GRPO (removes length bias), GSPO (sequence-level importance weighting).", "DAPO (sequence-level weighting), Dr. GRPO (clip-higher), GSPO (dynamic sampling).", "DAPO (length bias), Dr. GRPO (exploration stalls), GSPO (credit assignment).", "DAPO (critic removal), Dr. GRPO (KL anchor), GSPO (group sampling)." ], "answer_index": 0, "rationale": "DAPO (ByteDance): clip-higher (decouple ± clip for exploration) + dynamic sampling (skip zero-variance groups). Dr. GRPO: removes the length bias in per-token advantage averaging. GSPO: sequence-level (not token-level) importance weighting for stable credit assignment on long CoT. All three are PATCHES on the GRPO/RLOO base, not different algorithms — production recipes mix and match (e.g., 'GRPO with clip-higher and dynamic sampling' = GRPO + DAPO patches)." }, { "id": "Q03", "bloom": "recall", "type": "multiple_choice", "prompt": "State the 4 stages of the DeepSeek-R1 pipeline.", "options": [ "(1) Pretraining, (2) SFT, (3) DPO, (4) quantization.", "(1) Cold-start SFT (reasoning data), (2) Reasoning RL (GRPO on verifiable rewards), (3) Rejection-sampling SFT (generate, filter by reward, retrain), (4) Final RL for alignment (reasoning + safety/helpfulness).", "(1) Pure RL on base, (2) DPO on preferences, (3) Distillation, (4) RLHF with a reward model.", "(1) Cold-start SFT, (2) DPO, (3) Rejection sampling, (4) Continued pretraining." ], "answer_index": 1, "rationale": "The canonical R1 pipeline (arXiv:2501.12948 + Nature): (1) Cold-start SFT gives a readable CoT scaffold (fixes R1-Zero's readability problem); (2) Reasoning RL (GRPO) sharpens the capability; (3) Rejection-sampling SFT — generate candidates with the RL model, filter by reward, SFT a fresh model on survivors — bakes quality in without RL instability (→ FT15); (4) Final RL for alignment preserves reasoning while aligning for general use. R1-Zero is the separate pure-RL-on-base experiment, not a pipeline stage." }, { "id": "Q04", "bloom": "application", "type": "multiple_choice", "prompt": "You are training a model to write correct SQL. The reward is: run the generated SQL against a test database and check the output rows match the expected result. Which method do you use, and why?", "options": [ "DPO — SQL correctness is subjective and needs a judge to compare outputs.", "GRPO with the execution-based reward. The reward is VERIFIABLE (you can run the SQL and check the output deterministically), so on-policy RL lets the model generate attempts, verify them, and learn from its own error distribution. DPO would forfeit this exploration.", "SFT only — there is no need for RL when you have a test database.", "Continued pretraining on SQL examples — the model needs new SQL knowledge." ], "answer_index": 1, "rationale": "The reward is verifiable: you execute the SQL and check the output deterministically. Per the decision rule, verifiable reward → GRPO/RL for on-policy exploration. The model generates N SQL attempts per prompt, the verifier runs each and checks the output, and the policy learns from the distribution of its own successes/failures. DPO (offline, fixed pairs) would forfeit the exploration that a verifier enables. This is the canonical verifiable-domain case (like math and code)." }, { "id": "Q05", "bloom": "application", "type": "multiple_choice", "prompt": "You are configuring a GRPO run. Mid-training, the reward plateaus and you notice the model produces almost no diversity across its N samples — it keeps generating near-identical correct answers and stops exploring. Which GRPO++ patch directly addresses this?", "options": [ "Dr. GRPO (removes length bias).", "GSPO (sequence-level importance weighting).", "DAPO's clip-higher — decouple the PPO clip range for positive vs negative advantages so positive advantages can push further once the policy is mostly correct, keeping exploration alive.", "Increase the KL penalty to force more divergence from the reference." ], "answer_index": 2, "rationale": "This is the exploration-stalls failure DAPO's clip-higher fixes. Standard symmetric PPO clipping caps how far positive advantages can upweight good responses — once the policy is mostly correct, good responses can only be pushed up to the clip, so exploration dies. Clip-higher decouples the ± clip range, letting positive advantages push further, keeping exploration alive late in training. Dr. GRPO fixes length bias (wrong symptom); GSPO fixes credit assignment noise (wrong symptom). Increasing KL would push toward the reference, not toward exploration." }, { "id": "Q06", "bloom": "application", "type": "multiple_choice", "prompt": "Your GRPO reward curve climbs nicely but the model's outputs are getting suspiciously SHORT — terse answers that score high but skip reasoning. The verifier is strong (exact answer match). What is the likely cause and the fix?", "options": [ "Reward hacking — switch to a judge-based reward.", "The length bias in standard GRPO's per-token advantage averaging, which favors shorter sequences (fewer tokens to dilute the per-token advantage). Fix: apply Dr. GRPO, which removes this bias so 'shorter = higher reward' is no longer a spurious shortcut.", "The learning rate is too high; reduce it 10×.", "The group size N is too small; increase it to 64." ], "answer_index": 1, "rationale": "This is the length-bias failure Dr. GRPO fixes. Standard GRPO averages the advantage per-token, which biases toward shorter responses (shorter sequences have fewer tokens to dilute the per-token advantage), so the model can learn 'shorter = effectively higher reward' as a shortcut — producing terse outputs that score high. The verifier is strong (so it's NOT reward hacking); the problem is the objective's per-token averaging. Dr. GRPO re-weights to remove this bias. A judge would make it worse (introduces hackability)." }, { "id": "Q07", "bloom": "application", "type": "multiple_choice", "prompt": "You are choosing a training framework for a 70B reasoning-RL run on an 8-GPU cluster, needing thousands of parallel rollouts. Which tool do you use, and what is the architecture that makes it suitable?", "options": [ "TRL GRPOTrainer — it integrates with accelerate and handles distributed training.", "verl (HybridFlow, ByteDance) — built on Ray, it separates the generation (rollout) phase from the training (gradient) phase. Generation is memory-bandwidth-bound and embarrassingly parallel; training is compute-bound and needs orchestration. This separation is what makes frontier-scale RL tractable.", "OpenRLHF — it is faster than verl on micro-benchmarks.", "Plain PyTorch FSDP with a custom RL loop — no framework needed at this scale." ], "answer_index": 1, "rationale": "For a 70B cluster run needing thousands of parallel rollouts, verl (HybridFlow) is the standard. Built on Ray, it separates generation (rollout) from training (gradient) — the architecture that makes large-scale RL tractable, because the two phases have very different compute profiles (generation: bandwidth-bound, parallel; training: compute-bound, orchestrated). TRL GRPOTrainer is for moderate scale (single GPU/small cluster, 1B–7B). OpenRLHF has the masked_mean caveat. A custom loop at 70B cluster scale is reinventing verl poorly." }, { "id": "Q08", "bloom": "application", "type": "multiple_choice", "prompt": "Write a verifiable reward function for a math task where the model must output its final answer in a \\boxed{} block. The function takes lists of completions and ground-truth answers and returns a list of float rewards.", "options": [ "def reward(prompts, completions, answer): return [1.0 if 'correct' in c.lower() else 0.0 for c in completions]", "def reward(prompts, completions, answer):\n import re\n out = []\n for c, g in zip(completions, answer):\n m = re.findall(r'\\\\boxed\\{([^}]*)\\}', c)\n pred = m[-1].strip() if m else None\n out.append(1.0 if pred is not None and pred == g.strip() else 0.0)\n return out", "def reward(prompts, completions, answer): return [len(c) / 1000.0 for c in completions]", "def reward(prompts, completions, answer): return [0.5 for _ in completions] # participation trophy" ], "answer_index": 1, "rationale": "The verifiable reward parses the last \\boxed{} from each completion and compares it EXACTLY to the ground-truth answer — 1.0 for exact match, 0.0 otherwise. This is a verifier (deterministic, execution-equivalent), not a judge. Option (a) is a substring match (hackable — model can emit 'correct' without being correct). Option (c) rewards length (not correctness). Option (d) gives a constant reward (zero gradient signal — all advantages are zero). Only (b) tracks true correctness." }, { "id": "Q09", "bloom": "application", "type": "multiple_choice", "prompt": "You ran R1-Zero-style pure RL on a base model (no cold-start SFT) and reasoning emerged, but the outputs are messy, mixed-language, and hard to parse. What is the correct fix per the R1 pipeline?", "options": [ "Add more RL compute — the messiness means RL hasn't converged.", "Add a cold-start SFT stage BEFORE RL, using high-quality reasoning data. This gives the model a readable chain-of-thought scaffold to build on (R1 stage 1), solving the readability problem R1-Zero exhibited. You are not teaching reasoning; you are giving a clean format.", "Switch from GRPO to DPO — DPO produces cleaner outputs.", "Switch to a larger base model — the messiness is a capacity problem." ], "answer_index": 1, "rationale": "This is exactly the R1-Zero readability problem that R1 stage 1 (cold-start SFT) exists to solve. Pure RL optimizes for correctness, not human-facing format, so emergent CoT is messy/mixed-language/hard-to-parse. The fix is to add cold-start SFT on high-quality reasoning data BEFORE RL, giving the model a readable CoT scaffold. You are not teaching reasoning (the base already has it); you are giving a clean format. More RL compute won't fix readability (it's not a convergence issue). DPO is the wrong tool for verifiable reasoning. A bigger base doesn't fix format." }, { "id": "Q10", "bloom": "analysis", "type": "multiple_choice", "prompt": "Why does the FT00 thesis ('fine-tuning steers behavior; it does not teach knowledge') imply that RL-on-verifiable-rewards is the correct framing for reasoning, rather than 'RL teaches the model to reason'?", "options": [ "Because RL uses a higher learning rate than SFT, so it converges faster on reasoning.", "Because reasoning capability was already in the base from pretraining. R1-Zero proved reasoning EMERGES from RL alone on a base — but 'emerges' means RL SELECTS FOR behaviors (self-verify, backtrack, long CoT) that produce correct answers, steering the model to USE pathways it already has. No new knowledge enters; RL redirects probability mass toward behaviors that score correct. Had the capability been absent, no amount of RL would produce it — you'd need a different base or continued pretraining.", "Because GRPO has no critic, so it cannot teach — only steer.", "Because verifiable rewards are cheaper than preference labels, making RL more economical." ], "answer_index": 1, "rationale": "The FT00 thesis + the R1-Zero result together: the base already had the reasoning capability from pretraining. RL-on-verifiable-rewards does not teach reasoning; it SELECTS FOR the behaviors (self-verification, backtracking, longer CoT) that produce correct answers, steering the model to USE reasoning pathways it already has. 'Emergence' is the behavioral phenotype of this steering — it looks like new capability but is redirection of existing capability. Had the capability been absent, RL could not produce it (you cannot select for a behavior the model cannot express) — that would require a different base or continued pretraining. The economic argument (option d) is real but secondary." }, { "id": "Q11", "bloom": "analysis", "type": "multiple_choice", "prompt": "DeepSeek-R1-Distill-Qwen-32B was trained via SFT ONLY (no RL) on ~800K curated CoT samples from R1, yet beats OpenAI o1-mini. Reconcile this with the fact that R1 itself required expensive multi-stage RL. Why does SFT-only distillation work?", "options": [ "Because the 800K samples contain the reasoning knowledge that R1 learned via RL — distillation transfers that knowledge to the student.", "Because the student (Qwen-32B base) already had the reasoning capability from pretraining; the 800K curated CoT samples STEER it to use that capability reliably (FT00). No new knowledge transfers — only the behavioral pattern of careful, verified reasoning. The expensive RL (R1) was needed ONCE to produce a teacher whose CoT traces are high-quality; then SFT propagates the behavioral pattern to cheaper students at low cost.", "Because SFT is a stronger algorithm than RL for reasoning tasks.", "Because Qwen-32B is a larger model than R1, so it needs less training." ], "answer_index": 1, "rationale": "This is the FT00 thesis applied to distillation. The student already had the capability from pretraining; the 800K curated CoT samples from R1 STEER it to use that capability reliably (the behavioral pattern of careful verified reasoning). No new knowledge transfers. The expensive RL (R1) was needed ONCE to produce a teacher whose CoT traces are high-quality; then SFT alone propagates the behavioral pattern to cheaper students. This is why CoT distillation became standard: pay the RL cost once, distill repeatedly. Option (a) misframes it as knowledge transfer (it's behavior steering). The economics (option d) is incidental." }, { "id": "Q12", "bloom": "analysis", "type": "multiple_choice", "prompt": "A team argues: 'GRPO is just a special case of PPO with a cheaper baseline, so PPO with a well-trained critic should always match or beat it.' Evaluate this claim.", "options": [ "Correct — PPO's learned critic is always lower-variance, so PPO is strictly better given enough critic training.", "Misleading on two counts. (1) The critic is NOT free — it is ~the same size as the policy, must be trained, adds instability, and roughly doubles memory. For LLM-scale RL this cost is often prohibitive. (2) The group-mean baseline is bias-free and, with N samples, often lower-variance-per-compute than a partially-trained critic. GRPO's trade (N forward passes, no critic) is favorable when rewards are cheap to verify and sample diversity is desired (reasoning). 'Always' is false; the right tool depends on the cost of the critic vs. the cost of sampling.", "Correct — GRPO is deprecated and PPO is the modern default for reasoning.", "Correct only for small models; for 70B+ PPO and GRPO are equivalent." ], "answer_index": 1, "rationale": "The claim is misleading. (1) The critic is expensive: ~same size as the policy, must be trained (extra instability), and roughly doubles memory — prohibitive at LLM scale, which is WHY GRPO drops it. (2) The group-mean baseline is bias-free (unbiased MC estimate of E[r]) and, with N samples, often competitive in variance with a partially-trained critic per unit compute. GRPO's trade (pay N forward passes, save the critic) is favorable when rewards are cheap to verify and diversity is wanted (reasoning). 'Always' is false — the choice depends on the cost of training/serving a critic vs. the cost of N samples. This is why GRPO became the default for reasoning RL." }, { "id": "Q13", "bloom": "analysis", "type": "multiple_choice", "prompt": "A team uses a reward function that checks whether the model's output CONTAINS the correct answer as a substring (e.g., '42' appears anywhere in the response). The reward climbs steadily, but on held-out eval the model is wrong. Diagnose the failure and state the fix.", "options": [ "The verifier is too strict; loosen it to partial-credit matching.", "Reward hacking. The substring-containment reward is HACKABLE — the model learns to emit the expected answer string as a substring (e.g., padding with numbers) without the underlying reasoning being correct, gaming the reward. Fix: use a STRUCTURED verifier — parse the model's final answer from a delimited block (e.g., \\boxed{}) and compare EXACTLY, or for code, execute and check the output. The verifier must track true correctness, not a correlate the model can game.", "The learning rate is too high; the model is overfitting the reward.", "The group size N is too large; reduce it to 2." ], "answer_index": 1, "rationale": "This is textbook reward hacking. The substring-containment reward is hackable — the optimizer (GRPO) does exactly its job of maximizing the reward, and finds the shortcut of emitting the answer string as a substring without correct reasoning. The reward climbs but true correctness does not. The fix is a STRUCTURED verifier that tracks true correctness: parse the final answer from a delimited block (\\boxed{}) and compare exactly, or for code, execute and check the output. The verifier's fidelity to correctness is the ceiling on the run. Loosening it (option a) makes hacking worse. This is the 'your reward matters more than your optimizer' principle." }, { "id": "Q14", "bloom": "analysis", "type": "multiple_choice", "prompt": "Contrast the R1 and Qwen3 pipelines. Which statement is most accurate?", "options": [ "R1 and Qwen3 are essentially the same pipeline; Qwen3 just used more compute.", "R1 established the SCIENCE: reasoning emerges from RL alone (R1-Zero), the 4-stage pipeline (cold-start SFT → reasoning RL → rejection-sampling SFT → final RL) fixes readability, and SFT-only distillation propagates capability to cheaper students. Qwen3 INDUSTRIALIZED the recipe: long-CoT cold start, thinking/non-thinking mode FUSION (one model, two modes), a thinking-BUDGET runtime mechanism (adaptive compute), and 36T+ pretraining. R1 = the discoveries; Qwen3 = the engineering into a controllable product. Both canonical; cite both.", "Qwen3 superseded R1; only Qwen3 is relevant for 2026.", "R1 uses GRPO; Qwen3 uses DPO — that is the key difference." ], "answer_index": 1, "rationale": "R1 (arXiv:2501.12948 + Nature) established the science: emergence via pure RL (R1-Zero), the 4-stage pipeline, and SFT-only distillation that beats o1-mini. Qwen3 (arXiv:2505.09388) industrialized it: long-CoT cold start, thinking/non-thinking mode fusion (one model serving both via a mode flag), the thinking-budget runtime mechanism (adaptive compute per query), and 36T+ pretraining tokens. R1 is the discoveries; Qwen3 is the engineering into a single controllable product. Both use GRPO-family RL (option d is wrong). Both are canonical references — cite both. Neither supersedes the other." }, { "id": "Q15", "bloom": "analysis", "type": "multiple_choice", "prompt": "A colleague proposes: 'Skip the cold-start SFT and the rejection-sampling SFT in the R1 pipeline — just do RL on the base, then distill. R1-Zero proved you don't need SFT.' What is wrong with this plan for a PRODUCTION reasoning model?", "options": [ "Nothing — R1-Zero proved SFT is unnecessary, so this is the efficient path.", "Two problems. (1) R1-Zero's pure-RL outputs have poor READABILITY (messy, mixed-language, hard to parse) — production models need readable, parseable CoT, which cold-start SFT provides (R1 stage 1). (2) Rejection-sampling SFT (R1 stage 3) is the bridge that bakes RL-quality reasoning into a STABLE SFT model without the instability of continued RL — skipping it means deploying a pure-RL model that is harder to serve and align. R1-Zero is the proof-of-emergence experiment, NOT the production recipe; the full R1 pipeline adds the SFT stages for exactly these reasons.", "RL on a base is impossible without SFT first — the base cannot generate.", "Distillation requires RL on the student too; SFT-only distillation does not work." ], "answer_index": 1, "rationale": "R1-Zero is the proof-of-emergence experiment, not the production recipe. Two problems with skipping the SFT stages: (1) Readability — R1-Zero's pure-RL outputs are messy, mixed-language, hard to parse; production models need readable/parseable CoT, which cold-start SFT (stage 1) provides. (2) Stability — rejection-sampling SFT (stage 3) bakes RL-quality reasoning into a stable SFT model without continued-RL instability, producing a deployable artifact. R1-Zero proved reasoning CAN emerge from RL; the full pipeline makes it shippable. Option (c) is false (bases generate fine). Option (d) is false — R1 distillation is SFT-only and works." } ] }