{ "module": "FT15 — CoT Distillation and Rejection-Sampling FT", "course": "3 — LLM Fine-Tuning Masterclass", "version": "1.0.0", "duration_minutes": 45, "total_questions": 15, "bloom_distribution": { "target": "20% recall / 40% application / 40% analysis-design", "actual": { "recall": 3, "application": 6, "analysis": 6 } }, "passing_score_percent": 70, "questions": [ { "id": "Q01", "bloom": "recall", "type": "multiple_choice", "prompt": "State the two cheaper-than-RL routes to a reasoning small model covered in FT15.", "options": [ "DPO (preference optimization) and continued pretraining on math corpora.", "CoT distillation (SFT a smaller student on a strong teacher's traces — no RL) and rejection-sampling FT / RFT (generate candidates, filter by a verifiable reward, retrain on the winners — no policy gradient).", "QLoRA on math SFT data and full-parameter fine-tuning on code.", "Abliteration of the refusal direction and GRPO on preference rewards." ], "answer_index": 1, "rationale": "FT15 covers two cheaper-than-RL routes: (1) CoT distillation — a strong reasoning teacher emits traces, which are curated and SFT'd into a smaller student (no reward model, no policy gradient, no KL anchor — pure SFT); (2) RFT/RAFT — generate candidates with the current model, filter by a verifiable reward, retrain (SFT) on the winners only. Both share GRPO's rollout/reward machinery (RFT does) but avoid its policy-gradient training step. Both are terminal techniques, not just warmup for RL." }, { "id": "Q02", "bloom": "recall", "type": "multiple_choice", "prompt": "What is the headline result of DeepSeek-R1-Distill-Qwen-32B, and how was the student trained?", "options": [ "It matches GPT-4 on coding benchmarks; trained with GRPO on 800K preference pairs.", "It beats OpenAI o1-mini on several benchmarks (incl. AIME 2024, MATH-500); trained with SFT ONLY on ~800K curated reasoning traces distilled from R1 — NO RL on the student.", "It matches the R1 teacher exactly; trained with full RL on verifiable rewards identical to the teacher.", "It beats o1-mini but required PPO on 8M rollouts to reach that quality." ], "answer_index": 1, "rationale": "DeepSeek-R1-Distill-Qwen-32B was trained by using R1 (a frontier reasoner trained with full RL on verifiable rewards) as a TEACHER. R1 generated reasoning traces, curated to ~800K verified-correct samples (math + code + general). An off-the-shelf Qwen-32B base was then SFT'd on those traces — NO RL on the student, just supervised FT. The result beats OpenAI o1-mini on AIME 2024 and MATH-500. This single fact made distillation the default, accessible recipe for small reasoning models: SFT on a strong teacher's traces suffices to produce frontier-adjacent quality at small scale." }, { "id": "Q03", "bloom": "recall", "type": "multiple_choice", "prompt": "State the arXiv:2504.11343 (RAFT) finding and why it matters.", "options": [ "RFT is strictly dominated by GRPO on all math benchmarks; use GRPO always.", "A simple rejection-sampling fine-tuning baseline is SURPRISINGLY COMPETITIVE with GRPO and PPO on math reasoning (same base/data/compute) — close enough that RFT's simplicity and stability make it a legitimate terminal choice, not merely a warmup.", "RFT only works as a one-step initialization before GRPO; it cannot produce a shipping model.", "RFT requires a frontier teacher model to label candidates, making it as expensive as distillation." ], "answer_index": 1, "rationale": "arXiv:2504.11343 found that a simple rejection-sampling FT baseline is surprisingly competitive with GRPO and PPO on math reasoning — not always equal or better, but close enough that RFT's simplicity (no policy-gradient machinery, no critic, no KL tuning) and stability (SFT training step) make it a legitimate TERMINAL technique, not merely a stepping stone to full RL. This is load-bearing for FT15: it establishes RFT as a destination. The lesson is that simpler methods should not be dismissed — RFT is cheaper and more stable than full RL and a legit standalone technique." }, { "id": "Q04", "bloom": "application", "type": "multiple_choice", "prompt": "A team wants a small math-reasoning model. They have access to a frontier API with extended thinking and a constrained compute budget. Which FT15 technique do they use, and why?", "options": [ "GRPO — it is the most powerful and will give the best result regardless of budget.", "CoT distillation. They have a strong teacher (the frontier API with thinking) and a constrained budget. Distillation is the cheapest path that works at scale: generate traces from the API, curate (verify answers), SFT the small student. No RL, no policy-gradient instability, SFT compute profile. The ceiling is the teacher's, but matching a frontier teacher at small scale is already a win (cf. R1-Distill-Qwen-32B beating o1-mini).", "RFT — they should generate their own candidates and filter, since they have no teacher.", "DPO on preference pairs, since math rewards are not verifiable." ], "answer_index": 1, "rationale": "Distillation is the right call. The three-axis decision: (1) strong teacher available? YES — the frontier API with thinking. This puts distillation on the table as the cheapest option, considered first. (2) The constrained budget favors the SFT compute profile of distillation over the full RL loop of GRPO. (3) The goal does not require exceeding the teacher — matching a frontier teacher at small scale is already a win (R1-Distill-Qwen-32B beat o1-mini via this exact recipe). RFT/GRPO would be escalation if the distilled model falls short AND budget later allows. Option (c) is wrong — they DO have a teacher. Option (d) is wrong — math rewards ARE verifiable." }, { "id": "Q05", "bloom": "application", "type": "multiple_choice", "prompt": "A team wants to push a reasoning model PAST what any available teacher can produce, on math. They have a verifiable reward and a large budget willing to tune KL. Which technique, and why?", "options": [ "Distillation — copy the teacher's traces; it is the cheapest.", "RFT only — it is more stable than GRPO and 2504.11343 says it is competitive.", "GRPO (FT14). The goal is to EXCEED the teacher — distillation is capped at the teacher's ceiling. The reward is verifiable and the budget is large, so the conditions for full RL are met. RFT could be tried first as a stepping stone, but the terminal goal of pushing past the teacher requires learning from the model's own rollouts via policy gradient, which is GRPO's domain.", "DPO, because pushing past the teacher requires preference data." ], "answer_index": 2, "rationale": "GRPO is the right call when the goal is to push BEYOND the teacher. Distillation is capped at the teacher's ceiling (imitation). RFT is competitive and a legitimate first step (cheaper, stable), but its limit is that it ignores failures — it cannot learn the 'push down the bad' signal. GRPO's policy gradient learns from both successes and failures, which is the edge on the hardest exploration problems. The verifiable reward + large budget + willing-to-tune-KL conditions are exactly GRPO's prerequisites. Option (b) is defensible as a first attempt but wrong as the terminal answer for pushing past the teacher — RFT's expressivity limit means it may plateau where GRPO can still improve." }, { "id": "Q06", "bloom": "application", "type": "multiple_choice", "prompt": "A team runs RFT with a fuzzy LLM-judge reward on open-ended creative writing. The resulting model produces plausible-looking but unrewarding outputs. Diagnose.", "options": [ "The RFT loop ran too few iterations; increase to 10 rounds.", "The reward is NOT verifiable — RFT's entire signal is the filter. A noisy LLM judge with false positives means the team trained on INCORRECTLY-ACCEPTED candidates, producing a model that emits plausible-looking wrong/unrewarding outputs. RFT demands a verifiable reward (math checks, code passes strict tests). Open-ended writing preference is FT13 (DPO) territory, not reasoning FT. Fix: switch to a verifiable reward if one exists, or move to DPO/KTO.", "The learning rate was too high; reduce it 10×.", "The model needs more rollouts per problem; raise N to 32." ], "answer_index": 1, "rationale": "This is the 'RFT without reward verification' anti-pattern. RFT's signal is the FILTER — if the reward is a noisy LLM judge, false positives get accepted and the model trains on incorrectly-accepted candidates, learning to produce plausible-looking but unrewarding outputs. RFT demands a VERIFIABLE reward (math correctness, code passing strict tests, ground-truth match). An open-ended writing preference is a preference problem (FT13, DPO family), not reasoning FT. The fix is not more iterations, lower LR, or more rollouts (all of which amplify the bad signal) — it is to use a verifiable reward or switch to preference optimization." }, { "id": "Q07", "bloom": "application", "type": "multiple_choice", "prompt": "Order the defensible escalation stack for building a small reasoning model, from cheapest to most expensive, with the correct escalation trigger at each stage.", "options": [ "GRPO → RFT → distillation. Escalate down when budget runs out.", "Distill first (cheap, most of the way) → RFT to self-improve where the teacher was weak → GRPO ONLY if RFT plateaus and budget remains. Escalate when the current stage falls short of the target, not by default. Many production reasoners stop at distillation.", "RFT → distillation → GRPO. Always run all three for maximum quality.", "Distillation and GRPO simultaneously, then RFT to merge the results." ], "answer_index": 1, "rationale": "The defensible stack: distill first (cheapest, gets you most of the way using a strong teacher's traces, SFT compute profile) → RFT to self-improve on areas the teacher was weak (generate own candidates, filter by reward, retrain on winners — still SFT-stable, no teacher ceiling) → GRPO ONLY if RFT plateaus AND budget remains (full RL to push the frontier). Each stage is OPTIONAL — escalation is justified by measured shortfall against the target, not by default. Many production reasoning models stop at distillation (R1-Distill-Qwen-32B is shipped, not escalated). This avoids paying full RL cost for gains cheaper techniques already capture." }, { "id": "Q08", "bloom": "application", "type": "multiple_choice", "prompt": "You are building a hybrid (thinking-toggle) student via distillation. What must your distillation corpus contain that a pure-thinking distillation does not?", "options": [ "Only non-thinking examples, so the model learns to answer fast.", "Only thinking examples with very long traces, to maximize reasoning quality.", "A MIX of thinking examples (long trace, then answer) AND non-thinking examples (direct answer), plus ideally examples where the difficulty warrants one mode or the other — so the student learns BOTH modes and the ability to choose between them per request.", "Preference pairs comparing thinking vs non-thinking outputs." ], "answer_index": 2, "rationale": "A hybrid model (e.g., Qwen3) can toggle thinking on or off per request from the SAME weights. To distill a hybrid student, the corpus must include a MIX of thinking and non-thinking traces: long-trace examples (thinking on) for accuracy on hard problems, and direct-answer examples (thinking off) for low latency on easy ones, plus ideally examples demonstrating which mode is appropriate for which difficulty. This trains the student to perform in both modes and to select appropriately. Pure CoT distillation (only thinking traces) produces a model that always thinks — paying the latency tax on trivial queries, defeating the purpose of the hybrid. Option (d) is DPO territory, not distillation." }, { "id": "Q09", "bloom": "application", "type": "multiple_choice", "prompt": "A colleague distills from a 7B base model that itself struggles with the target math. The student confidently emits wrong reasoning chains. Diagnose and fix.", "options": [ "The student is overfitting; raise dropout and reduce epochs.", "This is the 'distilling from a weak teacher' anti-pattern. The student's ceiling is the teacher's — if the 7B teacher cannot actually do the math, the student imitates MEDIOCRE reasoning and confidently emits wrong chains. Garbage in, garbage out. Fix: use a genuinely strong teacher on the target domain (R1-Distill worked because R1 is a frontier reasoner). The teacher must be strong enough that imitating its traces produces correct reasoning.", "The LoRA rank is too low; raise r to 128.", "Switch from QLoRA to full fine-tuning to give the student more capacity." ], "answer_index": 1, "rationale": "This is the weak-teacher anti-pattern. Distillation is imitation; the student's reasoning ceiling is the teacher's ceiling minus whatever the smaller model forgets. If the 7B teacher itself struggles with the math, its traces encode incorrect or shaky reasoning, and the student learns to reproduce that — confidently wrong chains. The fix is NOT more capacity, higher rank, or full FT (those amplify the bad signal) — it is to use a genuinely strong teacher (a frontier reasoner or a model proven strong on the target domain). R1-Distill worked because R1 is frontier. The teacher must be strong on the target domain." }, { "id": "Q10", "bloom": "analysis", "type": "multiple_choice", "prompt": "Why is CoT distillation an instance of the FT00 thesis ('fine-tuning steers behavior; it does not teach knowledge'), and what is the implication for the student's ceiling?", "options": [ "Distillation teaches the student new reasoning ability the base lacked, so it can exceed the teacher.", "The student's base already saw the math/code during pretraining — the reasoning capability is LATENT. Distillation does not inject new ability; it steers the student to RELIABLY EMIT a structured trace and USE pathways it already has. The teacher's trace is the steering wheel. Implication: distillation CANNOT make the student exceed the teacher — imitation has a ceiling. To push past the teacher, you need RL on the student's own rollouts (RFT/GRPO).", "Distillation is knowledge injection, like continued pretraining, so the ceiling is unlimited.", "Distillation works by deleting a refusal direction, like abliteration." ], "answer_index": 1, "rationale": "Distillation is steering, not teaching (FT00). The student's base already possesses the math/code capability from pretraining — it is latent. Distillation routes probability mass through those existing pathways reliably and in a structured trace; the teacher's trace is the steering wheel. This is why a 32B distilled model punches above its pretraining weight class: the capability was there, distillation makes it reliable. The corollary is the ceiling: because it is imitation, the student cannot exceed the teacher. To push past the teacher, you need the student to learn from its OWN rollouts via RL (RFT/GRPO), where it can discover solutions the teacher never generated." }, { "id": "Q11", "bloom": "analysis", "type": "multiple_choice", "prompt": "What is the STRUCTURAL reason RFT is more stable than GRPO, and what does RFT trade away for that stability?", "options": [ "RFT uses a lower learning rate, which is inherently more stable.", "RFT's training step is SFT (cross-entropy on accepted candidates) — SFT does not diverge like policy gradients. GRPO's failure modes (rollout collapse, KL drift, flat-reward noise) are all POLICY-GRADIENT pathologies that can silently ruin the model while the reward looks fine. RFT's worst cases (small accepted set, wrong reward) are DIAGNOSABLE data-quality problems. RFT trades EXPRESSIVITY (it ignores failures, so it cannot learn the 'push down the bad' signal) for STABILITY + DIAGNOSABILITY.", "RFT uses a critic network that stabilizes training, unlike GRPO.", "RFT is more stable because it uses a larger batch size than GRPO." ], "answer_index": 1, "rationale": "The stability advantage is structural, not incidental. GRPO's policy gradient (advantage × log-prob) learns from both successes and failures, and its failure modes — rollout collapse (model finds one high-reward string and stops exploring), KL drift (model drifts from reference), flat-reward noise (degenerate advantage estimates) — are all policy-gradient pathologies that can silently ruin the model while the reward looks fine. RFT's training step is SFT — cross-entropy on accepted candidates — which does not diverge that way. RFT's worst cases (too few accepted candidates, wrong reward) are diagnosable data-quality problems you can inspect by reading the accepted set. The trade: RFT gives up expressivity (it ignores failures, so it cannot learn the 'push down the bad' signal) for stability + diagnosability. It is not about LR, critics, or batch size." }, { "id": "Q12", "bloom": "analysis", "type": "multiple_choice", "prompt": "A colleague says: 'Distillation is just a warmup for RL — real reasoning models need GRPO.' Evaluate against FT15.", "options": [ "Correct — GRPO is always required for a production reasoning model.", "Wrong. R1-Distill-Qwen-32B is a SHIPPING model trained with distillation ONLY (no student-side RL) that beats o1-mini. Distillation is a TERMINAL technique, not merely an init step. arXiv:2504.11343 shows the same for RFT — competitive with GRPO. Treating distillation as only a warmup leads to spending RL compute for marginal gains over a strong distilled baseline. Correct framing: distill, EVALUATE against your target, escalate to RL ONLY if it falls short. Many production reasoners stop at distillation. GRPO is for pushing past the teacher/plateau, not a default requirement.", "Partially correct — distillation is always inferior to GRPO but useful for small models.", "Correct only for code reasoning; distillation suffices for math." ], "answer_index": 1, "rationale": "The claim is wrong per FT15. R1-Distill-Qwen-32B is empirical proof that distillation is terminal: it is a shipped model (no student-side RL) that beats o1-mini. The arXiv:2504.11343 result extends this to RFT — competitive with GRPO, so RFT too can be the final technique. The anti-pattern is treating distillation/RFT as merely pre-RL initialization, which leads to spending RL compute for what may be marginal gains over a strong distilled/RFT'd baseline. The correct process: distill (or RFT), evaluate the result against your target, and escalate to GRPO ONLY if it falls short. GRPO is for pushing past the teacher or past an RFT plateau — not a default requirement for 'real' reasoning models." }, { "id": "Q13", "bloom": "analysis", "type": "multiple_choice", "prompt": "Both RFT and GRPO use rollouts from the current model plus a verifiable reward. What is the RADICAL difference in their training step, and what does it imply about when each wins?", "options": [ "There is no real difference; RFT and GRPO are the same algorithm with different names.", "GRPO computes a policy gradient (advantage × log-prob) over ALL rollouts — it learns from both successes and failures, pushing up the good and down the bad. RFT DISCARDS the failures and supervises (SFT) on the successes only. Implication: GRPO wins on the hardest exploration problems where learning from failures matters; RFT wins when you want SFT stability, diagnosability, and lower cost, and the task does not require the failure signal. RFT is competitive on many math tasks (2504.11343); GRPO is the escalation when RFT plateaus.", "RFT uses a policy gradient but GRPO uses SFT, so RFT is more powerful.", "GRPO discards failures and RFT learns from them, so RFT is more expressive." ], "answer_index": 1, "rationale": "The radical difference is the training step. GRPO keeps all rollouts and computes a policy gradient — advantage × log-prob — learning from BOTH successes (push up) and failures (push down). RFT DISCARDS the failures and supervises via SFT on the successes only. Same rollouts and reward; different training step. Implication: GRPO has an edge on the hardest exploration problems where the 'push down the bad' signal matters — it can learn what NOT to do. RFT wins on stability (SFT does not diverge), diagnosability (inspect the accepted set), and cost (no policy-gradient machinery), and is competitive on many math tasks per 2504.11343. RFT's expressivity limit (ignoring failures) is why it can plateau where GRPO can still improve — that is the escalation trigger." }, { "id": "Q14", "bloom": "analysis", "type": "multiple_choice", "prompt": "Why did R1-Distill-Qwen-32B beat o1-mini despite being trained with SFT only (no student-side RL)? Identify the three contributing factors.", "options": [ "Qwen-32B is a larger base than o1-mini, so it naturally wins.", "(1) The teacher (R1) is a FRONTIER reasoner whose traces encode genuinely high-quality reasoning — the student imitates good chains, not mediocre ones. (2) The curation was large and high-quality (~800K verified-correct samples across math/code/general). (3) The Qwen-32B base already had LATENT math/code capability from pretraining — distillation ROUTED probability mass through those pathways reliably and in a structured trace (FT00 steering). The base was capable; the traces made it reliable. Distillation's ceiling-is-teacher property is not a weakness when the teacher is frontier: matching a frontier teacher at 32B already beats a compressed-reasoning product.", "The R1 team used secret RL on the student that they did not disclose.", "o1-mini is a weak model that any 32B SFT run would beat." ], "answer_index": 1, "rationale": "Three factors. (1) Teacher quality: R1 is a frontier reasoner (trained with full RL on verifiable rewards) — its traces encode high-quality reasoning. The student imitates genuinely good chains; garbage-in-garbage-out works in reverse here (good traces in, good reasoning out). (2) Curation: ~800K samples, verified correct, balanced across math/code/general — scale and quality of the distillation corpus. (3) Latent capability: the Qwen-32B base already saw enormous math/code during pretraining; the capability was latent. Distillation is steering (FT00) — it routed probability mass through those existing pathways reliably and in a structured trace. The base was capable; the traces made it reliable. This shows distillation's ceiling-is-teacher is not a weakness when the teacher is frontier: matching a frontier teacher at 32B already beats a compressed-reasoning product like o1-mini. It is NOT because Qwen is larger (o1-mini is a strong product), NOT secret RL (the recipe is documented SFT), and NOT because o1-mini is weak." }, { "id": "Q15", "bloom": "analysis", "type": "multiple_choice", "prompt": "A team distills a small student from a strong teacher, but the teacher's traces are extremely long and use idiosyncratic formatting the student's tokenizer handles poorly. The student's reasoning quality is worse than expected. Diagnose and explain why, per FT15.", "options": [ "The teacher is too strong; use a weaker teacher so the student can keep up.", "This is the 'distilling traces the student cannot imitate' anti-pattern. Distillation is IMITATION, and imitation requires the target to be imitable. Excessively long traces exceed the small student's capacity (it truncates or drifts); idiosyncratic formatting wastes capacity on surface style rather than reasoning structure; poor tokenizer fit degrades the student's representation of the trace. Fix: CURATE the traces — trim to reasonable length, normalize formatting, ensure the format is something the student can reproduce. The target text must be learnable by the student, not just correct.", "The learning rate is too high; reduce it and the traces will fit.", "Switch to GRPO; RL does not require imitable traces." ], "answer_index": 1, "rationale": "This is the imitability anti-pattern. Distillation is imitation, and imitation requires the target to be imitable. If the teacher's traces are too long, the small student cannot reliably reproduce them (capacity limit) — it truncates, drifts, or spends all capacity on length rather than reasoning. If they are stylistically idiosyncratic (unusual tokens, erratic formatting), the student spends capacity mimicking surface style instead of reasoning structure. If the tokenizer handles the trace's tokens poorly (inefficient tokenization, unknown tokens), the student's representation is degraded. The fix is curation: trim to reasonable length, normalize formatting, ensure the format is something the student can reproduce. The target text must be LEARNABLE by the student, not just correct. A weaker teacher (option a) makes it worse. Lower LR (option c) does not fix imitability. GRPO (option d) avoids imitation but changes the entire technique and budget — not the diagnosed fix." } ] }