# Module FT15 — CoT Distillation and Rejection-Sampling FT **Course**: Course 3 — LLM Fine-Tuning Masterclass **Module**: FT15 — CoT Distillation and Rejection-Sampling FT **Duration**: 75 minutes **Level**: Senior Engineer and above **Prerequisites**: FT14 (GRPO and Verifiable Rewards) --- ## Learning Objectives After completing this module, you will be able to: 1. State the **two cheaper-than-RL routes** to a reasoning small model — CoT distillation and rejection-sampling fine-tuning (RFT/RAFT) — and why both matter even after GRPO (FT14) is on the table. 2. Describe the **DeepSeek-R1-Distill pipeline**: a strong reasoning teacher emits `…` traces, the traces are curated into ~800K samples, and a smaller student SFTs on them — **no RL required** — yielding R1-Distill-Qwen-32B, which beats OpenAI o1-mini on several benchmarks. 3. Distinguish **RFT/RAFT** from full RL: iteratively fine-tune on *positively-rewarded* candidates only (correct math, passing code), retrain on the winners — simpler and more stable than GRPO/PPO, and per arXiv:2504.11343, **surprisingly competitive** with them. 4. Make the **three-way decision** — distillation vs RFT vs GRPO — from compute budget, stability requirements, and whether a strong teacher is available. 5. Explain the **thinking-mode toggle** in hybrid models (Qwen3): a single model that can think or not, and thinking-budget mechanisms for adaptive compute. 6. Spot the anti-patterns: distilling from a weak teacher, RFT without reward verification, and ignoring RFT's stability advantage over GRPO. --- # 15.1 — The Problem: Reasoning Is Expensive FT14 gave you GRPO — Group Relative Policy Optimization on verifiable rewards. It works. It is also the most expensive, most unstable thing in this course. GRPO is a real RL loop: a policy model, a rollouts buffer, a reward signal, a critic-free advantage estimate, a KL anchor to a reference, and the whole machinery of policy-gradient training that loves to diverge when the reward landscape flattens, the KL coefficient is mis-set, or the rollout distribution collapses onto a single high-reward string. The question FT15 answers is direct: **can you get reasoning into a small model without paying the full RL cost?** The answer is yes, two ways, and the open ecosystem used both to catch up to the frontier reasoning labs in a matter of months: - **CoT distillation** — borrow reasoning traces from a strong teacher and SFT the student on them. No RL. This is the move that produced DeepSeek-R1-Distill-Qwen-32B. - **Rejection-sampling fine-tuning (RFT / RAFT)** — generate candidates with the current model, keep only the ones the reward accepts, retrain on the winners. A middle path between distillation and RL: you still need a reward, but you avoid the instability of policy gradients. Both are legitimate *terminal* techniques, not merely warmup steps. That distinction is load-bearing — it is the finding from arXiv:2504.11343 that this module hinges on. --- # 15.2 — CoT Distillation: Borrow the Reasoning ## The mechanism Chain-of-thought distillation is structurally simple. You have a strong reasoning teacher — a frontier model with thinking enabled (DeepSeek-R1, OpenAI o1, or a Claude/GPT-class model invoked with extended thinking). You give it problems. It produces solutions that include an explicit reasoning trace, conventionally wrapped in `…` tags. You collect those traces, curate them (filter for correctness, deduplicate, balance difficulty), and SFT a smaller student on the resulting `(problem, think-trace, answer)` tuples. The student learns to *emit* the reasoning trace as part of its generation. Two things to notice immediately. First, **this is SFT, not RL.** There is no reward model in the training loop, no policy gradient, no KL anchor. The teacher did the expensive search once at data-generation time; the student just imitates. The compute profile is that of an ordinary SFT run, not an RL run. Second, **the student is learning to reason in the teacher's style.** The trace is part of the target text, so the student is trained to *produce* the chain of thought token-by-token. At inference, the student generates its own `` block before committing to an answer. This is what makes distillation produce a genuine reasoning model, not a model that merely memorized answers. ## The headline result: DeepSeek-R1-Distill-Qwen-32B The result that made distillation the standard, accessible technique is DeepSeek-R1-Distill-Qwen-32B. DeepSeek took their full R1 model (a frontier reasoning model trained with RL on verifiable rewards — the FT14 lineage) and used it as a teacher. They generated a large corpus of reasoning traces, curated them down to roughly **800K samples** spanning math, code, and general reasoning, and SFT'd an off-the-shelf Qwen-32B base on those traces. No RL on the student. Just SFT on the distilled data. The result: **R1-Distill-Qwen-32B beats OpenAI o1-mini on several benchmarks**, including AIME 2024 and MATH-500. A 32B model, trained with nothing more than supervised fine-tuning on data generated by a stronger model, outperforms a frontier compressed-reasoning product. That single fact is why the open ecosystem got small reasoning models cheaply. Distillation became the default recipe: find or train a strong teacher, generate traces, curate, SFT. The R1-Distill family (Qwen-1.5B through 70B, Llama-8B and 70B) demonstrated that the recipe scales down to consumer-GPU sizes and up to frontier-adjacent quality. This is the clearest empirical statement in the module: **for getting reasoning into a small model, distillation is the cheapest path that works at scale, and it works without RL.** ## Why distillation is steering, not teaching Note the FT00 framing. The student is not learning *new reasoning ability* the base did not have — the Qwen-32B base already saw enormous amounts of math and code during pretraining. Distillation is steering the student to *reliably emit* a structured reasoning trace and to *use* the reasoning pathways the base already possesses. The teacher's trace is the steering wheel. This is why a 32B distilled model can punch above its pretraining-weight class: the capability was latent; distillation routes probability mass through it reliably. The corollary, and the limit: **distillation cannot make the student exceed the teacher.** You are imitating the teacher's traces; the student's ceiling is the teacher's ceiling (minus whatever the smaller model forgets). If you need to push *beyond* what the teacher can produce, you need RL on the student's own rollouts — that is FT14, or RFT (15.3). ## When distillation is the right call Distillation is the right call when: - You have access to a strong reasoning teacher (a frontier API with thinking, or your own R1/o1-class model). - You want a small, cheap reasoning model for deployment. - You do not need the student to exceed the teacher. - You want to avoid RL instability entirely. It is the wrong call when you have no strong teacher (you cannot distill reasoning the teacher does not have), or when the whole point is to push past the teacher's quality. --- # 15.2.1 — The Exception: Same-Size Expert-to-Merged Distillation (RI-003) *The one setting where the "student cannot exceed the teacher" corollary above is wrong. The GLM-5 pattern. Read this as the qualification to the ceiling claim, not a contradiction of the rest of the module.* ## Why the ceiling claim has a hole The statement "distillation cannot make the student exceed the teacher" is correct for the **single-teacher, small-student** setting this module has covered so far: R1-Distill-Qwen-32B imitates R1's traces, so its reasoning ceiling is R1's ceiling (minus forgetting). One teacher, one smaller student, behavior copied. That is the entire R1-Distill family. There is a second distillation regime that the ceiling claim does not cover, and that the open-weights frontier used in 2025–2026: **distillation *between same-size models* as a route to a *merged* model that exceeds any single expert.** Here the teacher is not one stronger model — it is several same-size peers, each a specialist, and the "student" is a merge that distills from all of them simultaneously. The merge can end up better than every expert on every expert's specialty, because each expert's weakness is covered by another expert's strength. This is the pattern behind the GLM-4.5 / GLM-5 family's reasoning post-training (Zhipu AI), and it appears in similar form across the 2025 same-size expert-merge literature. It is the distillation analog of mixture-of-experts, done at the *data* level rather than the *architecture* level. ## The pattern You have *k* same-size specialists — say a math specialist, a code specialist, a tool-use specialist, a multi-language specialist — each produced by domain-specific fine-tuning (SFT or RL) from the same base. None is a good general model on its own; each is excellent at its specialty and mediocre elsewhere. The job is to produce one general model that retains each specialty without catastrophic forgetting of the others. Distillation is the tool. ```mermaid flowchart TB Base["Shared base model
(same size for all)"] E1["Math specialist
(SFT/RL on math)"] E2["Code specialist
(SFT/RL on code)"] E3["Tool-use specialist
(SFT/RL on tools)"] Ek["... k specialists ..."] D1["Math traces
(verified)"] D2["Code traces
(tests pass)"] D3["Tool traces
(task success)"] Dk["... k trace sets ..."] Merge["Merged trace corpus
(balanced, de-duped,
coverage across all k)"] Student["Same-size merged student
SFT on the merged corpus"] Base --> E1 & E2 & E3 & Ek E1 --> D1 E2 --> D2 E3 --> D3 Ek --> Dk D1 & D2 & D3 & Dk --> Merge Base --> Student Merge --> Student style Base fill:#14141f,stroke:#9494a0,stroke-width:1.5px,color:#e4e4e8 style E1 fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8 style E2 fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8 style E3 fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8 style Ek fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8 style D1 fill:#08080c,stroke:#3a3a48,stroke-width:1px,color:#9494a0 style D2 fill:#08080c,stroke:#3a3a48,stroke-width:1px,color:#9494a0 style D3 fill:#08080c,stroke:#3a3a48,stroke-width:1px,color:#9494a0 style Dk fill:#08080c,stroke:#3a3a48,stroke-width:1px,color:#9494a0 style Merge fill:#14141f,stroke:#f0a868,stroke-width:1.5px,color:#e4e4e8 style Student fill:#14141f,stroke:#5eead4,stroke-width:2px,color:#5eead4 ``` **Type:** Architecture. **Purpose:** Contrast same-size expert-to-merged distillation with the single-teacher small-student distillation of 15.2. **Reading the diagram:** the teacher is not one stronger model — it is *k* same-size specialists. The student is the *same size* as each specialist. The merged corpus is where the anti-forgetting magic happens: each specialist contributes traces only in its area of strength, so the student is never asked to imitate a specialist outside its lane. The recipe: 1. **Train k same-size specialists** from one base, each on a single domain (math, code, tools, etc.). Verifiable-reward RL (FT14) or domain SFT (FT12) per specialist. The specialists share the base and the parameter count — they are *peers*, not a size hierarchy. 2. **Generate verified traces from each specialist, in its lane.** Math traces from the math specialist (checked against ground truth); code traces from the code specialist (tests pass); tool-use traces from the tool specialist (task succeeds). Critically — generate *only* in the lane each specialist is strong in. Do not ask the math specialist for code traces; those would be the mediocre traces the ceiling claim warns about. 3. **Merge the trace corpora.** Balance the domains (so the student sees math/code/tools in the ratio you want for the general model), deduplicate (FT06), and decontaminate (FT06). The merged corpus is the steering wheel. 4. **SFT the same-size student on the merged corpus.** The student has the same parameter count as each specialist. It is not smaller. It is learning the *union* of the specialties from the union of the verified traces. ## Why this beats any single expert (and why it is still distillation) The merged student can exceed *each individual specialist* on that specialist's own domain — counterintuitive, but the mechanism is simple. The math specialist, trained only on math, has partially forgotten code/tool/multi-language capability from the shared base (catastrophic forgetting, FT12.3). The merged student, trained on math traces *plus* code traces *plus* tool traces, never forgets any lane because every lane is in the steering data. So the merged student retains the base's general capability *and* acquires each specialty — and on math benchmarks it can match or exceed the math specialist, because the math specialist was operating with a partially-forgotten general substrate while the merged student operates with the full one. This is why the GLM-5 pattern inverts the ceiling claim. The ceiling claim assumes one teacher; here there are k. The ceiling claim assumes the student is smaller; here it is the same size. The ceiling claim assumes the student imitates one trace distribution; here it imitates the union of k verified distributions. Each assumption is what makes "student ≤ teacher" hold, and each is relaxed in the expert-merge regime. It is still distillation — SFT on traces generated by other models, no RL on the student — and it is still steering (FT00): the merged student routes probability mass through reasoning pathways the shared base already had, now reliably surfaced across all k lanes simultaneously. No new knowledge. Better coverage. ## When to use it Same-size expert-to-merged distillation is the right call when: - You need **one general model** that is strong across multiple distinct specialties, not a specialist. - You can afford to train **k specialists** (each is a single-domain SFT/RL run — cheaper than one giant mixed-domain run, and parallelizable). - **Catastrophic forgetting** is the enemy — the specialist-on-its-own-lane trace generation is precisely the anti-forgetting mechanism. - The specialties are **verifiable** (math, code, tools) so the trace generation can be filtered by reward. Without verification, you cannot guarantee each specialist contributes only its strong traces. It is the wrong call when you genuinely need one frontier reasoning teacher distilled into a smaller cheap model (use 15.2 R1-Distill-style distillation), or when your domains are not separable into specialties (there is no clean "math lane" to generate traces in). ## The corrected ceiling claim Restated precisely, the limit is: > **Distillation cannot make a student exceed the *best available teacher on a given task*.** In the single-teacher small-student setting (15.2), that bound is the one teacher's quality. In the same-size expert-merge setting (this section), the "best available teacher" on each task is the *specialist for that task* — and the merged student approaches the max over specialists on each task simultaneously, which no single specialist can do. The student does not exceed the best teacher; it approaches the *envelope* of all the teachers, and that envelope exceeds any one of them. The earlier module-level assertion — "distillation cannot make the student exceed the teacher" — is correct for the single-teacher case it was describing. Read this section as the qualification: the moment "the teacher" becomes a set of peer specialists and "the student" is a same-size merge, the relevant bound is the envelope, not any single teacher, and distillation reaches it. --- # 15.3 — Rejection-Sampling Fine-Tuning (RFT / RAFT) ## The mechanism Rejection-sampling fine-tuning — RFT, also called RAFT (Rejection sampling Fine-Tuning) in the arXiv:2504.11343 paper — is the middle path. You do not have a teacher whose traces you can copy, or you want the student to learn from *its own* attempts. So you let the current model generate. The loop, iterated: 1. **Generate.** For each problem in your dataset, sample N candidates from the current model (temperature > 0, several per problem). 2. **Filter by reward.** Score each candidate with a verifiable reward — does the math check, does the code pass tests, does the answer match. Keep only the positively-rewarded candidates. 3. **Retrain on the winners.** SFT the model on the accepted candidates. This becomes the new current model. 4. **Repeat.** Generate with the improved model, filter, retrain. Each round, the model is trained only on its own successes. This is the same data-generation machinery as GRPO (rollouts, verifiable reward) but a radically different training step. GRPO uses the rollouts to compute a policy gradient — it learns from *both* the successes and the failures, pushing up the good and down the bad, with all the instability that implies. RFT throws away the failures and supervises on the successes only. You keep the reward signal; you discard the gradient noise. ## The arXiv:2504.11343 finding: simpler than expected The finding that gives RFT its place in this module is from arXiv:2504.11343. The authors set up a careful comparison: rejection-sampling fine-tuning versus GRPO and PPO on math reasoning, same base model, same data, same compute. The expectation in the field was that full RL (GRPO, PPO) would dominate — that rejection sampling was a warmup, a poor man's RL. The result: **a simple rejection-sampling baseline is surprisingly competitive with GRPO and PPO on math reasoning.** Not always equal, not always better, but competitive — close enough that the simplicity and stability advantages of RFT make it a legitimate choice, not merely a stepping stone. The lesson, and it is the module's second load-bearing claim: **simpler methods should not be dismissed.** RFT is cheaper than full RL (no policy-gradient machinery, no critic, no KL tuning), more stable (you are doing SFT, which does not diverge the way policy gradients can), and a legitimate standalone technique. It is not just a warmup step before you "graduate" to GRPO. For many reasoning tasks, RFT is the destination. ## Why RFT is more stable than GRPO The stability advantage is not incidental — it is structural. GRPO's policy gradient is `advantage × log-prob`, summed over rollouts. When the rollout distribution collapses (the model finds one high-reward string and keeps generating it), the gradient becomes degenerate. When the KL coefficient is mis-set, the model drifts from the reference and the reward signal decouples from genuine improvement. When the reward landscape is flat (few rollouts get any reward), the advantage estimates are noise. Each of these is a real, observed failure mode of RL fine-tuning. RFT has none of these. The training step is SFT — cross-entropy on accepted candidates. SFT does not diverge in the way policy gradients do. The worst case in RFT is that you accept too few candidates and the training set is small, or you accept bad candidates because the reward is wrong — both are data-quality problems you can diagnose by inspecting the accepted set. The worst case in GRPO is a silent mode collapse that ruins the model while the reward looks fine. RFT trades expressivity (it cannot learn from failures) for diagnosability (what you train on is exactly what you accepted). ## When RFT is the right call RFT is the right call when: - You have a verifiable reward (math, code, unit-testable logic) but no strong teacher. - You want the model to improve on its own rollouts (self-improvement without a teacher ceiling). - You want to avoid RL instability but still push past plain SFT. - Your compute budget cannot absorb a full RL run's iteration cost. It is the wrong call when the reward is not verifiable (RFT needs the reward to *filter* — a noisy reward means you train on garbage), or when you specifically need the expressivity of learning from failures (GRPO's edge on the hardest exploration problems). --- # 15.4 — The Decision: Distillation vs RFT vs GRPO This is the judgment the module exists to teach. Three techniques, one goal — reasoning in a small model. Choose by three axes. ## Axis 1: Do you have a strong teacher? - **Yes.** Distillation is on the table. It is the cheapest path and you should consider it first. - **No.** Distillation is off the table. You must generate your own signal — RFT or GRPO. ## Axis 2: Is your reward verifiable? - **Yes (math, code, logic with checks).** RFT and GRPO are both available. - **No (preference, style, open-ended).** Neither RFT nor GRPO applies cleanly. You are in preference-optimization territory (FT13), not reasoning FT. ## Axis 3: Compute budget and stability requirements - **Constrained budget, must-not-diverge requirement.** RFT. SFT stability, reward-filtered data, no policy-gradient machinery. - **Large budget, willing to tune KL and babysit rollouts, need to push the frontier.** GRPO. The most powerful, the most expensive, the most likely to find improvements RFT cannot. ## The decision, in one line each - **Distillation** — cheapest, if you have a strong teacher and do not need the student to exceed it. - **RFT** — middle ground; generates its own data, filters by reward, no RL instability. - **GRPO** — most powerful, most expensive, for pushing beyond what distillation/RFT achieve. A common, defensible stack: distill first (cheap, gets you most of the way), then RFT to self-improve on the areas the teacher was weak, then GRPO only if RFT plateaus and you have the budget. Each stage is optional; many production reasoning models stop at distillation. --- # 15.5 — The Thinking-Mode Question: Hybrid Models A reasoning model that *always* emits a long `` block pays a latency tax on every query — including the trivial ones where the answer is obvious. The frontier's answer is the **hybrid model**: a single model that can toggle thinking on or off, controlled by a flag in the prompt or the generation config. Qwen3 is the canonical example. A Qwen3 model can be invoked in thinking mode (emit the trace, then answer) or non-thinking mode (answer directly), selected per request. The toggle is not a separate model — it is the same weights, steered by a control token or system instruction. This lets one deployed model serve both the cheap fast path (non-thinking, for "what's the capital of France") and the expensive accurate path (thinking, for "prove this integral converges"). The deeper idea is the **thinking budget** — a mechanism for adaptive compute. Rather than a binary toggle, the model is given a token budget for thinking: think for up to N tokens, then answer. This lets the model spend more compute on hard problems and less on easy ones, allocating inference cost to where it buys accuracy. Thinking-budget mechanisms are how hybrid models avoid the latency tax without giving up reasoning quality on the hard cases. The implication for fine-tuning: a hybrid model is trained on a *mixture* of thinking and non-thinking traces, so it learns both modes and the ability to choose. Distillation data for a hybrid model includes both long-trace examples (thinking on) and direct-answer examples (thinking off), plus ideally examples where the difficulty warrants one or the other. This is more involved than pure CoT distillation, but it produces a model that is deployable across the whole latency/accuracy tradeoff. --- # 15.6 — Anti-Patterns ### Distilling from a weak teacher Garbage in, garbage out. The student's reasoning ceiling is the teacher's. If you distill from a model that cannot actually reason well, you get a small model that imitates mediocre reasoning. The R1-Distill result worked because R1 is a frontier reasoner. Distilling from a 7B base that itself cannot do the math will produce a student that confidently emits wrong chains. **The teacher must be genuinely strong on the target domain.** ### RFT without reward verification RFT's entire signal is the filter. If the reward is noisy — a fuzzy matcher, a lenient LLM judge, a test suite with false positives — you train on incorrectly-accepted candidates and the model learns to produce plausible-looking wrong answers. RFT demands a *verifiable* reward: math that checks, code that passes strict tests, logic with a ground truth. A preference-style reward is not sufficient for RFT; it is a different regime (FT13). ### Ignoring RFT's stability advantage over GRPO The most expensive anti-pattern: reaching for GRPO by default because "RL is more powerful." On many reasoning tasks, RFT is competitive (arXiv:2504.11343), cheaper, and dramatically more stable. If you skip RFT and go straight to GRPO, you pay the full RL cost — compute, tuning, divergence risk — for gains that RFT may already capture. **Try RFT first when the reward is verifiable; escalate to GRPO only when RFT plateaus with budget remaining.** ### Treating distillation as a warmup for RL only Distillation is a terminal technique. R1-Distill-Qwen-32B is a shipping model, not a warmup checkpoint. If you treat distillation as merely a pre-RL initialization step, you will spend RL compute to gain what may be marginal improvements over a strong distilled baseline. Distill, *evaluate the distilled model against your target*, and only escalate if it falls short. ### Distilling traces the student cannot imitate If the teacher's traces are too long, too stylistically idiosyncratic, or use tokens the student's tokenizer handles poorly, the student will learn a degraded imitation. Curate the traces: trim, normalize, ensure the format is something the student can reproduce. Distillation is imitation; imitation requires the target to be imitable. --- ## Key Terms | Term | Definition | | --- | --- | | **CoT distillation** | SFT a smaller student on chain-of-thought reasoning traces generated by a stronger teacher model; no RL in the training loop | | **Teacher model** | The strong reasoning model (R1, o1, Claude/GPT with thinking) that generates the traces distilled into the student | | **`` trace** | The explicit reasoning block a reasoning model emits before its answer; the distillation target | | **DeepSeek-R1-Distill-Qwen-32B** | The 32B model distilled from R1 via SFT on ~800K curated reasoning samples; beats o1-mini on several benchmarks — the headline distillation result | | **Rejection-sampling FT (RFT / RAFT)** | Iteratively SFT on only positively-rewarded candidates generated by the current model; generate → filter by reward → retrain on winners | | **Verifiable reward** | A reward that can be checked exactly (math correctness, code passing tests); required for RFT and GRPO | | **arXiv:2504.11343** | The RAFT paper; finding that simple rejection-sampling FT is surprisingly competitive with GRPO and PPO on math reasoning | | **Hybrid model (thinking toggle)** | A single model that can run in thinking or non-thinking mode per request (e.g., Qwen3), avoiding the always-on latency tax | | **Thinking budget** | A token/length budget for thinking, allowing adaptive compute — more on hard problems, less on easy ones | | **GRPO** | Group Relative Policy Optimization (FT14); full RL on verifiable rewards; most powerful, most expensive, most unstable of the three | | **Same-size expert-to-merged distillation** | Distillation from k same-size peer specialists (each trained on one domain) into a same-size merged student via SFT on the union of verified traces. The GLM-4.5/5 pattern. Inverts the single-teacher ceiling: the merged student approaches the *envelope* of all specialists. | | **The envelope bound** | The corrected ceiling claim: distillation cannot make a student exceed the *best available teacher on a given task*. With one teacher that is the teacher's quality; with k peer specialists it is the max over specialists — and the merged student reaches it on all tasks simultaneously. | | **Catastrophic forgetting (specialist context)** | Why each same-size specialist is weak outside its lane: single-domain training partially erases the shared base's other capabilities. The merged student avoids this because every lane is in its steering data. | --- ## Lab Exercise See `07-lab-spec.md`. The "Distill Reasoning" lab: generate CoT traces from a strong teacher on 200 GSM8K math problems, SFT a small student (MiniCPM5-1B or Qwen2.5-1.5B) on those traces, and measure the GSM8K lift versus the un-distilled base. Consumer-GPU. Stretch goal: an RFT variant that filters the student's own candidates. --- ## References 1. **DeepSeek-AI (2025)** — *DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning*. arXiv:2501.12948. The R1 paper; the distillation recipe and the R1-Distill model family (incl. R1-Distill-Qwen-32B) are described here. The distilled weights and the distillation data methodology are on the [DeepSeek-R1 GitHub](https://github.com/deepseek-ai/DeepSeek-R1). 2. **Pang et al. (2025)** — *Iterative Reasoning Preference Optimization via Rejection Sampling Fine-Tuning* (RAFT). arXiv:2504.11343. The finding that rejection-sampling FT is surprisingly competitive with GRPO and PPO on math reasoning; the empirical basis for RFT as a terminal technique. 3. **Yang et al. (2025)** — *Qwen3 Technical Report*. The hybrid thinking-toggle model; thinking-budget mechanisms for adaptive compute. 4. **RLHF Book** — *Reasoning chapter*. The broader framing of reasoning fine-tuning, RLHF, and the distillation-vs-RL tradeoff in the open ecosystem. 5. **FT14 — GRPO and Verifiable Rewards** (prerequisite). The full-RL counterpart to this module; GRPO is the third option in the decision tree. 6. **FT00 — The Steering Stack**. The thesis this module rests on: distillation is steering (routing the student through reasoning pathways it already has), not teaching new reasoning ability. 7. **Zhipu AI / GLM Team (2025)** — *GLM-4.5 / GLM-5 technical reports and model cards*. The same-size expert-to-merged distillation pattern (15.2.1): peer specialists trained per-domain, verified traces generated in-lane, merged into a same-size general student. The open-weights existence proof that the merge can exceed any single specialist by reaching the envelope. 8. **TaskMixture / data-level merging literature (2024–2025)** — The broader finding that balancing verified traces from multiple same-size teachers mitigates catastrophic forgetting more effectively than training one model on a mixed corpus, because each teacher contributes only its strong traces. The empirical basis for the anti-forgetting claim in 15.2.1.