CoT Distillation and Rejection-Sampling FT

CoT distillation is the cheapest path to a reasoning small model that works at scale: a strong reasoning teacher (DeepSeek-R1, o1, Claude/GPT with thinking) emits ... traces, the traces are curated (verified correct, deduplicated, difficulty-balanced), and a smaller student SFTs on them. No RL in the training loop — no reward model, no policy gradient, no KL anchor. The teacher did the expensive search once at data-generation time; the student imitates. This is how the open ecosystem got small reasoning models cheaply. Distillation is steering (FT00): the student's base already has latent math/code capability from pretraining; the teacher's trace routes probability mass through those pathways reliably. The limit: the student cannot exceed the teacher (imitation has a ceiling) — to push past the teacher you need RL on the student's own rollouts (RFT or GRPO).

The headline result: DeepSeek-R1-Distill-Qwen-32B beats OpenAI o1-mini on several benchmarks (incl. AIME 2024 and MATH-500), achieved via SFT on ~800K curated reasoning samples distilled from R1 — NO RL on the student. A 32B model trained with nothing more than supervised fine-tuning on a stronger model's traces outperforms a frontier compressed-reasoning product. This single fact made distillation the default, accessible recipe. The R1-Distill family (Qwen 1.5B–70B, Llama 8B/70B) demonstrated the recipe scales from consumer-GPU sizes to frontier-adjacent quality.

Rejection-sampling fine-tuning (RFT / RAFT) is the middle path: iteratively fine-tune on only POSITIVELY-REWARDED candidates. Generate N candidates per problem with the current model, filter by a verifiable reward (math checks, code passes tests, answer matches), retrain (SFT) on the winners, repeat. Same data-generation machinery as GRPO (rollouts, verifiable reward) but a radically different training step: GRPO computes a policy gradient over ALL rollouts (learns from successes AND failures); RFT DISCARDS the failures and supervises on the successes only. You keep the reward; you discard the gradient noise.

The arXiv:2504.11343 finding: a simple rejection-sampling baseline is SURPRISINGLY COMPETITIVE with GRPO and PPO on math reasoning (same base/data/compute) — close enough that RFT's simplicity and stability make it a legitimate TERMINAL choice, not merely a warmup before 'graduating' to GRPO. RFT is cheaper (no policy-gradient machinery, no critic, no KL tuning) and structurally more stable (the training step is SFT, which does not diverge the way policy gradients do; RFT's worst cases — small accepted set, wrong reward — are diagnosable data problems, whereas GRPO's worst case is a silent mode collapse that ruins the model while the reward looks fine). RFT trades expressivity (ignores failures) for stability + diagnosability. Simpler methods should not be dismissed.

The three-way decision — distillation vs RFT vs GRPO — by three axes: (1) strong teacher available? Yes → distillation (cheapest, consider first). (2) reward verifiable? Yes → RFT & GRPO available; No → FT13 preference territory, not reasoning FT. (3) compute/stability: constrained/must-not-diverge → RFT; large budget/willing to tune KL/push frontier → GRPO. In one line: distillation = cheapest (have teacher, don't need to exceed it); RFT = middle (own data, reward filter, no RL instability); GRPO = most powerful/expensive (push beyond both). A defensible stack: distill first → RFT to self-improve → GRPO only if RFT plateaus and budget remains. Each stage optional; many production reasoners stop at distillation.

Hybrid models (Qwen3) can toggle thinking on or off per request from the SAME weights — controlled by a control token / system instruction, not a separate model — avoiding the always-on latency tax of a reasoning model that always emits a long block. Thinking budgets make this continuous: a token budget for thinking (think up to N tokens, then answer) enables adaptive compute — more on hard problems, less on easy ones. A hybrid student is trained on a MIX of thinking and non-thinking traces, so it learns both modes and the ability to choose.