DeepSeek-R1
The reasoning distillation reference. R1-Zero proves reasoning emerges from RL alone; R1 is the four-stage pipeline that made it reliable; the 800K-trace distillation made chain-of-thought transferable to any base.
Reasoning emerges from RL alone. R1-Zero applied GRPO directly to a base model with verifiable rewards (math, code, LeetCode) and no supervised reasoning data. Long, self-correcting chain-of-thought appeared spontaneously. This is the single strongest evidence that RL steers existing capability rather than teaching new ability.
The R1 pipeline is four stages: cold-start SFT, reasoning RL, rejection-sampling SFT, final RL. R1-Zero was a proof, not a product. It was messy (language mixing, non-readable chains). The cold-start seed anchors format and tone; reasoning RL scales capability; rejection sampling cleans the data; the final RL aligns the full distribution.
Distillation is SFT-only on curated traces — no RL required. R1 distilled its 800K best traces into six dense models (Qwen and Llama bases). R1-Distill-Qwen-32B beat o1-mini on AIME. The student inherits reasoning behavior without ever running an RL loop. This is why CoT distillation is now standard.
RL on the small distilled students did not help. DeepSeek explicitly reports that adding RL on top of the distilled models gave marginal gains — the teacher's traces carried the reasoning. Distillation beat RL for the small models, contradicting the assumption that RL is always necessary for reasoning.