DeepSeek-R1

Module FTDD-07 · Course 3 — LLM Fine-Tuning Masterclass

45 minutes · 4 sub-sections: R1-Zero · The R1 Pipeline · The Distillation · Why R1 Is The Reference

The reasoning distillation reference. Proof that reasoning is steering, not teaching.

Deep-Dives

The thesis, demonstrated

Reasoning is a steering problem, not a knowledge problem.

DeepSeek-R1 is the strongest published evidence for the course thesis:

R1-Zero — pure RL on a base model, zero supervised traces. Reasoning emerged.
R1 — the four-stage pipeline that made it shippable.
The distillation — SFT-only on 800K traces. R1-Distill-Qwen-32B beat o1-mini.

If reasoning were knowledge, none of this would work. It is capability the base already had.

R1-Zero — the proof

GRPO on a base model. Zero supervised reasoning data.

Input	What it was
Base	DeepSeek-V3-Base (671B MoE, 37B active). No instruction tuning.
Scaffold	A minimal `<think></think>` template. No demonstrations.
Rewards	Verifiable only: math correctness, code execution, LeetCode verdicts. No reward model.

What emerged: long chain-of-thought grew; self-correction ("wait, let me reconsider"); re-derivation; the "aha moment." Unprompted. The reward made deliberation pay off, and deliberation is what grew.

Why R1-Zero proves the thesis

If reasoning were KNOWLEDGE

GRPO on a base with no reasoning traces could not produce it.

You would need supervised demonstrations or continued pretraining on reasoning corpora.

If reasoning were STEERING

The base already has the capability (saw it in pretraining). RL redirects probability mass so the reasoning pathways become the chosen behavior.

This is exactly what happened.

R1-Zero was a proof, not a product. Mixed languages, run-on chains, correct-but-ugly output. R1 exists to fix this.

R1 — the four-stage pipeline

STAGE 1 · Cold-start SFT · anchors format, fixes R1-Zero's messiness

↓

STAGE 2 · Reasoning RL (GRPO) · verifiable rewards; scales capability

↓

STAGE 3 · Rejection-sampling SFT · generate, filter correct + readable → ~800K clean traces

↓

STAGE 4 · Final RL · whole distribution; turns reasoner into product

Each stage fixes a specific failure of the one before. Stage 3 is the source of the distillation data.

Why each stage exists

Stage	Fixes
1. Cold-start SFT	R1-Zero's unreadable chains. Anchors tone, single language, clean `<think>` structure.
2. Reasoning RL	The cold-start only shaped format; RL fills the shape with reward-driven deliberation. Capability scales.
3. Rejection-sampling SFT	The RL'd model is good but inconsistent. Rejection sampling yields clean traces that reliably reproduce its best behavior.
4. Final RL	Stage 2 optimized only reasoning. Stage 4 aligns the full assistant distribution (helpfulness, format).

The distillation

SFT-only. No RL on the students.

~800K curated traces from Stage 3 → plain SFT into six dense bases. Qwen-1.5B / 7B / 14B / 32B, Llama-8B / 70B.

The result that made the field notice:

R1-Distill-Qwen-32B beat OpenAI o1-mini on AIME 2024.
A 32B dense model, trained by SFT only, outperformed a frontier reasoning model.

This is why CoT distillation went from "a technique some labs used" to the default way to build a reasoning model.

Why RL on the students gave marginal returns

DeepSeek explicitly reports: applying RL on top of the distilled students yielded marginal gains.

The interpretation

The teacher's curated traces were already a near-optimal steering signal. They showed the student the high-reward trajectories directly.

The practitioner takeaway

Distill first. Reach for RL on the student only if you have a verifiable-reward domain the traces under-cover — and the compute to spend.

Distillation = the thesis, other direction

If reasoning were a capability the small models lacked, distilling traces could not transfer it.

The fact that SFT on traces did transfer reasoning is evidence the small bases already had latent capability. The traces steered them to express it.

The student inherits the behavior, not new ability.

R1 vs Qwen3 — the two references

R1 — distillation reference

Teacher pipeline + SFT-only transfer.

Use when: building a reasoner by distilling from a teacher.

Qwen3 — hybrid reference

One model, fused thinking + non-thinking modes, thinking-budget adaptive compute.

Use when: shipping a production model that adapts compute to the question.

Two canonical references. They prove different things. Know both.

What the field took from R1

CoT distillation is the practical path. Before R1: assume you need full RL. After R1: get a strong teacher, curate traces, SFT your student.
Verifiable rewards beat reward models for reasoning RL. If you can verify, do not approximate with a judge.
RL-on-base proves the capability is latent. R1-Zero is the reference everyone cites for "reasoning is steering."

Published in Nature (s41586-025-09422-z). Weights open at github.com/deepseek-ai/deepseek-r1.

Anti-patterns

Assuming RL is required for a reasoning student. R1's own ablation: SFT-only distillation captures most of the gain. Distill first; RL the student only with a verifiable domain and a clear gap.

Dismissing R1-Zero for its messiness. The mixed languages are irrelevant to what it proved. R1's cold-start stage is the fix — it builds on emergence, not contradicts it.

Distilling from a weak teacher. The teacher's quality is the ceiling. Weak teacher → mediocre traces → mediocre reasoner.

What you can now do

Explain why R1-Zero is the strongest evidence that reasoning is steering.
Draw the four-stage R1 pipeline and what each stage fixes.
Distinguish the distillation from the teacher pipeline and why it beat o1-mini.
Defend publishing both the proof (R1-Zero) and the product (R1).

Next: FTDD-08 — Qwen3 (the hybrid thinking/non-thinking reference)