Synthetic Data Generation

Module FT05 · Course 3 — LLM Fine-Tuning Masterclass

75 minutes · 7 sub-sections: thesis · teacher distillation · Magpie · Evol-Instruct · distilabel · decontamination · filtering

Pillar 1 (Data) · Prereq: FT04. Where SFT/preference data actually comes from.

Pillar 1 — Data

The thesis — one sentence

The gains are in filtering, not generation.
A 500-sample set you actually read and curated will steer better than 50,000 raw teacher outputs you did not look at.

The data-side complement of FT00. Fine-tuning steers via a low-rank update pointed by the data. A noisy direction (unfiltered data) is a noisy steer, no matter the optimizer.

The steering wheel is the curated set, not the raw pile. LIMA's 1k curated beat Alpaca's 52k unfiltered. Magpie's filtering 1M→300K improved quality more than doubling the raw.

Why this matters immediately

Mystery	Why the thesis explains it
Why LIMA worked with 1,000 examples	Hand-curated. Quality and diversity, not volume. The model was already capable.
Why "100k GPT-4, ship it" underperforms	Duplicates, low-info responses, style collapses, contamination. Optimizer steers toward all of it.
Why Magpie emphasizes its filtering pipeline	"From scratch" generation is half; disciplined filtering is the other half.
Why FT06 is load-bearing	This module generates; FT06 cleans. Both required. Neither alone is enough.

The two-approach landscape

Approach	Mechanism	When to use
Strong-teacher distillation	Seed prompts → strong model (GPT-4/Claude/R1/70B) generates responses	You have a target prompt distribution (real user queries, a domain)
Magpie self-synthesis	Prompt aligned model with pre-query template alone — it generates both sides, no seeds	You want broad diversity from scratch, no seed prompts

Both produce (instruction, response) pairs. Both require the same downstream cleaning. The choice: where does the distribution come from — an external teacher, or the aligned model's own prior?

Strong-teacher distillation

Shines

Target prompt distribution — real user queries, specific domain, specific format (JSON tool calls).

Leaks — style leakage

7B trained on 50k GPT-4 responses adopts GPT-4's verbosity, hedging, list-bias.

Mitigate: mix 2–3 teachers, post-process tells.

Leaks harder — contamination. GPT-4 has seen MMLU. Ask it for "exam-style questions" → eval-shaped items in your training set. This is the decontamination obligation (FT06).

Magpie — the key insight

arXiv:2406.08464 · ICLR 2025 · self-synthesis from nothing

Aligned instruct models will generate instructions when prompted with their pre-query template alone — the user turn is empty, no seed prompts.

RLHF/DPO made instruction-responding the default. With no user content, the model surfaces a plausible query from its instruction prior — diverse, natural, uncontaminated by any seed set.

The headline: Magpie-Pro-Filtered (300K from 1M raw) as SFT for Llama-3-8B beats the official Llama-3-8B-Instruct on MT-Bench and AlpacaEval 2 — self-generated data from a model in the same family. Magpie-Air from 8B (cheap); Magpie-Pro from 70B (quality).

Magpie — the two-pass mechanism

Same aligned model, both passes. No seeds, no external teacher.

Pass 1 — instruction generation
  feed pre-query template (user turn EMPTY)
  <|user|>
              <- empty
  <|assistant|>
  -> model samples: "Explain TCP handshake\n..."

Pass 2 — response generation
  place instruction in user turn PROPERLY
  <|user|>Explain TCP handshake<|assistant|>
  -> model samples: "The three-way handshake...\n..."

Result: a self-consistent (instruction, response) pair — same model generated both sides. Why Magpie beat Alpaca/Evol-Instruct/UltraChat: no seed-prompt bias (samples the full prior), natural self-consistency, and the filtering pipeline still applies (dedup, quality, decontam — half the contribution).

Evol-Instruct & the post-Magpie landscape

WizardLM (arXiv:2304.12201). Iterative LLM-driven complexification. Still used; now a special case, not a default.

In-depth evolving

Add constraints, deepen reasoning, require more steps.

In-breadth evolving

Topic-related variant → diversify.

Why superseded as default: (1) Magpie samples difficulty naturally — the prior spans easy to hard, no loop needed. (2) Direct teacher generation is simpler if you have prompts. Still right for hardness amplification (reasoning models, Pillar 4). distilabel has an EvolInstruct step.

distilabel — the standard framework

Argilla, open-source (MIT), HF-native. Four pluggable pipeline stages:

Stage	What it does
Generate	Magpie step OR teacher step. Encapsulates the pre-query-template trick per model.
Judge	Stronger model scores/ranks → preference pairs (DPO) + quality scores (filter).
Filter	Length · format · language · regex · decontamination. Where FT06 plugs in.
Dedup	MinHash on instruction n-grams; keep one per cluster. Biggest volume reduction.

Output: a Distiset (HF Dataset) ready for FT11. Use it: reproducibility (DAG w/ checkpointing), non-trivial judge step, fiddly per-model Magpie templates, composes with FT06.

The decontamination obligation

Before training, remove items that overlap with benchmarks you'll evaluate against (MMLU, GSM8K, MT-Bench, your held-out eval). Non-negotiable for honest evals. Detailed in FT06.

If a training item is near-identical to an eval item, the eval is compromised — the model has seen the answer. The number is fake.

Generation-time concern, not just post-hoc: Don't ask the teacher for "MMLU-style questions." Don't seed with benchmark items. Don't use them as few-shot examples. Fix the eval set BEFORE generation; check generation against it.

Generate first, pick evals later → you'll pick evals your data does well on. That's a press release, not an eval.

The data-quality funnel & anti-patterns

The funnel

Raw — ~20–50k

Dedup — ~10–20k (~50%)

Quality — ~5–10k (~50%)

Diversity — ~3–6k (~30%)

Decontam — ~2.8–5.8k

Clean — ~2.5–5k

Generate 4–10x; filter to clean. The ratio is the price of a clean steering wheel.

Anti-patterns

Generate-without-filter. Ship raw 50k. Cardinal sin.

No decontamination. Eval is fake.

Single-teacher style leakage. Mix 2–3 teachers.

Ignored diversity. 70% one topic. Cap per-cluster.

What you can now do

State the thesis — gains are in filtering — and defend it with LIMA and Magpie's ablations.
Contrast strong-teacher distillation and Magpie self-synthesis, and choose between them for a given target behavior.
Describe Magpie's two-pass mechanism and why the empty pre-query template works.
Place Evol-Instruct correctly — superseded as default, still right for hardness amplification.
Name distilabel as the standard framework and its four stages.
State the decontamination obligation and why it's non-negotiable.

Lab: generate 500 samples two ways (Magpie + teacher), dedup, compare on 20-sample review.

Next: FT06 — Dedup, Filter, Decontaminate (the cleaning half of the funnel).