# Teaching Script — Module FT05: Synthetic Data Generation

**Course**: Course 3 — LLM Fine-Tuning Masterclass
**Module**: FT05 — Synthetic Data Generation
**Duration**: ~40 minutes (spoken at ~140 wpm)
**Format**: Verbatim transcript with `[SLIDE N]` cues. Read aloud or use as speaker notes.

---

[SLIDE 1 — Title]

Welcome to module FT05, Synthetic Data Generation. This is the second module of Pillar one, Data — the steering wheel for everything you'll train in Pillars two and three.

In FT04 you learned the formats: how SFT, preference, and chat-template data are structured on disk. This module answers the question that comes before format: where does the data actually come from? For almost everyone watching this, the answer is: you generate it. And the single idea that organizes the whole module is this — the gains are in filtering, not generation.

Let's get to it.

[SLIDE 2 — The thesis]

Here is the thesis. Read it twice. The gains are in filtering, not generation. A five-hundred-sample set you actually read and curated will steer better than fifty-thousand raw teacher outputs you did not look at.

This is the data-side complement of FT00. FT00 said fine-tuning steers rather than teaches. This module adds: steering is only as good as the steering wheel, and the steering wheel is the curated set, not the raw pile. The data points the low-rank update in a direction. A noisy direction is a noisy steer, no matter how good the optimizer.

The evidence is consistent. LIMA showed one thousand hand-curated examples beat Alpaca's fifty-two-thousand unfiltered. Magpie's own ablations showed filtering one-million raw down to three-hundred-thousand improved downstream quality more than doubling the raw would have. Curation has a steeper quality curve than volume, for steering.

[SLIDE 3 — Why this matters immediately]

Once the thesis lands, four things become obvious.

First, why LIMA worked with one thousand examples. The thousand were hand-curated. Quality and diversity, not volume. The model was already capable; the curated set pointed the steering vector cleanly.

Second, why "generate a hundred-k with GPT-4 and ship it" underperforms. The hundred-k contains near-duplicates, low-information responses, style collapses, and often benchmark contamination. The optimizer steers toward all of it.

Third, why the Magpie paper emphasizes its filtering pipeline as much as its generation method. Magpie's headline is "self-synthesis from nothing," but the ablations show the filtering stage contributes a large share of the final quality.

Fourth, why FT06 — dedup, filter, decontaminate — is the load-bearing follow-on. This module teaches you to generate. The next module teaches you to clean. Both are required. Neither alone is enough.

[SLIDE 4 — The two-approach landscape]

Modern synthetic data comes from one of two approaches, often combined.

Strong-teacher distillation: you have prompts, you send them to a strong model — GPT-4, Claude, DeepSeek-R1, or a seventy-billion-parameter model you self-host — and you capture the responses. Optionally the teacher generates the prompts too, which is what Self-Instruct pioneered and Alpaca scaled.

Magpie self-synthesis: you prompt an aligned instruct model with its pre-query template alone — the user turn is empty, no seed prompts — and it generates both the instruction and the response.

Both produce instruction-response pairs. Both require the same downstream cleaning. The choice is about where you want the distribution to come from: an external teacher's notion of what's worth asking, or the aligned model's own latent instruction distribution.

[SLIDE 5 — Strong-teacher distillation]

The workhorse. You have prompts; a strong model answers them.

The pipeline is straightforward. Acquire or generate seed prompts — real user logs are best, hand-written seeds are good, LLM-bootstrapped is acceptable but watch for style collapse. Send each prompt to a strong teacher. Capture the response. Filter, dedup, decontaminate.

Where it shines: when you have a target prompt distribution. Real user queries from your product. A specific domain — medical, legal, code. A specific format — JSON tool calls. The teacher's job is to produce the gold response; your job is to make sure the prompts are representative.

Where it leaks: style leakage. If you train a seven-B on fifty-thousand GPT-4 responses, the seven-B will adopt GPT-4's verbosity, hedging, and list-heavy format. That is usually unwanted. Mitigations: mix teachers — GPT-4 plus Claude plus R1. Strip stylistic tells in post. Or use a smaller, cleaner teacher whose style is closer to your target.

Where it leaks harder: contamination. If your teacher has seen the benchmarks you intend to evaluate against — and GPT-4 has seen MMLU — and you ask it for exam-style questions, you will get eval-shaped items in your training set. That is the decontamination obligation, which we'll get to.

[SLIDE 6 — Magpie: the key insight]

Now the twenty-twenty-four finding that changed the defaults. Magpie. arXiv twenty-four-oh-six-point-zero-eight-four-six-four, ICLR twenty-twenty-five.

The key insight. Aligned instruct models — Llama-3-Instruct, Qwen-two-point-five-Instruct, Mistral-Instruct — are trained, via SFT then RLHF or DPO, to respond to user turns. Their chat template ends with a marker that says "the user is about to speak; produce a response." Magpie's insight: if you feed the model its pre-query template with the user turn empty, the model generates an instruction anyway. It cannot help itself. Its training made instruction-responding the default behavior. With no user content to respond to, it generates plausible user content first.

This is not a hack. It is a direct consequence of how alignment training shaped the distribution. The aligned model has a strong prior over "what kinds of things do users ask?" baked into its weights. Empty-prompt sampling surfaces that prior. You get, for free, the model's latent instruction distribution — diverse, natural, and uncontaminated by any seed set you might have written.

The paper released two main datasets. Magpie-Air, generated from Llama-3-8B-Instruct — cheaper, fits on a consumer GPU, comparable instruction diversity. Magpie-Pro, generated from Llama-3-70B-Instruct — higher-quality responses, the variant used in the paper's headline SFT results.

And here is the headline. Magpie-Pro-Filtered — three-hundred-thousand curated from the one-million raw — used as SFT data for Llama-3-8B produces a model that outperforms the official Llama-3-8B-Instruct on MT-Bench and AlpacaEval two, using only self-generated data from a model in the same family. The same eight-B, tuned on data the seventy-B generated about itself, beats the official eight-B-Instruct tuned on Meta's closed SFT mix.

[SLIDE 7 — Magpie: the two-pass mechanism]

Magpie generates a full instruction-response pair in two passes over the same aligned model.

Pass one, instruction generation. Feed the pre-query template — everything up to and including the user-turn opener, with no user content. Sample. The model generates an instruction, a plausible user query.

Pass two, response generation. Take that generated instruction, place it in the user turn properly — full chat template, instruction as user content, then the assistant-turn opener. Sample again. The model generates the response.

The result is a self-contained instruction-response pair, generated end-to-end by the same aligned model, with no seed prompts, no prompt engineering, and no external teacher.

Why did Magpie outperform Alpaca, Evol-Instruct, and UltraChat? Three reasons, in order of importance. First, no seed-prompt bias — the others start from human-written seeds whose biases propagate; Magpie samples the model's full prior, broader than any seed set. Second, natural response distribution — same model generates both sides, so the pair is self-consistent. Third — and this is the part people miss — the filtering pipeline still applies. Magpie's authors deduped with MinHash, filtered by quality, removed contaminated items. The "from scratch" generation is half the contribution. The disciplined filtering is the other half.

[SLIDE 8 — Evol-Instruct and the post-Magpie landscape]

Evol-Instruct. WizardLM, twenty-twenty-three. Still used, now a special case. Worth knowing because it appears in distilabel and in many open datasets.

The mechanism. Take a seed instruction and ask an LLM to make it harder. Two operators. In-depth evolving: add constraints, deepen the reasoning, require more steps. In-breadth evolving: generate a topic-related but different instruction, to diversify the set. Iterate. Each seed produces a tree of evolved descendants. Then dedup, filter, and use as SFT prompts.

WizardLM's contribution was the observation that hardness is a useful axis to push on. A model tuned on hard instructions follows instructions better than one tuned on easy ones, because the hard ones exercise more of the instruction-following capability.

Why is it now default-superseded? Two things. Magpie samples difficulty naturally — the aligned model's prior already spans easy to hard, you don't need an evolution loop. And direct teacher generation is simpler if you have a target prompt distribution. Evol-Instruct remains the right tool when you specifically want to amplify hardness — for a reasoning-tuned model in Pillar four. distilabel includes an EvolInstruct step for exactly this. But for general SFT, the twenty-twenty-four-onward default is Magpie-style self-synthesis or direct teacher distillation.

[SLIDE 9 — distilabel]

distilabel. Argilla's open-source framework. If you're building synthetic data pipelines in code, you're probably building them in distilabel.

What it standardizes. A Pipeline object with four stages, each pluggable. Generate — a step that calls an LLM, local or remote; Magpie-style two-pass, direct teacher, Evol-Instruct are all generate-step variants, and the Magpie step encapsulates the pre-query-template trick per model. Judge — a step that asks a usually stronger model to score or rank outputs, for preference pairs and quality filtering. Filter — rules: length, format, language, regex, decontamination; this is where FT06 plugs in. Dedup — MinHash on instruction n-grams, keep one per cluster; the biggest volume reduction.

The output is a Distiset — an HF Dataset ready for FT11's trainer.

Why use it instead of rolling your own? Reproducibility — a pipeline is a DAG with a config, you can checkpoint and resume. The judge step is non-trivial — handling ties and position bias is easy to get wrong. Magpie is fiddly — the pre-query template differs per model and getting it wrong is a silent bug. And it composes with FT06 — the filtering and dedup steps are distilabel Step subclasses.

[SLIDE 10 — The decontamination obligation]

Non-negotiable. Detailed in FT06, flagged here because it must be planned at generation time, not bolted on after.

The obligation. Before you train on any synthetic dataset, you must remove items that overlap with the benchmarks you intend to evaluate against. MMLU, GSM8K, HumanEval, MT-Bench, AlpacaEval, your own held-out eval set. If any item in your training data is near-identical to an eval item, your eval is compromised. The model has seen the answer. The number is fake.

This is not hypothetical. The original Alpaca dataset was found to contain near-duplicates of MMLU items. Multiple "model X beats GPT-4 on benchmark Y" claims from twenty-twenty-three and twenty-twenty-four were walked back when contamination was discovered. The Magpie paper explicitly decontaminates against MMLU, GSM8K, and others, and reports the decontamination step's effect on honest eval.

Why is it a generation-time concern? You might think "I'll decontaminate after generation, in FT06." You will — but you should also avoid generating contaminated items in the first place. Don't ask the teacher to generate MMLU-style questions. Don't seed with benchmark items. Don't use benchmark items as few-shot examples. Do plan your eval set first, hold it out, and decontaminate the generated data against it. The rule: the eval set is fixed before generation begins, and generation is checked against it. If you generate first and pick evals later, you will pick evals your data happens to do well on. That's not an eval. That's a press release.

[SLIDE 11 — The data-quality funnel and anti-patterns]

The thesis as a diagram. The data-quality funnel. Generate four-to-ten times what you'll train on. Filter aggressively to the clean subset.

The stages. Raw generation — twenty-to-fifty-thousand pairs, Magpie or teacher. After dedup — about half removed, near-duplicates. After quality filter — keep judge score four or above, about half of what remains gone. After diversity cap — cap per-cluster count, about thirty percent gone. After decontamination — about five percent gone. End with two-and-a-half to five-thousand clean pairs.

The ratio at the bottom is the load-bearing number. Five-thousand clean pairs steer better than fifty-thousand raw. The ratio is not waste. It's the price of a clean steering wheel.

Four anti-patterns to leave with.

First, generate-without-filter. The cardinal sin. Generate fifty-thousand pairs with GPT-4, ship them straight to the trainer. The result is a model steered toward the teacher's noise floor. Always filter. The filtering is where the gains are.

Second, no decontamination. Training on data that overlaps your eval. The eval number is then meaningless. Decontaminate before training, every time, against every benchmark you will report.

Third, single-teacher style leakage. Generating one hundred percent of your data from one teacher. The steered model adopts the teacher's surface style as much as the target behavior. Mitigate by mixing teachers — two or three minimum.

Fourth, ignored diversity. Generating twenty-thousand pairs that are seventy percent coding questions because the teacher was good at coding and the prompts drifted. The steered model becomes a coding assistant with degraded general instruction-following. Enforce diversity at the prompt-distribution level and at the filter level. Diversity is a steering-wheel property, not a nice-to-have.

[SLIDE 12 — What you can now do]

You can now state the thesis — the gains are in filtering, not generation — and defend it with LIMA and Magpie's ablations. You can contrast strong-teacher distillation and Magpie self-synthesis, and choose between them for a given target behavior. You can describe Magpie's two-pass mechanism and why it works. You can place Evol-Instruct correctly in the history. You can name distilabel as the standard framework and its four stages. And you can state the decontamination obligation and why it's non-negotiable.

The lab generates five-hundred samples both ways — Magpie and teacher — dedups both, and compares them on a twenty-sample manual review. You'll feel the two methods and the filtering step before you spend GPU on training.

Next, module FT06: dedup, filter, decontaminate. The cleaning half of the funnel. This is where the gains actually live.

---

*End of module FT05. Duration: approximately forty minutes at one-hundred-forty words per minute.*