# Module FT05 — Synthetic Data Generation **Course**: Course 3 — LLM Fine-Tuning Masterclass **Module**: FT05 — Synthetic Data Generation **Duration**: 75 minutes **Level**: Senior Engineer and above **Prerequisites**: FT04 — Dataset Formats --- ## Learning Objectives After completing this module, you will be able to: 1. State the thesis of this module — *the gains are in filtering, not generation* — and explain why a 500-sample filtered set routinely beats a 50k-sample unfiltered one for SFT steering. 2. Contrast the two dominant synthetic-data approaches — **strong-teacher distillation** (GPT-4/Claude/R1 generates instruction-response pairs from seed prompts) and **Magpie self-synthesis** (an aligned model generates *both* sides from its pre-query template alone, no seed prompts) — and choose between them for a given target behavior. 3. Describe how Magpie works at the mechanism level: the aligned model's pre-query template is itself a sufficient prompt because RLHF/DPO training made the model want to *respond* — so feeding it the empty user turn elicits an instruction, and a second pass elicits the response. 4. Explain why Magpie (arXiv:2406.08464, ICLR 2025) outperformed Alpaca, Evol-Instruct, and UltraChat for SFT, and why `Magpie-Air` (from Llama-3-8B-Instruct) and `Magpie-Pro` (from Llama-3-70B-Instruct) exist as separate artifacts. 5. Describe Evol-Instruct (WizardLM) as iterative LLM-driven complexification of seed instructions, and place it correctly in the history: a real advance that is now superseded by Magpie + direct teacher-generation as the defaults. 6. Name **distilabel** (Argilla) as the standard framework for building synthetic preference/SFT pipelines, and list the four pipeline stages it standardizes (generate → judge → filter → dedup). 7. State the **decontamination obligation**: before training, remove any SFT pair that overlaps with MMLU, GSM8K, MT-Bench, or your own held-out evals. Non-negotiable for honest evals. Detailed in FT06. 8. Identify the four anti-patterns — generate-without-filter, no-decontamination, single-teacher style leakage, ignored diversity — and explain how each corrupts the steering wheel. --- # 5.1 — The Thesis *The one idea this module is built on. Get it wrong and you will spend compute generating data you should have spent compute filtering.* ## The sentence > **The gains are in filtering, not generation.** A 500-sample set you actually read and curated will steer better than 50,000 raw teacher outputs you did not look at. Read it twice. The dominant failure mode in synthetic-data work is not "we generated too little" — it is "we generated plenty and then shipped the unfiltered pile." The field spent 2023 believing scale-of-generation was the lever. The Magpie paper, the LIMA paper (arXiv:2305.11206, 1k curated examples), and the AlpaGasaurus / DOLLY line of evidence all converged on the same finding: **for steering, curation beats volume by a wide margin.** A tiny, clean, diverse set reliably outperforms a large, noisy one. This is the data-side complement of FT00's thesis. FT00 said fine-tuning steers rather than teaches. This module adds: *steering is only as good as the steering wheel, and the steering wheel is the curated set, not the raw pile.* The steering happens in the low-rank update; the data points the update in a direction; a noisy direction is a noisy steer, no matter how sophisticated the optimizer. ### Why this matters immediately Once the thesis lands, four things that look mysterious become obvious: 1. **Why LIMA worked with 1,000 examples.** The 1,000 were hand-curated. Quality and diversity, not volume. The model was already capable; the curated set pointed the steering vector cleanly. 2. **Why "generate 100k with GPT-4 and ship it" underperforms.** The 100k contains near-duplicates, low-information responses, style collapses, and (often) benchmark contamination. The optimizer dutifully steers toward all of it. 3. **Why the Magpie paper emphasizes its filtering pipeline as much as its generation method.** Magpie's headline contribution is "self-synthesis from nothing," but the paper's ablations show the filtering stage contributes a large share of the final quality. 4. **Why FT06 (Dedup, Filter, Decontaminate) is the load-bearing follow-on.** This module teaches you to *generate*; the next module teaches you to *clean*. Both are required. Neither alone is enough. ## The two-approach landscape Modern synthetic SFT/preference data comes from one of two approaches (often combined): | Approach | Mechanism | Teacher | Cost | Default use | | --- | --- | --- | --- | --- | | **Strong-teacher distillation** | Seed prompt → strong model generates response (and sometimes the prompt too) | GPT-4 / Claude / DeepSeek-R1 / a 70B you control | API $$ or GPU $$ | When you have a target distribution of prompts and need high-quality responses | | **Magpie self-synthesis** | Prompt the aligned model with its *pre-query template alone*; it generates the instruction, then a second pass generates the response | Any aligned instruct model (Llama-3-8B-Instruct, Qwen2.5-Instruct, etc.) | Cheap — a single 8B model suffices | When you want broad diversity *from scratch*, no seed prompts, no API bills | Both produce instruction-response pairs. Both require the same downstream cleaning. The choice between them is a question of *where you want the distribution to come from*: an external teacher's notion of what's worth asking, or the aligned model's own latent instruction distribution. --- # 5.2 — Strong-Teacher Distillation *The workhorse. You have prompts; a strong model answers them.* ## The pipeline The strong-teacher approach is straightforward in concept and subtle in execution: 1. **Acquire or generate seed prompts.** From real user logs (best), from a seed set you hand-write (good), or from another LLM (acceptable, but watch for style collapse). 2. **Send each prompt to a strong teacher.** GPT-4o, Claude, DeepSeek-R1, or a 70B you self-host. Capture the response. 3. **(Optional) Generate the prompt too.** If you don't have prompts, ask the teacher to *also* generate plausible user queries. This is what Self-Instruct (arXiv:2212.10560) pioneered and what Alpaca scaled. 4. **Filter, dedup, decontaminate.** (FT06. Non-negotiable.) The pair `(prompt, teacher_response)` becomes one SFT row. If you generate *two* responses per prompt and label which is better (via a judge or a rubric), you have a preference pair for DPO. distilabel standardizes both flavors. ## Where it shines, where it leaks **Shines**: when you have a *target prompt distribution* — real user queries from your product, a specific domain (medical, legal, code), a specific format (JSON tool calls). The teacher's job is to produce the gold response; your job is to make sure the prompts are representative of what the steered model will actually face in deployment. **Leaks**: when the teacher's *style* dominates. If you train a 7B on 50k GPT-4 responses, the 7B will adopt GPT-4's verbosity, hedging, and list-heavy format. This is **style leakage** — the model steers toward the teacher's surface style as much as toward the target behavior. It is usually unwanted. Mitigations: mix teachers (GPT-4 + Claude + R1), strip stylistic tells in post, or use a smaller/cleaner teacher whose style is closer to your target. **Leaks harder**: when you train on a teacher that itself was trained on benchmarks you intend to evaluate against. GPT-4 has seen MMLU. If your SFT data contains MMLU-shaped Q&A (because you asked GPT-4 to generate "exam-style questions"), you have now contaminated your training set with eval-shaped items. This is the decontamination obligation, detailed in FT06. ## The Self-Instruct lineage Strong-teacher generation of *both* prompt and response traces to Self-Instruct (Wang et al., 2022): seed with ~175 hand-written instructions, prompt GPT-3 to generate more in the same vein, filter (rouge dedup + heuristics), then generate responses. Alpaca (Stanford, 2023) scaled this to 52k with GPT-3.5 and showed a 7B LLaMA could be instruction-tuned usefully on the result. The bootstrapping loop is real and still works — but as a *default* it has been superseded by Magpie (for the no-seeds case) and direct teacher generation (for the have-prompts case). Self-Instruct's loop is now a special case of distilabel's generate step. --- # 5.3 — Magpie: Self-Synthesis From Nothing *The 2024 finding that changed the defaults. arXiv:2406.08464, ICLR 2025.* ## The key insight Aligned instruct models — Llama-3-Instruct, Qwen2.5-Instruct, Mistral-Instruct — are trained (via SFT then RLHF/DPO) to *respond* to user turns. Their chat template ends with a marker that says "the user is about to speak; produce a response." Magpie's insight: **if you feed the model its pre-query template *with the user turn empty*, the model generates an instruction anyway.** It cannot help itself — its training made instruction-responding the default behavior, and with no user content to respond to, it generates plausible user content first. This is not a hack. It is a direct consequence of how alignment training shaped the distribution. The aligned model has a strong prior over "what kinds of things do users ask?" baked into its weights. Empty-prompt sampling surfaces that prior. You get, for free, the model's latent instruction distribution — diverse, natural, and uncontaminated by any seed set you might have written. ## The two-pass mechanism Magpie generates a full instruction-response pair in two passes over the same aligned model: 1. **Instruction pass.** Feed the pre-query template (everything up to and including the user-turn opener, with no user content). Sample. The model generates an instruction — a plausible user query. 2. **Response pass.** Take that generated instruction, place it in the user turn *properly* (full chat template, instruction as user content, then the assistant-turn opener). Sample again. The model generates the response. The result is a self-contained `(instruction, response)` pair, generated end-to-end by the same aligned model, with no seed prompts, no prompt engineering, and no external teacher. ``` Pass 1 (instruction generation): <|begin_of_text|><|start_header_id|>user<|end_header_id|> <- empty user turn <|eot_id|><|start_header_id|>assistant<|end_header_id|> -> model samples: "Explain how TCP handshake works\n..." (the instruction) Pass 2 (response generation): <|begin_of_text|><|start_header_id|>user<|end_header_id|> Explain how TCP handshake works <|eot_id|><|start_header_id|>assistant<|end_header_id|> -> model samples: "The TCP three-way handshake...\n..." (the response) ``` ## Magpie-Air and Magpie-Pro The paper released two main datasets, distinguished by the source model: - **Magpie-Air** — generated from **Llama-3-8B-Instruct**. Cheaper to reproduce (an 8B fits on a consumer GPU), lower per-response quality ceiling, but the *diversity* of generated instructions is comparable to the 70B variant because the 8B's instruction prior is itself rich. - **Magpie-Pro** — generated from **Llama-3-70B-Instruct**. Higher-quality responses (the 70B is a stronger teacher of itself), more expensive to reproduce, the variant used in the paper's headline SFT results. The paper's ablations: Magpie-Pro-Filtered (300K curated from the 1M raw) used as SFT data for Llama-3-8B produces a model that outperforms the official Llama-3-8B-Instruct on MT-Bench and AlpacaEval 2, *using only self-generated data from a model in the same family.* That is the headline. The same 8B, tuned on data the 70B generated about itself, beats the official 8B-Instruct tuned on Meta's (closed) SFT mix. ## Why Magpie beat Alpaca, Evol-Instruct, UltraChat Three reasons, in order of importance: 1. **No seed-prompt bias.** Alpaca/Evol-Instruct/UltraChat all start from human-written seeds whose biases — topic, difficulty, length, style — propagate. Magpie samples the aligned model's *full* instruction prior, broader than any seed set a human would write. 2. **Natural response distribution.** Because the response comes from the same model that generated the instruction, the pair is *self-consistent* — the response is exactly what that model would naturally say. UltraChat and Alpaca often have response/instruction mismatches (the teacher's style doesn't match the prompt's implied expectation). 3. **The filtering pipeline still applies.** Magpie's authors did not skip FT06. They deduped (MinHash), filtered by response quality, removed contaminated items, and curated down to the filtered subsets that actually work. The "from scratch" generation is half the contribution; the disciplined filtering is the other half. --- # 5.4 — Evol-Instruct: The Complexification Loop *Still used, now a special case. Worth knowing because it appears in distilabel and in many open datasets.* ## The mechanism Evol-Instruct (WizardLM, arXiv:2304.12201) takes a seed instruction and asks an LLM to make it *harder* — deeper, more constrained, more multi-step. Two evolution operators: - **In-depth evolving.** Add constraints, deepen the reasoning, require more steps. "Write a Python function to sort a list" → "Write a Python function to sort a list of dictionaries by a user-specified key, handling missing keys gracefully, with O(n log n) worst-case complexity, and include property-based tests." - **In-breadth evolving.** Generate a *topic-related but different* instruction, to diversify the set. Iterate. Each seed produces a tree of evolved descendants. The evolved set is then deduped, filtered, and used as SFT prompts (responses generated by a teacher). WizardLM's contribution was the observation that *hardness* is a useful axis to push on — a model tuned on hard instructions follows instructions better than one tuned on easy ones, because the hard ones exercise more of the instruction-following capability. ## Why it's now a default-superseded approach Evol-Instruct works. It is not wrong. But two things displaced it as the default: 1. **Magpie samples difficulty naturally.** The aligned model's instruction prior already spans easy to hard. You don't need an evolution loop to get hard instructions — you get them for free by sampling enough. Magpie's instruction distribution includes graduate-level questions without anyone having to *evolve* toward them. 2. **Direct teacher generation is simpler.** If you have a target prompt distribution (real user queries), you don't need to evolve — you already have the distribution. Send the real prompts to the teacher and ship the responses (after filtering). Evol-Instruct remains the right tool when you specifically want to *amplify hardness* — e.g., you're building a reasoning-tuned model (Pillar 4) and you want to over-sample multi-step problems. distilabel includes an EvolInstruct step for exactly this. But for general SFT data generation, the 2024-onward default is Magpie-style self-synthesis or direct teacher distillation, with Evol-Instruct as an optional complexity-amplification stage. --- # 5.5 — distilabel: The Standard Framework *Argilla's distilabel is the HF-native framework for synthetic preference/SFT pipelines. If you're building this in code, you're probably building it in distilabel.* ## What it standardizes distilabel (Argilla, open-source, MIT) gives you a `Pipeline` object with four standard stages, each pluggable: 1. **Generate.** A `Step` that calls a `LLM` (local via vLLM/transformers, or remote via an OpenAI-compatible API). Magpie-style two-pass generation, direct teacher generation, Evol-Instruct evolution — all are generate-step variants. The `Magpie` step in distilabel encapsulates the pre-query-template trick. 2. **Judge.** A `Step` that asks a (usually stronger) model to score or rank generated outputs. Used for preference-pair construction (judge picks the better of two responses → DPO pair) and for quality filtering (judge scores 1–5, you keep ≥4). 3. **Filter.** A `Step` that applies rules: length bounds, language detection, regex constraints, dedup (MinHash or exact), contamination removal. This is where FT06's machinery plugs in. 4. **Dedup.** Often folded into Filter, but worth treating as its own stage because it's where the biggest volume reduction happens. MinHash on n-grams of the instruction; cluster; keep one per cluster. The output is a `Distiset` — an HF Dataset with the columns your downstream trainer expects (`instruction`, `response`, `preference`, `score`, etc.), ready for FT11's training loop. ## Why use it instead of rolling your own You *can* write a 200-line Python script that loops over prompts, calls an API, and writes JSONL. Many teams do. The reasons to use distilabel instead: **reproducibility** (a `Pipeline` is a DAG with a config — checkpoint, resume, share); **the judge step is non-trivial** (handling ties, position bias, length bias is easy to get wrong — distilabel's `UltraFeedback` step implements the documented protocol); **Magpie is fiddly** (the pre-query template differs per model — Llama-3 vs Qwen2.5 vs Mistral — and getting it wrong is a source of silent bugs); and **it composes with FT06** (the filtering/dedup/decontaminate steps are distilabel `Step` subclasses — building generation in distilabel means your cleaning pipeline drops in directly). If you're doing this once, a script is fine. If you're iterating on data quality, regenerating as the base changes, or building preference data for DPO, distilabel pays for itself within the second run. --- # 5.6 — The Decontamination Obligation *Non-negotiable. Detailed in FT06; flagged here because it must be planned at generation time, not bolted on after.* ## The obligation Before you train on any synthetic dataset, you must remove items that overlap with the benchmarks you intend to evaluate against. MMLU, GSM8K, HumanEval, MT-Bench, AlpacaEval, your own held-out eval set — if any item in your training data is (near-)identical to an eval item, your eval is compromised. The model has seen the answer. The number is fake. This is not a hypothetical. The original Alpaca dataset was found to contain near-duplicates of MMLU items. Multiple "model X beats GPT-4 on benchmark Y" claims from 2023–2024 were walked back when contamination was discovered. The Magpie paper explicitly decontaminates against MMLU, GSM8K, and others, and reports the decontamination step's effect on honest eval. ## Why it's a generation-time concern You might think "I'll decontaminate after generation, in FT06." You will — but you should also *avoid generating contaminated items in the first place*: - **Don't ask the teacher to generate "MMLU-style questions."** It will, and they'll overlap. - **Don't seed with benchmark items** "to get the model started." That's literal contamination. - **Don't use benchmark items as few-shot examples** in your generation prompt. The teacher will pattern-match and produce near-duplicates. - **Do plan your eval set first**, hold it out, and decontaminate the generated data against it. FT06 covers the mechanics (n-gram overlap, MinHash, semantic similarity thresholds). The rule: **the eval set is fixed before generation begins, and generation is checked against it.** If you generate first and pick evals later, you will (intentionally or not) pick evals your data happens to do well on. That's not an eval; that's a press release. --- # 5.7 — Quality Filtering: The Preview *The deep dive is FT06. Here is what you need to know to generate data that's filterable.* ## What filtering buys you The Magpie paper's ablation: filtering 1M raw Magpie-Pro down to 300K *filtered* improved downstream SFT quality more than doubling the raw would have. LIMA's 1k hand-curated beat Alpaca's 52k unfiltered. **Curation has a steeper quality curve than volume, for steering.** The filtering axes (each a distilabel step, each covered in FT06): - **Deduplication.** MinHash on instruction n-grams; cluster; keep one per cluster. Removes the "same question phrased 40 ways" failure. - **Quality scoring.** A judge model scores each response 1–5. Keep ≥4. - **Diversity enforcement.** Cluster by topic/embedding; cap per-cluster count. Prevents the set from being 60% coding questions. - **Decontamination.** (Above.) Remove eval-overlap items. - **Length / format heuristics.** Drop degenerate outputs (empty, repetition loops, refusal-as-response when you didn't want refusals). The point: **generate enough that you have headroom to filter aggressively.** If you need 5k clean SFT pairs, generate 20–50k raw, filter to 5k. The ratio is not waste; it's the price of a clean steering wheel. --- ## Anti-Patterns ### Generate-without-filter The cardinal sin. Generate 50k pairs with GPT-4, ship them straight to the trainer. The result is a model steered toward the teacher's noise floor — duplicates, low-information responses, style collapses. Always filter. The filtering is where the gains are. ### No decontamination Training on data that overlaps your eval. The eval number is then meaningless — you cannot tell whether the model learned the behavior or memorized the answer. Decontaminate before training, every time, against every benchmark you will report. Detailed in FT06. ### Single-teacher style leakage Generating 100% of your data from one teacher (GPT-4, Claude, a single 70B). The steered model adopts the teacher's surface style — verbosity, hedging, list-format bias — as much as the target behavior. Mitigate by mixing teachers (2–3 minimum), by post-processing stylistic tells, or by using a teacher whose style matches your target. ### Ignored diversity Generating 20k pairs that are 70% coding questions because the teacher was good at coding and the prompts drifted that way. The steered model becomes a coding assistant with degraded general instruction-following. Enforce diversity at the prompt-distribution level (sample across domains) and at the filter level (cap per-cluster count). Diversity is a steering-wheel property, not a nice-to-have. --- ## Key Terms | Term | Definition | | --- | --- | | **Strong-teacher distillation** | Generating SFT data by sending seed prompts to a strong model (GPT-4, Claude, R1, a 70B) and capturing responses | | **Magpie (self-synthesis)** | Generating instruction-response pairs by prompting an aligned model with its pre-query template alone — no seed prompts. arXiv:2406.08464, ICLR 2025 | | **Pre-query template** | The portion of a chat template up to and including the user-turn opener, with no user content. Magpie's key lever — feeding this empty elicits an instruction | | **Magpie-Air** | Magpie dataset generated from Llama-3-8B-Instruct (cheaper, reproducible on consumer GPU) | | **Magpie-Pro** | Magpie dataset generated from Llama-3-70B-Instruct (higher response quality, the headline SFT results) | | **Evol-Instruct** | WizardLM's iterative LLM-driven instruction complexification (in-depth: add constraints; in-breadth: topic-related variant). Now a special case, not a default | | **Self-Instruct** | The 2022 bootstrapping loop (seed → LLM generates more instructions → filter → respond). Alpaca's ancestor; now a special case of distilabel's generate step | | **distilabel** | Argilla's open-source framework for synthetic preference/SFT pipelines. Standardizes generate → judge → filter → dedup | | **Decontamination** | Removing training items that overlap with eval benchmarks (MMLU, GSM8K, MT-Bench). Non-negotiable for honest evals. Detailed in FT06 | | **Style leakage** | The steered model adopting the teacher's surface style (verbosity, hedging, format bias) as a side effect of training on its outputs | | **The data-quality funnel** | The generate-lots → filter-aggressively pipeline. Typical ratio: generate 4–10x what you'll train on, filter to the clean subset | --- ## Lab Exercise See `07-lab-spec.md`. The "Magpie + Teacher" lab: generate 500 SFT samples two ways (Magpie self-synthesis from Llama-3-8B-Instruct or a similar aligned model, and strong-teacher distillation from an API or local 70B), dedup both, and compare quality on a 20-sample manual review against a rubric. Consumer-hardware friendly. Runnable with distilabel or direct transformers. --- ## References 1. **Xu et al. (2024)** — *Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing*. arXiv:2406.08464, ICLR 2025. The self-synthesis method that displaced seed-prompt bootstrapping as the default. Magpie-Air (8B) and Magpie-Pro (70B). 2. **WizardLM / Xu et al. (2023)** — *WizardLM: Empowering Large Language Models to Follow Complex Instructions*. arXiv:2304.12201. Evol-Instruct — the iterative complexification loop. 3. **Wang et al. (2022)** — *Self-Instruct: Aligning Language Models with Self-Generated Instructions*. arXiv:2212.10560. The bootstrapping lineage; Alpaca's ancestor. 4. **Taori et al. (2023)** — *Stanford Alpaca*. The 52k Self-Instruct scaling that made synthetic SFT mainstream. (And the contamination cautionary tale.) 5. **Zhou et al. (2023)** — *LIMA: Less Is More for Alignment*. arXiv:2305.11206. 1,000 curated examples beat 52k unfiltered. The curation-over-volume evidence. 6. **Argilla — distilabel documentation**. The standard framework for synthetic preference/SFT pipelines. `distilabel.argilla.io`. Magpie, UltraFeedback, EvolInstruct steps. 7. **Eugene Yan — on synthetic data**. Practitioner-oriented writing on synthetic-data pipelines, evaluation, and the filtering-over-generation principle. `eugeneyan.com`. 8. **Course 3, Module FT06** — *Dedup, Filter, Decontaminate*. The load-bearing follow-on. The cleaning half of the data-quality funnel.