{ "module": "FT05 — Synthetic Data Generation", "course": "3 — LLM Fine-Tuning Masterclass", "version": "1.0.0", "duration_minutes": 40, "total_questions": 15, "bloom_distribution": { "target": "20% recall / 40% application / 40% analysis-design", "actual": { "recall": 3, "application": 6, "analysis": 6 } }, "passing_score_percent": 70, "questions": [ { "id": "Q01", "bloom": "recall", "type": "multiple_choice", "prompt": "What is the central thesis of Module FT05?", "options": [ "Synthetic data is always lower quality than human-annotated data and should be avoided.", "The gains are in filtering, not generation. A 500-sample curated set steers better than 50,000 raw teacher outputs.", "More data is always better for SFT — generate as much as you can afford.", "Magpie is the only correct way to generate synthetic SFT data; teacher distillation is obsolete." ], "answer_index": 1, "rationale": "The thesis is 'the gains are in filtering, not generation.' LIMA (1k curated beat 52k unfiltered) and Magpie's own ablations (filtering 1M to 300K improved quality more than doubling raw) are the evidence. This is the data-side complement of FT00's 'steering not teaching' — the steering wheel is the curated set, not the raw pile." }, { "id": "Q02", "bloom": "recall", "type": "multiple_choice", "prompt": "What is the key insight of Magpie (arXiv:2406.08464, ICLR 2025)?", "options": [ "Aligned models generate better responses when given longer, more detailed seed prompts.", "An evolutionary loop that complexifies seed instructions produces harder, more useful training data.", "Aligned instruct models will generate instructions when prompted with their pre-query template alone — the user turn is empty, no seed prompts needed.", "A strong teacher model can be distilled into a smaller student by training on the teacher's logits." ], "answer_index": 2, "rationale": "Magpie's key insight: RLHF/DPO training made instruction-responding the aligned model's default behavior. Feeding the pre-query template with the user turn empty surfaces the model's instruction prior — it generates a plausible user query because it cannot help but 'respond.' Option B is Evol-Instruct; option D is general distillation, not Magpie." }, { "id": "Q03", "bloom": "recall", "type": "multiple_choice", "prompt": "What is the difference between Magpie-Air and Magpie-Pro?", "options": [ "Magpie-Air is filtered; Magpie-Pro is raw and unfiltered.", "Magpie-Air is generated from Llama-3-8B-Instruct; Magpie-Pro is generated from Llama-3-70B-Instruct.", "Magpie-Air uses seed prompts; Magpie-Pro uses the empty-template trick.", "Magpie-Air is for SFT; Magpie-Pro is for DPO preference pairs only." ], "answer_index": 1, "rationale": "Magpie-Air is generated from Llama-3-8B-Instruct (cheaper, reproducible on a consumer GPU, comparable instruction diversity). Magpie-Pro is generated from Llama-3-70B-Instruct (higher-quality responses, the variant used in the paper's headline SFT results where Magpie-Pro-Filtered tuned on Llama-3-8B beat the official Llama-3-8B-Instruct). Both use the same empty-template self-synthesis mechanism." }, { "id": "Q04", "bloom": "application", "type": "multiple_choice", "prompt": "You are building a coding assistant for your company's proprietary API. You have 5,000 real user queries from your support logs. Which generation approach do you use, and why?", "options": [ "Magpie self-synthesis — you want broad diversity from the model's instruction prior.", "Strong-teacher distillation from your real prompts — you have a target prompt distribution; you need high-quality responses to YOUR queries, not broad coverage.", "Evol-Instruct — evolve your 5,000 prompts into harder variants to improve instruction-following.", "Self-Instruct bootstrapping — generate more prompts from your seeds to scale up." ], "answer_index": 1, "rationale": "You HAVE a target prompt distribution (real user queries about your API). Strong-teacher distillation is the right tool: send your real prompts to a strong teacher (or your own 70B), capture responses, filter aggressively. Magpie is for when you want broad coverage from scratch with no seed distribution. Evol-Instruct is for hardness amplification (Pillar 4), not general SFT." }, { "id": "Q05", "bloom": "application", "type": "multiple_choice", "prompt": "You are building a general-purpose assistant and have no specific prompt distribution. You want maximum diversity. Which approach, and why?", "options": [ "Strong-teacher distillation from a hand-written seed set of 100 diverse prompts.", "Magpie self-synthesis — it samples the aligned model's full instruction prior with no seed bias, giving broader diversity than any human-written seed set.", "Evol-Instruct on a single seed, evolved 10,000 times.", "Self-Instruct bootstrapping from 175 hand-written seeds." ], "answer_index": 1, "rationale": "Magpie samples the aligned model's FULL instruction prior — broader than any seed set a human would write, with no seed-prompt bias (the failure mode of Alpaca/Evol-Instruct/UltraChat). For broad coverage from scratch, Magpie is the default. (The production pattern is often Magpie for diversity + teacher for quality, mixed and filtered.)" }, { "id": "Q06", "bloom": "application", "type": "multiple_choice", "prompt": "You generated 50,000 (prompt, response) pairs with GPT-4 and are about to ship them to the trainer. What must you do first, per FT05?", "options": [ "Nothing — GPT-4's outputs are high quality and don't need filtering.", "Split 80/20 into train/eval and start training.", "Filter: dedup (MinHash on instruction n-grams), quality-score (judge >=4), diversity-cap (per-cluster), and decontaminate (remove eval-overlap items). This is non-negotiable.", "Shuffle the data to ensure randomness, then train." ], "answer_index": 2, "rationale": "Generate-without-filter is the cardinal sin. The 50k contains near-duplicates, low-information responses, style collapses (GPT-4's verbosity/hedging), and likely benchmark contamination. The optimizer steers toward ALL of it. Filter aggressively — the gains are in the filtering. Decontamination is non-negotiable for honest evals (FT06)." }, { "id": "Q07", "bloom": "application", "type": "multiple_choice", "prompt": "You notice your fine-tuned 7B model has adopted GPT-4's verbose, hedging, list-heavy style — even on simple yes/no questions. What went wrong, and how do you fix it?", "options": [ "Your LoRA rank was too low; increase it to r=64.", "Style leakage: you trained on a single teacher (GPT-4) and the model adopted its surface style. Mitigate by mixing 2-3 teachers, post-processing stylistic tells, or using a teacher whose style matches your target.", "Your learning rate was too high; retrain at 1e-6.", "Your model is undertrained; double the dataset size and retrain." ], "answer_index": 1, "rationale": "Style leakage is the unwanted side effect of single-teacher distillation: the steered model adopts the teacher's surface style (verbosity, hedging, list-format bias) as much as the target behavior. Mitigations: mix teachers (2-3 minimum), post-process stylistic tells, or use a teacher whose style matches your target. More data or different LoRA rank won't fix a style-leakage problem." }, { "id": "Q08", "bloom": "application", "type": "multiple_choice", "prompt": "You need 5,000 clean SFT pairs. Using the data-quality funnel from FT05, approximately how many raw pairs should you generate, and why?", "options": [ "5,000 — generate exactly what you need to avoid waste.", "5,500 — a 10% buffer for filtering losses.", "20,000–50,000 — a 4–10x ratio. Dedup removes ~50%, quality filter ~50% more, diversity cap ~30%, decontamination ~5%. The ratio is the price of a clean steering wheel, not waste.", "100,000+ — more data is always better for steering." ], "answer_index": 2, "rationale": "The data-quality funnel: generate 4–10x what you'll train on, filter aggressively to the clean subset. Typical stage losses: dedup ~50%, quality filter ~50% of what remains, diversity cap ~30%, decontamination ~5%. To end with 5k clean, generate 20–50k raw. The ratio is not waste; it's the price of a clean steering wheel (the thesis of FT05)." }, { "id": "Q09", "bloom": "application", "type": "multiple_choice", "prompt": "You are designing your generation pipeline. Which of these would NOT violate the decontamination obligation (i.e., which is safe)?", "options": [ "Seeding your generator with 50 MMLU questions 'to get the model started on exam-style items.'", "Using 5 MMLU questions as few-shot examples in your teacher prompt to 'show the format you want.'", "Asking GPT-4 to 'generate MMLU-style multiple-choice questions about biology.'", "Holding out your eval set BEFORE generation begins, then checking your generated data for n-gram overlap against it and removing matches." ], "answer_index": 3, "rationale": "Options A, B, and C all but guarantee contamination: seeding with benchmark items is literal contamination; using them as few-shot examples causes the teacher to pattern-match and produce near-duplicates; asking for 'MMLU-style' questions produces eval-shaped items. Option D is the correct protocol: fix the eval set BEFORE generation, then decontaminate the generated data against it (FT06 covers the mechanics — n-gram overlap, MinHash, semantic similarity)." }, { "id": "Q10", "bloom": "analysis", "type": "multiple_choice", "prompt": "Why does Magpie outperform seed-prompt-based methods (Alpaca, Evol-Instruct, UltraChat) on instruction diversity? What is the underlying mechanism?", "options": [ "Magpie uses a stronger base model, so its instructions are more diverse.", "Seed-prompt methods inherit the biases of their human-written seed sets (topic, difficulty, length, style). Magpie samples the aligned model's FULL instruction prior, which is broader than any seed set a human would write — no seed bias.", "Magpie's evolution loop generates more variants per seed than Evol-Instruct does.", "Magpie filters more aggressively, which artificially inflates diversity scores." ], "answer_index": 1, "rationale": "The mechanism: seed-prompt methods (Alpaca, Evol-Instruct, UltraChat) all start from human-written seeds whose biases propagate. Magpie samples the aligned model's full instruction prior via the empty-template trick — broader than any seed set, with no human bias. Filtering does not 'inflate' diversity; it preserves what generation produced. Magpie does not use an evolution loop (that's Evol-Instruct)." }, { "id": "Q11", "bloom": "analysis", "type": "multiple_choice", "prompt": "How does the FT05 thesis ('gains are in filtering') follow from FT00's thesis ('steering not teaching')? What is the logical connection?", "options": [ "They are unrelated theses from different modules.", "FT00 says fine-tuning steers via a low-rank update pointed by the data. A noisy direction (unfiltered data: duplicates, low-quality, contaminated, style-collapsed) is a noisy steer, no matter the optimizer. So the leverage is in curation (filtering), not volume of generation. The steering wheel is the curated set.", "FT05 contradicts FT00 — if steering is low-rank, then more data (not less) is needed to find the right direction.", "FT00 is about the model; FT05 is about the harness. There is no logical connection." ], "answer_index": 1, "rationale": "The logical connection: FT00 established that fine-tuning steers via a low-rank update, and the data points the update in a direction. FT05 extends this: if the data is noisy (unfiltered), the direction is noisy, and no optimizer can fix a bad direction. Therefore the leverage is in curation (filtering), not in volume of generation. The two theses are the model-side and data-side of the same argument: steering is low-rank, and the steering wheel is the curated set." }, { "id": "Q12", "bloom": "analysis", "type": "multiple_choice", "prompt": "A team generates 100k Magpie pairs from Llama-3-8B-Instruct, skips filtering ('Magpie is already high quality'), and trains a 7B. The resulting model is worse than the base on MT-Bench. What is the most likely cause, and what does FT05 say about it?", "options": [ "Magpie from an 8B is too low quality; they should have used Magpie-Pro from the 70B.", "They committed the generate-without-filter anti-pattern. The 100k raw contains near-duplicates, low-quality responses, and possibly contaminated items. The optimizer steered toward the noise floor. FT05: the gains are in filtering — even Magpie's authors filtered (1M raw -> 300K filtered) and showed filtering contributed a large share of final quality.", "Their learning rate was wrong; SFT always needs lr=2e-5.", "Llama-3-8B is too small to be a Magpie source; only 70B works." ], "answer_index": 1, "rationale": "Generate-without-filter is the cardinal sin, and it applies to Magpie too. The Magpie paper's own ablations showed filtering 1M raw down to 300K filtered improved downstream quality more than doubling the raw would have. Magpie is not magic — it produces a noisy raw set that must be deduped, quality-scored, diversity-capped, and decontaminated, just like any other generation method. Option A is plausible but secondary; the primary failure is skipping filtering." }, { "id": "Q13", "bloom": "analysis", "type": "multiple_choice", "prompt": "Why is Evol-Instruct described as 'default-superseded' rather than 'wrong'? What changed between 2023 (when it was state-of-the-art) and 2024-onward?", "options": [ "Evol-Instruct was found to be mathematically incorrect and produces invalid instructions.", "Two things displaced it as a DEFAULT: (1) Magpie samples difficulty naturally — the aligned model's prior already spans easy to hard, no evolution loop needed; (2) direct teacher generation is simpler if you have a target prompt distribution. Evol-Instruct still works and is the right tool when you specifically want to amplify hardness (e.g., reasoning-tuned models, Pillar 4).", "Evol-Instruct was deprecated by Argilla and removed from distilabel.", "Evol-Instruct was found to always contaminate eval sets." ], "answer_index": 1, "rationale": "Evol-Instruct works and is not wrong. Two things displaced it as the DEFAULT: Magpie samples difficulty naturally (the prior spans easy to hard, no evolution loop needed), and direct teacher generation is simpler when you have a target prompt distribution. Evol-Instruct remains the right tool for SPECIFIC hardness amplification (reasoning models, Pillar 4) and is included in distilabel as an EvolInstruct step. It's a special case, not a default." }, { "id": "Q14", "bloom": "analysis", "type": "multiple_choice", "prompt": "A team trains a 7B on 50k GPT-4-generated pairs and reports it 'beats GPT-4 on MMLU.' On audit, 3% of their training set is near-duplicates of MMLU questions. What does FT05 say about this result, and what should they have done?", "options": [ "The result stands — 3% contamination is within acceptable noise.", "The result is invalid. Training on near-duplicates of MMLU means the model has seen the answers; the MMLU score is fake (memorization, not capability). They violated the decontamination obligation. They should have held out their eval set BEFORE generation, then decontaminated the generated data against it (FT06 mechanics: n-gram overlap, MinHash). The 3% must be removed and the eval re-run.", "They should report the contamination but keep the score — it's a minor footnote.", "GPT-4 has seen MMLU anyway, so the contamination doesn't matter." ], "answer_index": 1, "rationale": "The decontamination obligation is non-negotiable for honest evals. Training on near-duplicates of MMLU means the model memorized the answers; the MMLU score measures memorization, not capability. The result is invalid. Correct protocol: fix the eval set BEFORE generation begins, then decontaminate the generated data against it (FT06: n-gram overlap, MinHash, semantic similarity thresholds). Option D is a pernicious misconception — GPT-4 having seen MMLU is exactly why you must decontaminate teacher-generated data." }, { "id": "Q15", "bloom": "analysis", "type": "multiple_choice", "prompt": "In the FT05 lab, Magpie from a small aligned model (Qwen2.5-1.5B) is compared to teacher distillation from gpt-4o-mini. The Magpie set has higher instruction diversity and self-consistency but lower response quality. The teacher set has the opposite profile. What is the production pattern that captures both strengths, and why?", "options": [ "Pick whichever scores higher on the rubric and use only that one.", "Always use Magpie — diversity matters more than response quality for steering.", "Concatenate the two deduped sets, dedup the union, and you have a hybrid set with Magpie's diversity AND the teacher's response quality. Filter the union. This is the production pattern: Magpie for diversity + teacher for quality, mixed and filtered. Both approaches, combined.", "Generate Magpie first, then use the teacher to rewrite Magpie's responses for quality." ], "answer_index": 2, "rationale": "The production pattern: mix both. Magpie contributes broad instruction diversity (its strength); teacher distillation contributes high-quality responses (its strength). Concatenate the deduped sets, dedup the union (catching cross-method near-duplicates), and filter the whole. This captures both profiles. Option D (teacher rewriting Magpie responses) is a valid variant but more expensive and introduces a second round of style leakage risk. The straightforward mix-and-filter is the standard." } ] }