"What is the central thesis of FT05, stated in one sentence?"	"The gains are in filtering, not generation. A 500-sample set you actually read and curated will steer better than 50,000 raw teacher outputs you did not look at. This is the data-side complement of FT00's 'steering not teaching': the steering wheel is the curated set, not the raw pile."	c3::ft05::recall
"Name the two dominant synthetic-data approaches for SFT/preference data."	"(1) Strong-teacher distillation: seed prompts go to a strong model (GPT-4/Claude/R1/a 70B) which generates responses (and optionally the prompts too, Self-Instruct-style). (2) Magpie self-synthesis: an aligned model generates BOTH instruction and response when prompted with its pre-query template alone — no seed prompts."	c3::ft05::recall
"What is Magpie (arXiv:2406.08464, ICLR 2025) and what is its key insight?"	"Magpie is a self-synthesis method for alignment data. Key insight: aligned instruct models (Llama-3-Instruct, Qwen2.5-Instruct) will generate instructions when prompted with their pre-query template ALONE (the user turn empty). RLHF/DPO made instruction-responding the default behavior; with no user content, the model surfaces a plausible query from its instruction prior."	c3::ft05::recall
"What is the two-pass mechanism Magpie uses to generate a full (instruction, response) pair?"	"Pass 1 (instruction generation): feed the pre-query template with the user turn EMPTY; the model samples a plausible user query. Pass 2 (response generation): place that generated instruction in the user turn PROPERLY (full chat template), feed back to the SAME model; it samples the response. Two passes, same model, no seeds, no external teacher."	c3::ft05::application
"What is the difference between Magpie-Air and Magpie-Pro?"	"Magpie-Air is generated from Llama-3-8B-Instruct (cheaper, reproducible on a consumer GPU, comparable instruction diversity). Magpie-Pro is generated from Llama-3-70B-Instruct (higher-quality responses, the variant used in the paper's headline SFT results — Magpie-Pro-Filtered tuned on Llama-3-8B beats the official Llama-3-8B-Instruct on MT-Bench and AlpacaEval 2)."	c3::ft05::recall
"Why did Magpie outperform Alpaca, Evol-Instruct, and UltraChat for SFT?"	"Three reasons: (1) No seed-prompt bias — it samples the aligned model's FULL instruction prior, broader than any human-written seed set. (2) Natural response distribution — same model generates both sides, so (instruction, response) is self-consistent. (3) The filtering pipeline still applies — Magpie's authors deduped, quality-scored, and decontaminated; the 'from scratch' generation is half the contribution, disciplined filtering is the other half."	c3::ft05::analysis
"What is Evol-Instruct (WizardLM, arXiv:2304.12201) and how does it work?"	"Evol-Instruct takes a seed instruction and uses an LLM to make it HARDER via two operators: in-depth evolving (add constraints, deepen reasoning, require more steps) and in-breadth evolving (generate a topic-related but different instruction to diversify). Iterate to produce a tree of evolved descendants, then dedup/filter/respond. Contribution: hardness is a useful axis for instruction-following."	c3::ft05::recall
"Why is Evol-Instruct now a default-superseded approach, and when is it still the right tool?"	"Superseded because (1) Magpie samples difficulty naturally — the aligned model's prior already spans easy to hard, no evolution loop needed; (2) direct teacher generation is simpler if you have a target prompt distribution. Still the right tool when you specifically want to AMPLIFY hardness — e.g., building a reasoning-tuned model (Pillar 4) and over-sampling multi-step problems. distilabel includes an EvolInstruct step for this."	c3::ft05::analysis
"What is distilabel (Argilla) and what four pipeline stages does it standardize?"	"distilabel is the open-source (MIT) HF-native framework for synthetic preference/SFT pipelines. Four stages, each a pluggable Step: (1) GENERATE (Magpie step or teacher step), (2) JUDGE (a stronger model scores/ranks outputs — for preference pairs and quality filtering), (3) FILTER (length/format/language/regex/decontam rules), (4) DEDUP (MinHash on instruction n-grams). Output is a Distiset ready for FT11's trainer."	c3::ft05::recall
"What is the decontamination obligation, and why is it non-negotiable?"	"Before training, remove SFT items that overlap with benchmarks you'll evaluate against (MMLU, GSM8K, HumanEval, MT-Bench, AlpacaEval, your own held-out eval). If a training item is near-identical to an eval item, the eval is compromised — the model has seen the answer, the number is fake. Non-negotiable for honest evals. Detailed in FT06. Must be planned at generation time (don't seed with benchmark items, don't use them as few-shot examples)."	c3::ft05::application
"Why is decontamination a generation-time concern, not just a post-hoc filter?"	"You decontaminate after generation (FT06), but you should ALSO avoid generating contaminated items: don't ask the teacher for 'MMLU-style questions' (it will produce near-duplicates), don't seed with benchmark items, don't use benchmark items as few-shot examples (the teacher pattern-matches). Plan the eval set FIRST, hold it out, and check generation against it. If you generate first and pick evals later, you'll pick evals your data does well on — that's a press release, not an eval."	c3::ft05::analysis
"What is style leakage and how do you mitigate it?"	"Style leakage: when you train on a single teacher's outputs (all GPT-4, or all one 70B), the steered model adopts the teacher's SURFACE STYLE — verbosity, hedging, list-format bias — as much as the target behavior. Mitigations: mix 2-3 teachers, post-process stylistic tells, or use a teacher whose style matches your target. A common unwanted side effect of strong-teacher distillation."	c3::ft05::analysis
"What is the data-quality funnel and what is the typical generation-to-clean ratio?"	"The funnel: generate 4-10x what you'll train on, filter aggressively to the clean subset. Stages: raw generation -> dedup (remove near-duplicates, ~50% gone) -> quality filter (keep judge score >=4, ~50% gone) -> diversity cap (per-cluster count, ~30% gone) -> decontamination (~5% gone). Typical: generate 20-50k raw, end with 2.5-5k clean. The ratio is the price of a clean steering wheel, not waste."	c3::ft05::application
"What does the Magpie paper's ablation show about filtering vs generation, and how does it support the FT05 thesis?"	"Filtering 1M raw Magpie-Pro down to 300K filtered improved downstream SFT quality MORE than doubling the raw data would have. Combined with LIMA (1k curated beat Alpaca's 52k unfiltered), the pattern repeats: curation has a steeper quality curve than volume, for steering. This is direct evidence for 'the gains are in filtering, not generation.'"	c3::ft05::analysis
"What is Self-Instruct (Wang et al. 2022, arXiv:2212.10560) and how does it relate to Alpaca and to modern defaults?"	"Self-Instruct bootstraps instruction data: seed with ~175 hand-written instructions, prompt GPT-3 to generate more in the same vein, filter (rouge dedup + heuristics), then generate responses. Alpaca (Stanford 2023) scaled this to 52k with GPT-3.5 and showed a 7B LLaMA could be instruction-tuned usefully. The bootstrapping loop is real and still works, but as a DEFAULT it's superseded by Magpie (for the no-seeds case) and direct teacher generation (for the have-prompts case)."	c3::ft05::application
"Name the four anti-patterns in synthetic data generation and the one-line fix for each."	"(1) Generate-without-filter: always filter, the gains are there. (2) No-decontamination: decontaminate before training, every time, against every benchmark you'll report. (3) Single-teacher style leakage: mix 2-3 teachers or post-process stylistic tells. (4) Ignored diversity: enforce diversity at prompt-distribution level (sample across domains) and at filter level (cap per-cluster count). Diversity is a steering-wheel property."	c3::ft05::application
"For a domain-specific assistant (say, a coding assistant for your company's API), which generation approach do you pick and why?"	"Strong-teacher distillation from domain prompts. You HAVE a target prompt distribution (real user queries about your API), so you don't need Magpie's broad self-synthesis — you need high-quality responses to YOUR prompts. Send real prompts to a strong teacher (or your own 70B), capture responses, filter aggressively. Magpie is for when you want broad coverage from scratch with no seed distribution."	c3::ft05::analysis
"For a general-purpose assistant with no specific prompt distribution, which approach do you pick and why?"	"Magpie self-synthesis (or a mix). You want broad diversity from the aligned model's full instruction prior — Magpie samples it directly with no seed bias. Mix with teacher distillation for response quality (Magpie's responses from a small model are weaker than a strong teacher's). The production pattern: Magpie for diversity + teacher for quality, concatenated, deduped, filtered. Both approaches, mixed, filtered."	c3::ft05::analysis
"What is LIMA (arXiv:2305.11206) and why is it cited as evidence for the FT05 thesis?"	"LIMA (Less Is More for Alignment) showed that 1,000 HAND-CURATED examples beat Alpaca's 52,000 unfiltered for SFT steering. The 1,000 were read and curated by humans. This is direct evidence that curation has a steeper quality curve than volume for steering — the model was already capable; the curated set pointed the steering vector cleanly. A foundational citation for 'the gains are in filtering, not generation.'"	c3::ft05::analysis
"Why does the FT05 thesis ('gains are in filtering') follow from FT00's thesis ('steering not teaching')?"	"FT00: fine-tuning steers via a low-rank update; the data points the update in a direction. FT05: a noisy direction (unfiltered data: duplicates, low-quality, contaminated, style-collapsed) is a noisy steer, no matter how good the optimizer. The steering wheel is the CURATED set, not the raw pile. So the leverage is in curation (filtering), not in volume of generation. The two theses are the data-side and model-side of the same argument."	c3::ft05::analysis
"What is the pre-query template, and why is feeding it empty the core of Magpie?"	"The pre-query template is the portion of a chat template up to and including the user-turn opener, with NO user content (e.g., '<|start_header_id|>user<|end_header_id|>\n\n<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n'). Feeding it empty is the core of Magpie because RLHF/DPO training made 'respond to a user turn' the aligned model's default behavior — with no user content, the model generates plausible user content first (the instruction), then in a second pass responds to it. The empty template surfaces the model's instruction prior for free."	c3::ft05::application
"If you generate 50k pairs with GPT-4 and ship them straight to the trainer, what goes wrong?"	"The cardinal sin (generate-without-filter). The 50k contains near-duplicates, low-information responses, style collapses (GPT-4's verbosity/hedging/list-bias), and likely benchmark contamination. The optimizer dutifully steers toward ALL of it. Result: a model steered toward the teacher's noise floor. Always filter — dedup, quality-score, diversity-cap, decontaminate. The filtering is where the gains are."	c3::ft05::analysis
"Why does distilabel's Magpie step handle per-model chat templates, and what's the risk of rolling your own?"	"Getting the pre-query template exactly right per model (Llama-3 vs Qwen2.5 vs Mistral have different chat templates) is a source of SILENT bugs — if the template is wrong, the model returns empty or an assistant turn instead of an instruction, and you don't notice because it looks like a degenerate sample. distilabel's Magpie step handles this per-model. Rolling your own works but requires careful per-model template construction; Path B in the lab shows the explicit mechanism."	c3::ft05::application