Synthetic Data Generation
Where SFT and preference data actually comes from. The two dominant approaches — strong-teacher distillation and Magpie self-synthesis — and the one thesis that organizes them: the gains are in filtering, not generation.
The gains are in filtering, not generation. A 500-sample curated set steers better than 50,000 raw teacher outputs. The steering wheel is the curated set, not the raw pile — the data-side complement of FT00's 'steering not teaching.'
Magpie (arXiv:2406.08464, ICLR 2025) changed the defaults. Aligned instruct models generate instructions when prompted with their pre-query template alone — no seed prompts. Two passes over the same model produce a self-consistent (instruction, response) pair. Magpie-Pro-Filtered beats the official Llama-3-8B-Instruct using only self-generated data.
Strong-teacher distillation is the workhorse when you have a target prompt distribution. Real user queries → strong model → responses. Watch for style leakage (mix 2–3 teachers) and contamination (decontaminate against evals).
Evol-Instruct (WizardLM) is now a special case, not a default. Iterative LLM-driven complexification still works for hardness amplification (Pillar 4), but Magpie samples difficulty naturally and direct teacher generation is simpler when you have prompts.
distilabel (Argilla) is the standard framework for synthetic preference/SFT pipelines. Four stages: generate → judge → filter → dedup. Composes directly with FT06's cleaning machinery.
The decontamination obligation is non-negotiable. Remove eval-overlap items before training (MMLU, GSM8K, MT-Bench, your held-out eval). Plan it at generation time — don't seed with or few-shot with benchmark items. Detailed in FT06.