# Diagrams — Module FT05: Synthetic Data Generation

**Module**: FT05 — Synthetic Data Generation
**Diagram count**: 5
**Tool**: Mermaid (primary). Each diagram validated in [Mermaid Live Editor](https://mermaid.live).

---

## Diagram 1 — Strong-Teacher Distillation Pipeline

**Type**: Linear pipeline
**Purpose**: The workhorse synthetic-data approach. Seed prompts go in; a strong model generates responses; (optionally) the strong model also generates prompts; the pairs flow to filtering.
**Reading the diagram**: Left to right. The top branch is "I have prompts" (response-only generation). The bottom branch is "I need prompts too" (Self-Instruct-style bootstrap). Both converge on the filter/dedup/decontaminate stage that is non-negotiable (FT06).

```mermaid
flowchart LR
  Seeds["Seed prompts\n(real user logs OR\nhand-written OR\nLLM-bootstrapped)"]
  Teacher["STRONG TEACHER\nGPT-4 · Claude · R1 · 70B"]
  Resp["Responses"]
  Pair["(prompt, response)\nSFT pairs"]
  Clean["Filter · Dedup · Decontaminate\n(FT06 — non-negotiable)"]
  Out["Clean SFT set"]

  SelfInst["Self-Instruct bootstrap\n(optional)\nteacher generates\nprompts too"]
  Seeds --> Teacher
  SelfInst -.-> Seeds
  Teacher --> Resp
  Resp --> Pair
  Seeds --> Pair
  Pair --> Clean
  Clean --> Out

  style Seeds fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style Teacher fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style Resp fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style Pair fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style Clean fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style Out fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style SelfInst fill:#08080c,stroke:rgba(255,255,255,0.12),stroke-dasharray: 4 2,color:#9494a0
```

---

## Diagram 2 — Magpie Self-Synthesis (the two-pass mechanism)

**Type**: Sequence with annotation
**Purpose**: How Magpie generates a full (instruction, response) pair from an aligned model with NO seed prompts. The key insight: the aligned model's pre-query template is itself a sufficient prompt.
**Reading the diagram**: Top to bottom. Pass 1 feeds the empty user turn → model samples an instruction. Pass 2 places that instruction properly → model samples the response. Same model, both passes. No seeds, no teacher.

```mermaid
flowchart TD
  Pre["PRE-QUERY TEMPLATE\nchat template up to user-turn opener\nUSER CONTENT IS EMPTY"]
  Aligned["ALIGNED MODEL\nLlama-3-8B-Instruct\nQwen2.5-Instruct · Mistral-Instruct"]

  Pass1["PASS 1 — instruction generation\nfeed empty pre-query template\nmodel samples a plausible user query"]
  Instr["Generated instruction\n(e.g. 'Explain TCP handshake')"]

  Pass2["PASS 2 — response generation\nplace instruction in user turn properly\nfeed back to SAME model\nmodel samples the response"]
  Resp["Generated response\n(self-consistent with the instruction)"]

  Pair["(instruction, response) pair\nself-synthesized end-to-end\nno seed prompts · no external teacher"]

  Pre --> Aligned
  Aligned --> Pass1
  Pass1 --> Instr
  Instr --> Pass2
  Pass2 --> Resp
  Resp --> Pair
  Instr --> Pair

  style Pre fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style Aligned fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style Pass1 fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style Pass2 fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style Instr fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style Resp fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style Pair fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
```

---

## Diagram 3 — Evol-Instruct Complexification Loop

**Type**: Iterative loop with operators
**Purpose**: How WizardLM's Evol-Instruct turns a seed instruction into a tree of harder, more diverse descendants. Still used for hardness amplification; superseded as a default by Magpie + direct teacher generation.
**Reading the diagram**: A seed instruction enters. Two operators act on it: in-depth (add constraints, deepen reasoning) and in-breadth (topic-related variant). The loop iterates. Output is a tree of evolved instructions, then deduped and responded to.

```mermaid
flowchart TD
  Seed["SEED instruction\n'Write a Python sort function'"]
  Loop{"Evol-Instruct\nLLM-driven\niteration"}

  Depth["IN-DEPTH evolving\nadd constraints\ndeepen reasoning\nrequire more steps"]
  Breadth["IN-BREADTH evolving\ntopic-related variant\n(diversify)"]

  Evolved["Evolved instructions\n'tree' of harder descendants"]

  Seed --> Loop
  Loop --> Depth
  Loop --> Breadth
  Depth --> Evolved
  Breadth --> Evolved
  Evolved -.iterate.-> Loop

  Evolved --> Dedup["Dedup · filter\n(FT06)"]
  Dedup --> Respond["Generate responses\n(teacher or self)"]
  Respond --> Out["SFT set with\namplified hardness"]

  style Seed fill:#14141f,stroke:rgba(255,255,255,0.12),color:#e4e4e8
  style Loop fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style Depth fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style Breadth fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style Evolved fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style Dedup fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style Respond fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style Out fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
```

---

## Diagram 4 — The distilabel Pipeline (generate → judge → filter → dedup)

**Type**: DAG with parallel branches
**Purpose**: The four standard stages distilabel (Argilla) gives you. Generate (Magpie or teacher), Judge (preference pairs / quality scores), Filter (rules, decontam), Dedup (MinHash). Output is an HF Distiset ready for FT11's trainer.
**Reading the diagram**: Left to right. The generate stage has two interchangeable inputs (Magpie self-synthesis OR teacher distillation). The judge stage produces both quality scores (for filtering) and preference labels (for DPO). Filter and dedup are the FT06 machinery.

```mermaid
flowchart LR
  subgraph Gen["GENERATE (Step)"]
    Magpie["Magpie step\nself-synthesis\nno seeds"]
    Teacher["Teacher step\nseed prompts\n+ strong model"]
  end

  Pairs["(instruction, response)\npairs"]

  subgraph Jud["JUDGE (Step)"]
    Quality["Quality score\n1–5 per response"]
    Pref["Preference label\nbetter/worse\n-> DPO pair"]
  end

  subgraph Filt["FILTER (Step)"]
    Rules["Length · format\nlanguage · regex"]
    Decon["Decontaminate\nMMLU · GSM8K · MT-Bench\nyour held-out eval"]
    Diversity["Diversity cap\nper-cluster count"]
  end

  Dedup["DEDUP (Step)\nMinHash on instruction n-grams\nkeep one per cluster"]
  Distiset["Distiset (HF Dataset)\nready for FT11 trainer"]

  Magpie --> Pairs
  Teacher --> Pairs
  Pairs --> Quality
  Pairs --> Pref
  Quality --> Rules
  Pref --> Rules
  Rules --> Decon
  Decon --> Diversity
  Diversity --> Dedup
  Dedup --> Distiset

  style Magpie fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style Teacher fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style Pairs fill:#14141f,stroke:rgba(255,255,255,0.12),color:#e4e4e8
  style Quality fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style Pref fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style Rules fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style Decon fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style Diversity fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style Dedup fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style Distiset fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
```

---

## Diagram 5 — The Data-Quality Funnel

**Type**: Funnel / volume-reduction
**Purpose**: The thesis of this module as a diagram: generate 4–10x what you'll train on, filter aggressively to the clean subset. The gains are in the filtering, not the generation.
**Reading the diagram**: Top to bottom. Wide at the top (raw generation), narrow at the bottom (clean SFT set). Each stage removes a specific failure mode. The ratio at the bottom is the load-bearing number — 5k clean pairs steer better than 50k raw.

```mermaid
flowchart TD
  Raw["RAW GENERATION\n~20–50k pairs\n(Magpie or teacher)"]
  Dedup1["After DEDUP\n~10–20k\n(removed near-duplicates)"]
  Qual["After QUALITY FILTER\n~5–10k\n(kept judge score >= 4)"]
  Div["After DIVERSITY CAP\n~3–6k\n(capped per cluster)"]
  Decon["After DECONTAMINATION\n~2.8–5.8k\n(removed eval-overlap)"]
  Clean["CLEAN SFT SET\n~2.5–5k pairs\nthe steering wheel"]

  Raw --> Dedup1
  Dedup1 --> Qual
  Qual --> Div
  Div --> Decon
  Decon --> Clean

  Raw -.->|"~50% removed\nduplicates"| Dedup1
  Dedup1 -.->|"~50% removed\nlow-quality"| Qual
  Qual -.->|"~30% removed\nover-represented topics"| Div
  Div -.->|"~5% removed\ncontaminated"| Decon

  style Raw fill:#14141f,stroke:rgba(255,255,255,0.12),color:#e4e4e8
  style Dedup1 fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style Qual fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style Div fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style Decon fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style Clean fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
```

---

## Validation notes

- All five diagrams use the course design system colors: `#14141f` panel fill, `#5eead4` accent for primary, `rgba(255,255,255,0.12)` for secondary borders, `#e4e4e8` / `#9494a0` for text.
- Paste each into [Mermaid Live Editor](https://mermaid.live) to render. All use stable Mermaid syntax (`flowchart`) supported in current Mermaid (v10.4+).
- For the slide deck (artifact 03), these are rendered as static SVG/PNG captures from Mermaid Live, inlined into reveal.js.
- Diagram 4 uses `subgraph` to group the four distilabel stages; the subgraph styling is applied via node-level styles so it renders consistently across Mermaid versions.