# Lab Specification — Module FT05: Synthetic Data Generation

**Course**: Course 3 — LLM Fine-Tuning Masterclass
**Module**: FT05 — Synthetic Data Generation
**Lab**: "Magpie + Teacher" — generate 500 SFT samples two ways, dedup, compare
**Duration**: 60–90 minutes
**Environment**: Python 3.11+. A consumer GPU (RTX 4090 / 16–24GB) OR Apple Silicon (M-series) OR free Google Colab T4 OR an API key for an OpenAI-compatible endpoint (OpenAI, Together, Groq, OpenRouter). ~6GB free disk for an 8B model if you go local.

This lab is consumer-hardware friendly. You can run it three ways:

- **Path A — fully local, distilabel + vLLM/transformers.** Best on a 16–24GB GPU. Generates both Magpie and teacher data from local models.
- **Path B — fully local, direct transformers (no distilabel).** Best on Apple Silicon or a smaller GPU. Uses Llama-3.2-1B-Instruct or Qwen2.5-1.5B-Instruct so it fits in ~4GB.
- **Path C — API teacher + local Magpie, or fully API.** Best if you have no local GPU. Uses an OpenAI-compatible API for the teacher; Magpie can be done via a free inference endpoint (Together/Groq serve Llama-3-8B-Instruct).

The solution key covers all three. Pick one.

---

## Learning objectives

By the end of this lab you will have:

1. **Generated SFT data via Magpie self-synthesis** — fed an aligned model its pre-query template with the user turn empty, captured the generated instruction, then generated the response in a second pass. No seed prompts.
2. **Generated SFT data via strong-teacher distillation** — sent a fixed set of seed prompts to a stronger model (API or larger local) and captured responses.
3. **Deduplicated both sets** with MinHash on instruction n-grams (the FT06 preview step).
4. **Compared the two approaches on a 20-sample manual review** against a rubric (instruction diversity, response quality, self-consistency, contamination risk).
5. **Stated, in your own words, why the thesis holds**: the gains are in filtering, not generation — and which approach you'd pick for your own project, and why.

This lab does NOT train a model. The point is to feel the two generation methods and the filtering step before you spend GPU on training. Training is FT11.

---

## Phase 0 — Environment setup (5 min)

### Path A — local with distilabel (recommended if you have a 16GB+ GPU)

```bash
python3.11 -m venv ft05-env && source ft05-env/bin/activate
pip install -q "distilabel[transformers,vllm]" datasets datasketch
# vLLM is optional but ~5x faster; transformers is the fallback
```

### Path B — local with direct transformers (Apple Silicon or small GPU)

```bash
python3.11 -m venv ft05-env && source ft05-env/bin/activate
pip install -q transformers accelerate torch datasketch datasets
```

### Path C — API teacher + (local or API) Magpie

```bash
python3.11 -m venv ft05-env && source ft05-env/bin/activate
pip install -q openai datasketch datasets
# For local Magpie on a small machine, also:
pip install -q transformers accelerate torch
```

Verify the stack:

```python
import torch
print(f"PyTorch: {torch.__version__}")
print(f"MPS available: {torch.backends.mps.is_available()}")  # Apple Silicon
print(f"CUDA available: {torch.cuda.is_available()}")          # NVIDIA
```

Set up your output directory:

```python
import os, json
os.makedirs("ft05-out", exist_ok=True)
N_TARGET = 500  # raw pairs per method; we'll dedup down
```

---

## Phase 1 — Magpie self-synthesis (20–30 min)

The goal: generate 500 `(instruction, response)` pairs by prompting an aligned model with its pre-query template alone. No seed prompts.

### Path A — distilabel Magpie step (cleanest)

```python
# magpie_distilabel.py
from distilabel.llms import TransformersLLM
from distilabel.pipeline import Pipeline
from distilabel.steps import Magpie, StepResources

MODEL_ID = "meta-llama/Llama-3.1-8B-Instruct"  # or Qwen/Qwen2.5-7B-Instruct

with Pipeline(name="magpie-ft05") as pipeline:
    magpie = Magpie(
        llm=TransformersLLM(
            model=MODEL_ID,
            torch_dtype="bfloat16",
            device_map="auto",
            generation_kwargs={"temperature": 1.0, "max_new_tokens": 256, "do_sample": True},
        ),
        n_turns=1,                 # single-turn (instruction, response)
        n_rows=N_TARGET * 2,       # over-generate; we'll dedup down to 500
        batch_size=8,
        # Magpie over-samples then you filter; 2x is a safe ratio for 500 final
        only_instruction=False,    # we want both instruction AND response
        system_prompt=None,        # no system prompt — that's the point
    )
    # (no downstream step here — we save and dedup separately)

if __name__ == "__main__":
    ds = pipeline.run(use_cache=False)
    out = ds["default"]["train"]
    rows = [{"instruction": r["instruction"], "response": r["response"]} for r in out]
    with open("ft05-out/magpie_raw.jsonl", "w") as f:
        for r in rows:
            f.write(json.dumps(r) + "\n")
    print(f"Wrote {len(rows)} raw Magpie pairs")
```

### Path B/C — direct transformers (the mechanism, fully visible)

This is the explicit two-pass implementation. Use this if you want to *see* the pre-query-template trick, or if you're on a machine where distilabel's pipeline overhead isn't worth it.

```python
# magpie_direct.py
import torch, json
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL_ID = "Qwen/Qwen2.5-1.5B-Instruct"  # small enough for Apple Silicon / 4GB GPU
# On a 16GB+ GPU use "meta-llama/Llama-3.1-8B-Instruct" for better quality.

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID, torch_dtype=torch.float16, device_map="auto", trust_remote_code=True,
)
model.eval()

def gen(messages, max_new_tokens=256, temperature=1.0):
    text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
    inputs = tokenizer(text, return_tensors="pt").to(model.device)
    with torch.no_grad():
        out = model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            do_sample=True,
            temperature=temperature,
            top_p=0.95,
        )
    return tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)

# PASS 1 — instruction generation.
# The trick: feed the chat template with an EMPTY user turn.
# apply_chat_template with content="" produces the pre-query template.
def gen_instruction():
    empty_user = [{"role": "user", "content": ""}]
    raw = gen(empty_user, max_new_tokens=128, temperature=1.0)
    # The model samples a plausible user query. Strip any trailing assistant turn.
    instr = raw.strip().split("\n\n")[0].strip()
    return instr

# PASS 2 — response generation.
# Place the generated instruction in the user turn PROPERLY and re-sample.
def gen_response(instruction):
    proper = [{"role": "user", "content": instruction}]
    return gen(proper, max_new_tokens=512, temperature=0.7).strip()

rows = []
attempts = 0
while len(rows) < N_TARGET and attempts < N_TARGET * 3:
    attempts += 1
    try:
        instr = gen_instruction()
        if len(instr) < 8:           # too short, skip
            continue
        resp = gen_response(instr)
        if len(resp) < 16:           # degenerate, skip
            continue
        rows.append({"instruction": instr, "response": resp})
        if len(rows) % 25 == 0:
            print(f"  Magpie: {len(rows)}/{N_TARGET}")
    except Exception as e:
        print(f"  skip: {e}")
        continue

with open("ft05-out/magpie_raw.jsonl", "w") as f:
    for r in rows:
        f.write(json.dumps(r) + "\n")
print(f"Wrote {len(rows)} raw Magpie pairs to ft05-out/magpie_raw.jsonl")
```

### Path C (API-only Magpie via an OpenAI-compatible endpoint)

If you have no local GPU at all, you can run Magpie via an inference API that serves an aligned instruct model (Groq and Together both serve Llama-3-8B-Instruct). The trick: send an empty-string user message and capture what comes back as the "instruction," then re-send it properly.

```python
# magpie_api.py
from openai import OpenAI
import json, os

# Set GROQ_API_KEY or TOGETHER_API_KEY in env
client = OpenAI(
    api_key=os.environ["GROQ_API_KEY"],
    base_url="https://api.groq.com/openai/v1",  # or together
)
MODEL = "llama-3.1-8b-instant"  # Groq; Together uses different names

def gen(messages, max_tokens=256, temp=1.0):
    r = client.chat.completions.create(
        model=MODEL, messages=messages,
        max_tokens=max_tokens, temperature=temp,
    )
    return r.choices[0].message.content.strip()

rows = []
attempts = 0
while len(rows) < N_TARGET and attempts < N_TARGET * 3:
    attempts += 1
    instr = gen([{"role": "user", "content": ""}], max_tokens=128, temp=1.0)
    if not instr or len(instr) < 8:
        continue
    resp = gen([{"role": "user", "content": instr}], max_tokens=512, temp=0.7)
    if len(resp) < 16:
        continue
    rows.append({"instruction": instr, "response": resp})

with open("ft05-out/magpie_raw.jsonl", "w") as f:
    for r in rows:
        f.write(json.dumps(r) + "\n")
print(f"Wrote {len(rows)} raw Magpie pairs")
```

**Record**: open `ft05-out/magpie_raw.jsonl` and read 5 random instructions. Note their diversity. The Magpie thesis is that these came from *no seed prompts* — the model generated them itself from its instruction prior.

> **What just happened (the teaching moment):** You fed the aligned model its pre-query template with the user turn empty. It generated an instruction anyway — because RLHF/DPO training made instruction-responding the default, and with no user content to respond to, it surfaced a plausible user query from its prior. Then you placed that query properly and the model answered it. Two passes, same model, no seeds. That is Magpie.

---

## Phase 2 — Strong-teacher distillation (15–25 min)

The goal: generate 500 `(instruction, response)` pairs by sending a fixed set of seed prompts to a stronger model and capturing responses. We provide a small diverse seed set; you scale it up.

### Seed prompts

We use 50 seed prompts spanning 5 domains (10 each), and over-sample to get 500 pairs by generating 10 responses each with high temperature (diversity). Save as `seeds.jsonl`:

```python
# seeds.py
import json
SEEDS = [
    # Coding (10)
    "Write a Python function to compute the nth Fibonacci number iteratively.",
    "Explain the difference between a mutex and a semaphore with an example.",
    "Write a SQL query to find the second-highest salary in an employees table.",
    "Implement a simple LRU cache in Python with get and put methods.",
    "What is the time complexity of quicksort in the average and worst case?",
    "Write a regex to validate an email address and explain its parts.",
    "Explain what a closure is in JavaScript with a concrete example.",
    "Write a bash one-liner to find the 10 largest files in a directory.",
    "Implement binary search in Python and explain when it's preferable to linear search.",
    "Explain the CAP theorem and give an example of each trade-off.",
    # Reasoning / math (10)
    "A train travels 60 mph for 2 hours, then 80 mph for 1.5 hours. What is the average speed?",
    "Prove that the square root of 2 is irrational.",
    "If you flip 3 coins, what is the probability of getting at least 2 heads?",
    "Explain Bayes' theorem with a worked medical-testing example.",
    "Solve: 3x + 7 = 22. Show each step.",
    "What is the derivative of x^3 * sin(x)? Show the product rule.",
    "Explain the difference between correlation and causation with an example.",
    "A rectangle has perimeter 40 and area 96. Find its dimensions.",
    "Explain the Monty Hall problem and why switching is correct.",
    "What is a p-value, and what is it NOT?",
    # Writing / creative (10)
    "Write a one-paragraph product description for a smart water bottle.",
    "Draft a polite email declining a meeting invitation.",
    "Write a short poem about the ocean in the style of haiku.",
    "Summarize the plot of Romeo and Juliet in three sentences.",
    "Write a LinkedIn post announcing a job change, professional tone.",
    "Rewrite this sentence to be more concise: 'Due to the fact that it was raining, we decided to postpone the event.'",
    "Write a restaurant review for a fictional Italian place, 100 words.",
    "Compose a toast for a friend's wedding, warm and brief.",
    "Write a headline and lede for an article about a new battery technology.",
    "Draft a customer-service reply apologizing for a delayed shipment.",
    # Knowledge / explanation (10)
    "Explain how the TCP three-way handshake works.",
    "What is the difference between TCP and UDP? When would you use each?",
    "Explain how HTTPS encryption works at a high level.",
    "What is the Krebs cycle and why is it important?",
    "Explain the causes of World War I in four bullet points.",
    "What is quantum entanglement, explained for a non-physicist?",
    "Describe how a vaccine trains the immune system.",
    "What is the difference between machine learning and deep learning?",
    "Explain what inflation is and why central banks target ~2%.",
    "What is the carbon cycle? Name its main reservoirs.",
    # Open-ended / instruction-following (10)
    "List 5 healthy breakfast ideas that take under 10 minutes to prepare.",
    "What are the best practices for writing clean, maintainable code?",
    "Give me a weekly workout plan for a beginner, no equipment.",
    "Explain how to prioritize tasks when everything feels urgent.",
    "What should I look for when buying a used car?",
    "Give me 3 tips for improving sleep quality.",
    "How do I start learning to play the guitar as an adult?",
    "What are the key differences between renting and buying a home?",
    "Suggest 5 books for someone who liked 'Project Hail Mary'.",
    "How do I write a good resume summary with 5 years of experience?",
]
with open("ft05-out/seeds.jsonl", "w") as f:
    for s in SEEDS:
        f.write(json.dumps({"instruction": s}) + "\n")
print(f"Wrote {len(SEEDS)} seed prompts")
```

### Generate (local teacher — Path A/B)

```python
# teacher_local.py
import torch, json
from transformers import AutoModelForCausalLM, AutoTokenizer

# Use a STRONGER model than the Magpie source for a fair comparison.
# If Magpie ran on Qwen2.5-1.5B, run the teacher on Qwen2.5-7B-Instruct.
# If Magpie ran on Llama-3-8B, run the teacher on Llama-3-70B (needs ~40GB)
#  or use an API for the teacher (Path C below).
TEACHER_ID = "Qwen/Qwen2.5-7B-Instruct"

tokenizer = AutoTokenizer.from_pretrained(TEACHER_ID, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    TEACHER_ID, torch_dtype=torch.float16, device_map="auto", trust_remote_code=True,
)
model.eval()

seeds = [json.loads(l)["instruction"] for l in open("ft05-out/seeds.jsonl")]

def gen(prompt, max_new_tokens=512, temperature=0.8):
    msgs = [{"role": "user", "content": prompt}]
    text = tokenizer.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
    inputs = tokenizer(text, return_tensors="pt").to(model.device)
    with torch.no_grad():
        out = model.generate(
            **inputs, max_new_tokens=max_new_tokens,
            do_sample=True, temperature=temperature, top_p=0.95,
        )
    return tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True).strip()

rows = []
# 10 responses per seed (high temp) = 500 pairs
RESPONSES_PER_SEED = 10
for i, seed in enumerate(seeds):
    for j in range(RESPONSES_PER_SEED):
        resp = gen(seed, temperature=0.9 + 0.05 * (j % 3))  # vary temp slightly
        if len(resp) < 16:
            continue
        rows.append({"instruction": seed, "response": resp})
    if (i + 1) % 5 == 0:
        print(f"  Teacher: {len(rows)}/{len(seeds) * RESPONSES_PER_SEED}")

with open("ft05-out/teacher_raw.jsonl", "w") as f:
    for r in rows:
        f.write(json.dumps(r) + "\n")
print(f"Wrote {len(rows)} raw teacher pairs to ft05-out/teacher_raw.jsonl")
```

### Generate (API teacher — Path C)

```python
# teacher_api.py
from openai import OpenAI
import json, os, time

client = OpenAI()  # OPENAI_API_KEY in env; or point base_url at Together/OpenRouter
MODEL = "gpt-4o-mini"  # cheap, strong enough for a lab
# Or: "gpt-4o", "claude-3-5-sonnet" (via OpenRouter), "deepseek-chat"

seeds = [json.loads(l)["instruction"] for l in open("ft05-out/seeds.jsonl")]
RESPONSES_PER_SEED = 10

rows = []
for i, seed in enumerate(seeds):
    for j in range(RESPONSES_PER_SEED):
        for attempt in range(3):
            try:
                r = client.chat.completions.create(
                    model=MODEL,
                    messages=[{"role": "user", "content": seed}],
                    max_tokens=512, temperature=0.9 + 0.05 * (j % 3),
                )
                resp = r.choices[0].message.content.strip()
                if len(resp) >= 16:
                    rows.append({"instruction": seed, "response": resp})
                break
            except Exception as e:
                print(f"  retry: {e}")
                time.sleep(2)
    if (i + 1) % 5 == 0:
        print(f"  Teacher: {len(rows)}/{len(seeds) * RESPONSES_PER_SEED}")

with open("ft05-out/teacher_raw.jsonl", "w") as f:
    for r in rows:
        f.write(json.dumps(r) + "\n")
print(f"Wrote {len(rows)} raw teacher pairs")
```

**Record**: open `ft05-out/teacher_raw.jsonl`. Note that the *instructions* are the 50 seeds repeated 10x each — the diversity is in the responses (high-temp variation), not the prompts. This is the structural difference from Magpie, where the instructions themselves are diverse.

> **What just happened (the teaching moment):** You sent 50 fixed seed prompts to a stronger model and captured 10 high-temperature responses each. The teacher's responses are higher quality than what a small aligned model would self-generate — but the *instruction distribution* is whatever your 50 seeds covered, not the teacher's full prior. This is the trade: teacher distillation gives better responses on a narrower prompt distribution; Magpie gives broader prompt diversity with weaker responses.

---

## Phase 3 — Dedup (10 min)

Both sets will have near-duplicates. Magpie generates similar instructions across samples (the model's prior has modes); the teacher set has 10 responses to each of 50 prompts (some will be near-identical under high temp). Dedup with MinHash on instruction n-grams.

```python
# dedup.py
import json
from datasketch import MinHash, MinHashLSH

def load(path):
    return [json.loads(l) for l in open(path)]

def minhash_instruction(instr, num_perm=128):
    m = MinHash(num_perm=num_perm)
    # 3-grams on whitespace tokens
    tokens = instr.lower().split()
    for i in range(len(tokens) - 2):
        m.update(" ".join(tokens[i:i+3]).encode("utf-8"))
    return m

def dedup(rows, threshold=0.7):
    lsh = MinHashLSH(threshold=threshold, num_perm=128)
    kept = []
    for i, r in enumerate(rows):
        mh = minhash_instruction(r["instruction"])
        if lsh.query(mh):  # near-duplicate of something already kept
            continue
        lsh.insert(i, mh)
        kept.append(r)
    return kept

for name in ["magpie", "teacher"]:
    rows = load(f"ft05-out/{name}_raw.jsonl")
    print(f"{name}: {len(rows)} raw")
    deduped = dedup(rows, threshold=0.7)
    print(f"{name}: {len(deduped)} after dedup ({len(rows) - len(deduped)} removed)")
    with open(f"ft05-out/{name}_dedup.jsonl", "w") as f:
        for r in deduped:
            f.write(json.dumps(r) + "\n")
```

**Record**: the dedup ratios. Magpie typically loses 10–30% to near-duplicate instructions (the prior has modes). The teacher set loses very little on instructions (they're 50 distinct seeds) but you could also dedup on *responses* — try it: change `minhash_instruction` to hash the response and see how many near-duplicate responses the high-temp sampling produced. Usually 5–15%.

> **What just happened (the teaching moment):** You applied one stage of the FT06 funnel — dedup. This is where the thesis bites: you generated 500 each, and you're keeping fewer. If you needed 500 *clean* pairs, you should have generated 1000–1500 and filtered to 500. The ratio is the price of a clean steering wheel.

---

## Phase 4 — Manual review (15 min)

From each deduped set, sample 10 random pairs. Review all 20 against the rubric below. This is the qualitative counterpart to the quantitative filter — and it's where you, the engineer, develop the judgment that no automated filter can give you.

```python
# sample_review.py
import json, random
random.seed(42)

for name in ["magpie", "teacher"]:
    rows = [json.loads(l) for l in open(f"ft05-out/{name}_dedup.jsonl")]
    sample = random.sample(rows, min(10, len(rows)))
    print(f"\n=== {name.upper()} — 10 sample pairs ===\n")
    for i, r in enumerate(sample, 1):
        print(f"--- {name} sample {i} ---")
        print(f"INSTRUCTION: {r['instruction']}")
        print(f"RESPONSE: {r['response'][:400]}{'...' if len(r['response']) > 400 else ''}")
        print()
```

### Review rubric (score each pair 0–2 on each axis)

| Axis | 0 — fail | 1 — pass | 2 — strong |
| --- | --- | --- | --- |
| **Instruction clarity** | Garbled / not actually an instruction | Clear but trivial | Clear, non-trivial, answerable |
| **Response correctness** | Wrong or nonsensical | Mostly right, minor issues | Correct and complete |
| **Self-consistency** | Response doesn't match the instruction | Loosely matches | Directly answers what was asked |
| **Contamination risk** | Looks like a benchmark item (MMLU/GSM8K shape) | — | Clearly original, not eval-shaped |
| **Diversity** (set-level) | Same as the others | Some variety | Spans topics/styles |

For each set, compute the mean score per axis and the overall mean. Then answer:

1. Which set had higher *instruction diversity*? (Expect Magpie — its instructions came from the model's full prior; the teacher's came from 50 fixed seeds.)
2. Which set had higher *response quality*? (Expect the teacher — it's a stronger model answering.)
3. Which set had higher *self-consistency*? (Expect Magpie — same model generated both sides.)
4. Did you spot any *contamination-risk* items? (Benchmark-shaped Q&A. Flag them. This is the FT06 obligation in miniature.)
5. If you were building an SFT set for a *general-purpose assistant*, which would you start from? If for a *domain-specific* assistant (say, coding), which?

---

## Phase 5 — The thesis, in your own words (5 min)

No code. Write 4–6 sentences answering:

1. State the thesis of FT05 in one sentence. How did this lab demonstrate it?
2. Compare the two approaches on the four rubric axes. Where did each win?
3. If you needed 5,000 clean SFT pairs for a real project, what would your generation-to-filtering ratio be, and why?
4. Name one contamination risk you observed (or would expect) and how you'd mitigate it (FT06 preview).
5. Which approach would you pick for your own project, and why? (There is no wrong answer if you justify it — but "both, mixed, filtered" is often the right one.)

---

## Deliverables

Submit `ft05-lab-report.md`:

- [ ] Phase 1: `magpie_raw.jsonl` (≥500 raw pairs); 5 sample instructions quoted; note diversity
- [ ] Phase 2: `teacher_raw.jsonl` (≥500 raw pairs); the 50-seed prompt list; note response variation
- [ ] Phase 3: dedup output for both sets, with the before/after counts and the dedup ratios
- [ ] Phase 4: the 20-sample review table (5 axes × 20 pairs) with per-set means; your 5 answers
- [ ] Phase 5: your 4–6 sentence thesis statement
- [ ] (Optional) your choice of approach for a real project, with justification

Attach the two `_dedup.jsonl` files.

---

## Solution key

- **Phase 1 (Magpie)**: a correct run produces ~500 `(instruction, response)` pairs from the aligned model with no seed prompts. Instructions should span topics (coding, reasoning, writing, knowledge, open-ended) — the model's instruction prior is broad. Responses are self-consistent (the same model generated both). Quality varies; some responses will be weak — that's expected and is the point of Phase 3–4.
  - Common bug (Path B/C): the empty-user-turn trick produces an *assistant* turn instead of an instruction. Fix: ensure `apply_chat_template` with `content=""` produces the pre-query template ending in the assistant-turn opener, and that `skip_special_tokens=True` strips it. If the model returns nothing or returns `""`, the chat template is wrong for that model — switch to Llama-3 or Qwen2.5, both of which work.
  - Common bug: instructions come out as fragments ("Write a..."). Fix: increase `max_new_tokens` for the instruction pass to 256, and strip at the first `\n\n`.

- **Phase 2 (teacher)**: a correct run produces ~500 pairs from 50 seeds × 10 responses. Instructions are the 50 fixed seeds (lower diversity than Magpie). Responses are higher quality than the small aligned model's Magpie output (the teacher is stronger). Some near-duplicate responses appear under high temperature — Phase 3 catches them.

- **Phase 3 (dedup)**: Magpie typically retains 70–90% after instruction-level dedup (threshold 0.7 Jaccard on 3-grams). The teacher set retains ~100% on instructions (50 distinct seeds) but 85–95% on responses if you dedup responses too. If your Magpie retention is below 60%, your temperature is too low (the prior collapsed to modes) — bump to 1.0–1.2.

- **Phase 4 (review)**: typical results on a small aligned model (Qwen2.5-1.5B) vs an API teacher (gpt-4o-mini):
  | Axis | Magpie | Teacher |
  | --- | --- | --- |
  | Instruction diversity | **higher** (full prior) | lower (50 seeds) |
  | Response quality | lower (small model) | **higher** (strong teacher) |
  | Self-consistency | **higher** (same model both sides) | lower (different prompt/response generators) |
  | Contamination risk | low (model-generated, not benchmark-shaped) | medium (seeds may have eval-shaped items if you wrote them carelessly) |
  | Set-level diversity | **higher** | lower |
  Note: if you ran Magpie on Llama-3-8B and the teacher on a 70B or GPT-4, response quality narrows but the teacher still wins on average.

- **Phase 5 (thesis)**: a correct statement names (a) "the gains are in filtering, not generation" — demonstrated by the dedup ratio and the manual review surfacing items you'd never ship; (b) the trade: Magpie wins diversity and self-consistency, teacher wins response quality; (c) the ratio: generate 4–10x what you'll train on, filter to the clean subset; (d) a contamination observation (e.g., "one Magpie instruction was MMLU-shaped: 'Which of the following is a characteristic of...' — I'd flag it for FT06 removal"); (e) a justified choice — for general assistants, "Magpie + teacher mixed, filtered" is the most defensible; for domain-specific, "teacher distillation from domain prompts" is usually right because you control the prompt distribution.

---

## Stretch goals

1. **Run distilabel end-to-end.** Replace the Phase 1 + Phase 3 scripts with a single distilabel `Pipeline` that has `Magpie` → `Step` (quality filter via a judge LLM) → `Step` (MinHash dedup) → `Distiset`. Compare the output to your hand-rolled version. (Sets up FT06's full funnel.)
2. **Add a judge step.** For each Magpie pair, ask an API model to score the response 1–5. Keep only ≥4. How much did the set shrink? Did the manual-review quality of the survivors go up? (This is the FT06 quality-filter step, previewed.)
3. **Generate preference pairs.** For 50 of the teacher prompts, generate 2 responses each at different temperatures. Ask a judge which is better. You now have 50 DPO pairs. (Sets up FT13 — DPO.)
4. **Decontaminate.** Download MMLU (or a subset) and compute n-gram overlap between your Magpie instructions and MMLU questions. Remove any pair with overlap above a threshold. How many did you remove? (Sets up FT06 — the obligation.)
5. **Mix the two sets.** Concatenate the deduped Magpie and teacher sets, dedup the union, and you have a hybrid set with both diversity (Magpie) and quality (teacher). This is the production pattern. (Sets up the FT11 trainer, which will consume this set.)