Module FT06 — Dedup, Filter, Decontaminate

Dedup, Filter, Decontaminate

Data quality is where 80% of fine-tuning gains are. Generation is cheap; curation is the moat. MinHash dedup, semantic dedup, perplexity filtering, decontamination, and quality subset selection — the full clean pipeline that turns a noisy corpus into a steering wheel.

minutes

artifacts

sub-sections

Anyone can generate a million samples — almost nobody can clean them. This module teaches the full data-quality pipeline: MinHash + LSH for near-duplicate text, embedding-based semantic dedup for paraphrases, perplexity filtering to drop garbage (with the NeurIPS 2025 link to catastrophic forgetting), and decontamination to keep your evals honest. DCLM proved that curation beats compute. This is where that curation happens.

Key Claims

Load-Bearing Claims

Data quality is where 80% of fine-tuning gains are. DCLM held the model and training recipe constant and varied only the data pipeline — rigorous dedup + perplexity + quality selection beat the baseline by 6.5% absolute. Same model, same compute. Curation is the moat.

MinHash + LSH catches near-duplicates that exact hashing misses. Shingle → hash → band → verify. The LSH banding trick makes it scale beyond O(n²). Standard tools: datasketch, text-dedup.

MinHash is not enough — semantic dedup catches paraphrases. Embed every sample with a sentence-transformer, cluster, drop intra-cluster pairs with cosine ≥ 0.9. The same machinery enables quality subset selection (DCLM: quality filtering ≈ 4× compute scaling).

Perplexity filtering is two wins in one pass. Train a small LM on clean data; drop the high-perplexity tail as garbage. The NeurIPS 2025 link: low-perplexity tokens produce small gradients and mitigate catastrophic forgetting in continued pretraining.

Decontamination is non-negotiable. If your training set contains benchmark test items (MMLU, GSM8K, HumanEval, MT-Bench), every eval score you report is fiction. 13-gram matching against the test sets, every time, for every benchmark you will report.

After This Module

State the thesis — data quality is where 80% of gains are; generation is cheap, curation is the moat — and defend it with the DCLM evidence (6.5% absolute swing from curation alone).

Explain how MinHash + LSH finds near-duplicate text at scale (shingle → hash → band → verify) and contrast it with exact hashing and embedding-based semantic dedup.

Describe perplexity filtering: train a small LM on clean data, drop high-perplexity samples — and connect it to the NeurIPS 2025 finding that token-perplexity reduction mitigates catastrophic forgetting.

Define decontamination, explain why it is non-negotiable for honest evals, and execute an n-gram (13 text / 8 code) match against benchmark test sets.

Diagnose the four data-quality anti-patterns: skipping decontamination, dedup too aggressive, no quality filtering, and confusing curriculum learning with curriculum teaching.

Artifacts

Teaching Document

~3,200 words; 6 sub-sections — the thesis (DCLM anchor), MinHash+LSH mechanism, semantic dedup & quality subset selection, perplexity filtering with the NeurIPS 2025 link, decontamination, and curriculum learning

READ

Diagrams

5 Mermaid diagrams — the full clean pipeline, MinHash/LSH mechanism, decontamination flow, data-quality funnel (10K → 2K), perplexity filtering concept

READ

Slide Deck

14-slide reveal.js deck matching the FT00 design system (dark #08080c bg, #5eead4 accent, #f08080 danger for decontamination/garbage)

READ

Teaching Script

~40-minute teaching script with [SLIDE N] cues, ~2,000 words spoken at 140 wpm