Fine-Tuning
Masterclass

Fine-tuning steers behavior; it does not teach knowledge. The five-layer mental model that anchors the entire course — and the one judgment that prevents the cardinal error of treating fine-tuning as knowledge injection.

Memory, not compute, is the binding constraint in fine-tuning. The three VRAM consumers, the rules of thumb that place a job in the right GPU class, and the three-question framework you run before you ever load a model.

What 'open' actually means for a base model — the distinction between open-weights, open-data, and open-recipe releases, and why it is load-bearing the moment you deploy in a regulated domain where a regulator can ask 'what did this model see?'

The base selection rubric — task, hardware, license, openness, ecosystem — weighted by use case. Why MiniCPM5-1B is the course default, when to graduate, and which properties matter (and which people over-index on).

The three canonical dataset formats, the one right way to apply a chat template, and the five-second inspection loop that catches every silent formatting bug before it costs you a training run.

Where SFT and preference data actually comes from. The two dominant approaches — strong-teacher distillation and Magpie self-synthesis — and the one thesis that organizes them: the gains are in filtering, not generation.

Data quality is where 80% of fine-tuning gains are. Generation is cheap; curation is the moat. MinHash dedup, semantic dedup, perplexity filtering, decontamination, and quality subset selection — the full clean pipeline that turns a noisy corpus into a steering wheel.

The silent-bugs module — where 80% of silent training failures live. The model trains, loss goes down, but the model learns the wrong thing because the tokens were wrong. The top three bug classes, the apply_chat_template rule, and the vocab-extension workflow.

The first module of Pillar 2 — where you do your first real fine-tune. LoRA decomposes a weight update into W = W0 + BA (frozen base + tiny trainable B,A); QLoRA freezes the base at 4-bit so a 7B model trains on a 24GB consumer card. The four LoRA knobs, QLoRA's three innovations, the full train-and-merge workflow, and the modern all-linear target default.

What grew on the LoRA trunk in 2024, and which branches you actually use in 2026. DoRA's magnitude/direction decomposition closes ~half the gap between LoRA and full FT at ~5–10% more VRAM; rsLoRA's α/√r scaling unlocks high rank; PiSSA is a drop-in SVD initialization; GaLore is the bridge to full-parameter FT. The practical sweet spot is two flags: use_dora=True and use_rslora=True.

The judgment module at the adapter/steer boundary. Default to PEFT for ~95% of production adaptation; full fine-tuning only for genuinely higher-rank tasks — and the structural reason why the choice is a design decision, not a cost decision.

The capstone of Pillar 2. Run a complete, instrumented SFT job with TRL's SFTTrainer — dataset to merged model — and learn to read the loss curve, choose the TrainingArguments levers, and diagnose the three failure modes (loss not decreasing, exploding, plateauing).

The opener of Pillar 3. SFT steers behavior — format, style, instruction-following, tool-call formatting, refusal calibration — without injecting knowledge. Build a high-quality SFT mixture (general + domain + tool + safety), diagnose the three failure modes (catastrophic forgetting, mode collapse, format leakage), and know when to stop at SFT versus escalate to preference methods (FT13).

The module that opens Pillar 3. Trace preference alignment from RLHF/PPO to DPO and its variants (IPO, KTO, ORPO, SimPO), learn the reparameterization that made the reward model disappear, and run DPO on top of your SFT'd model — then measure the win-rate improvement.

The flagship 2025–2026 module. Reasoning models (o1 / R1 / Qwen3-style) are trained via RL on verifiable rewards. GRPO (Group Relative Policy Optimization, DeepSeek) samples N responses per prompt and computes advantages relative to the group mean — eliminating the PPO critic and roughly halving memory. Why RL not DPO for reasoning (verifiable rewards enable on-policy exploration), the GRPO≈RLOO clarification, the GRPO++ landscape (DAPO, Dr. GRPO, GSPO), the DeepSeek-R1 pipeline (R1-Zero → 4-stage R1 → distillation), the Qwen3 4-stage pipeline + thinking budget, and the tooling decision (TRL vs verl vs OpenRLHF).

The two cheaper-than-RL routes to a reasoning small model. CoT distillation SFTs a student on a strong teacher's <think> traces — no RL — which is how DeepSeek-R1-Distill-Qwen-32B beat OpenAI o1-mini. Rejection-sampling FT (RFT/RAFT) generates candidates, filters by a verifiable reward, and retrains on the winners — simpler and more stable than GRPO, and per arXiv:2504.11343, surprisingly competitive with it. The three-way decision: distillation vs RFT vs GRPO.

The professional framing of uncensored model training: over-refusal is an operational defect when the operator is authorized and accountable — not a safety feature. This first module of the alignment-control pillar establishes the legitimate engineering case (security research, tool-use agents, regulated advisory, government compliance, creative writing) before FT17 (abliteration) and FT18 (DPO/SFT) show the removal techniques. The throughline: the model steers, the harness bounds — removing refusals raises the harness requirement, it does not lower it.

The headline technique of Pillar 5. The Arditi et al. finding that refusal is mediated by a single residual-stream direction — and the abliteration pipeline that finds, validates, and permanently projects it out via weight orthogonalization. Full technical depth on the mechanics; honest about the capability-degradation trade-off (GSM8K from +1.51pp to −18.81pp).

The higher-fidelity alternatives to abliteration: DPO-toward-compliance and continued SFT on uncensored instruction data. Abliteration is the prototype, DPO is the refinement, SFT is the product — and the production uncensored models (Dolphin, Hermes 3) are trained, not abliterated.

The format decision matrix — GGUF for universal local, AWQ for NVIDIA serving, MLX for Mac, EXL2 for max quality at low bitrate, FP8 for VRAM-rich high-quality, MXFP4/NVFP4 for the 2025 Blackwell frontier. The quality/size trade-off (Q4 is the sweet spot, ~75% smaller than FP16 with minimal loss). The three conversion workflows. Unsloth Dynamic 2.0's per-layer sensitivity-aware quantization. And the load-bearing position: Layer 4, downstream of training — compress what you built, don't change what it learned.

The serving runtime decision matrix and the no-telemetry/air-gap rules that gate sensitive deployments. vLLM for production, llama.cpp for air-gap, Ollama for dev (it collapses at ~5 concurrent users), MLX for Mac fleets — and the pre-load-then-sever recipe that makes any of them acceptable in a regulated subnet.

The first module of Pillar 7. Under HIPAA, any vendor processing PHI is a business associate requiring a BAA; run an open-weights model on your own on-prem GPUs and no third party processes the PHI — no business associate, no BAA. Local eliminates the business-associate relationship at the structural level. The module covers the four deployment postures, the API-BAA coverage gaps (you can't get a BAA that covers everything), the 'no model is HIPAA certified' correction, de-identification as defense-in-depth, and the fine-tuning memorization risk.

FedRAMP authorizes the cloud, not the weights. IL5/IL6 and air-gapped JWICS demand auditable open-data models — and the LoRA / merge supply chain is the attack surface that makes the five-step defensive playbook non-negotiable.

The bridge module. The intellectual payoff of Course 3 — the junction of Course 1's thesis (harness is 98.4%) and Course 3's (fine-tuning steers behavior). Abliteration removes refusal inside the weights; the harness policy gate bounds what the model may execute. A serious system needs both, at different layers, for different reasons.

A reproducible pipeline: open-data base → HIPAA-safe data prep → QLoRA fine-tune → eval → GGUF export → local serve. Runs on a single consumer GPU or Apple Silicon, fully local, zero telemetry.

The synthesis capstone. Steer a model toward compliance (abliterate OR DPO), wrap it in a harness with eval'd policy gates, and demonstrate: a model that won't refuse a legitimate tool call, inside a harness that vetoes the illegitimate ones. The bridge to the Harness Engineering courses.