Module FT11 — The Training Loop with TRL

The Training Loop with TRL

The capstone of Pillar 2. Run a complete, instrumented SFT job with TRL's SFTTrainer — dataset to merged model — and learn to read the loss curve, choose the TrainingArguments levers, and diagnose the three failure modes (loss not decreasing, exploding, plateauing).

minutes

artifacts

sub-sections

TRL is the canonical post-training library: 75+ methods, ~3M downloads/month, a Stability Contract, and a production CLI. In this module you run the SFTTrainer end-to-end — dataset loading, model loading, PEFT attachment, TrainingArguments, trainer.train(), eval, save/merge — and you learn to read the loss curve as the EKG of your run. The three failure modes (not decreasing, exploding, plateauing) each map to a probable cause: LR, data, template, or numerical instability. Get the levers right — LR band for the method, effective batch via accumulation, BF16 not FP16, gradient checkpointing, FlashAttention 2 — and the loop converges.

Key Claims

Load-Bearing Claims

TRL is the canonical post-training library and SFTTrainer is the workhorse. v1.0 (March 31, 2026) ships 75+ methods, ~3M downloads/month, a Stability Contract, and a production CLI. It automates the steering mechanics — chat templates, completion masking, PEFT attachment, packing — so you spend time on data and hyperparameters, not glue code.

The loss curve is the EKG of your run, and three failure modes account for almost every bad run. Loss not decreasing → LR too low, data problem, or template bug (FT07). Loss exploding (NaN/spike) → LR too high, FP16 overflow (use BF16), or EOS/template bug. Loss plateauing → converged (eval flat and low) or stuck (eval rising = overfitting). The grad norm spike is the early-warning canary before the NaN.

The method sets the LR band, and mismatching them is the most common ticket. Full FT: 1e-5 to 5e-5. LoRA: 1e-4 to 5e-4 (an order of magnitude higher — adapters are small and random, needing faster learning against a frozen base). Full-FT LR on LoRA = no learning; LoRA LR on full FT = explosion.

Save, merge, and load are three operations with three purposes. Save writes what was trained (adapter or full). merge_and_unload folds the adapter into the base for single-model deployment. PeftModel.from_pretrained loads an adapter for hot-swapping — the FT00 swappability property in code.

After This Module

Run a complete, instrumented SFT job end-to-end with TRL's SFTTrainer — dataset loading, model loading, PEFT attachment, TrainingArguments, trainer.train(), evaluation, and save — and explain what each stage does.

Read a training loss curve and diagnose the three failure modes (loss not decreasing, loss exploding/NaN, loss plateauing), mapping each to its probable cause (LR, data, template, numerical instability).

Choose the standard TrainingArguments levers — learning rate, batch size + gradient accumulation, warmup ratio, LR scheduler, gradient checkpointing, FlashAttention 2/3, optimizer — for a given hardware budget and method.

Save, merge, and load a fine-tuned model — adapter-only save, merge_and_unload for deployment, PeftModel.from_pretrained for adapter loading — and explain when each is appropriate.

Wire up logging (W&B, Trackio, or JSONL) and know what to watch (loss, eval loss, LR, grad norm, throughput) and which anti-patterns (no eval, wrong LR for the method, FP16 instead of BF16) will silently ruin a run.

Artifacts

Teaching Document

~3,400 words; 6 sub-sections — TRL the canonical library, the SFTTrainer end-to-end pipeline, reading a loss curve (3 failure modes), the TrainingArguments levers, save/merge/load, logging & what to watch

READ

Diagrams

5 Mermaid diagrams — the SFTTrainer end-to-end pipeline, the loss-curve diagnostic chart (healthy vs 3 failures), the TrainingArguments decision tree, the save/merge/load workflow, the eval-during-training loop

READ

Slide Deck

14 reveal.js slides — title, TRL overview, why TRL, SFTTrainer pipeline, loss curve reading, LR & batch levers, schedule/memory/optimizer levers, save/merge/load, logging, anti-patterns, the lab, what you can now do

READ

Teaching Script

~2,200 words; ~50 min spoken at 140 wpm; 14 [SLIDE N] cues matching the deck