The Training Loop with TRL

Module FT11 · Course 3 — LLM Fine-Tuning Masterclass

90 minutes · The capstone of Pillar 2 — run a complete, instrumented SFT job end-to-end.

Prereq: FT08 (LoRA/QLoRA), FT10 (Full FT vs PEFT decision).

Pillar 2 — Parameter-Efficient Fine-Tuning

TRL — the canonical post-training library

Transformers Reinforcement Learning. v1.0 released March 31, 2026. The library you reach for to steer a transformer model.

75+

training methods & trainer variants

~3M

downloads / month

Ships a Stability Contract (pinned API) and a production CLI: trl sft, trl dpo, trl grpo.

Design principle = the FT00 thesis: every trainer is a steering tool. None inject knowledge.

Why TRL, not raw transformers

SFT is not just "language modeling on a new dataset." Three things make it different — TRL handles all three:

Problem	What TRL does
Chat templates	Applies the model's tokenizer template; trains only on the completion (masks the prompt). Learn to respond, not parrot.
PEFT integration	Attach a `LoraConfig` → model wrapped, base frozen, adapter trained & saved. No PEFT glue.
Packing	Concatenates short examples into one sequence (with attention masking). Often ~2x throughput.

Cost: learn the config surface (SFTConfig). Benefit: stop debugging glue code, start debugging data & hyperparameters.

The SFTTrainer, end to end

Six stages, in the order they run on your machine.

1. Dataset · chat-format, train/eval split

↓

2. Model · bf16 + FlashAttention 2

↓

3. PEFT config · LoraConfig (or None for full FT)

↓

4. SFTConfig · the TrainingArguments levers

↓

5. trainer.train() · loss curve, eval every N steps

↓

6. Save / Merge / Load · adapter or full model

Reading a training loss curve

The EKG of your run. Three failure modes + the healthy baseline.

Curve	Diagnosis	First check
Healthy — steep drop, gentle decline, flatten	Converging	Confirm eval tracks train
Not decreasing — flat from step 1	Not learning	LR too low · data/template bug (FT07)
Exploding — NaN / spike	Unstable	LR too high · FP16 overflow (use BF16) · EOS bug
Plateau — descended, then flat	Converged or stuck	Eval flat low = done · eval rising = overfitting

Watch grad norm too. A 10x spike between two logging steps = about to explode, even if loss looks fine. Kill it, lower LR, check clipping.

The levers — learning rate & batch

Learning rate (the big one)

Method	LR range
Full FT	`1e-5` – `5e-5`
LoRA / QLoRA	`1e-4` – `5e-4`

LoRA is 10x higher — adapters are small & random, need faster learning vs the frozen base.

Effective batch

effective_batch =
  per_device_batch_size
  × grad_accum_steps
  × world_size

VRAM-limited? Lower per-device batch, raise accumulation. Same optimizer math, slower wall-clock.

Cardinal error: full-FT LR on a LoRA run (loss barely moves) or LoRA LR on full FT (loss explodes). The method (FT10) sets the LR band.

The levers — schedule, memory, optimizer

Warmup & scheduler

Warmup ratio 0.03–0.1 — stabilize early training
cosine — the common default (smooth decay)
linear — straight-line to zero
constant_with_warmup — flat, pick best checkpoint

Memory & speed

gradient_checkpointing — ~60-70% mem, ~30% slower (FT01)
FlashAttention 2/3 — effectively mandatory, 2-4x speedup
bf16=True — NOT fp16 (overflow → NaN)

Optimizer

Option	When
`adamw`	Default. Robust. Most memory (2 momentum states/param).
`adamw_8bit`	Memory-tight. Quantized states, small accuracy cost.
`paged_adamw_8bit`	QLoRA default. CUDA paging offloads states to CPU.

Save · Merge · Load

Three operations, three purposes. Confuse them → ship the wrong artifact.

Save (what was trained)

trainer.save_model(path)

With PEFT: the adapter (~100s of MB). Without: the full model (GBs).

Merge (for deployment)

merged = model.merge_and_unload()
merged.save_pretrained(dir)

Adapter folded into base → single standalone model. Quantize (FT19) or serve (FT20).

Load (hot-swap at inference): PeftModel.from_pretrained(base, adapter) + load_adapter() to swap. One base, many adapters. Layer 2 detaches from Layer 1 — the FT00 swappability property, in code.

Logging — what to watch

If it isn't logged, it didn't happen. If you can't see the loss curve, you are not training — you are hoping.

Backends

W&B — report_to="wandb" — rich dashboards
Trackio — report_to="trackio" — HF Spaces-backed
TensorBoard/JSONL — zero dependencies

What to log

train_loss, lr, grad_norm — every step
eval_loss — every eval step
throughput — spot slowdowns

The one plot: train loss & eval loss on the same chart. The gap = overfitting signal. Stop at the eval-loss minimum. load_best_model_at_end=True.

Anti-patterns

No eval (flying blind). Can't tell convergence from overfitting, can't compare checkpoints, ship a memorizing model. Eval is not optional — 200 examples is enough.

Wrong LR for the method. Full-FT LR on LoRA = no learning. LoRA LR on full FT = explosion. The method sets the band.

No logging / ignoring grad norm. A NaN'd run and a converged run look identical from the final checkpoint. Grad norm spike = the canary before the NaN.

FP16 instead of BF16. Same memory, FP16 overflows, BF16 doesn't. On Ampere+ there is no reason to use FP16 for training.

The lab — the full SFT loop

The capstone of Pillar 2. Run a real SFT job.

Model: Qwen2.5-3B-Instruct or MiniCPM3-4B (your choice)
Data: your FT05/FT06 output, or the provided trl-lib/Capybara
Log to W&B or Trackio, eval every N steps
Produce: loss curve, eval curve, merged model
Generate before/after samples — see the steering

If your run NaNs — good. The failure is the lesson. Diagnose via the loss curve & grad norm, fix the lever, re-run. That diagnosis is the skill this module teaches.

Consumer GPU (RTX 4090 / 24GB) or Colab. Runnable Python in 07-lab-spec.md.

What you can now do

Run a complete SFT job with TRL end to end — dataset, model, PEFT, config, train, eval, save.
Read a loss curve and diagnose the three failure modes (not decreasing, exploding, plateau).
Choose the TrainingArguments levers for a given hardware budget and method.
Save, merge, and load a fine-tuned model — adapter, merge_and_unload, PeftModel.from_pretrained.
Wire up logging and know what to watch — loss, eval, LR, grad norm, throughput.

Next: FT12 — SFT: The Baseline