Module FTDD-04 — TRL — Transformers Reinforcement Learning

TRL — Transformers Reinforcement Learning

TRL v1.0 (released March 31, 2026) is the canonical post-training library — 75+ methods, 3M downloads/month, a Stability Contract, and a production CLI. It is the standard to teach first because every higher-level tool (Axolotl, Unsloth) either wraps it or competes with it on its terms.

minutes

artifacts

sub-sections

When you reach Layer 3 of the Steering Stack — the actual fine-tuning — there is one library the entire ecosystem converges on: TRL. It is downloaded three million times a month, it implements more than seventy-five post-training methods, and its v1.0 release shipped a Stability Contract that production teams can finally build on. If you understand the six trainers and the CLI, you understand the substrate of the open-weights post-training world.

Key Claims

Load-Bearing Claims

TRL is the substrate. SFTTrainer, DPOTrainer, KTOTrainer, RLOOTrainer, GRPOTrainer, reward modeling, and async RL are all thin wrappers over the HuggingFace Transformers Trainer — same optimizer, same data collator, same distributed story (DDP, DeepSpeed ZeRO, FSDP).

v1.0 turned TRL from an experiment into infrastructure. The Stability Contract means trainer signatures, config keys, and CLI flags are now maintained across releases. Three million monthly downloads means major downstream projects (Axolotl, Unsloth, HuggingFace Jobs) treat it as a build-time dependency.

The CLI is the production path. `trl sft`, `trl dpo`, `trl grpo` run the same trainers as the Python API, from a YAML config, with no training code to write or maintain. This is how most real jobs are launched.

Teach TRL first because everything else is defined relative to it. Axolotl wraps TRL under the hood (config + multi-GPU). Unsloth competes by replacing TRL's kernels for single-GPU speed. Knowing the TRL API is the Rosetta Stone for the entire ecosystem.

After This Module

Explain why TRL is the canonical post-training library and what the v1.0 Stability Contract guarantees (and what it does not).

Map the six core trainers (SFT, DPO, KTO, RLOO, GRPO, RewardModeling) to the Steering Stack layers they operate at, and state when each is the right tool.

Contrast the Python-API path against the production CLI (`trl sft`/`dpo`/`grpo`), and explain why the CLI is how most production jobs are launched.

Place TRL in the ecosystem: how Axolotl wraps it, how Unsloth competes with it, and when raw TRL is the correct choice (full control, research, custom rewards).

Artifacts

Teaching Document

~2,400 words; 4 sub-sections — why TRL is the substrate, the six trainers, the production CLI, TRL in the ecosystem

READ

Diagrams

4 diagrams — TRL as the substrate layer, the six trainers mapped to steering goals, the CLI vs API decision, the ecosystem (Axolotl wraps / Unsloth competes / raw TRL controls)

READ

Slide Deck

reveal.js slide deck, dark theme, design-system teal

READ

Teaching Script

Verbatim teaching transcript, [SLIDE N] cues