# Diagrams — Module FTDD-04: TRL

**Module**: FTDD-04 — TRL (Transformers Reinforcement Learning)
**Diagram count**: 4
**Tool**: Mermaid (primary). Each diagram validated in [Mermaid Live Editor](https://mermaid.live).

---

## Diagram 1 — TRL as the Substrate Layer

**Type**: Layered architecture
**Purpose**: The central picture. TRL sits on the HuggingFace `Trainer` and is itself the substrate that Axolotl, Unsloth, and HuggingFace Jobs build on.
**Reading the diagram**: Bottom-up. The Transformers Trainer is the foundation; TRL's trainers are thin wrappers; the ecosystem tools sit above. The arrows mean "builds on / wraps."

```mermaid
block-beta
  columns 1
  Ecosystem["THE ECOSYSTEM\nAxolotl (config + multi-GPU)\nUnsloth (kernel replacement)\nHuggingFace Jobs"]
  TRL["TRL v1.0\n75+ post-training methods · 3M downloads/month\nStability Contract + production CLI"]
  Trainer["HuggingFace Trainer / Transformers\noptimizer · schedulers · DDP · DeepSpeed ZeRO · FSDP"]

  Trainer --> TRL
  TRL --> Ecosystem

  style Trainer fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style TRL fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style Ecosystem fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
```

---

## Diagram 2 — The Six Trainers, Mapped to Steering Goals

**Type**: Mapping / decision table-as-diagram
**Purpose**: Which trainer steers which thing. This is the core of the module.
**Reading the diagram**: Left = trainer. Middle = what it steers. Right = the data shape it expects.

```mermaid
flowchart LR
  SFT["SFTTrainer"] -->|"format, instruction-following"| D1["prompt + completion"]
  DPO["DPOTrainer"] -->|"preference (better/worse)"| D2["chosen vs rejected pairs"]
  KTO["KTOTrainer"] -->|"preference (unpaired)"| D3["thumbs up / down"]
  RLOO["RLOOTrainer"] -->|"reasoning (verifiable reward)"| D4["reward function"]
  GRPO["GRPOTrainer"] -->|"reasoning (verifiable reward)"| D4
  RM["RewardTrainer"] -->|"learned reward for RLHF"| D5["preference pairs -> scorer"]

  style SFT fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style DPO fill:#14141f,stroke:#5eead4,color:#e4e4e8
  style KTO fill:#14141f,stroke:#5eead4,color:#e4e4e8
  style RLOO fill:#14141f,stroke:#5eead4,color:#e4e4e8
  style GRPO fill:#14141f,stroke:#5eead4,color:#e4e4e8
  style RM fill:#14141f,stroke:#5eead4,color:#e4e4e8
  style D1 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#9494a0
  style D2 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#9494a0
  style D3 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#9494a0
  style D4 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#9494a0
  style D5 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#9494a0
```

---

## Diagram 3 — Python API vs Production CLI

**Type**: Comparison / two-door
**Purpose**: Two equivalent surfaces into the same engine. When to use which.
**Reading the diagram**: Both doors run the same trainer classes. The CLI is configured; the API is programmed. The CLI wins for standard recipes and reproducibility; the API wins when you need custom logic.

```mermaid
flowchart TD
  Engine["TRL Trainers\nSFTTrainer · DPOTrainer · GRPOTrainer ..."]
  CLI["CLI path\ntrl sft --config x.yml\ntrl dpo --config x.yml\ntrl grpo --config x.yml"]
  API["Python API path\nSFTConfig + SFTTrainer\nfull control, custom rewards"]

  CLI -->|"same trainers, no code"| Engine
  API -->|"same trainers, full control"| Engine

  CLI -.->|"standard recipes, reproducible, CI-friendly"| Use1["Production jobs"]
  API -.->|"custom reward fns, research, non-standard loop"| Use2["Research + custom"]

  style Engine fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style CLI fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style API fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style Use1 fill:#08080c,stroke:rgba(255,255,255,0.08),color:#9494a0
  style Use2 fill:#08080c,stroke:rgba(255,255,255,0.08),color:#9494a0
```

---

## Diagram 4 — TRL in the Ecosystem: Wraps, Competes, Controls

**Type**: Three-relationship map
**Purpose**: How Axolotl and Unsloth relate to TRL, and when raw TRL is the answer.
**Reading the diagram**: Three relationships to the same substrate. Axolotl WRAPS (adds config + multi-GPU). Unsloth REPLACES kernels (single-GPU speed). Raw TRL is full control.

```mermaid
flowchart LR
  TRL["TRL\nthe substrate"]

  Axolotl["Axolotl\nWRAPS TRL\nYAML config + multi-GPU\n(FSDP/DeepSpeed orchestration)"]
  Unsloth["Unsloth\nREPLACES kernels\nTriton kernels, single-GPU speed\nTRL-compatible API"]
  Raw["Raw TRL\nFULL CONTROL\nPython API or CLI\nresearch, custom rewards, freshest methods"]

  Axolotl -->|"wraps"| TRL
  Unsloth -->|"competes: replaces kernels"| TRL
  Raw -->|"is"| TRL

  style TRL fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style Axolotl fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style Unsloth fill:#14141f,stroke:rgba(94,234,212,0.5),color:#e4e4e8
  style Raw fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
```

---

## Validation notes

- All four diagrams use the course design system colors: `#14141f` panel fill, `#5eead4` accent for primary, `rgba(255,255,255,0.08)`/`rgba(94,234,212,0.3)` for secondary borders, `#e4e4e8` / `#9494a0` for text.
- Paste each into [Mermaid Live Editor](https://mermaid.live) to render. All use stable Mermaid syntax (`block-beta`, `flowchart`) supported in current Mermaid (v10.4+).
- For the slide deck (artifact 03), these are rendered as static SVG/PNG captures from Mermaid Live, inlined into reveal.js.