{
  "module": "FTDD-04 — TRL (Transformers Reinforcement Learning)",
  "course": "3 — LLM Fine-Tuning Masterclass",
  "version": "1.0.0",
  "duration_minutes": 45,
  "total_questions": 10,
  "bloom_distribution": {
    "target": "40% recall / 30% application / 30% analysis",
    "actual": { "recall": 4, "application": 3, "analysis": 3 }
  },
  "passing_score_percent": 70,
  "questions": [
    {
      "id": "Q01", "bloom": "recall", "type": "multiple_choice",
      "prompt": "What is TRL, and why is it described as the substrate of the open-weights post-training ecosystem?",
      "options": [
        "A Reinforcement Learning library that only implements PPO; SFT and DPO live elsewhere.",
        "HuggingFace's full-stack post-training library (3M downloads/month, 75+ methods) that every higher-level tool either wraps (Axolotl) or competes with by replacing pieces (Unsloth).",
        "A model-serving library for deploying fine-tuned models with vLLM.",
        "A dataset-curation tool for building preference datasets."
      ],
      "answer_index": 1,
      "rationale": "TRL is the full-stack post-training library and the substrate: Axolotl wraps it, Unsloth replaces its kernels. Its trainer API is the Rosetta Stone for the ecosystem. Despite its RL name it long ago absorbed SFT and the preference family."
    },
    {
      "id": "Q02", "bloom": "recall", "type": "multiple_choice",
      "prompt": "What does the TRL v1.0 Stability Contract guarantee, and what does it NOT cover?",
      "options": [
        "It freezes ALL defaults (learning rates, schedulers) so configs never need updating.",
        "It guarantees trainer names, constructor signatures, config/YAML keys, and CLI flags are maintained across releases; breaking changes need a major version bump. It does NOT freeze defaults.",
        "It guarantees every trainer stays in the library forever, including experimental ones.",
        "It guarantees TRL will never release a v2.0."
      ],
      "answer_index": 1,
      "rationale": "The Stability Contract covers the API surface (trainer names, signatures, config keys, CLI flags). It does NOT freeze defaults — a default LR/scheduler may improve between minor versions with a deprecation cycle. In production, pin every hyperparameter explicitly."
    },
    {
      "id": "Q03", "bloom": "recall", "type": "multiple_choice",
      "prompt": "What is the 'thin-wrapper principle' behind TRL's trainers?",
      "options": [
        "Each TRL trainer re-implements the training loop from scratch for maximum speed.",
        "Each trainer subclasses the HuggingFace Trainer, overrides compute_loss to inject the SFT/DPO/GRPO objective, and inherits everything else (optimizer, schedulers, DDP/DeepSpeed/FSDP).",
        "Each trainer is a standalone binary with no dependency on the Transformers library.",
        "Each trainer wraps PyTorch Lightning instead of the HuggingFace Trainer."
      ],
      "answer_index": 1,
      "rationale": "TRL trainers are thin wrappers over the HuggingFace Trainer. They override compute_loss and inherit the optimizer, schedulers, data collation, chat templates, and the distributed story (DDP, DeepSpeed ZeRO, FSDP). This is why TRL scales to cluster-scale training — scaling is DeepSpeed/FSDP's job."
    },
    {
      "id": "Q04", "bloom": "recall", "type": "multiple_choice",
      "prompt": "Which data shape does KTOTrainer expect, and how does it differ from DPOTrainer?",
      "options": [
        "KTO needs (prompt, chosen, rejected) triples, same as DPO.",
        "KTO needs unpaired BINARY feedback (thumbs up/down per response), not chosen/rejected pairs. This matters because paired preference data is expensive; binary feedback is what production thumbs-up buttons produce.",
        "KTO needs a verifiable reward function, like GRPO.",
        "KTO needs a learned reward model trained by RewardTrainer first."
      ],
      "answer_index": 1,
      "rationale": "KTO (Kahneman-Tversky Optimization) turns unpaired binary feedback into a preference gradient. DPO needs chosen/rejected pairs, which are expensive to collect. KTO is the right trainer when all you have is thumbs-up/thumbs-down signals."
    },
    {
      "id": "Q05", "bloom": "application", "type": "multiple_choice",
      "prompt": "You have a dataset of (prompt, chosen_response, rejected_response) triples and want to steer the model to prefer the chosen style. Which TRL trainer, and why?",
      "options": [
        "SFTTrainer — it handles any preference data.",
        "DPOTrainer — it consumes chosen/rejected pairs and applies Direct Preference Optimization with a reference-model KL anchor, no reward model needed.",
        "GRPOTrainer — it is the most modern preference trainer.",
        "RewardTrainer — you must train a reward model first, then do RLHF."
      ],
      "answer_index": 1,
      "rationale": "DPOTrainer is the direct fit: it consumes (prompt, chosen, rejected) triples and derives the preference gradient directly from the policy with a reference-model anchor, requiring no learned reward model. GRPO needs a verifiable reward function; RewardTrainer is for classical RLHF when you lack a verifiable reward."
    },
    {
      "id": "Q06", "bloom": "application", "type": "multiple_choice",
      "prompt": "You want to sharpen a model's math reasoning using 'did the answer match the ground truth' as the reward. Which trainer, and what must you provide?",
      "options": [
        "DPOTrainer with a preference dataset of correct vs incorrect solutions.",
        "GRPOTrainer (or RLOOTrainer) with a Python reward function that checks answer correctness. These RL trainers consume a verifiable reward, not preference pairs.",
        "SFTTrainer with a dataset of correct solutions only.",
        "KTOTrainer with thumbs up/down on solutions."
      ],
      "answer_index": 1,
      "rationale": "GRPO/RLOO take a verifiable reward FUNCTION (a Python callable) and reinforce completions that score well. 'Answer matches ground truth' is a textbook verifiable reward. DPO/KTO need preference labels; SFT only distills fixed completions. The verifiable-reward path is strictly more reliable than a learned reward for math."
    },
    {
      "id": "Q07", "bloom": "application", "type": "multiple_choice",
      "prompt": "Your team runs a standard SFT recipe in production and wants zero training code, reproducibility-as-a-file, and CI-friendliness. Which TRL surface, and what is its limitation?",
      "options": [
        "The Python API (SFTConfig + SFTTrainer) — full control, but you must write and maintain training code.",
        "The production CLI (trl sft --config x.yml) — no training code, reproducible YAML, CI-validatable. Limitation: it is configured not programmed, so custom rewards/data/interleaved-eval require dropping to the Python API.",
        "Axolotl — it is the only CI-friendly option.",
        "Unsloth — it is the only reproducible option."
      ],
      "answer_index": 1,
      "rationale": "The CLI runs the same SFTTrainer from a YAML config with no training code, giving reproducibility-as-a-file and CI-friendliness (lint/schema-validate the YAML). Its scope is the ~80% of jobs fitting standard recipes; the moment you need custom logic you drop to the Python API. The wrappers are not required for these properties post-v1.0."
    },
    {
      "id": "Q08", "bloom": "analysis", "type": "multiple_choice",
      "prompt": "Why does the thin-wrapper principle explain TRL's ability to scale to 405B full-parameter training on clusters?",
      "options": [
        "Because TRL reimplements model parallelism more efficiently than DeepSpeed.",
        "Because TRL's trainers only override compute_loss and inherit DDP/DeepSpeed ZeRO/FSDP from the HuggingFace Trainer — scaling is DeepSpeed/FSDP's battle-tested job, and TRL gets out of their way rather than reimplementing it.",
        "Because TRL uses Triton kernels that are inherently faster on clusters.",
        "Because TRL trains 405B models on a single GPU via aggressive quantization."
      ],
      "answer_index": 1,
      "rationale": "TRL's trainers subclass the HuggingFace Trainer and inherit its distributed story (DDP, DeepSpeed ZeRO, FSDP). Scaling is the Transformers library's problem, already battle-tested. TRL only overrides compute_loss, so it rides the existing machinery to cluster scale without reimplementation — which is precisely why a 'thin wrapper' can power 405B training."
    },
    {
      "id": "Q09", "bloom": "analysis", "type": "multiple_choice",
      "prompt": "A team skips SFT and runs GRPO directly on a base model whose reasoning is incoherent. The result is worse than the base. What does this illustrate?",
      "options": [
        "GRPO is the wrong algorithm for reasoning; they should have used DPO.",
        "GRPO reinforces whatever the base can already do — it does not install capability. Reinforcing incoherent reasoning produces more confidently incoherent reasoning. SFT must come first to establish format and basic competence, then GRPO to sharpen.",
        "The base model was too large for GRPO.",
        "GRPO requires a learned reward model, not a verifiable one."
      ],
      "answer_index": 1,
      "rationale": "GRPO amplifies existing capability; it does not create it. If the base produces incoherent reasoning, GRPO faithfully reinforces that incoherence. SFT first to establish format and competence, then GRPO to sharpen reasoning. This is the 'reach for GRPO before SFT' anti-pattern (Module FT14)."
    },
    {
      "id": "Q10", "bloom": "analysis", "type": "multiple_choice",
      "prompt": "Contrast how Axolotl and Unsloth relate to TRL, and explain why a practitioner might choose each.",
      "options": [
        "Both wrap TRL identically; the choice is purely about speed.",
        "Axolotl WRAPS TRL (YAML config + opinionated defaults + battle-tested multi-GPU orchestration via FSDP/DeepSpeed) — chosen for production multi-GPU and reproducibility. Unsloth REPLACES TRL's kernels with hand-tuned Triton and exposes a TRL-compatible API — chosen for ~2x single-GPU throughput / half the VRAM, sub-30B, where its limited multi-GPU story is acceptable.",
        "Both replace TRL entirely; neither depends on it.",
        "Axolotl replaces TRL's kernels; Unsloth wraps TRL's config layer."
      ],
      "answer_index": 1,
      "rationale": "Axolotl is a declarative wrapper over TRL adding config ergonomics and multi-GPU orchestration — the production multi-GPU path. Unsloth competes by replacing the performance-critical kernels with Triton for single-GPU speed, with a TRL-compatible API. The choice is multi-GPU production (Axolotl) vs single-GPU speed (Unsloth). Raw TRL remains the full-control/freshest-methods option."
    }
  ]
}