{
  "module": "FT01 — VRAM Math: Can I Actually Run This?",
  "course": "3 — LLM Fine-Tuning Masterclass",
  "version": "1.0.0",
  "duration_minutes": 35,
  "total_questions": 15,
  "bloom_distribution": {
    "target": "20% recall / 40% application / 40% analysis-design",
    "actual": { "recall": 3, "application": 6, "analysis": 6 }
  },
  "passing_score_percent": 70,
  "questions": [
    {
      "id": "Q01", "bloom": "recall", "type": "multiple_choice",
      "prompt": "In fine-tuning LLMs, what is almost always the binding constraint — the thing that decides whether the job runs at all?",
      "options": [
        "Compute (FLOPs). A modern GPU's math throughput is the limiting factor.",
        "Memory (VRAM). You run out of it long before you run out of FLOPs; stepping over the budget OOMs the job rather than slowing it.",
        "Disk I/O. Loading checkpoints is the bottleneck.",
        "Network bandwidth for multi-GPU gradient sync."
      ],
      "answer_index": 1,
      "rationale": "The central claim of FT01: memory is a cliff, not a budget. Compute is a budget (slower is still progress). Step over the VRAM cliff and the job dies with an OOM. This is why VRAM math is a foundations module — it is the gatekeeper on every other technique in the course."
    },
    {
      "id": "Q02", "bloom": "recall", "type": "multiple_choice",
      "prompt": "Name the three consumers of training VRAM.",
      "options": [
        "CPU RAM, GPU VRAM, and disk cache.",
        "Model weights, optimizer states + gradients, and activations.",
        "Forward pass, backward pass, and the dataloader.",
        "Pretraining cost, fine-tuning cost, and inference cost."
      ],
      "answer_index": 1,
      "rationale": "Every training OOM is one of: (1) MODEL WEIGHTS at their stored precision; (2) OPTIMIZER STATES + GRADIENTS (AdamW's FP32 master + first moment m + second moment v + FP16 gradient — ~16 bytes/trainable-param for full FT); (3) ACTIVATIONS, which scale with context length, batch size, and layers. Memorize these three; the rest of the module is about which dominates for which method."
    },
    {
      "id": "Q03", "bloom": "recall", "type": "multiple_choice",
      "prompt": "What does full AdamW cost per trainable parameter, and why is this the line separating 'fine-tuning' from 'full fine-tuning'?",
      "options": [
        "~2 bytes/param — just the FP16 weight. Negligible.",
        "~16 bytes/param — FP16 weight + FP16 gradient + FP32 master copy + first moment m + second moment v. For a fully-trainable 7B that's ~112 GB before activations, which is why full FT is a datacenter event while PEFT is cheap.",
        "~4 bytes/param — only the FP32 gradient.",
        "~0.5 bytes/param — quantized 4-bit states."
      ],
      "answer_index": 1,
      "rationale": "Full AdamW keeps, per trainable param: FP16 weight (2) + FP16 gradient (2) + FP32 master copy (4) + first moment m (4) + second moment v (4) = ~16 bytes. For full FT ALL params are trainable, so 7B × 16 = ~112 GB of weights+grads+optimizer alone. PEFT escapes this because only the ~1% adapter is trainable — optimizer states collapse to rounding error. This 10x gap is the entire economic argument for PEFT."
    },
    {
      "id": "Q04", "bloom": "application", "type": "multiple_choice",
      "prompt": "Using the three-question framework: 7B model, QLoRA, 2–4K context. What GPU class do you need?",
      "options": [
        "Laptop / Apple Silicon / Colab T4 free tier.",
        "RTX 4090 24GB ($1,500). 7B QLoRA needs ~10–16 GB.",
        "multi-A100 80GB (~$50K of GPUs).",
        "8–16× H100 multi-node."
      ],
      "answer_index": 1,
      "rationale": "7B QLoRA at 2–4K context runs ~10–16 GB — fits comfortably on an RTX 4090 24GB. This is the sweet spot for most real steering work and the 'I start tonight' path. The 4090 is the consumer card that made QLoRA viable for 7B; before QLoRA (arXiv:2305.14314), 7B fine-tuning needed a datacenter GPU."
    },
    {
      "id": "Q05", "bloom": "application", "type": "multiple_choice",
      "prompt": "Using the three-question framework: 70B model, QLoRA, 4K context. What GPU class do you need?",
      "options": [
        "RTX 4090 24GB.",
        "1× A100 80GB. 70B QLoRA needs ~48–60 GB (4-bit weights ~35 GB + activations).",
        "8–16× H100 multi-node (~1.0–1.4 TB).",
        "A laptop with 32GB unified memory."
      ],
      "answer_index": 1,
      "rationale": "70B QLoRA runs ~48–60 GB — 4-bit frozen weights (~35 GB) plus activations. It fits on a single A100 80GB. Note this is NOT 10× a 7B QLoRA (~15 GB), only ~3.4×, because activation memory scales with hidden dim (~params^(1/3)) not linearly with params. The bottom-up estimator captures this; a naive linear multiplier would over-provision."
    },
    {
      "id": "Q06", "bloom": "application", "type": "multiple_choice",
      "prompt": "Using the three-question framework: 7B model, FULL fine-tuning, 4K context. What GPU class and roughly what cost?",
      "options": [
        "RTX 4090 24GB ($1,500).",
        "1× A100 80GB.",
        "multi-A100 80GB (~$50K of GPUs). 7B full FT needs ~100–160 GB.",
        "Colab T4 free tier."
      ],
      "answer_index": 2,
      "rationale": "7B full FT runs ~100–160 GB — ~16 bytes/param (weights + grads + AdamW states) over all 7B params plus activations. A single 80GB A100 cannot hold it; you need multiple 80GB cards (~$50K). This is the expensive path and the reason the course default is 'start at QLoRA, escalate to full FT only with evidence (Module FT10).'"
    },
    {
      "id": "Q07", "bloom": "application", "type": "multiple_choice",
      "prompt": "Your training job OOMs at step 3. You budgeted a 7B QLoRA at '2K context' (~12 GB) on a 16 GB card, but your real data's longest sequences are 8K. What went wrong, and what is the most important fix?",
      "options": [
        "The model is too big — switch to a 3B base.",
        "You underestimated context length. Naive attention is N×N, so 8K vs 2K is ~16× the attention activation memory. Turn on FlashAttention 2 (quadratic→linear) and size for the 99th-percentile sequence length.",
        "The batch size is too large — reduce effective batch to 1.",
        "QLoRA is the wrong method — switch to full fine-tuning for stability."
      ],
      "answer_index": 1,
      "rationale": "The classic OOM: planning for the mean context and hitting the tail. Naive attention stores an N×N matrix per layer per head — 8K vs 2K is a 16× activation blow-up. Two fixes: (1) FlashAttention 2 fuses attention so memory is linear (effectively mandatory); (2) always size for the 99th-percentile sequence length. Also: gradient checkpointing + lower physical batch via gradient accumulation."
    },
    {
      "id": "Q08", "bloom": "application", "type": "multiple_choice",
      "prompt": "A job OOMs and you suspect batch size. Physical batch is 8. What is the standard fix, and what does it cost you?",
      "options": [
        "Reduce the effective batch size to 1 and accept noisier gradients.",
        "Gradient accumulation: run micro-batches of size 1–2 and sum gradients over N steps before the optimizer step. You get effective batch N at physical memory cost of 1. It costs you nothing but wall-clock time.",
        "Switch to a smaller model.",
        "Disable the optimizer to free memory."
      ],
      "answer_index": 1,
      "rationale": "Gradient accumulation decouples EFFECTIVE batch size (the gradient noise profile you want) from PHYSICAL batch size (what fits in VRAM). Dropping physical batch 8→1 with grad-accum of 8 cuts activation memory ~8× while preserving the effective batch. It trades wall-clock time (more, smaller forward/backward passes) for memory — the cleanest OOM fix there is."
    },
    {
      "id": "Q09", "bloom": "application", "type": "multiple_choice",
      "prompt": "FlashAttention is OFF on a job with 4K+ context. What is the memory consequence and the fix?",
      "options": [
        "No memory consequence — FlashAttention only affects speed, not memory.",
        "Naive attention materializes an N×N matrix per layer per head. Double the context → quadruple the attention activation memory (quadratic). At 8K on a 7B it can exceed the weight footprint. Fix: set attn_implementation='flash_attention_2' — memory goes quadratic→linear, plus ~20–30% faster. Effectively mandatory.",
        "FlashAttention increases memory usage; turning it OFF is correct.",
        "Switch to INT4 quantization to compensate."
      ],
      "answer_index": 1,
      "rationale": "Without FlashAttention, attention activation memory is O(N²) because the full N×N attention matrix materializes per layer/head. FlashAttention 2/3 (Dao et al., arXiv:2205.14135) fuses the computation so the matrix never materializes — memory becomes O(N). It is free, stable, and faster. If your config lacks attn_implementation='flash_attention_2' and your context is >2K, treat it as a bug."
    },
    {
      "id": "Q10", "bloom": "analysis", "type": "multiple_choice",
      "prompt": "A team budgets a single 80GB A100 for a 7B full fine-tune, reasoning '14 GB weights, plenty of room.' Why will this OOM, and what is the real budget?",
      "options": [
        "It won't OOM — 14 GB fits comfortably in 80 GB.",
        "They forgot optimizer states in full FT. The 14 GB is the INFERENCE footprint (FP16 weights only). Training adds FP16 gradient (14 GB) + FP32 master (28 GB) + AdamW m (28 GB) + AdamW v (28 GB) ≈ 98 GB of optimizer/grads alone, before activations. Real 7B full FT budget: ~100–160 GB → multiple A100 80GB cards.",
        "The A100 is too slow; they need H100s for throughput.",
        "They need to quantize the base to 4-bit first."
      ],
      "answer_index": 1,
      "rationale": "The 'forgetting optimizer states' anti-pattern. The 14 GB inference footprint is irrelevant to the training budget. Full FT carries ~16 bytes/trainable-param (weights + grads + AdamW) over ALL params: ~112 GB before activations. The real budget is ~100–160 GB — a single 80GB card will OOM. This is why the module hammers 'optimizer states are the consumer people forget' and why PEFT (only ~1% trainable) is the default."
    },
    {
      "id": "Q11", "bloom": "analysis", "type": "multiple_choice",
      "prompt": "For 7B full fine-tuning (~118 GB), which consumer dominates, and what is the single cheapest knob to shrink the budget by ~10×?",
      "options": [
        "Activations dominate; the knob is gradient checkpointing.",
        "4-bit weights dominate; the knob is 2-bit quantization.",
        "Optimizer states + gradients dominate (~98 GB of ~118 GB). The knob: switch from full FT to QLoRA — trainable params drop from 100% to ~1%, optimizer states collapse from ~98 GB to <0.5 GB, total falls from ~118 GB to ~10–16 GB. Same model, ~10× cheaper.",
        "The KV cache dominates; the knob is paged attention."
      ],
      "answer_index": 2,
      "rationale": "For 7B full FT, optimizer states + gradients (~98 GB — the FP32 master + AdamW m,v + FP16 gradient over all 7B params) dwarf both weights (14 GB) and activations (~6–18 GB). The single highest-leverage knob is the METHOD choice: full FT → QLoRA makes only ~1% of params trainable, so the ~98 GB of optimizer states collapse to <0.5 GB. The total drops ~10× for the same model. This is the entire economic argument for PEFT."
    },
    {
      "id": "Q12", "bloom": "analysis", "type": "multiple_choice",
      "prompt": "Why does a 70B model NOT cost 10× the activation memory of a 7B model, despite having ~10× the parameters?",
      "options": [
        "Because 70B models use a different attention mechanism.",
        "Because activation memory scales with layers × batch × seq × hidden, and hidden grows as ~params^(1/3) (not linearly). 7B → hidden~4096/layers~34; 70B → hidden~8900/layers~74 — only ~2× the per-token activation cost despite 10× the params. So 70B QLoRA (~53 GB) is only ~3.4× a 7B QLoRA (~15 GB) total.",
        "Because 70B models are always trained at lower precision.",
        "Because larger models use fewer layers."
      ],
      "answer_index": 1,
      "rationale": "Activation memory is per-layer × per-token, and transformer architecture scales hidden ~params^(1/3) with layers ~hidden/120. So a 70B (hidden~8900, layers~74) has only ~2× the activation cost per token of a 7B (hidden~4096, layers~34), despite 10× the params. This is why a naive linear multiplier (e.g., 'QLoRA = 1.5× 4-bit size') over-provisions large models. A bottom-up estimator (the FT01 lab) is more trustworthy than any single multiplier."
    },
    {
      "id": "Q13", "bloom": "analysis", "type": "multiple_choice",
      "prompt": "The field rule says 'QLoRA ≈ 1.5–2× the 4-bit model size.' Why is this shorthand potentially misleading, and what is more trustworthy?",
      "options": [
        "It is exactly correct and should always be trusted.",
        "It reproduces the 7B number tolerably (4-bit ~3.5 GB → ~10–16 GB incl. overhead) but gets looser at 70B because activation overhead grows sub-linearly with params. It also conflates bare NF4 weights (~3.5 GB) with the deployed 4-bit footprint (~6 GB). More trustworthy: a bottom-up sum of the three consumers (weights + optimizer + activations). The multiplier is a sanity check; the sum is the plan.",
        "It is only correct for FP16 models, not 4-bit.",
        "It underestimates because it ignores the harness."
      ],
      "answer_index": 1,
      "rationale": "Shorthand multipliers break across model sizes because activation memory scales sub-linearly with params (hidden ~ params^(1/3)). The 1.5–2× rule works for 7B but over-provisions 70B. It also conflates bare NF4 weights with deployed footprints. The FT01 lab builds a bottom-up estimator (sum of three consumers with derived architecture) that stays accurate across sizes — that is the trustworthy planning tool."
    },
    {
      "id": "Q14", "bloom": "analysis", "type": "multiple_choice",
      "prompt": "Rank the highest-leverage decision for reducing a 7B training budget from ~118 GB (full FT) to ~10–16 GB. What single choice delivers the ~10× cut?",
      "options": [
        "Turning on gradient checkpointing (~30% slower for ~65% activation cut).",
        "Switching from full AdamW to AdamW 8-bit (halves optimizer states).",
        "Turning on FlashAttention (quadratic→linear attention memory).",
        "Switching the METHOD from full FT to QLoRA: the frozen 4-bit base drops weight footprint (14 GB → 3.5 GB) AND, more importantly, only ~1% of params are trainable, so optimizer states collapse (~98 GB → <0.5 GB). This single decision is worth ~$48,500 and ~10× the hardware."
      ],
      "answer_index": 3,
      "rationale": "For 7B full FT, optimizer states dominate (~98 GB of ~118 GB). No memory knob (checkpointing, FlashAttention, AdamW 8-bit) can touch the ~98 GB of optimizer states because they exist over ALL trainable params. Only changing the METHOD — making just ~1% of params trainable via QLoRA — collapses that consumer. The method choice is the 10× lever; the knobs are 1.3–2× levers. This is why the course default is 'start at QLoRA.'"
    },
    {
      "id": "Q15", "bloom": "analysis", "type": "multiple_choice",
      "prompt": "You want to fine-tune a 1B model on a laptop you already own. Using VRAM math, which statement is correct about Apple Silicon (M-series) as a training path?",
      "options": [
        "Apple Silicon is never viable for training — you must rent an NVIDIA GPU.",
        "M-series Macs have unified memory (GPU and CPU share one pool), so a 64GB Mac presents ~48–58 GB of usable GPU memory. A 7B QLoRA (~10–14 GB) fits on a 32GB Mac; a 1B QLoRA on a 16GB Mac. The catch: MPS throughput is lower than a dedicated NVIDIA card and not every kernel is optimized. For iteration and small steering experiments, it is a first-class target, not a fallback.",
        "Apple Silicon can only do inference, never training.",
        "You must disable MPS and use CPU-only training for correctness."
      ],
      "answer_index": 1,
      "rationale": "Apple Silicon's unified memory makes it a genuinely viable small-model training path — a 64GB Mac gives ~48–58 GB of usable GPU memory with no duplication. 1B–7B QLoRA fits comfortably (1B on 16GB, 7B on 32GB). The trade-off is throughput (slower than a dedicated NVIDIA card) and kernel coverage, not capability. For the course's orientation work and many PEFT experiments (Pillar 2), Apple Silicon via MLX or PyTorch MPS is a first-class target."
    }
  ]
}