Module FT01 — VRAM Math: Can I Actually Run This?

VRAM Math: Can I Actually Run This?

Memory, not compute, is the binding constraint in fine-tuning. The three VRAM consumers, the rules of thumb that place a job in the right GPU class, and the three-question framework you run before you ever load a model.

minutes

artifacts

sub-sections

You will run out of VRAM long before you run out of FLOPs. Compute is a budget; memory is a cliff. Full fine-tuning a 7B needs ~$50K of H100s; QLoRA does it on a $1,500 RTX 4090. That ~10× spread is the price of optimizer states — and it is the single decision that separates 'I need a budget cycle' from 'I start tonight.' This module makes the math reflexive.

Key Claims

Load-Bearing Claims

Memory is the binding constraint, not compute. A modern GPU can do the math for a 7B fine-tune in reasonable time; whether you can hold the working set in memory is what decides whether the job runs at all. Step over the VRAM cliff and the job doesn't go slower — it OOMs and dies.

Training VRAM is three consumers: weights, optimizer states + gradients, and activations. Every OOM is one of these. For full FT the optimizer states (~16 bytes/trainable-param) dominate; for QLoRA the frozen 4-bit base plus a tiny adapter collapses the optimizer bill to rounding error. That is the entire economic argument for PEFT.

The QLoRA-vs-full-FT decision is a ~10× lever on the same model. 7B QLoRA ~10–16 GB → RTX 4090; 7B full FT ~100–160 GB → multi-A100 (~$50K). 70B QLoRA ~48–60 GB → 1× A100 80GB; 70B full FT ~1.0–1.4 TB → 8–16× H100 multi-node. The course default: start at QLoRA, escalate only with evidence.

Three questions pick the GPU class: model size × method × context length. Context is the quadratic lever and the most underestimated — naive attention is N×N, so FlashAttention 2/3 (quadratic→linear) is effectively mandatory. Always size for your 99th-percentile sequence length, not the mean.

After This Module

Explain why memory, not compute, is the binding constraint in fine-tuning, and name the three VRAM consumers (weights, optimizer states + gradients, activations) and which dominates for QLoRA vs full FT.

Apply the field rules of thumb to estimate training VRAM for QLoRA, LoRA, and full fine-tuning, and place a job in the right GPU class (consumer card / single datacenter GPU / multi-GPU / multi-node).

Apply the three-question framework — model size × method × context length — to pick a GPU class before opening a model loader.

Explain how context length, batch size, gradient checkpointing, optimizer choice (AdamW 8-bit), and FlashAttention 2/3 move the numbers, and rank the knobs by leverage.

Artifacts

Teaching Document

~3,200 words; 6 sections — the binding-constraint problem, the three consumers + rules of thumb, the five knobs, the three-question framework, inference VRAM, Apple Silicon; with anti-patterns, key terms, references

READ

Diagrams

5 Mermaid diagrams — the three VRAM consumers, the method×model→GPU-class decision flowchart, context length quadratic-vs-linear, the three-question framework decision tree, four concrete examples mapped to hardware

READ

Slide Deck

reveal.js slide deck, dark theme, design-system teal

READ

Teaching Script

Verbatim teaching transcript, [SLIDE N] cues