Module FT08 — LoRA & QLoRA

LoRA & QLoRA

The first module of Pillar 2 — where you do your first real fine-tune. LoRA decomposes a weight update into W = W0 + BA (frozen base + tiny trainable B,A); QLoRA freezes the base at 4-bit so a 7B model trains on a 24GB consumer card. The four LoRA knobs, QLoRA's three innovations, the full train-and-merge workflow, and the modern all-linear target default.

minutes

artifacts

sub-sections

Fine-tuning steers behavior; it does not teach knowledge (FT00). And steering, it turns out, is low-rank. LoRA (Hu et al. 2021) exploits this: it freezes the pretrained weights W0 and learns a low-rank perturbation B·A on top — an adapter often under 1% of the parameters. QLoRA (Dettmers et al. 2023) freezes the base itself at 4-bit via three innovations (NF4, double quantization, paged optimizers), so a 7B model trains on a $1,500 RTX 4090 instead of a multi-A100 node. This module covers the one equation, the four config knobs (rank, alpha, target modules, dropout — modern default all-linear), the three QLoRA innovations, and the full workflow from 4-bit load through merge_and_unload. The lab is your first real fine-tune.

Key Claims

Load-Bearing Claims

LoRA's low-rank decomposition W = W0 + BA — where W0 is frozen and B (zero-init), A (random-init) are tiny trainable matrices — works because steering is intrinsically low-rank. The intrinsic dimension hypothesis (Aghajanyan 2020, arXiv:2007.07784) showed useful fine-tuning changes live in a low-rank subspace; you can fine-tune to within 90% of full-FT by training ~0.5% of params. This is the FT00 thesis made rigorous — steering is low-rank, knowledge injection would be high-rank. B's zero-init makes the adapter a residual steer that cannot degrade the base at step zero (a guarantee full FT lacks).

A LoRA adapter is four knobs: rank r (default 16), alpha α (convention α≈2×r, so the effective magnitude is stable across rank and you don't retune the LR), target modules (MODERN default = ALL attention + ALL MLP projections: q,k,v,o_proj + gate,up,down_proj — NOT attention-only), and dropout (0.05). Target modules is the single biggest quality lever: all-linear beats attention-only (the 2021 default) at comparable trainable-param budgets because behavior steering lives in the MLP feature pathway too, not just attention.

QLoRA (Dettmers 2023, arXiv:2305.14314) freezes the base at 4-bit via three innovations, each necessary: (1) NF4 (NormalFloat 4-bit) — quantile bins matched to the normal distribution, information-theoretically optimal for Gaussian weights; (2) double quantization — quantize the quantization constants themselves to 8-bit, saving ~0.37 bits/param; (3) paged optimizers — NVIDIA Unified Memory pages optimizer state to CPU to absorb the OOM spikes during checkpointing (the killer that crashes runs sized to steady-state). Together they fit 7B on a 24GB card at ~10–14 GB — a ~10× spread vs full FT's ~100–160 GB.

The full workflow is five steps: load base in 4-bit (BitsAndBytesConfig, nf4, double_quant, bf16) → prepare_model_for_kbit_training (the forgotten step) → get_peft_model with the LoraConfig (print_trainable_parameters → expect <1%) → train adapters only (4-bit base never moves) → merge_and_unload() to bake the adapter into the base for deployment (one artifact, then re-quantize to GGUF/AWQ), OR keep the adapter separate for hot-swapping (one base, many adapters). Merge = deploy default; keep separate = experimentation/multi-tenant default.

After This Module

State the LoRA mechanism — the low-rank decomposition W = W0 + BA (W0 frozen; B zero-init, A random-init) — and connect it to the intrinsic dimension hypothesis (Aghajanyan) and the FT00 thesis that fine-tuning steers behavior; it does not teach knowledge.

Configure a LoRA adapter from its four knobs — rank (r), alpha (α), target modules, dropout — justifying each choice, including the modern default of targeting ALL attention + MLP projections (q,k,v,o_proj + gate,up,down_proj), not attention-only.

Explain why steering is low-rank and why trainable params often land under 1% of the model — and why B's zero-init makes LoRA a robust residual steer.

Name QLoRA's three innovations (NF4, double quantization, paged optimizers) and justify why each is necessary and why they compose.

Derive, from the FT01 VRAM math, why QLoRA unlocks 7B training on a 24GB card (~10–14 GB) and where the ~10× spread vs full FT comes from.

Execute the full QLoRA workflow — load 4-bit → prepare → attach → train → merge or save — and decide between merge_and_unload() (baked-in, deploy) and a standalone adapter (hot-swappable).

Artifacts

Teaching Document

~3,400 words; 6 sub-sections — the LoRA mechanism (W=W0+BA, intrinsic dimension), the four config knobs (rank, alpha, target modules, dropout — modern all-linear default), QLoRA's three innovations (NF4, double quantization, paged optimizers), the VRAM math made concrete (7B in ~10–14 GB), the full workflow (load 4-bit → prepare → attach → train → merge), anti-patterns and key terms

READ

Diagrams

5 Mermaid diagrams (dark #14141f/#5eead4) — the W=W0+BA decomposition with init annotations; the four-knob LoRA config parameter map; QLoRA's three innovations and why each is necessary; the full 5-step workflow with the merge-vs-save branch; attention-only vs all-linear target comparison

READ

Slide Deck

14 reveal.js slides using the exact course dark template — equation, intrinsic dimension, four knobs, PEFT config, three innovations, VRAM math table, workflow, merge-vs-save, target comparison, anti-patterns, lab, objectives

READ

Teaching Script

~2,400-word teaching script (~45 min at 140 wpm) with [SLIDE N] cues — verbatim transcript covering all six sub-sections; speaker-ready