LoRA & QLoRA

Module FT08 · Course 3 — LLM Fine-Tuning Masterclass

75 minutes · the first module of Pillar 2 — where you do your first real fine-tune

Fine-tuning steers behavior; it does not teach knowledge. And steering is low-rank — so a tiny adapter, often under 1% of the params, is enough.

Pillar 2 — Parameter-Efficient Fine-Tuning

The one equation: W = W₀ + BA

LoRA (Hu et al. 2021) freezes the pretrained W₀ and adds a low-rank update B·A.

Frozen

W₀ — pretrained weight (d×k). No gradient. The base, untouched.

Trainable (the adapter)

A (r×k) — random-init · B (d×r) — ZERO-init

B starts at zero → BA = 0 at step 0. The adapter is the identity at init — it cannot degrade the base. It learns a residual perturbation on top. Full FT has no such guarantee.

Dimensional collapse: a 4096×4096 ΔW is 16.7M numbers; the LoRA update at r=8 is 65K — a 256× reduction. Across every layer: adapters under 1% of the model.

Why it works: steering is low-rank

Intrinsic dimension hypothesis (Aghajanyan 2020): the useful changes during fine-tuning live in a low-rank subspace. Fine-tune RoBERTa to within 90% of full-FT by training ~0.5% of params — because the effective dimension of the optimization is tiny.

This is the FT00 thesis made rigorous. If fine-tuning were injecting knowledge, you would need high rank. But it is redirecting probability mass the base already has — and that redirection is low-rank.

Biechler et al. 2024: LoRA and full-FT produce structurally different weight matrices. Not approximations — different geometry, similar behavior. So "more rank = closer to full FT" is a category error.

The LoRA config — four knobs

KnobWhat it doesModern default
r (rank)# directions of change16 (8 narrow, 64 complex)
α (alpha)scaling: applied as α/rα ≈ 2×r (r=16→α=32)
target_moduleswhich weights get adaptersALL linear — q,k,v,o + gate,up,down
lora_dropoutregularization0.05 (0.1 small data/high r)
Biggest quality lever: target modules. Attention-only (q,v_proj) was the 2021 default — cheap, weak. All-linear is the modern default — it targets the MLP feature pathway too, where behavior steering actually lives.

The PEFT config, made concrete

from peft import LoraConfig

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,               # convention: alpha ≈ 2×r
    target_modules=[
        "q_proj", "k_proj", "v_proj", "o_proj",   # attention
        "gate_proj", "up_proj", "down_proj",       # MLP (modern default)
    ],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)
Memorize this as the starting point. get_peft_model(model, lora_config) then print_trainable_parameters() → expect <1% trainable.

QLoRA — three innovations (Dettmers 2023)

LoRA made adapters cheap. QLoRA freezes the base at 4-bit so the whole job fits a consumer GPU.

#InnovationWhat it does
1NF4 (NormalFloat 4-bit)Quantile bins matched to the normal distribution. Info-theoretically optimal for Gaussian weights.
2Double quantizationQuantize the quantization constants to 8-bit. Saves ~0.37 bits/param (~0.3 GB on 7B).
3Paged optimizersNVIDIA Unified Memory pages optimizer state to CPU — absorbs the OOM spikes during checkpointing.
All three are necessary. Drop any one and the method degrades. They compose: NF4 shrinks the base, double quant trims overhead, paged optimizers absorb the spikes.

The VRAM math, made concrete (7B QLoRA, 4K ctx)

ConsumerRule7B
Base weights (4-bit NF4)0.5 B/param~3.5 GB
Quant constants (double-quant'd)~0.18 bits/param~0.16 GB
LoRA adapter (r=16, all-linear)~40M × ~10 bytes~0.4 GB
Activations (4K, batch 1, ckpt, FA2)rule of thumb~4–7 GB
CUDA context + fragmentationfixed overhead~1–2 GB
Total~10–14 GB → fits RTX 4090 (24GB)
vs full FT (~100–160 GB, multi-A100). The spread is ~10×. Almost no steering task justifies paying it.

The full QLoRA workflow

  1. Load base in 4-bit — BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=bf16)
  2. Prepare k-bit — prepare_model_for_kbit_training(model) — the step everyone forgets
  3. Attach adapters — get_peft_model(model, lora_config) → print_trainable_parameters() → expect <1%
  4. Train adapters only — SFTTrainer (TRL); the 4-bit base never moves
  5. Merge or save — see next slide
FT01 knobs all apply: gradient_checkpointing=True, attn_implementation="flash_attention_2", optim="paged_adamw_8bit", decouple batch via gradient_accumulation_steps.

Merge vs keep the adapter

merge_and_unload() — deploy

Bakes the adapter into the base: W₀ := W₀ + (α/r)BA. One self-contained model, no PEFT at serve time. Re-quantize (GGUF/AWQ) afterward.

keep separate — hot-swap

Save adapter_model.safetensors (<100 MB). Load base + PeftModel.from_pretrained. One base, many adapters, swap at runtime.

Caveat: merging a 4-bit base dequantizes → merged model is larger than the 4-bit base. Production path: merge THEN re-quantize. (Layer 2 → Layer 4.)

Attention-only vs all-linear — the biggest lever

Two teams, same trainable-param count. Who generalizes better?

TeamConfigResult
Ar=16, all-linear (7 modules)generalizes better on unseen prompts
Br=56, attention-only (2 modules)worse — missed the MLP pathway
Capacity distribution > total count. All-linear targets attention and the MLP feature pathway, where behavior steering lives. Attention-only concentrates capacity where the change mostly isn't. This is why all-linear is the modern default — not the 2021 q,v_proj convention.

Anti-patterns

  • Attention-only targeting on a modern model — leaves quality on the table. Use all-linear.
  • Too-low rank (underfit) — r=2/4, loss plateaus high, style never lands. Raise to 16.
  • Too-high rank (overfit + memory) — r=128 "to be safe." Overfits, approaches full-FT cost for LoRA geometry. Start at 16.
  • Alpha poorly tuned — α divorced from r. Keep α ≈ 2×r.
  • Forgetting prepare_model_for_kbit_training — one line, always include.
  • Treating QLoRA as inferior — for steering it is the correct tool. Full FT is the exception (FT10), not the default.

The lab — "First Fine-Tune"

QLoRA-fine-tune a 1–1.5B base on 500 style-steering examples. Merge. Run inference. <30 min on a consumer GPU or Colab T4.

The steer

Make the model a terse, contrarian pirate. Pure steering — the base already knows the vocabulary; you make it reliable and in-format.

The loop

Load 4-bit → prepare → attach → train 3 epochs → merge → infer on an unseen question. Compare to the un-steered base. The gap is your adapter.

Default: Qwen/Qwen2.5-1.5B-Instruct (open, ChatML). Fallbacks: MiniCPM5-1B, Llama-3.2-1B. Expect ~0.6% trainable, ~6–10 GB peak VRAM, loss 1.6 → ~0.9.

Next: FT09 — DoRA, rsLoRA & Modern PEFT

What you can now do

  1. State the LoRA mechanism — W = W₀ + BA — and tie it to the intrinsic dimension hypothesis & the FT00 thesis.
  2. Configure a LoRA adapter from its four knobs — rank, alpha, target modules, dropout — using the all-linear default.
  3. Name QLoRA's three innovations — NF4, double quantization, paged optimizers — and why each is necessary.
  4. Derive why 7B QLoRA fits a 24GB card from the FT01 VRAM math.
  5. Execute the full workflow — load 4-bit → prepare → attach → train → merge or save — and choose between merge and hot-swap.

The steering wheel is built. The tokens are verified. Now you have built the steer itself.

Course 3 — LLM Fine-Tuning Masterclass · FT08 · Pillar 2