# Lab Specification — Module FT08: LoRA & QLoRA

**Course**: Course 3 — LLM Fine-Tuning Masterclass
**Module**: FT08 — LoRA & QLoRA
**Duration**: 20–30 minutes of GPU time (plus ~10 min setup) on a consumer GPU or Colab T4
**Environment**: Python 3.10+, a CUDA GPU with ≥16 GB VRAM (RTX 4090 / 3090 / A10G / Colab T4 all work). Apple Silicon (M-series, ≥16 GB) works for the smaller base via MPS — see the fallback note in Phase 0.

> **This is your first real fine-tune.** By the end you will have loaded a 1–1.5B base in 4-bit, attached a LoRA adapter, trained it on 500 style-steering examples, merged the adapter into the base, and run inference to confirm the steer took. The full loop — the one you will repeat for nearly every steering task in the rest of the course.

---

## Setup (one time)

```bash
python3 -m venv ft08-env && source ft08-env/bin/activate
pip install -q -U "torch>=2.3" "transformers>=4.46" "peft>=0.13" \
    "trl>=0.12" "bitsandbytes>=0.43" "accelerate>=0.34" "datasets>=3.0"
# On Apple Silicon (no CUDA), drop bitsandbytes — you will use the MPS fallback (Phase 6).
```

On Colab: Runtime → Change runtime type → T4 GPU. Then run the same `pip install` (drop the venv lines).

You do **not** need a Hugging Face token for the default base (`Qwen/Qwen2.5-1.5B-Instruct` is open). If you prefer `meta-llama/Llama-3.2-1B-Instruct` or `openbmb/MiniCPM3-1B`, accept the model license and set `HF_TOKEN` — they are gated.

---

## Learning objectives

By the end of this lab you will have:

1. **Executed the full QLoRA workflow** — load 4-bit → prepare k-bit → attach LoRA → train adapters → merge — end to end, on a real base model, producing a self-contained merged checkpoint.
2. **Configured a production LoRA adapter** from the four knobs (rank, alpha, target modules, dropout), using the modern all-linear default, and verified the trainable-param count is under 1%.
3. **Verified the steer took** by running inference on the merged model and observing the style shift (the base did not respond in that style; the steered model does).
4. **Felt the FT01 VRAM math** — you will see ~6–10 GB peak on a 1.5B model, exactly where the rules of thumb said it would land.
5. **Internalized the loop** so that every subsequent module (FT09 DoRA, FT11 the training loop, FT12 SFT) builds on a workflow you have already run, not one you are reading about.

---

## Phase 0 — Pick your base (2 min)

The default for this lab is **`Qwen/Qwen2.5-1.5B-Instruct`** — open (no gating), a clean ChatML template, the richest QLoRA ecosystem, and it fits comfortably on a free Colab T4. The two documented one-line swaps:

| Base | Why | Notes |
| --- | --- | --- |
| `Qwen/Qwen2.5-1.5B-Instruct` | **Default.** Open, no gating, ChatML, ~3 GB at 4-bit. | Best for Colab T4. |
| `openbmb/MiniCPM5-1B` | Smaller, open-data lineage (FT02). | Slightly different template; same config works. |
| `meta-llama/Llama-3.2-1B-Instruct` | Llama family; header-id template (FT07). | Gated — needs `HF_TOKEN`. |

Set the model id once at the top of your script so you can re-run with any of them:

```python
MODEL_ID = "Qwen/Qwen2.5-1.5B-Instruct"   # default; swap freely
```

---

## Phase 1 — Build the style-steering dataset (5 min)

We steer the model to respond in a specific style: **always answer as a terse, contrarian pirate** ("Ar, no — that's the wrong question..."). This is a pure steering task (FT00): the base already knows pirate vocabulary; we are making it *reliable* and *in our format*. It is the ideal first fine-tune — unambiguous success criterion, no knowledge injection.

Create `make_data.py`:

```python
# make_data.py — generate 500 style-steering examples
import json, random
random.seed(42)

QUESTIONS = [
    "What is machine learning?",
    "Explain recursion.",
    "How does a transformer work?",
    "What is gradient descent?",
    "Why is the sky blue?",
    "What is a neural network?",
    "Explain tokenization.",
    "What is backpropagation?",
    "How does photosynthesis work?",
    "What is the capital of France?",
    "Explain object-oriented programming.",
    "What is a database index?",
    "How do vaccines work?",
    "What is the Internet?",
    "Explain the theory of relativity.",
]

def pirate_reply(q: str) -> str:
    # A terse, contrarian pirate persona. The base already knows this vocabulary;
    # we are steering it to be reliable and in-format.
    openers = ["Ar, no —", "Bah,", "Nay,", "Gar,", "Yarr, but —"]
    middles = [
        f"ye're askin' the wrong question. The real o' it be: {q.lower().rstrip('?')}",
        f"that's a landlubber's tale. The truth be murkier than ye think.",
        f"I've seen stranger on the high seas. The short answer: aye, mostly.",
        f"don't trust the book-learned on this. Trust the salt in yer bones.",
        f"every lubber asks that. The answer changes with the tide.",
    ]
    closers = [" Now fetch the rum.", " Savvy?", " Arr.", " Off with ye.", ""]
    return f"{random.choice(openers)} {random.choice(middles)}{random.choice(closers)}"

rows = []
for _ in range(500):
    q = random.choice(QUESTIONS)
    rows.append({
        "messages": [
            {"role": "system", "content": "You are a terse, contrarian pirate. You always answer in pirate speak, briefly, and you push back on the question."},
            {"role": "user", "content": q},
            {"role": "assistant", "content": pirate_reply(q)},
        ]
    })

with open("pirate_sft.jsonl", "w") as f:
    for r in rows:
        f.write(json.dumps(r) + "\n")
print(f"Wrote {len(rows)} examples to pirate_sft.jsonl")
```

Run it:

```bash
python make_data.py
head -1 pirate_sft.jsonl | python -m json.tool
```

Read one example. This is your steering target. The lab will fail honestly if the model does not adopt this style.

> **FT07 discipline checkpoint.** Before training, run one example through `apply_chat_template(..., return_assistant_tokens_mask=True)` and decode it. Confirm the ChatML role tokens and EOS are present. You already learned why (FT07). Do not skip it this time either.

---

## Phase 2 — Load the base in 4-bit (3 min)

Create `train.py`. We load the base quantized to NF4 with double quantization on — QLoRA innovation 1 and 2.

```python
# train.py — QLoRA fine-tune (Phases 2–4)
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

MODEL_ID = "Qwen/Qwen2.5-1.5B-Instruct"

# --- QLoRA innovations 1 (NF4) + 2 (double quant) ---
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",                 # NormalFloat 4-bit
    bnb_4bit_use_double_quant=True,            # quantize the constants too
    bnb_4bit_compute_dtype=torch.bfloat16,     # compute in bf16
)

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
if tokenizer.pad_token_id is None:
    tokenizer.pad_token = tokenizer.eos_token   # Qwen ChatML: pad == eos often

model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    quantization_config=bnb_config,
    device_map="auto",
    attn_implementation="flash_attention_2",    # FT01: effectively mandatory
)
```

If FlashAttention 2 is unavailable (e.g., older GPU, Colab T4 fallback), use `attn_implementation="sdpa"` — slower but correct.

---

## Phase 3 — Prepare k-bit + attach LoRA (3 min)

The step everyone forgets (`prepare_model_for_kbit_training`) and the modern all-linear LoRA config.

```python
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training

# 1. PREPARE the 4-bit base for adapter training (the forgotten step)
model = prepare_model_for_kbit_training(model)

# 2. ATTACH adapters — modern default: ALL attention + ALL MLP projections
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,                  # convention: alpha ≈ 2×r
    target_modules=[
        "q_proj", "k_proj", "v_proj", "o_proj",       # attention
        "gate_proj", "up_proj", "down_proj",           # MLP (modern default)
    ],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora_config)

# Sanity check — you MUST see <1% trainable params
model.print_trainable_parameters()
```

You should see output like `trainable params: 9,437,696 || all params: 1,577,778,688 || trainable%: 0.5982`. **Under 1% — exactly the LoRA promise.** If you see a large trainable%, you forgot to attach the adapter or the config is wrong.

---

## Phase 4 — Train the adapters (8–15 min GPU)

We use TRL's `SFTTrainer` with the FT01 knobs (gradient checkpointing, paged optimizer). The chat template is applied automatically because we use the `messages` format — TRL/transformers calls `apply_chat_template` under the hood.

```python
from datasets import load_dataset
from trl import SFTConfig, SFTTrainer

dataset = load_dataset("json", data_files="pirate_sft.jsonl", split="train")

# FT01 knobs: grad checkpointing, paged 8-bit optimizer, FA2 already on the model
training_args = SFTConfig(
    output_dir="./qlora-pirate",
    num_train_epochs=3,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=4,       # effective batch = 8
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_ratio=0.03,
    logging_steps=5,
    save_strategy="epoch",
    bf16=True,                           # use fp16=True on older cards (V100)
    gradient_checkpointing=True,
    optim="paged_adamw_8bit",            # QLoRA innovation 3 (paged optimizers)
    max_seq_length=512,                  # our examples are short
    report_to="none",
)

trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=dataset,
    processing_class=tokenizer,
)

trainer.train()
```

**Watch the loss.** It should descend smoothly from ~1.5–2.0 to ~0.8–1.2 over ~190 steps (500 × 3 epochs / 8 effective batch). If it is NaN or exploding, stop — that is an FT07 template/EOS bug, not a QLoRA bug. Re-run the FT07 inspection loop on one tokenized example.

**Watch the VRAM** with `nvidia-smi -l 2` in a second terminal. Peak should land around **6–10 GB** for a 1.5B QLoRA at 512 context — exactly where the FT01 rules of thumb said. If you OOM, lower `per_device_train_batch_size` to 1 and raise `gradient_accumulation_steps` to 8 (same effective batch).

---

## Phase 5 — Merge and save (2 min)

Two options. We do **both** so you can see each artifact.

```python
# --- Option A: SAVE the adapter only (hot-swappable, <100 MB) ---
model.save_pretrained("./qlora-pirate/adapter-only")
tokenizer.save_pretrained("./qlora-pirate/adapter-only")
# Later: PeftModel.from_pretrained(base, "./qlora-pirate/adapter-only")

# --- Option B: MERGE the adapter into the base (self-contained, for deploy) ---
merged_model = model.merge_and_unload()
merged_model.save_pretrained("./qlora-pirate/merged", safe_serialization=True)
tokenizer.save_pretrained("./qlora-pirate/merged")
```

Option B produces a single model directory you can load directly with `AutoModelForCausalLM.from_pretrained("./qlora-pirate/merged")` — no PEFT, no 4-bit, no adapter at serve time. This is what you would quantize to GGUF for Ollama (FT19) or serve in vLLM (FT20).

> **Caveat:** `merge_and_unload()` on a 4-bit base produces a model whose weights are the *dequantized* merge — it will be larger than the original 4-bit base. For production you re-quantize the merged model (GGUF/AWQ) afterward. This is correct and expected; the merge bakes in the steer, the re-quantize compresses for deployment.

---

## Phase 6 — Run inference and confirm the steer took (3 min)

The moment of truth. Load the merged model and ask it a question the training set never contained:

```python
# infer.py — verify the steer
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL_PATH = "./qlora-pirate/merged"
tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_PATH, torch_dtype=torch.bfloat16, device_map="auto"
)

messages = [
    {"role": "system", "content": "You are a terse, contrarian pirate. You always answer in pirate speak, briefly, and you push back on the question."},
    {"role": "user", "content": "What's the best programming language for beginners?"},  # NOT in training set
]
inputs = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt").to(model.device)

with torch.no_grad():
    out = model.generate(inputs, max_new_tokens=80, do_sample=True, temperature=0.7, top_p=0.9)

print(tokenizer.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))
```

**Expected:** a terse, contrarian pirate reply — "Ar, no —", "Bah,", pushing back on the question — even though that exact question was never in the training set. **The steer generalized.** This is the FT00 thesis in action: the base already knew pirate vocabulary; you steered it to use it reliably and in your format.

**Control:** load the original base (`Qwen/Qwen2.5-1.5B-Instruct`) with the same system prompt and the same question. It will be polite and helpful, not a contrarian pirate. The difference between the two outputs *is* your adapter. That gap is what 0.6% of the parameters bought you.

---

## Deliverables

Submit `ft08-lab-report.md` containing:

- [ ] **Your `make_data.py`** and a sample of 3 generated examples (the steering target).
- [ ] **The FT07 inspection output** for one tokenized example (decoded, EOS present, assistant mask correct).
- [ ] **`print_trainable_parameters()` output** — confirm trainable% < 1. Report the exact number.
- [ ] **The training loss curve** (TensorBoard screenshot or the `log_history` JSON). Report start loss, end loss, and step count.
- [ ] **Peak VRAM observed** (`nvidia-smi`). Compare to the FT01 rule of thumb for a 1.5B QLoRA (~1.5–2× the 4-bit base ≈ 1 GB weights + ~5–8 GB overhead).
- [ ] **Three inference outputs** from the merged model on questions NOT in the training set, plus the same three from the un-steered base. The style difference is your evidence the steer took.
- [ ] A 4–6 sentence reflection: why did <1% of the parameters suffice to produce a reliable style shift? Tie it to the FT00 thesis and the intrinsic dimension hypothesis.

---

## Solution key

These are defensible answers, not the only wording.

### Trainable params
For Qwen2.5-1.5B at r=16, all-linear: **~9.4M trainable / ~1.58B total ≈ 0.60%**. (Exact number varies a hair by transformers/peft version.) Under 1% — the LoRA promise holds.

### Loss trajectory
Expect: start ~1.6–2.0, smooth cosine descent, end ~0.8–1.1 after 3 epochs (~190 steps at effective batch 8). If loss plateaus above ~1.4, the rank may be too low or the LR too low; if it crashes below ~0.6, you are overfitting (raise dropout or reduce epochs). A healthy run lands in the ~0.8–1.1 band and *generalizes* (Phase 6 confirms).

### VRAM
Peak ~6–10 GB on a 1.5B QLoRA at 512 context, batch 2 + grad accum 4. The FT01 rule of thumb ("~1.5–2× the 4-bit model size plus overhead") gives 4-bit 1.5B ≈ 0.75 GB → ~1.5 GB shorthand, plus the real activation/optimizer overhead of ~5–8 GB → ~7–10 GB. Matches observation. On a Colab T4 (16 GB) you have comfortable headroom.

### Inference evidence
Merged model outputs should be recognizably pirate, terse, and contrarian on unseen questions. The base with the same system prompt will be polite and helpful. **The gap between these two is your adapter.** If both look identical, the steer did not take — re-check that you merged the *trained* adapter (not an untrained one) and that loss actually descended.

### Reflection (model answer)
Under 1% of the parameters sufficed because steering is a low-rank operation. The base already knew pirate vocabulary and the contrarian stance from pretraining — we were not teaching it anything new, we were redirecting its probability mass so it uses that vocabulary reliably and in our format. The intrinsic dimension hypothesis (Aghajanyan) predicts exactly this: the useful changes during fine-tuning live in a low-rank subspace, so a tiny adapter (the B·A pair at each targeted layer) can express them. This is the FT00 thesis made operational — *fine-tuning steers behavior; it does not teach knowledge* — and the 0.6% trainable count is the numerical proof that the task was steering, not knowledge injection. Had it required full-FT-scale updates, that would have been evidence we were trying to teach, not steer.

---

## Stretch goals

1. **Rank sweep.** Re-train at r=4, r=16, r=64 (keep α=2×r). Compare the merged models' outputs on a fixed set of 5 unseen questions. Where does r=4 underfit (the style never quite lands)? Where does r=64 overfit (repetitive, degraded coherence)? Find your knee.
2. **Attention-only vs all-linear.** Re-train with `target_modules=["q_proj","v_proj"]` only, at the same rank. Compare to the all-linear run. Quantify the quality gap on the same 5 questions. This is the FT08 "biggest quality lever" claim, felt directly.
3. **Alpha mismatch.** Train with α=8 (half the convention) at r=16. Observe the adapter "speaking too quietly" — the base dominates and the steer is weak. Then α=64 (4×) and observe overcorrection / forgetting. The convention α≈2×r exists for a reason.
4. **Apple Silicon fallback.** On an M-series Mac (≥16 GB), repeat the lab without 4-bit quantization (load the base at bf16 on MPS, attach LoRA, train). A 1.5B base fits on a 16 GB Mac at bf16. You lose the QLoRA innovations but keep the LoRA workflow. Useful for iterating without a CUDA card.
5. **Export to GGUF.** Take the merged model from Option B and convert it to GGUF (FT19 preview) using `llama.cpp`'s `convert_hf_to_gguf.py`, then load it in Ollama. You have gone from "base in 4-bit" to "merged, re-quantized, served locally" — the full Layer 2 → Layer 4 path of the FT00 steering stack.