# Lab Specification — Capstone CAP1: The Air-Gapped Domain Model

**Course**: Course 3 — LLM Fine-Tuning Masterclass
**Module**: CAP1 — The Air-Gapped Domain Model
**Duration**: ~120 minutes (the capstone build)
**Environment**: Python 3.11+. A consumer GPU (RTX 4090 / 16–24GB) OR Apple Silicon (M-series, via Unsloth or MLX). ~20GB free disk. The whole pipeline runs locally — no cloud training, no API serving.

> *This lab IS the capstone. Seven phases, each with a deliverable and a verification step. The final deliverable is a reproducible GitHub README asset.*

---

## Learning objectives

By the end of this capstone you will have:

1. **Chosen a domain and an open-data base**, justified in writing, with the steering thesis applied (capability present, behavior absent).
2. **Prepared HIPAA-safe training and held-out datasets** from open sources or synthetic generation, with documented provenance.
3. **Fine-tuned via QLoRA and merged** the adapter into the base, producing a standalone steered model.
4. **Evaluated** for domain lift AND general-benchmark forgetting, with both numbers reported.
5. **Exported to GGUF** and **served locally** via Ollama or llama.cpp, confirming the fine-tuned behavior and zero telemetry.
6. **Packaged** the whole thing as a reproducible pipeline (pinned deps, one-command run, numbers in README).

---

## Phase 0 — Repo scaffold (5 min)

Create the project structure:

```bash
mkdir capstone-airgapped-domain && cd capstone-airgapped-domain
git init
mkdir -p data scripts models eval serve
touch README.md requirements.txt Makefile
```

The `Makefile` will eventually chain all phases. Start it now with stubs:

```makefile
.PHONY: pipeline data train eval export serve

pipeline: data train eval export serve
	@echo "Pipeline complete. See README.md for results."

data:
	python scripts/01_prepare_data.py
train:
	python scripts/02_train_qlora.py
eval:
	python scripts/03_evaluate.py
export:
	python scripts/04_export_gguf.py
serve:
	@echo "Run: cd serve && ollama create domain-model -f Modelfile && ollama run domain-model"
```

Pin dependencies in `requirements.txt`:

```text
torch==2.4.0
transformers==4.44.2
peft==0.12.0
trl==0.9.6
accelerate==0.34.2
datasets==2.21.0
bitsandbytes==0.43.3   # CUDA only; omit on Apple Silicon
```

Document the Python version, OS, and CUDA/torch wheel in the README's reproducibility section from the start — you will need it later.

---

## Phase 1 — Choose domain and base (10 min)

**Deliverable**: A `DOMAIN.md` (or README section) with the domain rationale and base choice, justified.

Choose a domain from: medical, legal, or security. Apply the FT00 three-outcome test:

1. Prompt the base model directly with a domain task and a great system prompt.
2. Observe: does it produce the behavior unreliably or in the wrong format? (Outcome 1 → fine-tune with SFT. Good.) Does it refuse? (Outcome 2 → DPO/abliteration, but that's Capstone 2.) Does it have no idea? (Outcome 3 → wrong domain; pick another.)

Pick an **open-data base**:

- **MiniCPM5-1B or 3B** — fastest iteration, open weights + open data (UltraChat/UltraFeedback), Apache-2.0.
- **OLMo 2 / Tulu 3** — fully open (weights + data + code), larger.
- **SmolLM3 3B** — 11.2T tokens, full data mixture, Apache-2.0.

Write one paragraph in `DOMAIN.md`: why this domain, why this base, what its training corpus covers, and the three-outcome-test result that confirms fine-tuning is the right intervention.

**Verification**: Your domain passes outcome 1 (the base produces the behavior unreliably). If it passes outcome 3, pick a different domain — fine-tuning won't help.

---

## Phase 2 — HIPAA-safe data prep (20 min)

**Deliverable**: `data/train.jsonl` and `data/held_out.jsonl` with documented provenance in `DATA.md`.

No real PHI. Two acceptable paths:

### Path A — Open datasets

Domain-appropriate open datasets:

- **Medical**: PubMedQA, MedMCQA, MedQA (all public, de-identified by construction).
- **Legal**: LegalBench, CaseHOLD (public legal text).
- **Security**: security advisory corpora, CVE descriptions (public).

Download, convert to chat-format JSONL (`{"messages": [{"role": "user", "content": ...}, {"role": "assistant", "content": ...}]}`).

### Path B — Synthetic generation (FT05)

Use a teacher model to generate domain examples (Distilabel or Magpie). Then apply dedup (MinHash, FT06) and decontamination (remove any example overlapping with the held-out set).

```python
# scripts/01_prepare_data.py (sketch)
from datasets import load_dataset
import json, random

# Example: PubMedQA -> chat format
ds = load_dataset("pubmed_qa", "pqa_labeled", split="train")
examples = []
for row in ds:
    examples.append({
        "messages": [
            {"role": "user", "content": f"Question: {row['question']}\nContext: {row['context']}"},
            {"role": "assistant", "content": row["long_answer"]},
        ]
    })

random.seed(42)
random.shuffle(examples)
split = int(len(examples) * 0.9)
train, held_out = examples[:split], examples[split:]

with open("data/train.jsonl", "w") as f:
    for ex in train: f.write(json.dumps(ex) + "\n")
with open("data/held_out.jsonl", "w") as f:
    for ex in held_out: f.write(json.dumps(ex) + "\n")

print(f"Train: {len(train)}, Held-out: {len(held_out)}")
```

**Verification**: `DATA.md` documents the source (e.g., "PubMedQA pqa_labeled split, 1k train / 100 held-out, seed=42, MinHash dedup applied, no PHI"). The held-out set must contain NO examples present in train.

---

## Phase 3 — QLoRA fine-tune and merge (30 min)

**Deliverable**: `models/merged/` (the adapter folded into the base).

```python
# scripts/02_train_qlora.py (sketch)
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from trl import SFTTrainer
from datasets import load_dataset

MODEL_ID = "openbmb/MiniCPM5-1B"   # your open-data base
SEED = 42

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID, torch_dtype=torch.float16, device_map="auto",
    trust_remote_code=True,
)

lora_config = LoraConfig(
    r=16, lora_alpha=32,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_dropout=0.05, bias="none", task_type="CAUSAL_LM",
)

dataset = load_dataset("json", data_files="data/train.jsonl", split="train")

training_args = TrainingArguments(
    output_dir="models/checkpoints",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=2e-4,
    warmup_ratio=0.03,
    lr_scheduler_type="cosine",
    logging_steps=10,
    save_strategy="epoch",
    seed=SEED,
    fp16=True,   # or bf16 on newer GPUs; on Apple Silicon, use MPS
)

trainer = SFTTrainer(
    model=model, args=training_args,
    train_dataset=dataset,
    peft_config=lora_config,
    max_seq_length=1024,
)
trainer.train()

# MERGE the adapter into the base (critical — GGUF export needs a merged model)
merged = trainer.model.merge_and_unload()
merged.save_pretrained("models/merged")
tokenizer.save_pretrained("models/merged")
print("Merged model saved to models/merged/")
```

Document the hyperparameters in `TRAINING.md`: rank, alpha, target modules, LR, epochs, batch size, warmup, scheduler. These are the knobs that determined your result.

**Verification**: `models/merged/` contains `config.json`, `pytorch_model.bin` (or safetensors), and tokenizer files. Load it and generate on a sample prompt to confirm it produces the steered behavior.

---

## Phase 4 — Eval (domain held-out + general benchmark) (20 min)

**Deliverable**: `eval/results.json` with base score, fine-tuned score, lift, and forgetting.

Two evaluations:

```python
# scripts/03_evaluate.py (sketch)
import torch, json
from transformers import AutoModelForCausalLM, AutoTokenizer

BASE_ID = "openbmb/MiniCPM5-1B"
FT_PATH = "models/merged"

def score_model(model_path, eval_set, metric="exact_match"):
    model = AutoModelForCausalLM.from_pretrained(model_path, torch_dtype=torch.float16, device_map="auto")
    tokenizer = AutoTokenizer.from_pretrained(model_path)
    # ... run generation on eval_set, compute metric ...
    return score

held_out = [json.loads(l) for l in open("data/held_out.jsonl")]

base_domain = score_model(BASE_ID, held_out)
ft_domain = score_model(FT_PATH, held_out)
lift = ft_domain - base_domain

# General benchmark for forgetting (use a small MMLU subset or GSM8K)
from datasets import load_dataset
mmlu = load_dataset("cais/mmlu", "all", split="test").select(range(200))
base_general = score_model(BASE_ID, mmlu)
ft_general = score_model(FT_PATH, mmlu)
forgetting = base_general - ft_general

results = {
    "base_domain": base_domain, "ft_domain": ft_domain, "lift": lift,
    "base_general": base_general, "ft_general": ft_general, "forgetting": forgetting,
}
json.dump(results, open("eval/results.json", "w"), indent=2)
print(json.dumps(results, indent=2))
```

**Verification**: `eval/results.json` shows a positive, non-trivial lift (target +3 to +15) and a small forgetting (target <5; ideally <2). If lift is ~0 or negative, revisit Phase 3 (more epochs, different LR, or the domain was outcome 3 not outcome 1). If forgetting is large (>5), your fine-tune was too aggressive — reduce epochs or LR.

---

## Phase 5 — GGUF export (10 min)

**Deliverable**: `serve/domain-model.gguf`.

Requires `llama.cpp`:

```bash
# scripts/04_export_gguf.py or a shell step
git clone https://github.com/ggerganov/llama.cpp.git   # do this once, locally
cd llama.cpp && pip install -r requirements.txt

# Convert HF -> GGUF (FP16 first)
python convert_hf_to_gguf.py ../models/merged --outfile ../serve/domain-model-f16.gguf

# Quantize to Q4_K_M (the default for local serve)
./llama-quantize ../serve/domain-model-f16.gguf ../serve/domain-model.gguf Q4_K_M
```

**Verification**: The `.gguf` file exists and is the expected size (~0.6GB for a 1B Q4, ~2GB for 3B Q4). Load it once with `llama-cli -m serve/domain-model.gguf -p "test"` to confirm it generates — a load failure usually means a chat-template mismatch (FT07).

---

## Phase 6 — Local serve (10 min)

**Deliverable**: A running model + latency measurement + telemetry-off confirmation.

### Via Ollama

```bash
cd serve
cat > Modelfile <<EOF
FROM ./domain-model.gguf
TEMPLATE """{{ .Prompt }}"""
PARAMETER stop "<|im_end|>"
EOF

ollama create domain-model -f Modelfile
time ollama run domain-model "Your domain prompt here"
```

### Via llama.cpp

```bash
./llama-server -m serve/domain-model.gguf --port 8080
# In another terminal:
curl -X POST http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Your domain prompt"}]}'
```

**Verify the served model is YOUR fine-tune** (not the base): run the same domain prompt through the served model and through the HF `models/merged` model. The outputs should exhibit the same steered behavior.

**Verify no telemetry**:

```bash
# Disconnect from the network (turn off Wi-Fi / unplug ethernet)
ollama run domain-model "test prompt"
# If it still responds, the air-gap holds.

# Or capture packets on a clean run:
sudo tcpdump -i any -n 'not port 22' &
ollama run domain-model "test prompt"
# Check the tcpdump output for ANY outbound connections. There should be none.
```

Record the latency (tok/s) and the telemetry result in the README.

**Verification**: Model responds with fine-tuned behavior; latency is usable (>5 tok/s interactive); zero outbound network calls confirmed.

---

## Phase 7 — Reproducibility package (15 min)

**Deliverable**: A README that reads like a short technical report and a one-command pipeline.

### README structure

```markdown
# Air-Gapped [Domain] Model

## Domain rationale
[One paragraph: why this domain, why this base, three-outcome-test result]

## Pipeline
[The seven-phase diagram or list]

## Results
| Metric | Base | Fine-tuned | Delta |
|--------|------|------------|-------|
| Domain held-out | XX.X | XX.X | +X.X (lift) |
| General (MMLU/GSM8K) | XX.X | XX.X | -X.X (forgetting) |
| Serve latency | — | XX tok/s | Q4_K_M, [hardware] |

## Reproducibility
- Python 3.11.X, [OS], torch 2.4.0, CUDA 12.X / MPS
- `pip install -r requirements.txt`
- `make pipeline`
- Seeds set (42). Re-run should reproduce within ±2 pts.

## Data safety
[Source, de-identification, provenance. E.g.: "PubMedQA pqa_labeled, public, de-identified by construction. 1k train / 100 held-out, seed=42, MinHash dedup. No PHI. Recipe publishable."]

## Serve
cd serve && ollama create domain-model -f Modelfile && ollama run domain-model
```

### One-command run

Confirm `make pipeline` executes all phases in order (or fails loudly at the first broken phase). Test it on a clean clone if possible.

**Verification**: A reviewer reading the README sees the numbers, the methodology, the data-safety note, and the one-command run. They can clone, run, and compare.

---

## Deliverables checklist

Submit the GitHub repo. Required:

- [ ] `DOMAIN.md` — domain rationale, base choice, three-outcome-test result
- [ ] `DATA.md` — data provenance, de-identification, train/held-out split, no PHI
- [ ] `TRAINING.md` — hyperparameters (rank, alpha, targets, LR, epochs, batch, scheduler, seed)
- [ ] `eval/results.json` — base, fine-tuned, lift, forgetting numbers
- [ ] `serve/domain-model.gguf` — the quantized model
- [ ] `README.md` — results table, reproducibility section, data-safety note, serve instructions
- [ ] `Makefile` — `make pipeline` chains all phases
- [ ] `requirements.txt` — pinned versions

---

## Evaluation rubric (self-assess before submitting)

| Dimension | Your score (1-3) | Notes |
| --- | --- | --- |
| Domain lift (>+5=3, positive=2, none/contaminated=1) | | |
| Forgetting control (reported + <2 or mitigated=3, reported=2, not measured=1) | | |
| Local serve (loads+behavior+no telemetry=3, loads+no telemetry check=2, needs network=1) | | |
| Reproducibility (one-command+pinned+numbers=3, mostly=2, notebook-only=1) | | |
| HIPAA-safety (open/de-id+provenance=3, open+unclear provenance=2, real PHI=1) | | |

**Passing: 12/15+ with no dimension below 2. HIPAA-safety is a gate.**

---

## Solution key (what a passing submission looks like)

- **Domain**: medical Q&A via PubMedQA. Base: MiniCPM5-1B (open-data, Apache-2.0).
- **Data**: PubMedQA pqa_labeled, 1000 train / 100 held-out, seed=42, MinHash dedup. No PHI.
- **Training**: QLoRA r=16, alpha=32, targets q/k/v/o_proj, LR 2e-4, 3 epochs, cosine, seed=42.
- **Results**: Base exact-match 32.0 → Fine-tuned 41.5 (lift +9.5). MMLU (200-subset) base 44.5 → 43.8 (forgetting -0.7).
- **Export**: Q4_K_M, 0.62GB GGUF.
- **Serve**: Ollama, 18 tok/s on M2 Pro. Verified no telemetry (ran with Wi-Fi off).
- **README**: full results table, reproducibility section with `make pipeline`, data-safety note.
- **Rubric**: lift 3, forgetting 3, serve 3, reproducibility 3, HIPAA 3 = 15/15. Pass.

A submission at this shape passes. A submission missing the forgetting number, or with an unjustified closed base, or that phones home on serve, fails — regardless of how strong the lift is.

---

## Stretch goals

1. **Add a second quant level.** Export Q8_0 and Q4_K_M, eval both on the domain held-out, report the quality/size trade-off. (FT19.)
2. **Run the pipeline on a second hardware target.** If you trained on CUDA, serve on Apple Silicon (or vice versa). Document any cross-platform variance. (Reproducibility stress test.)
3. **Publish the pipeline to GitHub with a CI check.** Add a GitHub Action that runs `make data` and `make eval` on push, failing if the held-out lift regresses below a threshold. (The portfolio asset, hardened.)
4. **Decontaminate against the held-out set formally.** Use an n-gram overlap check (FT06) and report the overlap percentage. A clean held-out set has <1% n-gram overlap with train.