# Lab Specification — Module FT04: Dataset Formats

**Course**: Course 3 — LLM Fine-Tuning Masterclass
**Module**: FT04 — Dataset Formats
**Duration**: 30–45 minutes
**Environment**: Python 3.11+. **CPU only** — we use the tokenizer, never the model. ~200MB free disk for the MiniCPM5-1B tokenizer files.

---

## Learning objectives

By the end of this lab you will have:

1. **Loaded a raw Q/A CSV** and converted it into all three canonical formats — raw messages, instruction/response, and ShareGPT — proving you can recognize each on sight.
2. **Applied the model's chat template** with `apply_chat_template()` and compared it against a hand-concatenated string to see exactly why hand-concat is a silent bug.
3. **Run the inspection loop** — `tokenizer.decode(inputs["input_ids"][0])` — and read the decoded text to verify formatting *before* training.
4. **Caught a deliberate formatting bug** planted in the starter data and named its category from the bug taxonomy.

No GPU, no model weights, no training. The point is to feel the format substrate that every later module builds on.

---

## Phase 0 — Environment setup (5 min)

```bash
python3.11 -m venv ft04-env && source ft04-env/bin/activate
pip install -q transformers datasets
```

Verify (CPU-only is fine — we never instantiate the model):

```python
import transformers, datasets
print(f"transformers: {transformers.__version__}")
print(f"datasets:     {datasets.__version__}")
```

Load only the tokenizer (this downloads the tokenizer files, ~50MB, not the full model):

```python
from transformers import AutoTokenizer

MODEL_ID = "openbmb/MiniCPM5-1B"   # open weights + open data, Apache-2.0
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, trust_remote_code=True)
print("chat template present:", tokenizer.chat_template is not None)
```

If `chat template present: True`, you are ready. (If False, your transformers version is stale — upgrade it.)

---

## Phase 1 — The starter data (raw Q/A) (5 min)

Here is the starter data. **One of these rows contains a deliberate formatting bug.** Do not scroll ahead looking for it — you will find it in Phase 4 by inspecting the tokenized output, which is the whole point.

```python
import pandas as pd

raw_rows = [
    {"question": "What is 2+2?",                    "answer": "4"},
    {"question": "Name a primary color.",            "answer": "Blue"},
    {"question": "What is the capital of France?",   "answer": "Paris"},
    {"question": "Translate 'hello' to Spanish.",    "answer": "Hola "},   # <- trailing space (planted bug)
    {"question": "What gas do plants absorb?",       "answer": "Carbon dioxide"},
]
qa = pd.DataFrame(raw_rows)
print(qa)
```

This is the most common starting point in the real world: a CSV/JSONL of question-answer pairs, with no notion of roles or templates.

---

## Phase 2 — Convert to all three formats (10 min)

Convert the same source data into the three canonical shapes. The point is to feel that they are three different *representations* of the same underlying conversation.

**Format 1 — Raw messages:**

```python
def to_messages(row):
    return [{
        "role": "user",      "content": row["question"],
    }, {
        "role": "assistant", "content": row["answer"],
    }]

messages_examples = [to_messages(r) for _, r in qa.iterrows()]
print("=== FORMAT 1: raw messages ===")
print(messages_examples[0])
```

**Format 2 — Instruction/response (the Alpaca shape):**

```python
instruction_response = [
    {"instruction": r["question"], "input": "", "output": r["answer"]}
    for _, r in qa.iterrows()
]
print("=== FORMAT 2: instruction/response ===")
print(instruction_response[0])
```

**Format 3 — ShareGPT (the community shape, `from`/`value`):**

```python
sharegpt = [
    {"conversations": [
        {"from": "human", "value": r["question"]},
        {"from": "gpt",   "value": r["answer"]},
    ]}
    for _, r in qa.iterrows()
]
print("=== FORMAT 3: ShareGPT ===")
print(sharegpt[0])
```

**Now normalize ShareGPT back to messages** — the canonical conversion you will run on every ShareGPT dataset you download from the Hub:

```python
ROLE_MAP = {"human": "user", "gpt": "assistant", "system": "system", "tool": "tool"}

def sharegpt_to_messages(conv):
    return [{"role": ROLE_MAP[t["from"]], "content": t["value"]} for t in conv["conversations"]]

normalized = [sharegpt_to_messages(c) for c in sharegpt]
assert normalized == messages_examples, "ShareGPT normalization should reproduce Format 1 exactly"
print("ShareGPT -> messages conversion verified (matches Format 1).")
```

**Record**: all three formats produce the same underlying conversation once normalized. The `messages` shape is the lingua franca; the other two are just import conventions.

---

## Phase 3 — Apply the chat template vs hand-concat (10 min)

This is the heart of the lab. Take one example and render it two ways: the right way (template) and the wrong way (hand-concat). Compare the bytes.

**The RIGHT way:**

```python
example = messages_examples[0]

text_template = tokenizer.apply_chat_template(
    example,
    tokenize=False,
    add_generation_prompt=False,   # full conversation (training shape)
)
print("=== TEMPLATE OUTPUT ===")
print(repr(text_template))
```

**The WRONG way (the anti-pattern):**

```python
text_handconcat = (
    "<|im_start|>user\n" + example[0]["content"] + "<|im_end|>\n"
    "<|im_start|>assistant\n" + example[1]["content"] + "<|im_end|>"
)
print("=== HAND-CONCAT OUTPUT ===")
print(repr(text_handconcat))
```

**Compare the bytes:**

```python
print("strings identical?", text_template == text_handconcat)
```

If your tokenizer's template adds a trailing newline (most do) or differs on whitespace, these strings will *not* be identical even though they look the same to the eye. That is the lesson: the eye cannot catch these bugs; `repr()` and tokenization can.

**Now compare the token sequences** — the real test:

```python
ids_template   = tokenizer(text_template,   add_special_tokens=False)["input_ids"]
ids_handconcat = tokenizer(text_handconcat, add_special_tokens=False)["input_ids"]

print(f"template tokens:   {len(ids_template)}")
print(f"hand-concat tokens: {len(ids_handconcat)}")
print("token sequences identical?", ids_template == ids_handconcat)

# Find the first divergence, if any
for i, (a, b) in enumerate(zip(ids_template, ids_handconcat)):
    if a != b:
        print(f"first divergence at position {i}:")
        print(f"  template   token {a}: {tokenizer.decode([a])!r}")
        print(f"  handconcat token {b}: {tokenizer.decode([b])!r}")
        break
```

**Record**: how many tokens differ, and where. Even a one-token difference means the model sees a different input than its instruction-tuning data used.

> **Teaching moment:** If the token sequences differ, the model is being trained on a slightly out-of-distribution sequence. The loss will still drop. The model will still "work." It will just be subtly worse, and you will spend days blaming your data quality, your LoRA rank, your learning rate — never the whitespace. This is why the rule is absolute: **always `apply_chat_template`, never hand-concat.**

---

## Phase 4 — The inspection loop and the planted bug (10 min)

Now run the inspection loop on the full dataset. This is the habit.

```python
def inspect(example, idx):
    text = tokenizer.apply_chat_template(example, tokenize=False, add_generation_prompt=False)
    inputs = tokenizer(text, return_tensors="pt", add_special_tokens=False)
    decoded = tokenizer.decode(inputs["input_ids"][0])
    print(f"--- Example {idx} ---")
    print(decoded)
    print()
    return decoded

for i, ex in enumerate(messages_examples):
    inspect(ex, i)
```

**Read each decoded example carefully.** Look for the bug taxonomy signatures:

- Missing EOS (no `<|im_end|>` at the end)?
- Specials rendered as literal text (look for `<`, `|`, `i`, `m` as separate characters)?
- **Trailing spaces** before the closing special token?
- Anything else that looks off?

**The planted bug is in Example 3** (the "Translate 'hello' to Spanish" row). The answer is `"Hola "` with a trailing space. In the decoded output you will see something like:

```
Hola <|im_end|>
```

That trailing space *before* `<|im_end|>` is a real, common bug. It changes the tokenization of the boundary (the space may merge with the closing token or with `Hola`), and it teaches the model that assistant turns end with a space — which drifts its stopping behavior at inference.

**Verify it** by checking the raw token IDs around the boundary:

```python
buggy = messages_examples[3]
text = tokenizer.apply_chat_template(buggy, tokenize=False)
ids = tokenizer(text, add_special_tokens=False)["input_ids"]
# show the last 6 tokens
print("last 6 tokens:", ids[-6:])
print("decoded:", [repr(tokenizer.decode([t])) for t in ids[-6:]])
```

**Fix it** and confirm the fix:

```python
fixed = [
    {"role": "user",      "content": "Translate 'hello' to Spanish."},
    {"role": "assistant", "content": "Hola"},   # trailing space removed
]
text_fixed = tokenizer.apply_chat_template(fixed, tokenize=False)
ids_fixed = tokenizer(text_fixed, add_special_tokens=False)["input_ids"]
print("last 6 tokens (fixed):", ids_fixed[-6:])
print("decoded:", [repr(tokenizer.decode([t])) for t in ids_fixed[-6:]])
```

**Record**: the difference in the boundary tokens between buggy and fixed. The fix is one character; the silent effect on training would be a model that stops generating slightly unreliably.

---

## Phase 5 — Bonus: spot-check the special tokens (5 min)

Confirm that the special tokens the template uses are actually registered as special tokens (not ordinary text) in this tokenizer:

```python
for tok in ["<|im_start|>", "<|im_end|>", "<|endoftext|>"]:
    if tok in tokenizer.added_tokens_encoder:
        print(f"{tok!r:18} -> single token ID {tokenizer.added_tokens_encoder[tok]}  (CORRECT)")
    else:
        print(f"{tok!r:18} -> NOT a special token here  (would be a bug if the template used it)")
```

If a token the template relies on is missing from `added_tokens_encoder`, your tokenizer and your model are mismatched — the most catastrophic format bug. Stop and reconcile before any training.

---

## Deliverables

Submit `ft04-lab-report.md`:

- [ ] Phase 2: the three format shapes for one example; the assertion that ShareGPT normalization reproduces Format 1
- [ ] Phase 3: whether template vs hand-concat strings are identical; the first token divergence (if any) and its position
- [ ] Phase 4: the decoded output of all 5 examples; identification of the planted bug (which example, what category); the before/after boundary tokens
- [ ] Phase 5: the special-token registry output

---

## Solution key

- **Phase 2**: All three formats represent the same conversation. `sharegpt_to_messages` reproduces `to_messages` exactly (the assertion passes).
- **Phase 3**: On MiniCPM5-1B, the template typically adds a trailing newline that hand-concat omits, so the strings differ and the token sequences differ by 1+ tokens. The exact divergence depends on the template version; the point is that *they differ*, demonstrating why hand-concat is unsafe. If a student's strings happen to match exactly, have them try a different example or note that the match is fragile (it breaks the moment the template changes).
- **Phase 4**: The planted bug is the trailing space in Example 3's answer (`"Hola "`). In the bug taxonomy it is the **trailing spaces** category — "whitespace before closing token." The boundary tokens differ: with the space, the token sequence near `<|im_end|>` may include a space-bearing token; without it, a clean boundary. After fixing (strip the trailing space), the boundary is clean.
- **Phase 5**: MiniCPM5-1B registers `<|im_start|>`, `<|im_end|>`, `<|endoftext|>` as single special tokens. If any are missing, the student has the wrong tokenizer for the template — the catastrophic mismatch bug.

---

## Stretch goals

1. **Plant your own bug.** Take a clean example and introduce a missing-EOS bug (delete the closing `<|im_end|>` from the answer). Run the inspection loop and confirm you can spot it by eye in the decoded text. (Sets up the habit of always inspecting.)
2. **Try a Llama-3 template.** Load the tokenizer for a Llama-3 family model (e.g., `meta-llama/Meta-Llama-3-8B-Instruct`) and run the same `apply_chat_template` call. Confirm the output uses `<|start_header_id|>`/`<|eot_id|>` instead of `<|im_start|>`/`<|im_end|>` — proving every family is different and the template is the contract. (Sets up Module FT07.)
3. **Build a preference pair.** Take one Q/A example and construct a `{prompt, chosen, rejected}` preference example by writing a worse answer for `rejected`. Confirm the shape matches the DPO format from section 4.4. (Sets up Module FT13.)