# Module FT04 — Dataset Formats

**Course**: Course 3 — LLM Fine-Tuning Masterclass
**Module**: FT04 — Dataset Formats
**Duration**: 45 minutes
**Level**: Senior Engineer and above
**Prerequisites**: FT00 (The Steering Stack)

---

## Learning Objectives

After completing this module, you will be able to:

1. Recognize the three canonical dataset formats — **raw messages**, **instruction/response**, and **ShareGPT conversational** — and load each correctly with HuggingFace `datasets`.
2. Apply the model's **chat template** with `apply_chat_template()` and explain why hand-concatenating strings is the single most common silent bug in fine-tuning (token-boundary errors, special-token merging).
3. Structure **SFT data** (one correct response) versus **preference data** (chosen/rejected pairs) and place each behind the correct trainer.
4. Run the **end-to-end inspection** — decode the tokenized `input_ids` back to text — to catch format bugs *before* they cost you a training run.

---

# 4.1 — The Three Canonical Formats

*Pillar 1 is the substrate layer. The substrate is the steering wheel. Get the wheel crooked and every later module steers into a wall.*

There are exactly three shapes your data can arrive in. Every fine-tuning dataset in the field is one of these (or a preference-flavored variant covered in 4.4). Learn to recognize each on sight.

## Format 1 — Raw messages (the lingua franca)

A Python list of `{"role", "content"}` dicts. This is what the model's tokenizer actually consumes internally, and it is what `apply_chat_template` expects.

```python
example = {
    "messages": [
        {"role": "system",    "content": "You are a concise assistant."},
        {"role": "user",      "content": "What is 2+2?"},
        {"role": "assistant", "content": "4"},
    ]
}
```

Roles are constrained. The four you will see: `system`, `user`, `assistant`, `tool`. Anything else is a bug. The order matters — `system` first (optional), then alternating `user`/`assistant`, with `tool` messages inserted where tool calls resolved.

Load it with `datasets`:

```python
from datasets import Dataset
ds = Dataset.from_list([example, another_example, ...])
# or from disk:
ds = Dataset.from_json("data.jsonl")
```

If your data is already in messages form, you are done with format conversion. Skip to 4.2.

## Format 2 — Instruction/response (the two-column CSV)

The classic Alpaca/Vicuna shape: two columns, `instruction` and `output` (sometimes `input` as an optional third). This is what most "fine-tuning datasets" on the Hub look like because it is trivial to produce.

```python
example = {
    "instruction": "What is 2+2?",
    "input": "",              # optional context
    "output": "4",
}
```

Load and *convert to messages* before training:

```python
from datasets import Dataset

ds = Dataset.from_json("alpaca.jsonl")

def to_messages(row):
    user = row["instruction"] + ("\n\n" + row["input"] if row.get("input") else "")
    return {"messages": [
        {"role": "user",      "content": user},
        {"role": "assistant", "content": row["output"]},
    ]}

ds = ds.map(to_messages, remove_columns=ds.column_names)
```

The single most common mistake at this step: leaving the data as instruction/response and handing it to a trainer that expects messages. TRL's `SFTTrainer` will accept it via its `formatting_func`, but you are now responsible for getting the formatting right — and you will get it wrong. Convert once, sleep at night.

## Format 3 — ShareGPT / conversational (the multi-turn shape)

Used by most community-shared chat datasets. Top-level key `conversations` (sometimes `messages`); each turn is `{"from": "human"/"gpt"/"system", "value": "..."}`. The `from` field uses OpenAI-era role names, not the modern `role` vocabulary.

```python
example = {
    "conversations": [
        {"from": "human", "value": "What is 2+2?"},
        {"from": "gpt",   "value": "4"},
        {"from": "human", "value": "And 3+3?"},
        {"from": "gpt",   "value": "6"},
    ]
}
```

Convert it. Map `human → user`, `gpt → assistant`, `system → system`, `tool → tool`. Strip the `from`/`value` keys and rename to `role`/`content`:

```python
ROLE_MAP = {"human": "user", "gpt": "assistant", "system": "system", "tool": "tool"}

def sharegpt_to_messages(row):
    return {"messages": [
        {"role": ROLE_MAP[t["from"]], "content": t["value"]}
        for t in row["conversations"]
    ]}

ds = ds.map(sharegpt_to_messages, remove_columns=ds.column_names)
```

The pattern across all three: **whatever shape your data arrives in, you normalize it to a single `messages` column.** From there every downstream tool — `apply_chat_template`, TRL, the tokenized inspection loop — works identically. One internal representation; one conversion; one bug surface.

---

# 4.2 — `apply_chat_template()` — The One Right Way

*This section is the highest-leverage five minutes in the whole module. Most silent training failures trace back to skipping it.*

## The rule

> **Always apply the model's chat template. Never hand-concatenate strings.**

A chat template is a Jinja2 file shipped *with the tokenizer* (in `tokenizer_config.json`). It encodes the exact byte sequence the model was instruction-tuned against: the special tokens, their order, the whitespace, the newline placement, the system prompt slot. For ChatML models it produces `<|im_start|>user\n...<|im_end|>`. For Llama-3 it produces `<|start_header_id|>user<|end_header_id|>\n\n...<|eot_id|>`. For Qwen it is `<|im_start|>` again but with different EOS handling. **Every family is different, and the differences are invisible to the eye but catastrophic to training.**

```python
text = tokenizer.apply_chat_template(
    example["messages"],
    tokenize=False,          # get a string back, not token IDs
    add_generation_prompt=False,  # full conversation (training). True = prompt only (inference).
)
```

That single call replaces every hand-rolled `f"<user>{q}</user><assistant>{a}</assistant>"` you have ever written. It is the contract between the model and your data.

## Why hand-concatenation is the #1 silent bug

Three failure modes, each invisible until you decode the tokenized output (which is why nobody catches them).

**1. Token-boundary errors.** Tokenizers merge text greedily at whitespace. The string `"<|im_start|>user"` and the string `"<|im_start|> user"` tokenize to *different* token sequences because the leading space changes how the BPE merges happen. When you hand-concat, you almost always get the whitespace wrong by one character. The model sees a token sequence it never saw during its own instruction tuning, and behaves as if it is slightly out of distribution. The training still runs; the loss still goes down; the model is just *worse* than it should be, and you will blame the data, the rank, the optimizer — anything but the whitespace.

**2. Special-token merging.** Write `"<|im_start|>user"` as a raw string and the tokenizer treats `<|im_start|>` as *literal text*, not a special token — unless you remember `tokenize=False` plus `add_special_tokens=True` plus the tokenizer's `added_tokens_decoder` is set up correctly, which it only is if you used the template. With the template, `<|im_start|>` becomes a single token ID (e.g., 151644). Without it, it becomes six or seven ordinary tokens, and the model learns a completely different vocabulary boundary. This is the bug that produces "the model trained fine but talks like a different person."

**3. Wrong role tokens entirely.** A Llama-3 model has no `<|im_start|>` token. A ChatML model has no `<|start_header_id|>`. If you copied a formatting snippet from a Qwen tutorial and applied it to a Llama-3 model, the tokenizer falls back to treating those as unknown text. The model trains on garbage delimiters and you wonder why it cannot follow instructions.

`apply_chat_template` solves all three at once. It reads the tokenizer's own Jinja file — the same file the model's publisher used during instruction tuning — and produces the exact bytes the model expects. **There is no scenario in which hand-concatenation is safer, faster, or "simpler." It is a bug waiting one training run to fire.**

---

# 4.3 — Tokenizing and the Inspection Loop

*The five-second check that catches every formatting bug. Do this once per dataset and you have removed 80% of the silent-failure surface.*

After `apply_chat_template`, the next call is `tokenizer(...)`. That produces `input_ids`, `attention_mask`, and (if you ask) `labels`. Before you hand those to a trainer, **decode them back to text and read the result.**

```python
# Build the text
text = tokenizer.apply_chat_template(messages, tokenize=False)

# Tokenize
inputs = tokenizer(text, return_tensors="pt", add_special_tokens=False)

# THE INSPECTION — do this every time
print(tokenizer.decode(inputs["input_ids"][0]))
```

What you are looking at is what the model will actually see during training — token by token, special token by special token. Read it. The bugs are obvious once you look:

- A missing `<|im_end|>` / `<|eot_id|>` at the end means the model never learns to stop. It will generate forever at inference.
- A `<|im_start|>assistant` token that should mark where the assistant starts is missing — so the loss mask will train on the user's text too, and the model learns to echo.
- A literal `<|im_start|>` rendered as `<`, `|`, `im`, `_`, `start`, `_`, `|`, `>` instead of one token ID — the special-token-merging bug from 4.2.
- A trailing space before `<|im_end|>` — drifts the model's stopping behavior.

A second, sharper inspection: print the token IDs of the special tokens specifically.

```python
for tok in ["<|im_start|>", "<|im_end|>", "<|start_header_id|>", "<|eot_id|>"]:
    if tok in tokenizer.added_tokens_encoder:
        print(f"{tok!r:25} -> {tokenizer.added_tokens_encoder[tok]}")
    else:
        print(f"{tok!r:25} -> NOT A SPECIAL TOKEN in this tokenizer")
```

If a token the template uses does not appear here, your model and your template are mismatched. Stop. Fix it before training.

The inspection loop is the single habit that separates engineers who ship fine-tuned models from engineers who debug them. Ten seconds of reading decoded text saves ten hours of "why is the model broken."

---

# 4.4 — SFT Data vs Preference Data

*Two distinct training objectives, two distinct data shapes. Mixing them is a bug.*

## SFT data — one correct response

The shape from 4.1: a list of `messages` ending in an `assistant` turn. There is exactly one right answer per example. The trainer (TRL `SFTTrainer`) computes cross-entropy loss on the assistant turns and ignores the rest via the loss mask.

```python
sft_example = {
    "messages": [
        {"role": "system",    "content": "You are a concise assistant."},
        {"role": "user",      "content": "What is 2+2?"},
        {"role": "assistant", "content": "4"},
    ]
}
```

SFT steers *format and instruction-following*. It does not compare alternatives. (Module FT12.)

## Preference data — chosen vs rejected

Two responses to the same prompt; one is preferred. Used by DPO, KTO, IPO, and the rest of the preference-optimization family (Module FT13). Two acceptable shapes:

**Shape A — explicit prompt:**

```python
pref_example = {
    "prompt":   [{"role": "user", "content": "What is 2+2?"}],
    "chosen":   [{"role": "assistant", "content": "4"}],
    "rejected": [{"role": "assistant", "content": "I refuse to do math."}],
}
```

**Shape B — full conversations (each ending in the assistant turn):**

```python
pref_example = {
    "chosen":   [{"role": "user", "content": "What is 2+2?"},
                 {"role": "assistant", "content": "4"}],
    "rejected": [{"role": "user", "content": "What is 2+2?"},
                 {"role": "assistant", "content": "I refuse to do math."}],
}
```

TRL's `DPOTrainer` accepts both. The defining property: **`chosen` and `rejected` share the same prompt and differ only in the final assistant turn.** If they differ earlier in the conversation, your dataset is malformed and DPO will produce nonsense.

The mistake to avoid: feeding SFT data to a DPO trainer, or vice versa. SFT data has no `chosen`/`rejected` split, so DPO has nothing to contrast. Preference data has no single "correct" response, so SFT would train on whichever you happened to put first. The shape signals the objective.

---

# 4.5 — The Bug Taxonomy

*A checklist of the format bugs you will encounter, ranked by how silently they fail.*

| Bug | Symptom | Why it happens |
| --- | --- | --- |
| **Missing EOS** | Model never stops generating | Last assistant turn not closed with `<\|im_end\|>` / `<\|eot_id\|>` |
| **Wrong role tokens** | Model can't follow instructions | Copied a ChatML template onto a Llama-3 model (or vice versa) |
| **Special tokens as text** | Model trains on literal `<\|im_start\|>` text | Hand-concatenated instead of `apply_chat_template` |
| **Trailing spaces** | Stopping behavior drifts | A `"\n "` or `" "` before the closing special token |
| **Loss on user turns** | Model learns to echo the prompt | `completion_only_loss` / loss mask not set; `labels` not `-100`-masked |
| **Mixed formats** | Half the dataset is instruction/response, half is messages | Didn't normalize before `map()` |
| **Wrong `add_generation_prompt`** | Inference prompt is incomplete / training data is truncated | Set `True` at training or `False` at inference |
| **Tokenizer mismatch** | Tokens appear as `[UNK]` or render as text | Loaded a tokenizer from a different model than the one you're training |

Every one of these is caught by the inspection loop in 4.3. None is caught by loss curves. That asymmetry — silent in training, obvious in inspection — is why inspection is non-negotiable.

---

## Anti-Patterns

### Hand-concatenating strings

`f"<user>{q}</user><assistant>{a}</assistant>"`. It looks simpler than `apply_chat_template`. It is not — it is a bug deferred. Token-boundary errors, special-token-as-text, wrong role tokens. The model trains, the loss drops, the model is subtly wrong. Always use the template.

### Not inspecting tokenized output

If you have never called `tokenizer.decode(inputs["input_ids"][0])` on your dataset, you are flying blind. The decoded text is the ground truth of what the model sees. The raw messages are not. Inspect once per dataset; the bugs you catch cost you zero training runs.

### Mixing formats

Half the dataset as instruction/response, half as messages, fed to the same trainer. Either the trainer errors out (the lucky case) or it silently trains on malformed rows. Normalize to one format up front.

### Assuming the tokenizer handles roles

The tokenizer does not know what a "role" is. It knows tokens. The mapping from role to token is what the chat template encodes. Skip the template and you are guessing at that mapping, and the model was tuned against a specific guess.

### Treating SFT and preference data as interchangeable

SFT data has one response; preference data has two. The trainer for each expects the matching shape. Swapping them silently produces a model that learns the wrong objective.

---

## Key Terms

| Term | Definition |
| --- | --- |
| **Raw messages format** | A list of `{"role", "content"}` dicts; the internal lingua franca for all trainers |
| **Instruction/response format** | Two columns, `instruction` and `output`; the Alpaca/Vicuna shape; must be converted to messages |
| **ShareGPT format** | `{"conversations": [{"from", "value"}]}` with role names `human`/`gpt`; must be converted to messages |
| **Chat template** | A Jinja2 file shipped with the tokenizer that encodes the exact byte sequence the model expects per family |
| **`apply_chat_template`** | The tokenizer method that renders messages into the model's native string format; the only correct way to format |
| **Special token** | A token (e.g., `<\|im_start\|>`) the tokenizer treats as a single ID, not ordinary text; encoded in `added_tokens_decoder` |
| **EOS (end-of-sequence)** | The special token that signals the model to stop; missing it is the most common "won't stop generating" bug |
| **SFT data** | Messages with one correct assistant response; trains cross-entropy on the assistant turn |
| **Preference data** | `{prompt, chosen, rejected}` or two conversations differing only in the final turn; trains DPO/IPO/KTO |
| **Inspection loop** | `tokenizer.decode(inputs["input_ids"][0])` — read the decoded text before training to catch format bugs |
| **Loss mask** | Setting `labels` to `-100` on non-assistant tokens so the trainer ignores them; prevents learning to echo the prompt |

---

## Lab Exercise

See `07-lab-spec.md`. The "Format, Template, Inspect" lab: take a raw CSV of Q/A, convert it to all three canonical formats, apply the model chat template, decode the tokenized result, and catch the deliberate formatting bug planted in the starter data. CPU-only, MiniCPM5-1B tokenizer, ~30 minutes.

---

## References

1. **HuggingFace Transformers — Chat Templates** — *the official documentation for `apply_chat_template`*. The canonical reference for why templates exist and how to use them.
2. **HuggingFace TRL — Dataset Formats** — *SFT, DPO, and preference data shapes accepted by TRL trainers*.
3. **OpenAI — ChatML Format** — *the `<|im_start|>`/`<|im_end|>` convention used by Qwen, Yi, and many open models*.
4. **Meta — Llama 3 Prompt Format** — *the `<|start_header_id|>`/`<|eot_id|>` convention; the reminder that every family differs*.
5. **Taori et al. (2023) — Stanford Alpaca** — *the instruction/response format that became the default for community datasets*.
6. **Module FT07 — Tokenizers & Chat Templates** — *the deep dive on tokenization, BPE merges, and special-token handling*.
7. **Module FT00 — The Steering Stack** — *the prerequisite; data is the steering wheel, and format errors make the wheel crooked*.