# Module FT07 — Tokenizers & Chat Templates

**Course**: Course 3 — LLM Fine-Tuning Masterclass
**Module**: FT07 — Tokenizers & Chat Templates
**Duration**: 60 minutes
**Level**: Senior Engineer and above
**Prerequisites**: FT04 (Dataset Formats)

---

## Learning Objectives

After completing this module, you will be able to:

1. State the module thesis — *tokenizers and chat templates are where most SILENT training failures live; the model trains, loss goes down, but the model learns the wrong thing because the tokens were wrong* — and defend it with the top three bug classes.
2. Diagnose the three canonical silent-bug families from their symptoms: cross-family template misuse, EOS mishandling, and packing-without-attention-masking.
3. Explain why `apply_chat_template()` is mandatory and hand-concatenation is an anti-pattern, and distinguish the role-token conventions of the major families (Llama-3 vs Qwen ChatML vs Mistral/Gemma).
4. Decide, for a given domain, whether to train a new tokenizer, reuse the base tokenizer, or extend the vocab with domain tokens — and execute the vocab-extension workflow correctly (resize embeddings + lm_head, warm up new rows).
5. Inspect a tokenized training example end-to-end: encode, decode, and verify that the role boundaries, EOS, and assistant-only loss mask are correct before launching a run.

---

# 7.1 — The Thesis

*The data pipeline is not complete when the JSONL is clean. It is complete when the tokens are correct. FT07 is where that last step lives — and where 80% of silent training failures hide.*

## The sentence

> **The model trains. The loss goes down. The checkpoint loads. And the model learned the wrong thing — because the tokens were wrong.**

This is the silent-failure module. Every prior module in Pillar 1 was about *what* you feed the model: the format (FT04), the synthetic generation (FT05), the dedup and decontamination (FT06). This module is about *how that data becomes tokens* — the last transformation between your JSONL and the model's embedding layer. It is the single most under-scrutinized step in the pipeline, and it is where most silent training failures live.

A silent failure is not a crash. The training loop runs. The loss curve descends. TensorBoard looks healthy. You ship the adapter. And the model produces malformed output, ignores role boundaries, never stops generating, or quietly degraded on the capability you actually cared about. The bug is not in the optimizer, the data, or the hyperparameters. It is in the tokenizer and the chat template — the layer you assumed was handled.

### Why this is the final data module

The Steering Stack from FT00 has five layers. Pillar 1 sits at the substrate — the data and tokens that every steer (SFT, DPO, GRPO, abliteration) trains on. If the tokens are wrong, the steer is wrong. A brilliant algorithm on mis-tokenized data steers you precisely in the wrong direction. This is why FT07 is the capstone of Pillar 1: it is the last gate between "clean data" and "correct tokens," and getting it wrong invalidates everything downstream.

### The debugging checklist

The field has converged on three bug classes that account for the overwhelming majority of silent tokenizer/template failures. Every fine-tuning run you ever do should be checked against this list *before* you look at the loss curve:

1. **Cross-family template misuse** — applying a Llama-3 template to a Qwen model, or vice versa.
2. **EOS mishandling** — missing or wrong end-of-sequence tokens during SFT.
3. **Packing-without-attention-mask** — concatenating examples without masking cross-boundary attention or loss.

Each is a silent killer. Each has a tell-tale symptom. Each has a one-line fix once you know to look. The rest of this module is structured around these three, with the supporting machinery (templates, base models with templates, tool-calling, vocab extension, tokenizer training) built around them.

---

# 7.2 — The Top Three Silent Bugs

*The debugging checklist. Memorize the symptoms.*

## Bug 1 — Cross-family template misuse

Every modern chat model has its own role-token convention. The model was *pretrained and instruction-tuned* to recognize a specific sequence of special tokens as the boundary between system, user, and assistant turns. Use the wrong convention and you corrupt the model's understanding of who is speaking and where turns end.

The two conventions you will meet most often:

- **Llama-3 family** uses `<|begin_of_text|>` to start a document, then `<|start_header_id|>` / `<|end_header_id|>` to bracket a role name (`system`, `user`, `assistant`), and `<|eot_id|>` to end a turn.
- **Qwen family** uses ChatML: `<|im_start|>` followed by the role name and a newline, then the content, then `<|im_end|>`.

These are not interchangeable. They are not "close enough." A Qwen model trained on Llama-3-formatted tokens sees unfamiliar header tokens where it expects `<|im_start|>`, and its persona and format understanding — the thing you are trying to steer — degrades. The loss may still go down (the model is learning *something*), but the model learns a distribution contaminated by the wrong scaffolding.

The failure mode is usually subtle: the model produces text that is *almost* right but role boundaries are fuzzy, the assistant sometimes continues the user's turn, or the format drifts at inference. The diagnosis is simple: decode a tokenized training example and *read it*. If the special tokens do not match the model's family, you have bug 1.

## Bug 2 — EOS mishandling

The end-of-sequence token tells the model when to stop. During SFT, every assistant turn must end with the EOS token the model was trained on. Three ways this breaks:

1. **Missing EOS** — you strip it, or your data pipeline drops it, or the chat template does not append it. The model never learns to stop. At inference it generates until it hits the max-length limit, producing run-on text or hallucinated follow-up turns. This is the most common single cause of "my model won't stop generating" support threads.
2. **Wrong EOS** — you use the generic `</s>` (token id 2, the old T5/BART convention) on a model whose EOS is `<|im_end|>` or `<|end_of_text|>`. The model sees an out-of-distribution token and the stop signal is never registered.
3. **EOS in the wrong place** — you append EOS after the *whole packed sequence* instead of after *each assistant turn*. The model learns one EOS per hundred examples and again never internalizes when a single turn ends.

Bug 2 has a famous public footprint. The Hugging Face forums thread where Qwen SFT loss "explodes" (training loss jumps to absurd values or NaN) repeatedly traces back to EOS and chat-template misconfiguration — the model is being asked to predict tokens it cannot represent cleanly because the turn boundaries are wrong. The fix is almost always: use `apply_chat_template()` (which appends the correct EOS for the family), and verify by decoding the last tokens of an assistant turn.

## Bug 3 — Packing-without-attention-masking

To use GPU efficiently, you pack multiple short examples into a single fixed-length sequence (e.g., four 512-token conversations concatenated into one 2048-token sequence). This is correct and standard — *if and only if* you do two things:

1. **Mask cross-boundary attention.** Example B must not attend to example A's tokens. Without this, the model learns spurious correlations across unrelated conversations. Modern solutions: FlashAttention's document-level (variable-length) attention via the `position_ids` / cu_seqlens mechanism, or TRL's `packing=True` with the attention-mask handling built in. The wrong solution: a single `attention_mask` of all ones.
2. **Mask the loss on non-assistant tokens.** The model should only be trained to predict *assistant* tokens — the system prompt, the user turns, and the role scaffolding are context, not targets. If you compute loss on every token, the model spends capacity learning to imitate the user, which is not what you want and silently degrades assistant quality.

Bug 3 is the quietest of the three. The model trains fine. The loss looks great. The checkpoint produces *plausible* output. But the model has been subtly contaminated by cross-example attention and by learning to predict user turns. You only notice when you compare against a correctly-packed baseline and the quality gap appears. This is the silent quality regression that the checklist exists to catch.

---

# 7.3 — `apply_chat_template()`: Always Use It

*The single rule that prevents 80% of the bugs above.*

## The rule

> **Never hand-concatenate role strings. Always use the model's `apply_chat_template()`.**

FT04 introduced this rule; FT07 enforces it. Every tokenizer loaded from a modern checkpoint carries a Jinja2 chat template in `tokenizer.chat_template`. The template knows:

- The correct special tokens for this family (`<|im_start|>` vs `<|start_header_id|>`).
- Where to put the system prompt, the user turns, the assistant turns.
- Which EOS / EOT token to append, and where.
- How to handle tool calls and tool responses (where the template supports it).

Calling `tokenizer.apply_chat_template(messages, tokenize=True)` produces the correct token sequence for *that specific model*. Hand-concatenating `f"<user>{msg}</user>"` does not. The hand-concatenated string may look identical when you eyeball it, but it will be missing the special tokens, use the wrong ones, or place them wrong — and that is bug 1 or bug 2 waiting to happen.

### Inspect before you train

The discipline that makes `apply_chat_template` safe is the inspection loop:

1. Take one conversation from your dataset.
2. `ids = tokenizer.apply_chat_template(conv, tokenize=True, return_assistant_tokens_mask=True)`.
3. `print(tokenizer.decode(ids))` — *read it*. Confirm the role tokens match the family, the system prompt is present, EOS is appended.
4. Inspect the assistant-only mask — confirm only assistant tokens are unmasked.
5. Only then launch the full run.

This takes two minutes. It catches bugs 1, 2, and 3 before they cost you a GPU-hour. The number of teams that skip this step and ship a mis-tokenized model is the reason this module exists.

### A common template gotcha

Some tokenizers ship with no `chat_template` set (older base checkpoints) or with a default that does not match the model card's documented format. If `apply_chat_template` raises or produces unexpected output, the fix is to set the template explicitly from the model's documentation — never to fall back to hand-concatenation. For Qwen, the ChatML template is canonical; for Llama-3, the header-id template; for Mistral and Gemma, their respective `[INST]` and `<start_of_turn>` conventions. Match the template to the family.

---

# 7.4 — Base Models Now Ship With Templates

*The assumption "base = no template" is outdated.*

A historical assumption: base models (raw pretrained weights) have no chat template, and instruct/chat models do. This was largely true through 2023. It is no longer reliable.

Modern base checkpoints increasingly ship with chat templates and the corresponding special tokens already in the tokenizer vocab. Qwen3 base, for example, includes the ChatML special tokens (`<|im_start|>`, `<|im_end|>`) and a default chat template, even on the base checkpoint. The reason is operational: the tokenizer is shared across the base and instruct variants of the same release, so the special tokens are present from the start.

This has two implications:

1. **Do not assume base = no template.** Always check `tokenizer.chat_template`. If it is present, use it (or override it deliberately). If it is absent, set it explicitly from the model card.
2. **The special tokens are real vocabulary entries** in the base tokenizer, with trained embeddings. You are not "adding" them when you format SFT data — you are *using* embeddings the model already has. This is why base→SFT formatting works cleanly on Qwen3: the scaffolding tokens are not cold-started.

The takeaway: the base/instruct/chat distinction (FT03) is about *weights*, not about *template presence*. Check the tokenizer regardless of which checkpoint you loaded.

---

# 7.5 — Tool-Calling Templates

*The default template is not always correct for tools. Verify empirically.*

When your SFT data includes tool calls (the assistant emits a structured function call; a tool response comes back; the assistant continues), the chat template must handle four roles: system, user, assistant, and tool. Each family has its own convention for how a tool call is wrapped — Qwen uses a `<tool_call>...</tool_call>` block inside the assistant turn; Llama-3.1 uses a specific JSON schema inside the assistant content; Hermes uses a `<tool_call>` tag with its own escaping.

The default `apply_chat_template` handles the common case. But the *common case is not every case*. Known issues:

- The default Qwen template can mishandle some multi-turn tool-call scenarios where the tool response itself contains text that looks like a special token. The model may truncate or mis-parse.
- Some templates do not correctly append EOS after a tool-call turn when the call is the final turn in the conversation.
- The assistant-tokens mask (for loss) can be wrong on tool-call turns — the tool *response* is not assistant-generated and should be masked, but naive templates sometimes unmask it.

The rule: if your data includes tool calls, decode several examples including the assistant mask, and verify the boundaries are correct. If the default template is wrong for your case, set a corrected `chat_template` explicitly rather than fighting the default. Tool-calling is the area where `apply_chat_template` is most likely to need your intervention.

---

# 7.6 — Train Your Own Tokenizer vs Reuse the Base

*Almost always: reuse the base. Train your own only for grossly inefficient domain tokenization.*

## The decision

Training a new tokenizer is one of the most over-prescribed interventions in fine-tuning. The default — and the correct answer for the overwhelming majority of domains — is to **reuse the base tokenizer**. The base tokenizer was trained on a large, diverse corpus and handles most natural language, code, and common scripts efficiently.

You train your own tokenizer only when the base tokenizer is *grossly inefficient* for your domain. The four canonical cases:

1. **DNA / protein sequences** — the alphabet is tiny (A, C, G, T / 20 amino acids) and BPE tokenizers trained on natural language fragment these sequences into single characters, inflating sequence length 3-5x and destroying the model's ability to see local motifs.
2. **Chemistry SMILES strings** — dense notation with characters like `(`, `)`, `=`, `#` that natural-language tokenizers split into single tokens.
3. **Non-Latin scripts underrepresented in pretraining** — a tokenizer trained mostly on English and Chinese will fragment a low-resource script (e.g., some African or Southeast Asian scripts) badly.
4. **Heavy domain code in an unusual language** — rare esoteric languages or heavy domain-specific DSLs.

In each case, the symptom is the same: *tokens-per-character is abnormally high*, sequence lengths balloon, VRAM multiplies, and the model sees fragmented context. Measure tokens-per-character on a representative domain sample before deciding.

When you do train a new tokenizer, you are committing to continued pretraining (FT12 sidebar territory), not just SFT — because the model's embeddings for the new token merge rules are cold. This is why it is almost never the right move for a steering-only task. Reuse the base.

## The efficient tokenizer adaptation research (2025)

A 2025 line of work (arXiv:2512.03989) offers a middle path: *efficient tokenizer adaptation* rather than training from scratch. The idea is to carefully re-tokenize a domain corpus *respecting the SentencePiece/BPE merge rules already in the base tokenizer*, adding only the high-value domain merges that compress the corpus most. This preserves compatibility with the base model's existing embeddings (the merges are additive) and avoids the cold-start problem of a full retrain. The method is careful: it does not naively retrain BPE (which would invalidate existing merges) but extends the merge table in a controlled way.

For most readers this is research-awareness, not a default workflow. The practical takeaway: if your domain tokenization is bad but not catastrophic, tokenizer adaptation may be cheaper than a full tokenizer retrain plus continued pretraining. Watch this space.

---

# 7.7 — Extending the Vocab With Domain Tokens

*When you must add tokens. And the warm-up that everyone forgets.*

Sometimes you do not need a new tokenizer, but you do need a handful of *domain-specific tokens* — a special `<|citation|>` marker, a `<|tool_result|>` tag, a set of domain control tokens. The workflow:

1. **Add the tokens.** `tokenizer.add_tokens(["<|citation|>", ...])` or `add_special_tokens(...)`.
2. **Resize the model's embedding matrix.** `model.resize_token_embeddings(len(tokenizer))`. This adds new rows for the new tokens.
3. **Resize `lm_head` consistently.** For most architectures, `resize_token_embeddings` handles both the input embeddings and the lm_head (output projection) together. Verify both are resized — a mismatch between embedding rows and lm_head rows is a silent bug that produces NaNs or garbage logits.
4. **Warm up the new embeddings.** This is the step everyone forgets. The new token rows are *randomly initialized*. If you train SFT immediately, the model has to learn the new token embeddings *while also learning the steering task — and the random rows inject noise into the loss*. The fix: warm up the new embeddings first. Either (a) do a short embedding-only training pass on domain text that contains the new tokens, freezing the rest of the model, or (b) initialize the new rows from a meaningful average (e.g., the mean of existing token embeddings) to give them a non-random starting point.
5. **Re-save the tokenizer.** The new tokens must be in the saved tokenizer that ships with the adapter. A model trained with an extended tokenizer but served with the original tokenizer will produce token-id mismatches at inference.

The anti-pattern — extending the vocab without warming up the embeddings — produces a model that trains but behaves erratically on the new tokens. The warm-up is cheap insurance.

---

# 7.8 — The Debugging Decision Tree

*Symptom → bug → fix. The whole module on one page.*

| Symptom | Likely bug | Diagnosis | Fix |
| --- | --- | --- | --- |
| Model won't stop generating; runs to max length | EOS missing or wrong (bug 2) | Decode last tokens of an assistant turn; is the family EOS present? | Use `apply_chat_template` (appends correct EOS); re-tokenize |
| Loss explodes / goes to NaN on Qwen SFT | EOS / template misconfig (bug 2) | Check template family; check EOS token id matches model config | Set correct `chat_template`; verify `tokenizer.eos_token_id == config.eos_token_id` |
| Role boundaries fuzzy; assistant continues user turn | Cross-family template (bug 1) | Decode a tokenized example; do special tokens match the family? | Use the family's native template; never hand-concatenate |
| Model almost works but quality subtly degraded | Packing without attention mask (bug 3) | Check `attention_mask` and loss mask on a packed sequence | Use document/variable-length attention; mask loss on non-assistant tokens |
| New domain tokens behave erratically after training | Vocab extended without warm-up (7.7) | Check if new token embeddings were warmed up | Warm up new rows; re-train |
| Tool calls malformed or truncated | Tool-template edge case (7.5) | Decode tool-call examples incl. assistant mask | Override `chat_template` with corrected version |

When in doubt: **encode, decode, read.** Every silent tokenizer bug is visible if you decode one example and look at it. The discipline is not sophisticated; it is just routinely skipped.

---

## Anti-Patterns

### Hand-concatenation of role strings

Building the prompt by string-formatting `f"User: {x}\nAssistant: {y}"` instead of `apply_chat_template()`. Produces missing special tokens, wrong EOS, mismatched family conventions. The single most common source of bugs 1 and 2. Always use `apply_chat_template`.

### Not decoding to inspect

Running a full training job without ever calling `tokenizer.decode(ids)` on a sample. You cannot diagnose a tokenizer bug from the loss curve alone. The two-minute inspection loop catches the bugs that cost GPU-hours.

### Mixing template families

Applying a Llama-3 template to a Qwen model (or any cross-family application). The role tokens are not interchangeable. The model's persona and format understanding degrades silently. Match the template to the family — `apply_chat_template` does this automatically.

### Extending the vocab without warming up embeddings

Adding domain tokens, resizing embeddings, and immediately training SFT on the full task. The random new rows inject noise. Warm up the new embeddings first (short embedding-only pass or mean initialization).

### Packing without attention masking

Concatenating examples with `attention_mask` all ones, or computing loss on every token. Silently degrades quality. Use document/variable-length attention and mask loss on non-assistant tokens.

---

## Key Terms

| Term | Definition |
| --- | --- |
| **Chat template** | The Jinja2 template in `tokenizer.chat_template` that converts a list of `{role, content}` messages into the model's native token sequence |
| **`apply_chat_template()`** | The tokenizer method that applies the family-correct template; the only correct way to format chat data |
| **Role tokens** | The special tokens a family uses to mark role boundaries (Llama-3 `<|start_header_id|>`; Qwen `<|im_start|>`) |
| **EOS (end-of-sequence) token** | The token signaling the end of generation; must be appended to every assistant turn during SFT |
| **ChatML** | The role-token convention used by Qwen and others: `<|im_start|>role\n...<|im_end|>` |
| **Packing** | Concatenating multiple short examples into one fixed-length sequence for GPU efficiency; requires attention masking |
| **Document / variable-length attention** | FlashAttention's mechanism (via position_ids / cu_seqlens) that prevents examples in a packed sequence from attending across boundaries |
| **Assistant-tokens mask** | A mask over token positions marking only assistant-generated tokens for loss; system/user/tool tokens are masked out |
| **Cross-family template misuse** | Bug 1 — applying a template from one model family to a model of another family |
| **Vocab extension** | Adding new tokens to the tokenizer and resizing embeddings + lm_head; requires warm-up of new rows |
| **Efficient tokenizer adaptation** | (arXiv:2512.03989) Carefully extending the base tokenizer's merge table respecting SentencePiece rules, avoiding a full retrain |

---

## Lab Exercise

See `07-lab-spec.md`. The "Debugging Checklist" lab: three deliberately-broken training scripts (one per top bug), diagnose each from the loss curve / generation output, fix it, confirm the fix. CPU-only, runnable Python.

---

## References

1. **Hugging Face** — *Applying Chat Templates*. The canonical `apply_chat_template` documentation; why hand-concatenation is an anti-pattern.
2. **Qwen Team** — *Qwen Key Concepts / ChatML format*. The ChatML convention and the EOS handling specifics for the Qwen family.
3. **arXiv:2512.03989 (2025)** — *Efficient Tokenizer Adaptation*. Careful re-tokenization respecting SentencePiece merge rules; the middle path between reuse and retrain.
4. **Hugging Face Forums** — *Qwen SFT loss exploding thread*. The recurring public trace of bug 2 (EOS / chat-template mishandling) in the wild.
5. **Module FT04** — *Dataset Formats*. Introduced `apply_chat_template` and the conversation schema; FT07 enforces and debugs it.
6. **Module FT03** — *Base Model Selection*. The base/instruct/chat checkpoint distinction; why tokenizer presence is now independent of checkpoint type.
7. **FlashAttention / TRL** — *Document-level and variable-length attention*. The mechanism that makes correct packing possible.