# Teaching Script — Module FT07: Tokenizers & Chat Templates

**Course**: Course 3 — LLM Fine-Tuning Masterclass
**Module**: FT07 — Tokenizers & Chat Templates
**Duration**: ~40 minutes (spoken at ~140 wpm)
**Format**: Verbatim transcript with `[SLIDE N]` cues. Read aloud or use as speaker notes.

---

[SLIDE 1 — Title]

Welcome to module FT seven, Tokenizers and Chat Templates. This is the final module of Pillar One, Data. You have built the steering wheel across FT zero-four through FT zero-six — the formats, the synthetic generation, the dedup and decontamination. The data is clean. And now I am going to tell you that clean data is not enough. Because between your clean JSONL and the model's embedding layer, there is one last transformation — the tokenizer and the chat template — and that is where most silent training failures live.

This is the silent-bugs module. Memorize the thesis: the model trains, the loss goes down, the checkpoint loads, and the model learned the wrong thing — because the tokens were wrong. A silent failure is not a crash. The training loop runs. TensorBoard looks healthy. You ship the adapter. And the model produces malformed output, ignores role boundaries, never stops generating, or quietly degraded on the capability you cared about. The bug is not in the optimizer. It is in the tokenizer.

[SLIDE 2 — The thesis]

Here is why this module exists as the capstone of Pillar One. The FT zero-zero thesis is that fine-tuning steers behavior — it does not teach knowledge. Steering is only as good as the steering wheel, and the steering wheel is your tokenized data. A brilliant algorithm on mis-tokenized data steers you precisely in the wrong direction. The optimizer faithfully minimizes loss on the wrong token stream.

So the data pipeline is not complete when the JSONL is clean. It is complete when the tokens are correct. FT seven is where that last step lives. And the field has converged on three bug classes that account for the overwhelming majority of silent failures here. We will spend the rest of the module on those three.

[SLIDE 3 — The top three silent bugs]

The debugging checklist. Three bugs. Memorize the symptoms.

Bug one: cross-family template misuse. Every modern chat model has its own role-token convention. Llama-three uses begin-of-text, then start-header-id and end-header-id to bracket the role name, then eot-id to end the turn. Qwen uses ChatML: im-start, role name, newline, content, im-end. These are not interchangeable. They are not close enough. A Qwen model trained on Llama-three tokens sees unfamiliar header tokens where it expects im-start, and its persona and format understanding degrades. The symptom: role boundaries fuzzy, the assistant continues the user's turn, format drifts at inference. The diagnosis is simple — decode a tokenized example and read it.

Bug two: EOS mishandling. The end-of-sequence token tells the model when to stop. During SFT, every assistant turn must end with the EOS token the model was trained on. Three ways this breaks. Missing — you stripped it or the template dropped it. Wrong — you used the generic slash-s on a model whose EOS is im-end or end-of-text. In the wrong place — appended once per packed sequence instead of per turn. The symptom is the most-discussed one in the field: the model won't stop generating. It runs to max length every time. The Hugging Face forums thread where Qwen SFT loss explodes to NaN repeatedly traces back to this. The fix is almost always use apply-chat-template and verify the EOS token id matches the model config.

Bug three: packing without attention masking. To use the GPU efficiently, you pack multiple short examples into one fixed-length sequence. This is correct and standard — if and only if you do two things. First, mask cross-boundary attention. Example B must not attend to example A's tokens. Modern solution: FlashAttention's document-level or variable-length attention via the position-ids or cu-seqlens mechanism. Second, mask the loss on non-assistant tokens. The model should only learn to predict assistant tokens — the system prompt and user turns are context, not targets. Bug three is the quietest. The model trains fine. The loss looks great. The output is plausible. You only notice when you compare against a correctly-packed baseline. This is the silent quality regression the checklist exists to catch.

[SLIDE 4 — apply_chat_template, always]

The single rule that prevents about eighty percent of these bugs. Never hand-concatenate role strings. Always use the model's apply-chat-template.

FT zero-four introduced this rule; FT seven enforces it. Every modern tokenizer carries a Jinja-two chat template in tokenizer dot chat-template. The template knows the correct special tokens for the family, where to put the system prompt, the user turns, the assistant turns, which EOS to append and where, and how to handle tool calls. Calling apply-chat-template produces the correct token sequence for that specific model. Hand-concatenating does not.

The hand-concatenated string may look identical when you eyeball it. It will be missing the special tokens, or use the wrong ones, or place them wrong. That is bug one or bug two waiting to happen.

The discipline that makes apply-chat-template safe is the inspection loop. Take one conversation. Call apply-chat-template with tokenize true and return-assistant-tokens-mask true. Decode the ids. Read them. Confirm the role tokens match the family, the system prompt is present, EOS is appended. Inspect the assistant-only mask. Confirm only assistant tokens are unmasked. Then launch the full run. This takes two minutes. It catches bugs one, two, and three before they cost you a GPU-hour. The number of teams that skip this step and ship a mis-tokenized model is the reason this module exists.

[SLIDE 5 — Base models now ship with templates]

A historical assumption, now dangerous. The assumption was: base models have no chat template, instruct and chat models do. This was largely true through twenty-twenty-three. It is no longer reliable.

Modern base checkpoints increasingly ship with chat templates and the corresponding special tokens already in the tokenizer vocab. Qwen-three base, for example, includes the ChatML special tokens — im-start and im-end — and a default chat template, even on the base checkpoint. The reason is operational: the tokenizer is shared across the base and instruct variants of the same release, so the special tokens are present from the start.

Two implications. First, do not assume base equals no template. Always check tokenizer dot chat-template. Second, the special tokens are real vocabulary entries in the base tokenizer with trained embeddings. You are not adding them when you format SFT data — you are using embeddings the model already has. This is why base-to-SFT formatting works cleanly on Qwen-three: the scaffolding tokens are not cold-started. The base-instruct-chat distinction from FT zero-three is about weights, not about template presence.

[SLIDE 6 — Tool-calling templates]

When your SFT data includes tool calls, the chat template must handle four roles: system, user, assistant, and tool. Each family has its own convention. Qwen uses a tool-call block inside the assistant turn. Llama three-point-one uses a specific JSON schema. Hermes uses its own tag with its own escaping.

The default apply-chat-template handles the common case. But the common case is not every case. Known issues. The default Qwen template can mishandle multi-turn tool-call scenarios where the tool response itself contains text that looks like a special token — the model truncates or mis-parses. Some templates do not correctly append EOS after a tool-call turn when the call is the final turn. And the assistant-tokens mask can be wrong on tool-call turns — the tool response is not assistant-generated and should be masked, but naive templates sometimes unmask it.

The rule: if your data includes tool calls, decode several examples including the assistant mask and verify the boundaries are correct. If the default template is wrong for your case, set a corrected chat-template explicitly. Never fall back to hand-concatenation. Tool-calling is the area where apply-chat-template is most likely to need your intervention.

[SLIDE 7 — Train your own tokenizer or reuse the base]

Training a new tokenizer is one of the most over-prescribed interventions in fine-tuning. The default, for the overwhelming majority of domains, is to reuse the base tokenizer. The base tokenizer was trained on a large, diverse corpus and handles natural language, code, and common scripts efficiently.

You train your own only when the base tokenizer is grossly inefficient for your domain. Four canonical cases. DNA and protein sequences — the alphabet is tiny and natural-language BPE fragments these into single characters, inflating sequence length three-to-five-x and destroying local motifs. Chemistry SMILES strings — dense notation that tokenizers split into single tokens. Non-Latin scripts underrepresented in pretraining — a tokenizer trained mostly on English and Chinese fragments a low-resource script badly. And heavy domain code in an unusual language or esoteric DSL.

The symptom in every case is the same: tokens-per-character is abnormally high. Measure it before deciding. And note: when you train a new tokenizer you are committing to continued pretraining, not just SFT, because the model's embeddings for the new merge rules are cold. This is why it is almost never the right move for a steering-only task. Reuse the base.

There is a twenty-twenty-five middle path — efficient tokenizer adaptation, arXiv twenty-five-twelve-zero-three-nine-eight-nine. The idea is to carefully re-tokenize a domain corpus respecting the existing SentencePiece merge rules, adding only the high-value domain merges that compress the corpus most. This preserves compatibility with the base embeddings — the merges are additive — and avoids the cold-start problem of a full retrain. For most of you this is research-awareness, not a default workflow. But if your domain tokenization is bad but not catastrophic, watch this space.

[SLIDE 8 — Extending the vocab]

Sometimes you do not need a new tokenizer, but you do need a handful of domain-specific tokens — a citation marker, a tool-result tag, a set of control tokens. The workflow has five steps.

One: add the tokens with tokenizer dot add-tokens. Two: resize the model's embedding matrix with model dot resize-token-embeddings of the new tokenizer length. This adds new rows for the new tokens. Three: resize the lm-head consistently. For most architectures, resize-token-embeddings handles both the input embeddings and the lm-head together — verify both. A mismatch produces NaNs or garbage logits. Four — and this is the step everyone forgets — warm up the new embeddings. The new rows are randomly initialized. If you train SFT immediately, the model has to learn the new token embeddings while also learning the steering task, and the random rows inject noise into the loss. Warm them up: either a short embedding-only training pass on domain text containing the new tokens, freezing the rest of the model, or initialize the new rows from a meaningful average of existing embeddings. Five: re-save the tokenizer. The new tokens must be in the saved tokenizer that ships with the adapter, or you get token-id mismatches at inference.

The anti-pattern — extending the vocab without warming up the embeddings — produces a model that trains but behaves erratically on the new tokens. The warm-up is cheap insurance.

[SLIDE 9 — The debugging decision tree]

Symptom to bug to fix, on one diagram. If the model won't stop generating, runs to max length — bug two, EOS missing or wrong. Use apply-chat-template. If the loss explodes or goes to NaN on Qwen SFT — bug two, template or EOS misconfig. Verify tokenizer dot eos-token-id equals config dot eos-token-id. If role boundaries are fuzzy and the assistant continues the user turn — bug one, cross-family template. Use the family's native template. If the model trains fine but quality is subtly degraded versus a baseline — bug three, packing without attention mask. Use document or variable-length attention and mask loss on non-assistant tokens. If new domain tokens behave erratically — vocab extended without warm-up. Warm up the new embeddings. If tool calls are malformed or truncated — a tool-template edge case. Override chat-template with a corrected version.

When in doubt: encode, decode, read. Every silent tokenizer bug is visible if you decode one example and look at it. The discipline is not sophisticated. It is just routinely skipped.

[SLIDE 10 — The lab]

The lab is called the Debugging Checklist. Three deliberately-broken training scripts — one per top bug. A cross-family template applied to the wrong tokenizer. An EOS-stripping pipeline that trains a model to never stop. A packing routine with a single all-ones attention mask and loss on every token. Your job: for each, diagnose the bug from the decoded token output, write the fix using apply-chat-template correctly, and confirm the fix by re-inspecting.

The lab is CPU-only and deliberately training-loop-free. The point is that the bugs are diagnosable before you launch the run. The inspection loop is the skill. You get a helper script that does encode-decode-read with the assistant mask. You calibrate on a known-good tokenizer first, then break and fix each of the three cases. By the end, the inspection discipline is muscle memory.

[SLIDE 11 — What you can now do]

You can now state the module thesis and defend it with the three bug classes. You can diagnose cross-family template misuse, EOS mishandling, and packing without attention masking from their symptoms. You can explain why apply-chat-template is mandatory and distinguish the role-token conventions of Llama-three and Qwen. You can decide whether to train a new tokenizer, reuse the base, or extend the vocab — and execute the vocab-extension workflow with the warm-up. And you have internalized the inspection loop: encode, decode, read, before every run.

That closes Pillar One, Data. Next is Pillar Two, PEFT. Module FT zero-eight, LoRA and QLoRA. You have built the steering wheel and verified the tokens. Now we build the adapter — the lightweight, swappable steer that sits on top of the base. Because a brilliant algorithm on correctly-tokenized data still needs a cheap, swappable way to steer. Let's build the adapter.

---

*End of module FT07. Duration: approximately forty minutes at one-hundred-forty words per minute.*