Tokenizers & Chat Templates
The silent-bugs module — where 80% of silent training failures live. The model trains, loss goes down, but the model learns the wrong thing because the tokens were wrong. The top three bug classes, the apply_chat_template rule, and the vocab-extension workflow.
Tokenizers and chat templates are where most SILENT training failures live: the model trains, loss descends, the checkpoint loads, and the model learned the wrong thing because the tokens were wrong. The bug is not in the optimizer or the data — it is in the last transformation between clean JSONL and the embedding layer, and it is invisible to the loss curve by design.
The top three silent-bug classes are: (1) cross-family template misuse — applying a Llama-3 template to a Qwen model breaks role tokens; Llama-3 uses <|begin_of_text|>/<|start_header_id|>/<|eot_id|>, Qwen uses ChatML <|im_start|>/<|im_end|>, and they are NOT interchangeable. (2) EOS mishandling — missing or wrong EOS during SFT causes run-on generation or loss explosion (the HF forums Qwen loss-exploding thread traces here). (3) Packing-without-attention-mask — concatenating examples without document/variable-length attention and without loss masking on non-assistant tokens is a silent quality regression.
Always use apply_chat_template(); never hand-concatenate role strings. The tokenizer's Jinja2 chat_template knows the family-correct special tokens, role placement, EOS, and tool-call handling. The inspection loop (encode one example, decode it, read it, check the assistant mask) takes two minutes and catches all three bugs before a GPU-hour is spent — and teams that skip it are the ones posting 'my model won't stop generating' on the forums.
Base models now ship with chat templates — the assumption 'base = no template' is outdated. Qwen3 base includes ChatML tokens as real vocab entries with trained embeddings; the base/instruct/chat distinction (FT03) is about weights, not template presence. Train your own tokenizer ONLY for grossly inefficient domain tokenization (DNA, SMILES, non-Latin scripts, esoteric code); for most domains reuse the base. Extending the vocab with domain tokens requires resizing embeddings AND lm_head consistently and WARMING UP the new rows (the step everyone forgets) — otherwise random embeddings inject noise into the loss.