Key Principle
The tokenizer determines what the model can "see." Subword tokenization (BPE) balances vocabulary coverage with compression efficiency, but tokenization choices create direct, often invisible failure modes — from arithmetic errors to cross-lingual cost disparities to representational inequality. "A non-negligible number of LLM failures can be attributed to the way the text was tokenized" (Chapter 3).
Why This Matters
Engineers typically debug model behavior at the prompt or architecture level when the root cause is tokenization. These failures are invisible at the prototype stage but surface in production on edge-case inputs — another instance of the prototype-to-production gap. If you do not understand BPE tokenization, you cannot predict why your model fails on domain-specific terminology (Ch. 3).
Good Examples
- BPE mechanism: Starts with individual characters and iteratively merges the most frequent adjacent pairs until reaching target vocabulary size. Byte-level BPE (GPT family) starts with 256 byte tokens, guaranteeing every Unicode character can be encoded (Chapter 3).
- Domain adaptation: Domain-specific terms (medical, legal, financial) are split into semantically meaningless fragments with general-purpose tokenizers, degrading both cost efficiency and downstream performance. The fix: add domain tokens and continue pre-training (Chapter 3).
- Larger models are more robust to misspellings because misspellings already appear in training data (e.g., "insufficent" occurs 1,100+ times in C4). Robustness comes from data exposure, not architectural sophistication (Chapter 3).
Counterpoints
- Arithmetic failures: Numbers like 934 have no dedicated token in GPT-NeoX, partially explaining poor arithmetic. The model cannot reason about numbers it cannot represent as coherent units (Chapter 3).
- Representational inequality: Popular names/places (Boston, Ahmed, Donald) get their own tokens while others (Mesa, Chennai, Suhas, Maryam) do not, creating unequal representation in the model's vocabulary (Chapter 3).
- Glitch tokens: "SolidMagiGoldkarp" — a Reddit username that became a GPT-2 vocabulary token — causes anomalous behavior when the tokenizer is reused with a different pre-training corpus. These "undertrained tokens" have vocabulary entries but minimal training signal (Chapter 3).
- Cross-lingual inequality: Fertility (tokens/words) and parity (token ratio between languages) reveal non-English languages require several times more tokens, creating cost and performance disparities from English-dominated tokenizer training corpora (Chapter 3).
Key Quotes
"For a given task, if you observe strange behavior from LLMs on only a subset of your inputs, it is worthwhile to check how they have been tokenized... a non-negligible number of LLM failures can be attributed to the way the text was tokenized." — Suhas Pai, Chapter 3
Rules of Thumb
- When debugging unexpected model behavior on specific inputs, check tokenization first
- For domain-specific applications, evaluate tokenizer fertility on your domain text
- Budget for higher token counts in non-English languages (use parity metrics)
- A single typo causes completely different tokenization — consider this for user-facing input
- Five new words appear in the New York Times daily for the first time — subword tokenization exists for this reason
Related References
- Pre-Training Data: The Most Important Ingredient - Data composition determines tokenizer quality
- Selecting and Evaluating LLMs - Tokenizer choice as a model selection criterion
- Embeddings, Document Parsing, and Semantic Search - Tokenization affects embedding quality