Every large language model starts with a step nobody draws: converting text into token IDs. That step determines cost, fairness, and whether the model can do arithmetic at all. Here's the full picture — with live animations, not equations.
Byte Pair Encoding starts from individual characters and iteratively merges the most frequent adjacent pair. Step through the classic example below — each click performs one merge operation.
Text: "low lower newest widest" — the textbook BPE example (Sennrich et al. 2016)
Notice how each step collapses one frequent pair into a new token. By step 4, l·o·w is a
single unit. By step 8, low·er is one token because it appeared multiple times in the corpus.
The algorithm never looks ahead — it greedily takes the highest-frequency pair at each step.
That greedy choice is, it turns out, optimal BPE is NP-complete, so greedy is all we have.
BPE and WordPiece build up from characters. Unigram LM prunes down from a large set.
Every major lab expanded their vocabulary in 2024–2025. More tokens = better compression = fewer tokens per sentence = faster inference. The tradeoff: embedding tables grow linearly with vocabulary size. At 4096 embedding dimensions, going from 32K to 256K tokens adds about 1B parameters before any transformer layer.
Bar widths are proportional to vocabulary size. Gemini's 256K = 5× more tokens than BERT.
The phrase "I love India" (or its equivalent) tokenizes into very different numbers of tokens depending on the language. With GPT-4's cl100k_base tokenizer, Arabic text costs 3.3× more per word than English. Every non-English user pays a silent tax on every API call.
Colors represent different tokens. More boxes = more tokens = higher compute cost and API bill.
The 4o upgrade (o200k_base, 200K vocabulary) improved CJK by ~40%. But Arabic, Bengali, and Tamil remain penalized. The vocabulary budget went wherever the training data was most concentrated — which was English.
Fertility = tokens produced ÷ meaningful units in the original text. A perfect tokenizer would score 1.0 for every language. The multilingual BPE tokenizer I built for ERA v5 coursework achieves between 1.258 and 1.291 — a spread of just 0.034 across four scripts.
Lower fertility = better compression. The spread (max − min) = 0.034 determines the score.
Meta's Byte Latent Transformer (ACL 2025) is the first byte-level model to match tokenized LLMs at fixed compute. The key insight: allocate more compute where text is complex (high entropy), and use long, cheap patches where text is predictable.
Bar height = byte-level entropy (how uncertain the model is about the next byte). BLT places patch boundaries at entropy peaks. BPE cuts at fixed vocabulary-learned positions regardless of difficulty.
Result: 8B params / 4T bytes — matches tokenized LLM performance at fixed FLOP cost (Meta AI, ACL 2025)
The number 12345 doesn't have a consistent token representation. Depending on what surrounds it, it could split three different ways. The model never "sees" digits — it sees opaque numeric blobs. There's no place-value structure a neural network could learn arithmetic from.
"Say Anything but This" (arXiv 2601.14658) proved arithmetic failures are systematic, not random — caused by tokenizer inconsistency.
MrT5 (ICLR 2025) and BLT (ACL 2025) are the two credible paths to production byte-level models.
Tokenization is NP-complete. Whittington, Bachmann, and Pimentel proved this at ACL 2025 (arXiv:2412.15210) using a reduction from set cover. Both forms of the problem — choosing which vocabulary to build, and encoding text given a fixed vocabulary — are computationally intractable in general.
This means every tokenizer you've ever used — GPT-4o, LLaMA 3, Gemma — is a greedy approximation to a problem that cannot be solved optimally. There is no formula for the best vocabulary. There is no closed-form for the best corpus weights. Grid search is not a shortcut. It is provably the correct approach.