Tokenization is the part of an LLM nobody talks about — until they realize it's why Arabic API calls cost 3× more than English ones, why GPT-4 fails at multi-digit arithmetic, and why LLaMA 3 needed a billion extra parameters before a single transformer layer. This is a research review of where the field is now, what every major lab changed, and what's coming next.
After building a multilingual BPE tokenizer from scratch for ERA v5 coursework — achieving a compression spread of 0.034 across English, Hindi, Telugu, and Bengali — I went down the rabbit hole of what the rest of the field is actually doing. What I found was a field quietly in the middle of a major architectural rethink.
Every major lab doubled or quadrupled their vocabulary in the last 18 months. Meta rewrote their tokenizer entirely between LLaMA 2 and LLaMA 3. Google committed to 256,000 tokens. Researchers at ACL 2025 proved that optimal tokenization is NP-complete. And Meta published a byte-level model that matches tokenized LLMs at fixed compute cost — the first credible result that suggests subword tokenization may not be a permanent fixture of the architecture.
Here's what the landscape looks like.
BPE (Byte Pair Encoding) is the workhorse. Sennrich et al. adapted it from data compression in 2016. Start with individual characters. Find the most frequent adjacent pair across your training corpus. Merge it into a new token. Repeat until you hit your vocabulary size. Used by GPT-4o, LLaMA 3, Qwen, DeepSeek, and Mistral.
WordPiece modifies the merge criterion: instead of merging the most frequent pair, it merges
the pair with the highest pointwise mutual information — co-occurrence relative to chance. score(A,B) = freq(AB) / (freq(A) × freq(B)).
This tends to produce more linguistically coherent subwords. Used by BERT, DistilBERT, and older Google models.
Largely superseded in new work by larger-vocabulary BPE.
Unigram Language Model runs the algorithm in reverse: start with a huge over-complete vocabulary, then iteratively prune the token whose removal hurts corpus likelihood the least. Keep pruning until you hit your target size. This has a proper objective function (maximizing log-likelihood), supports probabilistic tokenization for training-time data augmentation, and is language-agnostic by construction. Used by Google's T5, Gemma, Gemini, PaLM, and LLaMA 1/2.
SentencePiece is a framework, not an algorithm — it can run BPE or Unigram internally. Its defining feature: it treats whitespace as a regular character (using ▁ as a marker), so it never needs a pre-tokenization step that splits on spaces. This makes it genuinely language-agnostic by default — crucial for languages like Chinese, Japanese, and Thai that have no word boundaries.
A larger vocabulary is a direct compression bet. With 32K tokens, common 5-character Chinese phrases split into 3–4 tokens because the budget ran out of Chinese entries. With 256K tokens, those phrases get dedicated slots. Fewer tokens per sentence = shorter sequences = less attention compute = more effective context window.
The tradeoff is real. Embedding tables scale as vocab_size × embedding_dim. At 4096 dimensions,
a 256K vocabulary requires 1.05B parameters just for embeddings — roughly 15% of a 7B model's parameters
dedicated to the lookup table before any computation happens.
The LLaMA 2→3 jump from 7B to 8B parameters is almost entirely explained by the vocabulary expansion from 32K to 128K.
Google's bet with Gemma/Gemini is the largest: 256K tokens shared across the entire model family. The Gemma 3 Technical Report (March 2025) confirms this design was specifically chosen to reduce over-tokenization of non-English text across 100+ supported languages.
The most impactful change: GPT-4o's tokenizer (May 2024) halved the token count for CJK text compared to GPT-4. The 200K vocabulary gives Chinese, Japanese, and Korean characters enough budget that common multi-character phrases become single tokens rather than character-by-character sequences. Independent measurement confirmed approximately 40% fewer tokens for CJK text with o200k_base.
The pre-tokenization regex also changed — the pattern that splits raw text before BPE merges begin now handles runs of spaces, digit sequences, and mixed-script text more gracefully. This matters especially for code (indentation, numeric literals) and multilingual documents.
The most dramatic change. LLaMA 2 used SentencePiece with a 32K vocabulary. LLaMA 3 switched to tiktoken-style BPE with 128,256 tokens — a 4× jump — and produced approximately 15% fewer tokens for the same text. The switch also broke the SentencePiece API that many downstream pipelines had been built around, requiring migrations across the ecosystem. Meta locked the LLaMA 3 tokenizer unchanged through the 3.1, 3.2, and 3.3 series.
Two decisions stand out. First: 256K vocabulary, the largest in production. Second: split-digit tokenization —
every number is tokenized as individual digits. 123 always becomes [1, 2, 3], never
[12, 3] or [123]. This is a direct architectural fix for the arithmetic failure problem
(more on this below). No other major production tokenizer does this consistently.
Anthropic has never published tokenizer details. No technical report, no vocabulary size, no algorithm disclosure. Empirical measurement via the API can infer token counts but not the underlying design. This is unusual — every other major lab has at least a technical report section on their tokenizer.
Qwen (Alibaba) takes the most pragmatic approach: start with OpenAI's cl100k_base vocabulary (100K tokens), then augment it with the most frequent Chinese characters and words that were underrepresented. Final size: 151,936 tokens. The cl100k_base already handles English, code, and general multilingual patterns well. By extending rather than replacing, Qwen preserves compatibility with tiktoken tooling while specifically targeting the Chinese coverage gap. Qwen 2.5 claims support for 29+ languages with this approach.
DeepSeek tracked different use cases with different vocabularies:
Chinese research labs are also pushing on something more fundamental: sub-character tokenization for Chinese text. Chinese characters are not atomic — they decompose into radicals (semantic/phonetic components). The character 語 (language) shares the radical 言 (speech) with 話 (speech/words) and dozens of other characters. A tokenizer that understands radicals understands something about semantic structure that BPE, merging bytes blindly, completely misses. Work from Chinese NLP labs (TACL 2024, MDPI 2025) shows that incorporating radical information improves representation quality, though the approach requires structured knowledge about character decomposition that makes it harder to apply beyond Chinese.
The motivation for going byte-level is compelling: subword tokenizers bake in English-centric assumptions, create arbitrary vocabulary boundaries that hurt morphologically rich languages, and produce inconsistent tokenizations for the same semantic content depending on surrounding context. Bytes are universal — every language, code file, and data format is just a byte sequence.
The challenge has always been sequence length. Byte sequences are 4–6× longer than token sequences for the same text. Standard Transformer attention scales as O(n²) with sequence length. A byte-level model processing English text needs 4–6× longer sequences, which means 16–36× more compute for attention. This killed early byte-level work.
Three architectures have now credibly solved this at different scales:
MrT5 augments ByT5's encoder with a learned "delete gate." After several encoder layers, the model decides which tokens carry redundant information and removes them — progressively compressing the byte sequence through the encoder stack. By the final layers, the sequence is at 20% of its original length: byte-level input, token-level processing cost. The deletion pattern emerges from training without supervision and is multilingual — different languages are compressed at different rates based on their orthographic structure. Up to 80% sequence length reduction while matching ByT5 accuracy.
BLT is the most important result. Instead of learning to delete, it uses entropy-based dynamic patching: a small local model estimates the entropy of the next byte at every position. Patch boundaries are placed where entropy is high (the model is uncertain — genuinely complex text). Easy text (predictable English words, repetitive boilerplate, common code) gets long patches and little compute. Hard text (rare words, non-Latin scripts, complex code logic) gets short patches and more attention.
The fixed-FLOP comparison is key. Previous byte-level work showed "byte models can match tokenized models" without controlling for compute — of course they can match if you give them more compute. BLT controls for this and still matches. It also shows qualitative improvements: typos, OCR errors, unusual encodings, and rare Unicode characters all degrade gracefully because there's no vocabulary to violate.
▁import▁ (with space), ▁def▁,
and named entities like ▁New▁York to become single tokens — a 15% improvement in bytes-per-token.
Cost: training takes 4.7 CPU-days on 1GB of text vs. 59 seconds for standard BPE. Research result, not yet production.
The "token tax" has been formalized in a growing body of research. With GPT-4's cl100k_base tokenizer:
This has a second effect beyond cost: the effective context window. A 128K-token context window holds roughly 30,000 English words but only 8,000–10,000 Arabic words, because each Arabic word consumes 3–4 tokens instead of one. Non-English users are paying more per query and getting a fraction of the context window they're billed for.
GPT-4o's o200k_base reduced the CJK tax by approximately 40%. But Arabic, Bengali, Tamil, and most African languages remain substantially penalized even with larger vocabularies, because the training data used to construct those vocabularies is still English-dominated. The vocabulary budget gets allocated to where the training data is most concentrated.
Recent research (arXiv 2025–2026) has proposed algorithmic fixes:
With GPT-4o's o200k_base tokenizer, the number 12345 might tokenize as [123, 45]
or [12, 345] or [1, 2345] depending on context. There is no consistent relationship
between token IDs and numeric values. The single-digit '2' is token 17.
The two-digit '20' is token 455. '200' is token 1179. '201' is token 667.
No pattern. No structure. No way for a model to learn place-value arithmetic from these representations.
A 2025 paper ("Tokenization Constraints in LLMs," arXiv 2505.14178) formally confirmed that arithmetic failure rates scale with operand size and are directly attributable to tokenization artifacts, not to reasoning capability. The model is not bad at math — it literally cannot see the digits.
Production workarounds range from clean (Google's split-digit tokenizer in Gemma) to messy (pre-processing all numeric inputs to add spaces between digits, routing arithmetic to Python interpreters, using chain-of-thought to reconstruct digit structure from patterns). The fact that "route all arithmetic to a Python interpreter" is now standard engineering advice is a remarkable admission that the tokenizer is broken for an entire category of tasks.
Code tokenization has its own set of issues. Python indentation (4 spaces as semantic syntax) may collapse to a
single token, erasing nesting information. getHTTPResponseCode splits at arbitrary capitalization
boundaries rather than at conceptual word boundaries. Common code patterns like import , -> ,
and == cross word boundaries and get split when they should be single tokens.
A 2024 paper ("Fishing for Magikarp") documented vocabulary entries that almost never appear in training data —
"undertrained tokens" whose embeddings are essentially random. These cause erratic model behavior when they
appear in prompts.
Agglutinative languages — Turkish, Finnish, Hungarian, Korean, Swahili — attach suffix chains to root words to encode tense, case, number, possession, and more. The Turkish word çekoslavakyalılaştıramadıklarımızdanmışsınızcasına is a grammatically valid single word meaning roughly "as if you were one of those we could not make Czechoslovakian." BPE sees a unique byte sequence and merges it at whatever character boundaries happened to be most frequent in training. The resulting tokens have no morphological coherence.
Active research in this space (2024–2025) includes:
The pattern across all this work: incorporating linguistic structure into tokenization improves performance on morphologically rich languages. The challenge is that building the linguistic resources (morpheme analyzers, morphologically annotated corpora) requires domain expertise that is expensive to produce for low-resource languages — exactly the languages that most need better tokenization.
"Tokenisation is NP-Complete" (arXiv 2412.15210) is the cleanest theoretical result about tokenization in years. It formalizes what practitioners had long suspected: you cannot compute the optimal vocabulary analytically. Both forms of the optimization — (1) which vocabulary to construct and (2) how to encode text given a fixed vocabulary — are computationally intractable in the worst case.
This explains a lot. It explains why every BPE implementation is greedy (merge the most frequent pair right now, don't look ahead). It explains why the optimal corpus weights for multilingual training cannot be computed from a formula — they're found by grid search. It explains why two tokenizers trained on similar data with similar algorithms produce different vocabularies, and why you can always find a better tokenizer by running more experiments but never prove you've found the best one.
The reduction used to prove NP-completeness is from set cover — a classical combinatorial optimization problem. The proof establishes that designing the optimal vocabulary is at least as hard as set cover, which is NP-complete. This means unless P = NP, there is no polynomial-time algorithm for optimal tokenization. Every tokenizer in production, from GPT-4o to LLaMA 3 to Gemma, is an approximation to an intractable problem.
Beyond multilingual equity, three domain-specific tokenization problems come up repeatedly in production deployments.
KL3M (arXiv 2503.17247, March 2025) is the most significant domain-specific tokenizer paper of the year.
A family of BPE tokenizers trained specifically on legal, financial, and governmental text from US federal sources.
Despite a vocabulary 35% smaller than GPT-4o, KL3M achieves 9% fewer tokens overall for domain documents —
and for specific legal terminology: up to 83% fewer tokens than LLaMA 3, because legal terms like
"notwithstanding" and "indemnification" that appear thousands of times in legal text get dedicated tokens
rather than being split into 6+ subwords.
All tokenizers are open-source on HuggingFace under alea-institute/.
A February 2026 paper from Peking University (arXiv 2602.04706) found that approximately 10% of tokens in major vocabularies — including o200k_base, Qwen, DeepSeek, LLaMA 3, Gemma 3, and BLOOM — are "intermediate merge residues": tokens that are frequent during vocabulary construction but almost never emitted at inference because they are always further merged in practice. These dead entries waste embedding space and introduce adversarial vulnerabilities (unusual inputs that force the model through undertrained token embeddings). Removing them reduces vocabulary size without performance loss.
"Getting the Most Out of Your Tokenizer for Pre-training and Domain Adaptation" (ICML 2024) showed how to extend existing tokenizers to new domains without full retraining — adding medical, legal, or scientific terminology to a general-purpose vocabulary while preserving compatibility with a pretrained model's embedding table. AdaptBPE (arXiv 2601.21665) formalizes this into a reusable framework.
The field is moving in two directions simultaneously, and they may eventually converge.
Short term: vocabulary expansion will continue. 256K is probably not the ceiling. The embedding table overhead is manageable, and the compression benefits for multilingual models are substantial. Parity-aware training objectives that explicitly penalize cross-language fertility gaps will likely be adopted by labs that care about non-English markets.
Medium term: byte-level models will enter production for specialized use cases first — probably code (where tokenization artifacts cause the most measurable harm) and multilingual applications (where the token tax is most expensive). The engineering for efficient byte-level inference is still being worked out, but BLT's result makes the direction clear.
The open question: can a byte-level model trained at frontier scale outperform a tokenized model trained on the same compute budget? BLT showed parity at 8B parameters. Nobody has tried at 70B or 400B. The answer to that question will determine whether we're looking at a gradual transition or a hard architectural break.