How Google Fixed Arithmetic: Inside Gemini's 256K Unigram Tokenizer

When I built a multilingual BPE tokenizer from scratch for ERA v5 coursework — a 10,000-token vocabulary covering English, Hindi, Telugu, and Bengali — the hardest design choices were about what to optimize. Fertility across scripts? Compression ratio? Vocabulary coverage? Every choice trades off against another.

Google faced the same tradeoffs at a completely different scale, with a model family covering 100+ languages and applications ranging from code generation to multilingual chat to mathematical reasoning. The Gemini tokenizer is an existence proof that you can push vocabulary size to 256K without breaking the model — if you're willing to pay the parameter cost and think carefully about what the vocabulary represents.

The root cause of the token tax — Arabic paying 3.3× more than English, Bengali ~4.5× more — is vocabulary budget exhaustion. When GPT-4's cl100k_base tokenizer was trained, English and code dominated the corpus. The BPE merge process allocated most of its 100K vocabulary slots to English words, subwords, and code patterns. Arabic characters, Bengali akshara, Telugu conjuncts — these got whatever budget was left after English filled up. Not enough for common words to become single tokens. They fragment into characters.

The solution is more vocabulary. With 256K tokens, you can dedicate tens of thousands of slots to non-Latin scripts and still have room for a comprehensive English and code vocabulary. This is what the Gemma/Gemini tokenizer does: the 256K vocabulary covers 100+ languages with enough slots that common words in all of them can be single tokens.

The Gemma 3 Technical Report (March 2025) is explicit: the 256K vocabulary was specifically designed to reduce over-tokenization of non-English text. Fertility measurements across their evaluation languages showed substantially more uniform token rates than smaller-vocabulary baselines.

The tradeoff is embedding table size. Vocabulary is not free.

At 256K vocabulary and 4096 hidden dimensions, the embedding table alone requires 1.05 billion parameters — before the first transformer layer runs. That's roughly 15% of a 7B model's total parameter budget, or 1.5% of a 70B model's budget. Google's bet is that the multilingual compression benefit — more content per context window, better language parity, reduced API cost for non-English users — is worth that upfront cost.

BPE and Unigram LM produce superficially similar outputs — both give you a subword vocabulary with frequencies that correlate with corpus statistics. But they work oppositely: BPE starts small and grows through merges, while Unigram starts enormous and shrinks through pruning.

The key property this gives Gemini's tokenizer: principled pruning for multilingual vocabularies. With BPE at 256K, the merge process's greedy frequency criterion can over-allocate to high-frequency English patterns at the expense of rare-but-important morphological units in other languages. Unigram LM's log-likelihood objective naturally keeps vocabulary items that are essential for encoding the training corpus, regardless of absolute frequency — a rare morpheme in Telugu that appears in many root forms gets to stay because removing it would hurt Telugu corpus likelihood even if it barely affects English likelihood.

The other advantage: Unigram supports probabilistic tokenization. Instead of one deterministic segmentation per sequence, you can sample from the distribution of valid segmentations. This was used as a data augmentation technique in T5 and PaLM — training on multiple tokenizations of the same text makes the model less sensitive to tokenization boundaries. Google's models have used this across their entire family.

GPT-4 fails at multi-digit arithmetic. Not because the model is bad at math — it's because the tokenizer destroys the structure of numbers before the model ever sees them.

In standard BPE, frequently co-occurring digit sequences get merged. "12" might be a single token. "345" might be another. So "12345" tokenizes as ["12", "345"] — and the tens place of the "12" block is spatially adjacent to the hundreds place of the "345" block in token space, but they represent different place values. The model has no consistent positional structure to learn from. Each number has an essentially arbitrary tokenization that depends on which merges happened during vocabulary training.

With split-digit tokenization, a model can learn that "the token at position N+2 from the start of a number represents the hundreds place" — because that relationship is now stable across all numbers. Addition, subtraction, carry propagation — these become learnable patterns rather than arithmetic performed on an inconsistent symbol soup.

The cost is sequence length. "127849" is 6 tokens with split-digit tokenization vs 2–3 tokens with BPE merges. Documents with lots of numbers — financial data, scientific texts, code — get longer token sequences in Gemini than they would in GPT-4. Google judged this a worthwhile tradeoff, and the evidence from Gemini's performance on mathematical benchmarks supports that judgment.

Most BPE implementations (including OpenAI's tiktoken and Meta's LLaMA 3 tokenizer) use a pre-tokenization step: a regex splits raw text into candidate words before BPE merges run. This is where language-specific assumptions creep in. A space-splitting regex is wrong for Chinese. A punctuation splitter is wrong for Thai. Even a carefully written Unicode-aware regex bakes in assumptions about what constitutes a "word."

SentencePiece avoids this entirely. Its key innovation: treat whitespace as a regular Unicode character (represented as ▁) and run tokenization on the raw text stream without any pre-splitting. This makes it genuinely language-agnostic — the tokenizer sees Japanese text the same way it sees English, because it doesn't try to identify word boundaries before tokenizing.

import sentencepiece as spm

# Training a SentencePiece Unigram model (simplified)
spm.SentencePieceTrainer.train(
    input='multilingual_corpus.txt',
    model_prefix='gemini_tok',
    vocab_size=256000,
    model_type='unigram',          # Unigram LM, not BPE
    character_coverage=0.9999,     # cover 99.99% of Unicode chars
    split_digits=True,             # Gemini's arithmetic fix
    byte_fallback=True,            # unknown bytes → byte tokens
    normalization_rule_name='nfkc' # Unicode normalization
)

Two parameters in that training call are particularly important for Gemini's design:

split_digits=True — this is the arithmetic fix. It instructs SentencePiece to add special handling that prevents digit sequences from merging. Each digit gets its own token, regardless of what the Unigram LM objective would otherwise do.

byte_fallback=True — for any byte sequence that cannot be represented by the learned vocabulary, fall back to individual byte tokens (256 possible byte tokens are always in the vocabulary). This makes the tokenizer complete: no input can ever produce an unknown token. Previously, encountering an out-of-vocabulary character would produce a special [UNK] token, losing information. With byte fallback, the model always has access to the raw bytes.

This is a significant operational commitment. Meta locked their LLaMA 3 tokenizer unchanged across the 3.1, 3.2, and 3.3 series. Google has done the same for their 256K vocabulary across a broader family. The reason is compatibility: downstream fine-tunes, embeddings, RLHF pipelines, and retrieval systems that use the tokenizer's indices need to remain stable. Every tokenizer change breaks everything upstream.

The 256K vocabulary choice was partly motivated by wanting a design that wouldn't need to change. A larger vocabulary front-loads the compression problem — spending more parameters on embeddings now avoids having to retokenize everything when you inevitably need better multilingual coverage later.

This is the same reasoning behind why LLaMA 2's 32K vocabulary was a pain point: when Meta built LLaMA 3, they had to change the tokenizer to fix multilingual coverage, and that change — 32K to 128K — added roughly a billion parameters and broke compatibility with everything trained on LLaMA 2.

What's interesting when you look at Gemini's tokenizer as a whole is that every unusual decision is a deliberate fix for a known failure mode. 256K vocabulary: fix the token tax for non-English languages. Unigram LM: principled multilingual coverage, probabilistic augmentation during training. Split-digit: fix arithmetic. Byte fallback: fix unknown tokens. SentencePiece without pre-tokenization: fix language-specific assumptions.

Compare this to the "extend cl100k_base" approach used by Qwen — which preserves English/code tokenization quality at the cost of adding CJK vocabulary on top of a system that wasn't designed for it — or the "switch to tiktoken BPE" approach used by LLaMA 3, which achieved 4× vocabulary expansion while accepting a different set of limitations on multilingual coverage.

None of these approaches is strictly better. Gemini's tokenizer trades parameter efficiency for breadth. LLaMA 3's trades breadth for speed (tiktoken is significantly faster than SentencePiece). Qwen's trades architectural elegance for backwards compatibility. The right choice depends entirely on what you're optimizing for — and that's before you consider whether byte-level models like BLT make the entire vocabulary question obsolete at the next scale.