Building a Multilingual BPE Tokenizer That Treats Four Scripts Fairly

Every LLM has a tokenizer, and almost every tokenizer has a bias. Feed it a sentence in Hindi and it produces 2–4× as many tokens as the English equivalent — which means 2–4× the compute, 2–4× the cost, and a model that's learned to think of Hindi as somehow more "complex" than it actually is.

The ERA v5 Session 2 assignment forces you to confront this directly: train a single BPE vocabulary of exactly 10 000 tokens over the Wikipedia "India" article in four languages, then measure how evenly it compresses each one. The scoring metric is:

The reference solution achieves 6 503 with a spread of 0.154. Ours hits 28 280 with a spread of 0.034. Here's how.

A "faithful unit" is defined as: one contiguous run of Unicode letters, marks, or digits — or one visible non-space punctuation character. So "India's" has three faithful units: India, ', s. The number "1,428,627,663" has seven: each digit-run and each comma.

Fertility = tokens / faithful_units. If the tokenizer maps "India's" to a single token ▁India's, that word contributes 1 token for 3 faithful units — a fertility of 0.33 for that word. If it maps "क" to a single token and "क" is one faithful unit, that contributes 1.0. Fertility can go below 1.0 when the tokenizer learns to bundle punctuation into the adjacent word.

The Wikipedia India article in English is 43 000 faithful units. Telugu's is 8 000. If you throw them all into BPE training without adjustment, English word-pairs fire in every merge round, soaking up most of the 10 000-token budget. Telugu gets a handful of merges. Hindi and Bengali sit somewhere in between.

My first naive run confirmed this exactly. English fertility: 1.29. Telugu: 1.29 (happened to be similar). But Maithili — a smaller article — came out at 1.34 and dragged the spread to 0.10, cutting the score in half. The numbers told me the fix wasn't in the algorithm; it was in the training data balance.

1. Metaspace pre-tokeniser, not Whitespace

Standard whitespace splitting turns "India's" into three pre-tokens: India, ', s. Each must become at least one BPE token — so you can never get below 3 tokens for 3 faithful units. Fertility floor = 1.0.

Metaspace marks word boundaries with ▁ inside the word: "India's" becomes a single pre-token ▁India's. If "India's" is common enough (it is — it's the India article), BPE merges it into one token. Now 1 token covers 3 faithful units: fertility contribution = 0.33. The overall fertility can drop below 1.0, which pulls the denominator of the score formula further from zero.

2. NFKC normalisation

Indic scripts have multiple Unicode representations for the same glyph: different ordering of combining marks, compatibility vs. canonical forms. NFKC canonicalises all of them before tokenisation, so the same word in two different encodings produces the same tokens. Without this, half your Devanagari character pairs appear twice in the merge statistics and get split across two different merge rules.

3. min_frequency = 1

The default HuggingFace BPE trainer uses min_frequency=2, discarding any pair that appears only once. For Telugu (8 000 faithful units) and Bengali (19 000), many valid character-pair patterns appear only once in a single article. Setting min_frequency=1 keeps all of them eligible for merging, which matters enormously for lower-resource scripts.

4. Corpus balancing by article size

Telugu's Wikipedia India article is 5× shorter than English's. If I train on each article once, Telugu gets 5× fewer pair statistics. I compensate by replicating each article proportionally:

English  × 3   (43 K faithful units)
Hindi    × 5   (20 K faithful units)
Telugu   × 8    (8 K faithful units)
Bengali  × 8   (19 K faithful units)

These weights were found empirically by grid search. The sweet spot for Telugu is exactly ×8 — at ×9, it overshoots and drops below Bengali, widening the spread again. The relationship isn't linear and you really do have to run the experiments.

5. Bengali as the fourth language

The assignment requires a fourth language of your choice. I tried Maithili first (same as the reference solution) — but its Wikipedia India article is only 4 KB. Even at ×8 replication it contributes less than 2% of the training corpus, so BPE learns almost nothing specific to Maithili. Score: 4 959.

Bengali (ভারত) has a 96 KB article — 24× larger. Bengali script also belongs to the same Brahmic family as Devanagari (Hindi), so the two scripts share structural patterns: character categories, vowel sign positions, common conjuncts. Cross-language merges benefit both. Score jumped to 28 280.

Hindi sits at 1.2636 — just above the 1.2 threshold that triggers the assignment's "Hindi penalty factor" of exp(max(0, fertility/1.2 − 1)). The penalty here is 1.054, reducing the adjusted score from 29 820 to 28 280. Getting Hindi below 1.2 requires a larger Hindi corpus weight, but that widens the spread in the other direction. It's a genuine tradeoff and 28 280 sits in the Pareto-optimal zone.

How It's Built

The tokenizer is trained with HuggingFace's tokenizers library (Rust backend, fast) using the BPE model class. Configuration:

from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.trainers import BpeTrainer
from tokenizers.pre_tokenizers import Metaspace
from tokenizers.normalizers import NFKC
from tokenizers.decoders import Metaspace as MetaspaceDecoder

tok = Tokenizer(BPE(unk_token="[UNK]"))
tok.normalizer    = NFKC()
tok.pre_tokenizer = Metaspace(replacement="▁", prepend_scheme="always")
tok.decoder       = MetaspaceDecoder(replacement="▁", prepend_scheme="always")

trainer = BpeTrainer(
    vocab_size=10_000,
    special_tokens=["[UNK]"],
    min_frequency=1,
    show_progress=True,
)
tok.train_from_iterator(corpus, trainer=trainer)

The corpus is built by replicating each language's article according to the tuned weights. Wikipedia articles are fetched via the action=parse API endpoint, parsed with BeautifulSoup to extract all visible text (tables, infoboxes, navboxes included — more text means richer pair statistics), then split into paragraph-level lines for the training iterator.

The faithful-unit counter uses the regex module's Unicode property classes for correct handling of Indic combining marks:

import regex
def count_faithful_units(text):
    return len(regex.findall(
        r'[\p{L}\p{M}\p{N}]+|[^\s\p{L}\p{M}\p{N}]', text
    ))

The web widget re-implements the BPE encoding in JavaScript so tokenisation runs in-browser with no server. The Metaspace pre-tokeniser is reconstructed as: prepend ▁ to the first character of each whitespace-delimited word, then apply merges in priority order using a Map lookup.

Reproducing the Results

cd ses2
pip install tokenizers requests beautifulsoup4 lxml regex
python3 train_bpe.py

The script fetches the four Wikipedia articles, trains the tokenizer, evaluates fertility, saves tokenizer.json, vocab.json, merges.json, and corpus/*.txt snapshots. The Wikipedia articles change over time, so the corpus snapshots in the repo are the ones used to produce the reported scores.