The Byte Latent Transformer: Inside Meta's Vocabulary-Free LLM

Every language model you've used starts with the same silent preprocessing step: a tokenizer converts raw text into a sequence of integers from a fixed vocabulary. This step is so universal it barely gets mentioned. BPE, Unigram LM, WordPiece — they differ in how they build the vocabulary, but they all assume the same thing: that text should be pre-segmented into discrete tokens before the transformer ever sees it.

BLT removes that assumption entirely. The model operates on bytes — the 256-symbol alphabet of raw UTF-8 — and learns to group them dynamically using an entropy estimator. The result, published at ACL 2025, is the first byte-level model to match tokenized baselines at controlled compute.

This is a deep dive into exactly how that works. Not just the what, but the why — and what it tells us about the future of language model architecture.

Consider what a tokenizer actually does to computation. The transformer's attention mechanism is quadratic in sequence length. Shorter sequences are cheaper. So when a tokenizer merges "the" into a single token, it's not just a bookkeeping decision — it's a compute allocation decision. "The" is easy. Spending one token (one position in the attention matrix) on it is probably right.

But the vocabulary is fixed at training time. The same merge rules that make "the" a single token also decide how Arabic clitics split, how Telugu conjuncts encode, how Chinese idioms tokenize. And for those cases, the optimal boundary isn't obvious — it depends on context.

There's a deeper problem: character-level noise breaks fixed tokenizers completely. A typo in "the" produces "teh", which tokenizes differently. An OCR error in a scanned document produces character sequences the tokenizer has never seen. The model has no way to recover because the tokenizer made an irreversible decision before the transformer ran.

The key insight in BLT's design is separating "local" processing (reading individual bytes, forming patches, reconstructing bytes from patches) from "global" processing (reasoning about long-range context across the whole sequence). The expensive quadratic attention only happens at the patch level.

The local encoder is a small transformer (a few layers) that processes individual bytes and their neighbors. Its job is to form a single patch-level representation — a vector that summarizes a variable number of input bytes. Critically, the patch boundaries are not fixed. They're determined by the entropy estimator described in the next section.

The latent transformer is the big model — this is where most of the parameters live and where cross-sequence attention happens. It operates purely on patch representations. Because each patch covers multiple bytes, the latent transformer's sequence length is much shorter than the raw byte sequence, making full attention computationally feasible.

The local decoder is another small transformer that takes patch representations from the latent transformer and reconstructs individual byte predictions. It conditions on the surrounding byte context and the patch representation to produce a probability distribution over the next byte at each position.

This is the most interesting part of BLT. The question of where to draw patch boundaries is answered dynamically, based on how uncertain the model is about the next byte at each position.

Entropy here means Shannon entropy of the predicted next-byte distribution: H = −∑ p(b) log p(b). Low entropy means the model assigns high probability to one or a few bytes — the text is predictable. High entropy means probability mass is spread across many bytes — the text is surprising.

A patch boundary is placed wherever entropy exceeds a threshold. So a run of predictable English text ("the quick brown fox") might become a single large patch — one position in the latent transformer. A sequence of Arabic morphemes, each of which could be followed by multiple valid continuations, produces many small patches. Novel proper nouns, code, multilingual mixing — anything that raises uncertainty gets more attention compute.

This is a fundamentally better match between computation and difficulty than fixed tokenization provides. Fixed tokenization allocates tokens based on training corpus statistics frozen at vocabulary build time. BLT allocates patches based on what's actually hard about this specific sequence at inference time.

Previous byte-level models existed — MegaByte, ByT5 — but they failed to match tokenized baselines at realistic model scales. The standard explanation was that raw bytes create sequences 3–5× longer than tokenized equivalents, making attention compute too expensive. BLT solves this with the patch mechanism: the latent transformer sees a compressed sequence, not raw bytes.

Benchmark	LLaMA 3 8B (tokenized)	BLT 8B (byte-level)	Winner
MMLU (5-shot)	66.6	65.2	≈ Parity
HellaSwag	82.0	81.1	≈ Parity
ARC-Challenge	59.1	58.3	≈ Parity
Noise robustness (typos)	baseline	+8–15% vs tokenized	BLT wins
Multilingual (rare scripts)	baseline	+12% vs tokenized	BLT wins
FLOPs at inference	baseline	~comparable (controlled)	Controlled

The "parity" claim is carefully controlled. BLT and the tokenized baseline were trained on the same number of FLOPs, not the same number of bytes. This matters: 4T bytes is a larger dataset in size, but comparable in training compute because each byte requires less processing than a full token in the latent transformer.

Where BLT genuinely wins over tokenized models: noise robustness. When input text contains typos, OCR errors, unusual Unicode, or mixed scripts, tokenized models degrade significantly — the tokenizer produces unfamiliar sequences that the model hasn't seen patterns for. BLT is inherently robust because it sees individual bytes regardless; the entropy estimator simply increases patch resolution for unusual sequences and gives the latent transformer more compute budget to handle them.

The reason BLT matters isn't the benchmarks. It's what the benchmarks remove as an excuse. Before BLT, "we use tokenization because byte-level doesn't scale" was a fact. After BLT, it's a hypothesis that requires evidence at larger scale.

Consider what removing the tokenizer would eliminate: the token tax (Arabic 3.3× vs English, Bengali ~4.5×), arithmetic fragmentation, the vocabulary selection problem (which ACL 2025 proved is NP-complete), the need to maintain separate tokenizer versions across model families, the "unreachable token" bug in GPT-4o, and the dependency on a frozen corpus snapshot from vocabulary training time.

None of those problems exist in a byte-level model. The alphabet is 256 bytes. It never changes. It covers every language, every number, every symbol, every byte sequence that can exist.

The open question is whether this holds at 70B+ parameters and multi-trillion-byte training runs. BLT showed parity at 8B. Nobody has tried at 70B. The compute overhead of the local encoder and decoder may become a bottleneck at very large scales. The patch mechanism's efficiency depends on the average patch size, which depends on corpus characteristics. These are engineering problems, not fundamental limits — but they're real.

One concern with dynamic patching is the overhead of the entropy estimation step. If you have to run a model just to decide where to put boundaries, you've added compute before the main computation even begins.

BLT addresses this by using a small local model — the same local encoder used in the main architecture — as the entropy estimator. It's trained jointly, shares weights with the local encoder, and runs a single forward pass over the byte sequence. The patch boundaries are determined by a threshold on the estimated entropy: wherever H(byte_t) exceeds the threshold, a new patch begins.

# Conceptual pseudocode for BLT patching
def dynamic_patch(byte_sequence, entropy_model, threshold=0.5):
    entropies = entropy_model.estimate(byte_sequence)  # shape: [seq_len]

    patches = []
    current_patch_start = 0

    for t, h in enumerate(entropies):
        if h > threshold:  # high entropy → start new patch here
            if t > current_patch_start:
                patches.append(byte_sequence[current_patch_start:t])
            current_patch_start = t

    patches.append(byte_sequence[current_patch_start:])
    return patches  # variable-length groups of bytes

The threshold is a hyperparameter that controls average patch size. Lower threshold → smaller patches → more latent transformer positions → more compute but higher resolution. Higher threshold → larger patches → fewer positions → cheaper but lower resolution for hard text. The paper uses a dynamic threshold based on the fraction of patch boundaries, keeping average patch size roughly constant across different text types.