One-Line Summary: BLT replaces tokenization entirely — it operates directly on raw UTF-8 bytes, dynamically grouping them into variable-length patches based on local entropy.
Prerequisites: Understanding of tokenization (BPE, WordPiece), the concept of fixed-vocabulary embeddings, and the trade-offs of operating on bytes vs. tokens.
What It Is
A byte-latent transformer (BLT) — introduced by Pagnoni et al. (2024) at Meta FAIR — is a tokenizer-free architecture with three components:
- A lightweight local encoder that ingests bytes and groups them into variable-length patches using entropy-based boundaries — predictable runs (whitespace, common prefixes) get long patches, surprising regions get short ones.
- A heavyweight global transformer that operates on patch representations, doing the bulk of the modeling work.
- A lightweight local decoder that takes the global model's output and predicts individual bytes.
The split is deliberate: the global transformer (the expensive part) sees a compressed sequence of high-information patches; the local encoder/decoder (the cheap parts) handle the byte-level details.
Why It Matters
Tokenizers are a leaky abstraction. They introduce a long list of headaches: whitespace sensitivity, brittle handling of rare words, inconsistent number tokenization (1234 vs 12 34), poor multilingual coverage, vocabulary lock-in after pre-training, and the fact that you can never tokenize a typo the way it should be tokenized.
BLT eliminates all of them. By starting from raw bytes, the model becomes natively robust to typos, adversarial perturbations, mixed scripts, and arbitrary non-text byte sequences. Even better, the entropy-based dynamic patching is a form of adaptive computation — the model spends more compute on hard text and less on easy text, something fixed tokenizers cannot do.
The headline result: BLT matches tokenizer-based models at equivalent compute budgets while gaining all of the above robustness for free.
Key Technical Details
Patch boundaries are decided by a small auxiliary entropy predictor, not by a fixed vocabulary. There's no separate tokenizer training step, no pre-decided vocab size, and no "oov" tokens. The architecture is harder to serve efficiently than a standard tokenized model (because patch boundaries vary per input), so adoption has been slower than the research results would suggest — but the architectural direction is increasingly compelling.