Before a language model can "think", it must cut the world into pieces. That cut — tokenization — decides what it can read, what it bills you, what it understands… and where it can be attacked.
Every tokenizer in this deck is genuinely trained in your browser, on an embedded corpus. Nothing is simulated.
"How many times does the letter r appear in strawberry?"
For years, the best models in the world answered "two". Confidently. The correct answer is three.
This is not a failure of intelligence. It is a failure of vision.
Three opaque blocks. None of them is a letter. Asking this model to count the r's is like asking someone who only reads Chinese characters to count the strokes in an English word.
The model does not read letters. It reads tokens — and it has never seen what is inside them.
The thesis of this deck: tokenization is the most underrated layer of generative AI. It is simultaneously an engineering decision, an economic decision, a political decision (which languages cost more?) and a security decision. Everything that follows flows from a single trade-off.
From the fundamental problem to concrete attacks, by way of eight families of algorithms.
A neural network only manipulates numbers. So "Hello world" has to become a list of integers. The tokenizer is that translator: it cuts text into pieces, and each piece has an index in a frozen dictionary — the vocabulary.
A ? marks a symbol absent from the vocabulary learned on this small corpus — digits, rare punctuation, foreign alphabets. That is exactly the out-of-vocabulary problem, and the reason byte-level fallback exists.
The closed list of every known piece. Typically 30,000 to 260,000 entries. Anything not in it must be rebuilt from fragments.
It is counted in tokens, not words. An inefficient tokenizer literally steals memory from you.
APIs bill per token. Same text, two tokenizers, two different invoices.
Every algorithm that follows does exactly one thing: negotiate a compromise between two opposing forces. There is no perfect solution, only equilibrium points.
<UNK> — information lostA BPE is trained on the embedded corpus, then applied with a growing number of merges. Watch the curve: every token you save costs more and more vocabulary.
The equilibrium the industry settled on: the subword. Big enough that "hello" is a single token, small enough that "antidisestablishmentarianism" stays representable.
The naive approach — and an instructive one, through its failures.
<UNK>.Analogy: a paper dictionary. If the word is not in it, you have no way to read it — not even letter by letter.
The opposite extreme — elegant, but expensive.
Analogy: reading a sentence by spelling it out. You will never be blocked by an unknown word, but you will read ten times slower.
The reigning algorithm. Repurposed in 2015 from a 1994 compression technique.
Analogy: a stenographer. After writing "-ment" a thousand times a day, they invent a single sign for it. Then one for "-tion". The vocabulary emerges from repetition, not from a linguistic theory.
eating → eat|ing, but eaten → ea|tenThe embedded corpus is real (English text, ~6,000 characters). Start the training and watch the vocabulary being born.
Notice the diminishing returns: the first merges shave off an enormous amount (they capture "th", "in", "er"…), the later ones less and less. That is Zipf's law showing itself — and the reason vocabularies plateau around 100,000–200,000 entries.
The trick that finally killed the <UNK> token.
Analogy: instead of a dictionary of words, a universal alphabet of 256 bricks. An emoji, an ideogram, a runic character, a binary file: everything can be rebuilt. Nothing is ever "unknown".
BPE's statistical cousin — the one behind BERT.
f(ab) / (f(a)·f(b)).##.The difference in one sentence: BPE asks "which pair do I see most often?". WordPiece asks "which pair appears most often relative to what chance would predict?". The second favors genuinely meaningful associations over ubiquitous letters.
## marker makes word boundaries explicit and readable[UNK] token — no byte-level fallbackThe probabilistic approach: start from too much, then prune.
Analogy: BPE is a bricklayer stacking bricks upward. Unigram is a sculptor: it starts from an oversized block and removes everything that serves no purpose. Crucially, it knows several valid cuts of the same word and which one is most likely.
▁) → language-independentAll of these tokenizers are trained on the same embedded corpus, with the same vocabulary target. The differences you see are therefore purely algorithmic.
ByT5, CANINE, MambaByte — remove the problem rather than optimize it.
The underlying argument: the tokenizer is the last unlearned piece of the pipeline. It is a frozen artifact, trained on a different corpus than the model, never updated. The history of deep learning suggests hand-built components always end up absorbed.
Meta, 2024-2025: cut by difficulty, not by frequency.
The idea is elegant: a classical tokenizer spends as much effort on the predictable ending of a common word as on an unexpected proper noun. BLT allocates its compute budget dynamically, based on actual uncertainty. Meta's work reports scaling laws at least comparable to Llama 3 at equal compute, with markedly better robustness to noise.
Rather than going below the word, go above it. SuperBPE (COLM 2025) lifts the constraint forbidding a token from crossing a space, and learns "superwords" like "by accident" or "depend on". Reported result: roughly 33% fewer tokens at equal vocabulary, an average gain of +4% across thirty tasks, and ~27% faster inference — with a few regressions, notably on code. Two opposite schools, one shared conclusion: there is nothing sacred about the word.
| Algorithm | Principle | Vocabulary | Main strength | Main weakness | Models |
|---|---|---|---|---|---|
| Words | Split on whitespace | 10⁵–10⁶ | Immediate meaning | Out-of-vocabulary wall | Word2Vec |
| Characters | One token = one character | ~10² | Zero unknowns | Sequences 4-5× longer | small models |
| BPE | Greedy merge by frequency | 30k–100k | Best overall trade-off | Non-morphological boundaries | GPT-2, RoBERTa |
| Byte-level BPE | BPE applied to bytes | 50k–260k | Universal coverage | Tax on non-Latin languages | GPT-4o, Llama 3 |
| WordPiece | Merge by likelihood | 30k | More meaningful segments | Exploitable greediness | BERT |
| Unigram | Probabilistic pruning + Viterbi | 32k–256k | Regularization, multilingual | Expensive training | T5, Gemma |
| Raw bytes | No learning at all | 256 | Fairness and robustness | Compute cost | ByT5, MambaByte |
| Patches (BLT) | Cut by entropy | 256 | Adaptive compute | Complexity, immaturity | BLT (Meta) |
The underlying trend is clear: vocabularies keep growing. More vocabulary means fewer tokens per sentence — hence more usable context and less compute per word — at the price of heavier embedding and output layers.
A tokenizer trained mostly on English compresses English very well. Other languages pay — in money, in latency, and in available context.
The BPE tokenizer here is trained on the embedded English corpus. Watch how it behaves on languages it has never seen — which is exactly the situation most of the world's languages are in when facing the tokenizers of large models.
Economic consequence. For equivalent content, some languages consume several times more tokens than English. Usage-based billing, context ceiling, latency: the bill is multiplied accordingly, for speakers often located in lower-income countries.
Qualitative consequence. A more fragmented segmentation means a more diluted signal and more fragile reasoning. The performance gap between languages is therefore not only a training-data problem: part of it is baked into the tokenizer.
Almost all LLM security attention goes to the prompt and to the weights. Yet there is a layer between the two — frozen, rarely audited, often shared between the model and its guardrails.
The tokenizer and the model are not trained on the same data. A forum username that was very active when the vocabulary was built gets its own token. Then that data is filtered out before the model is trained.
The result: the token exists, but its embedding vector was never learned. It stays close to its random initialization.
Feeding that token in amounts to injecting pure noise into the heart of the network. Models respond with insults, gibberish, unexplained refusals, language switches — or by ignoring the instruction entirely.
SolidGoldMagikarp — a Reddit username, discovered in 2023. Asking GPT-3 to repeat it produced "distribute", insults, or a refusal.
The work "Fishing for Magikarp" (2024) automated the detection of such tokens and found them in nearly every major model family: code fragments, spam-derived words, cleanup artifacts, configuration strings. Anomalous tokens have again been documented in recent models.
An out-of-distribution internal state can break learned behaviors — alignment among them.
Incoherent responses or infinite loops triggered by a short string.
The reaction to these tokens precisely identifies the model and its version behind an opaque API.
Unicode contains several characters visually indistinguishable from the Latin a. To a human, it is the same word. To a tokenizer working on bytes, it is a completely different object.
Published in 2025, this attack exploits a disconcerting asymmetry: adding a single letter to a word is enough to derail the segmentation of a BPE or WordPiece classifier — while the target LLM still understands perfectly.
The greedy segmentation can no longer anchor on the start of the word. The classifier receives fragments it has never associated with problematic content. The LLM, trained on billions of human typos, reads it without difficulty.
The attack targets the guardrail's tokenization strategy, not the model. BPE and WordPiece classifiers proved vulnerable; Unigram-based ones markedly less so, because Viterbi segmentation is global rather than greedy.
This is the first concrete security criterion for choosing a tokenizer: your filter and your model must see the same text.
Slipping through an instruction the human cannot see but the model reads perfectly.
Defense: strip non-printable Unicode categories (Cf, Co, tag characters) by default on input, log their presence as a strong alert signal, and base the security decision on the text after normalization.
The characters/tokens asymmetry is a denial-of-service lever.
Defense: cap in tokens, never in bytes or characters. Count tokens before calling the model. Monitor fertility (tokens ÷ characters): beyond a threshold, reject or degrade.
The common thread across all five attacks: they all exploit the same gap — between what the human sees, what the filter sees and what the model sees. Narrowing that gap is an architectural goal, not a one-off patch.
Nine concrete measures, ordered from the highest return to the most demanding.
Guiding principle: treat the tokenizer as a first-class security component — versioned, tested, logged, audited — rather than an implementation detail inherited from the original Hugging Face repository.
We started from a joke — a model unable to count the r's in strawberry. We arrive at an infrastructure layer that determines what a model can perceive, what its user pays, which languages are favored, and where an attacker can get in.
1. Every tokenizer is a compromise between vocabulary size and sequence length. There is no universal optimum.
2. The tokenizer is the last unlearned component of the pipeline — hence the next candidate to disappear (BLT, MambaByte).
3. It is a real attack surface, actively exploited, and almost never audited. Hardening starts there.
The natural next step for the AI WATCHTOWER project: build a tokenization test bench — glitch-token detection, fertility measurement, Unicode fuzzing, and verification that guardrail and model stay aligned.
Key sources: Sennrich et al. (BPE, 2015) · Kudo (Unigram, 2018) · Kudo & Richardson (SentencePiece, 2018) · Radford et al. (byte-level BPE, GPT-2, 2019) · Xue et al. (ByT5, 2022) · Land & Bartolo ("Fishing for Magikarp", 2024) · Pagnoni et al. (Byte Latent Transformer, Meta, 2024) · Liu et al. (SuperBPE, COLM 2025) · HiddenLayer (TokenBreak, 2025).