BPE, glitch tokens, and why the model can't count the r's in strawberry

Everything upstream of a transformer’s first matrix multiply depends on one decision made before any training begins: how text gets cut into tokens. Get it wrong and the failures are strange in a specific way, not crashes, not NaN, but a model that refuses to repeat a string, or one that quietly costs a specific language four times as much per query as English.

BPE, worked to an actual merge, not just described#

Byte-pair encoding starts with a base vocabulary of individual bytes or characters, counts every adjacent pair of tokens across the training corpus, merges the single most frequent pair into one new token, and repeats until reaching a target vocabulary size. That’s the whole algorithm. Here it is on a toy corpus, "low low lower lowest", underscores marking word boundaries the way real tokenizers do:

Starting vocabulary: individual characters, l, o, w, _, e, r, s, t.

Round 1. Counting adjacent pairs across l-o-w-_-l-o-w-_-l-o-w-e-r-_-l-o-w-e-s-t, the pair (l,o) appears 4 times, once in every occurrence of “low”, more than any other pair. Merge l+o into lo.

Round 2. Re-tokenized with lo as an atomic unit: lo-w-_-lo-w-_-lo-w-e-r-_-lo-w-e-s-t. Now (lo,w) appears 4 times, the most frequent remaining pair. Merge into low.

Round 3. low-_-low-_-low-e-r-_-low-e-s-t. The pair (low,e) now appears twice, in “lower” and “lowest”. Merge into lowe.

After three merges, the vocabulary holds low as a single token covering the standalone word entirely, and lowe as a shared prefix for “lower” and “lowest”. That’s the entire subword mechanism BPE is built to discover: common whole words collapse into single tokens, rarer word families share a subword prefix instead of needing entirely separate tokens each. Production BPE runs this identical loop over billions of characters instead of twenty. Nothing about the mechanism changes at scale, only the corpus size and the merge count.

Vocabulary size is secretly a decision about attention’s cost#

A larger vocabulary means each token covers more characters on average, so the same sentence encodes into fewer tokens. This isn’t just a storage convenience. Attention’s compute cost scales as $O(S^2 \cdot d)$ and its memory cost as $O(S^2)$, where $S$ is sequence length in tokens. Halving the token count for the same text roughly quarters attention’s compute and memory cost, a vocabulary-size decision made once, before training even starts, that directly multiplies or divides the cost of every single attention operation for the rest of the model’s life.

Llama 3’s actual, verified improvement from expanding Llama 2’s 32K vocabulary to 128K was a compression increase from 3.17 to 3.94 characters per token on English data, roughly 24% more text per token, which by the $O(S^2)$ relationship above translates into a meaningfully larger reduction in attention cost specifically, not just a proportional one. Worth being precise about how that result gets described, though: it’s commonly paraphrased as a specific inference-cost percentage, but the paper’s own framing is that the improved tokenizer lets the model “read” more text for the same training compute, fewer tokens, same content. The direction is accurate; a specific invented percentage attached to it usually isn’t the paper’s own number.

Glitch tokens: a mystery with a genuinely rigorous explanation#

In early 2023, researchers found that asking GPT-3 to simply repeat certain specific strings, " SolidGoldMagikarp" among the most famous, produced bizarre outputs: evasive non-answers, apparent insults, complete non-sequiturs, anything but the string itself. This isn’t folklore. It has a mechanistic explanation built from nothing more than embeddings and the fact that gradients only update what training data actually exercises.

The mechanism. A tokenizer is trained on one corpus, often a large, noisy web scrape. The model is then trained on a separate, more carefully curated corpus. SolidGoldMagikarp was, in fact, a frequently posting Reddit username, appearing constantly in the raw web text used to build the tokenizer, so it earned its own dedicated token. But that string is rare or entirely absent from the model’s actual training corpus. The token exists in the vocabulary and gets assigned an embedding at initialization, and then that embedding never receives a meaningful gradient update, because the token essentially never appears during training for the model to learn anything about it.

What an untrained embedding actually looks like, verified rather than assumed. The original investigators found these anomalous tokens cluster near the centroid, the center of mass, of the entire embedding space, rather than being randomly scattered. A later, independent, falsifiable experiment explains precisely why: feeding the model 3 million randomized internal states and recording which tokens it ever selects as output, 510 of 50,257 possible tokens were never once produced, and 85 of those 510 were among 133 previously identified “weird” tokens. A control group of 50,257 genuinely random embedding vectors, run through the identical experiment, had every single one predicted at least 16 times. The chance of that overlap being coincidence is vanishingly small. The geometric explanation: these tokens sit in the interior of the convex hull formed by all token embeddings. There is no direction the model’s internal state can point in for which that specific embedding is the closest match, because it’s surrounded on every side by other tokens’ embeddings, not sitting at an extreme, “winnable” position the way a well-trained, distinctive token would.

This is now a formally studied class of problem, with peer-reviewed work specifically building detection methods for it, precisely because the disconnect between tokenizer-training data and model-training data is a structural property of how frontier models actually get built, two separate pipelines, two separate datasets, often two separate teams on two separate timelines, not a one-off bug in a single model.

Why “strawberry” is architectural, not a reasoning failure#

A transformer never sees individual characters inside a multi-character token. It sees one continuous embedding vector representing the whole token. If "strawberry" tokenizes as one or two tokens rather than ten individual letters, the model has no direct, guaranteed access to the sequence r-r-y at the end of the word. Any ability to answer “how many r’s are in this word” has to come from indirect statistical association learned during training, not from something architecturally equivalent to actually looking at each letter. This isn’t a reasoning failure in the sense people usually mean it. It’s a direct, predictable consequence of the tokenization boundary hiding character-level structure from every layer above it, and it degrades specifically for longer or rarer words precisely because those get split across fewer, larger tokens with less consistent internal structure for the model to have statistically absorbed.

Numbers: the failure named everywhere, and the subtler one that isn’t#

The commonly cited failure: a vocabulary too small forces numbers to be tokenized digit by digit, so 1234 becomes four separate tokens with no inherent representation of place value. The model has to learn that the token for 1 in the thousands position means something different from the token for 1 in the ones position, purely from statistical exposure, with no architectural help.

The subtler failure, and arguably the worse one: a large BPE vocabulary doesn’t fix this cleanly, it can make it inconsistent instead. Since BPE merges are driven purely by frequency, a common number like 2024 might merge into a single token, because it appeared constantly during tokenizer training as a year, while an equally valid but less frequent number like 2027 might not, splitting into 20+27 or some other arbitrary split. The same architecture, on the same kind of input, ends up representing two numbers with completely different token structures, purely depending on how often each specific numeral string happened to appear during tokenizer training. This is exactly why some frontier tokenizers now deliberately special-case digit tokenization, forcing consistent, fixed-size digit groupings, rather than leaving numeral tokenization to emerge naturally from the same frequency-driven merge process used for ordinary words.

Four failures that look similar and aren’t#

Failure	Mechanism	What actually distinguishes it
Tokenizer mismatch, training vs. serving	One environment normalizes Unicode (NFKC), another doesn’t; a small fraction of tokens decode differently	An encoding-time inconsistency between two pipelines processing the same text differently, the token IDs themselves disagree
Glitch tokens	Tokenizer-training data and model-training data disagree on which strings are common	Both pipelines agree on the token ID; the problem is the model was never exposed to that ID during training, a training-data gap, not an encoding disagreement
Out-of-vocabulary bytes	Certain Unicode sequences aren’t in the vocabulary at all	A missing mapping, not an undertrained one
Numbers as digits	Vocabulary too small, or inconsistent frequency-driven merges	A structural representation problem, present even with a perfectly consistent, fully-trained tokenizer

These four get lumped together as “tokenizer bugs” constantly, and the fix for one does nothing for the others. Re-training the tokenizer with more data fixes glitch tokens, does nothing for digit inconsistency, which needs a special-cased rule instead.

Common mistakes#

Treating a model’s refusal to repeat a specific string as a safety-training artifact or a reasoning failure before checking whether the string maps to a rare or undertrained token: the glitch-token mechanism above produces symptoms that are easy to misattribute to something far more complicated than one never-trained embedding sitting at the center of the space.

Assuming a bigger vocabulary is free, or purely a quality improvement: it directly changes attention’s compute and memory cost through sequence length, a decision made once at tokenizer-design time with consequences for every attention call the model will ever make.

Blaming multilingual underperformance on the model’s actual capability before checking tokenizer fertility for that language: a model can be equally capable per token while the tokenizer itself taxes one language several times more per unit of meaning than another.

Expecting a larger vocabulary to fix numeral inconsistency: it can just make the inconsistency less predictable, since which numbers get single-token treatment is a function of tokenizer-training frequency, not of any principled rule about numbers.

What this takes to be frontier-job-ready#

Given a model behaving erratically on specific, narrow inputs, refusing to repeat a string, producing non-sequiturs on one particular phrase while working normally elsewhere, know to check whether the input touches a rare or unusual token before assuming a reasoning failure or a safety-training artifact. And given a multilingual product underperforming for one specific language, know to check tokenizer fertility for that language before assuming the model’s actual capability differs, the model may be equally capable per token while the tokenizer quietly taxes that language more per unit of meaning.

Try it yourself#

Beginner. Run one more BPE merge round on the worked example above (after lowe is formed) and identify the next most frequent pair by hand.

Intermediate. For a fixed piece of text, estimate the attention compute cost at two different tokenizations, one producing 100 tokens, one producing 400 tokens for the same content, using the $O(S^2)$ scaling above, and compute the resulting cost ratio.

Advanced. Explain, in terms of convex hull geometry, why an embedding vector that never receives a gradient update is more likely to end up literally inside the hull formed by other tokens’ embeddings rather than at an extreme point, and why that specific position (interior, not extreme) is what makes it nearly impossible for the model to ever select as output.

---

The one-sentence version: BPE is nothing but repeatedly merging the most frequent adjacent pair until you hit a target vocabulary size, that vocabulary size decision directly sets how expensive every attention call will be for the model’s entire life, and glitch tokens exist because a tokenizer trained on one corpus and a model trained on another can agree on a token’s existence while the model never learns anything about it, leaving that embedding stranded at the one point in the entire vector space the model can never actually point to. Those tokens are what actually get cached, once per layer, at every step of generation, which is the constraint that decides how the attention mechanism itself has to be shaped.