THE MERGE HAPPENING NOW #0
press run — it will find the most common adjacent pair and fuse it
THE CORPUS, RE-TOKENIZED every symbol is a letter
VOCABULARY
letters (1.0)chunks (→3+)
MERGES, IN ORDER DISCOVERED
(base: space + a…z)27
BPE learns what a “symbol” should be. It starts knowing only the 27 characters. Then it repeats one move: scan the whole corpus, find the most frequent adjacent pair of symbols, and fuse that pair into a single new symbol added to the vocabulary. Do it a few hundred times and a vocabulary of chunks emerges — not designed, discovered.
Watch the order. The first merges are the tightest bonds in English: e␣, s␣, th, t␣. Within a dozen it fuses the␣ as one token; soon you, ing, and, that. Frequency alone rebuilds the word boundaries. The corpus panel re-tokenizes live — letters clumping into teal pairs, then violet whole-words, as the merges apply.
Why it beats the letter field. The bigram field stalled near 50% because one letter of memory can't predict much. A BPE token already is several letters — so predicting the next token reaches further than predicting the next letter, and the compression climbs from 1 char/token toward 3+. Same corpus, richer alphabet: this is how real models get past the character wall before attention even enters.