Top Pair Frequencies
Merge History
Base Vocabulary 256
Learned Merges 0
Merges will appear here as training progresses
Tokenized Corpus Preview
Corpus will appear here after initialization
How BPE Works
1
Vocabulary Initialization
Start with 256 byte values (0-255). Using UTF-8, any text maps to these bytes, solving the "out of vocabulary" problem.
2
Pretokenization
Split text using GPT-2's regex pattern. This preserves word boundaries and prevents merging across words (e.g., "the" in "the cat" vs "other").
3
Iterative Merging
Find the most frequent adjacent pair → merge into new token → repeat. This greedily maximizes compression of the training corpus.