The central mechanism
A ranked pair rule replaces two adjacent symbols with one reusable symbol. Repeating that operation creates a vocabulary that balances sequence length against coverage.
Text becomes a sequence the model can price.Interactive language-model explainer
Watch text become tokens, merge by merge
Type a sentence, compare character, word, and subword views, then scrub through ranked byte-pair merge rules. The 3D blocks are deterministic teaching tokens, not IDs or output from a production tokenizer.
View
A model does not read the string you see
Before a transformer receives text, a tokenizer maps the input into a sequence of discrete symbols from a vocabulary. Those symbols are converted into integer IDs and then embedding vectors. The vocabulary and segmentation rules therefore sit directly on the path between human text and model computation.
Character tokenization is simple and open-ended, but long words produce long sequences. Whole-word tokenization makes common words compact, but a fixed vocabulary cannot cover every spelling, language, name, typo, URL, or code fragment. Subword tokenization uses reusable pieces: common sequences stay compact while rare strings can still be represented from smaller units.
This simulator begins each word as a boundary marker plus characters. Its ranked teaching rules merge adjacent pairs such as t + o -> to or i + n -> in. A production BPE tokenizer learns a much larger ranked merge table from a corpus and usually operates on bytes or Unicode-aware text representations. The blocks here are intentionally transparent and are not a replica of any commercial model vocabulary.
Scrub the merge slider slowly. The text does not change, but the sequence length and block boundaries do. That changes how many positions enter the transformer, how much context remains, and which fragments share a learned embedding lookup. Tokenization is not cosmetic preprocessing; it changes the computational interface.
Vocabulary is a tradeoff
A larger vocabulary can shorten common sequences but requires a larger embedding table and still reflects the corpus used to learn it.
Boundaries matter
Many tokenizers encode whether a piece begins a word or follows whitespace. The same letters can map differently in different positions.
Rare text decomposes
Names, identifiers, code, and uncommon spellings often split into smaller pieces. Coverage remains possible, but sequence cost can rise.
Normalization changes input
Unicode normalization, whitespace treatment, casing, and pre-tokenization decisions can affect segmentation before merge rules run.
How a BPE vocabulary is learned
Start from guaranteed coverage
A byte-level system can represent arbitrary input because every byte has a base symbol. Character-level systems rely on their chosen character set. A boundary-aware representation also records where words begin so the model can distinguish internal pieces from new-word pieces.
Count adjacent pairs
The trainer scans segmented corpus entries and counts neighboring symbol pairs, weighted by occurrence. A pair that appears across many common words can be a strong candidate. Different implementations may constrain where pairs are allowed or how ties are resolved.
Add a merged symbol
The selected pair becomes a new vocabulary item. Occurrences are replaced, counts are updated, and the process repeats. Earlier rules can enable later rules: merging t + h creates th, which can then merge with e.
Freeze the model interface
After training, the vocabulary and merge order become part of the model contract. Changing them changes token IDs, embedding lookup, sequence lengths, and compatibility with model weights. A tokenizer version must travel with the model that expects it.
Three views of the same sentence
Character view
Every visible character and space becomes a teaching token. Coverage is straightforward, but sequences are long and the model must compose frequent patterns repeatedly.
T | o | k | e | n | i | z | a | t | i | o | n
Subword view
Frequent fragments become reusable pieces while rare strings fall back to smaller units. This is the practical middle ground used by many language-model tokenizers.
Token | iz | ation | changes | inference | cost | .
Word view
Whole words are easy to interpret, but an open vocabulary is impossible to enumerate. New words, typos, code, and multilingual input create unknown-token pressure.
Tokenization | changes | inference | cost | .
BOUNDARYCHARACTERPAIR COUNTMERGE RANKVOCABULARYTOKEN IDEMBEDDING LOOKUPCONTEXT POSITIONBOUNDARYCHARACTERPAIR COUNTMERGE RANKVOCABULARYTOKEN IDEMBEDDING LOOKUPCONTEXT POSITION
Token count becomes system cost
Attention sees positions
A transformer processes token positions, not human-perceived words. For full self-attention, the number of pairwise attention relationships grows quadratically with sequence length, although modern implementations and architectures change practical memory and compute behavior.
Context is shared capacity
Instructions, conversation history, retrieved documents, tool results, and generated output compete for a finite context budget. A verbose serialization or poorly tokenized identifier can crowd out evidence that matters more.
Billing follows provider rules
Many APIs meter input and output tokens, sometimes with separate cached-input or reasoning categories. Never estimate a production bill from this teaching tokenizer; count with the exact tokenizer and pricing policy for the deployed model.
Languages can differ
A vocabulary learned from uneven data can tokenize languages and scripts at different efficiencies. Compare real task distributions rather than assuming the same words-per-token ratio across languages, domains, or formatting styles.
Why personal agents need tokenizer awareness
A personal agent assembles instructions, user messages, memories, browser observations, and tool results into model input. Tokenization determines how much of that evidence fits and where truncation pressure appears. The application should manage context intentionally instead of treating a character count as a token budget.
For a text-message AI assistant, short messages can still contain long URLs, order identifiers, emoji, or code-like fragments that tokenize unexpectedly. Count with the deployed model's tokenizer before deciding how much history to retain.
A computer-use cache may store accessibility trees, DOM text, screenshots descriptions, or action receipts. Structured compression, deduplication, and retrieval can preserve useful evidence while avoiding repeated token-heavy boilerplate.
When an AI website-building agent reads source files and browser logs, minified code or generated markup can consume context quickly. Chunk by semantic ownership, retrieve the files relevant to the requested edit, and keep exact paths and error messages even when surrounding text is summarized.
Super applies these constraints to messaging and computer-use workflows where an agent must preserve enough state to act, verify, and report without letting stale or duplicated context dominate the prompt.
Use the tokenizer version paired with the deployed model.
Measure real prompts, tool outputs, and multilingual inputs.
Reserve output capacity before filling the input context.
Keep special tokens and chat-template overhead in the count.
Retrieve relevant evidence instead of replaying all history.
Deduplicate repeated tool schemas and browser observations.
Test truncation behavior at boundaries and maximum lengths.
Track token count separately from characters and words.
Frequently asked questions
Is this the tokenizer used by a specific production model?
No. It is a deterministic teaching implementation with a small, visible merge list. Production tokenizers have model-specific vocabularies, normalization, pre-tokenization, special tokens, and much larger learned rule sets.
Does BPE always merge the globally most frequent pair?
The original training intuition counts frequent adjacent pairs, but implementations differ in data representation, constraints, tie-breaking, byte handling, and optimization. At inference time, a learned rank table determines which compatible merges are applied.
Why include a word-boundary marker?
A boundary marker makes word starts explicit. It helps distinguish a piece that begins after whitespace from the same character sequence inside a word. SentencePiece commonly exposes an underscore-like boundary symbol; byte-level BPE systems often use their own representation.
Are tokens the same as syllables or morphemes?
Not reliably. Some pieces align with meaningful linguistic units because frequent patterns recur, but the algorithm optimizes a statistical vocabulary objective rather than a linguistic analysis. A token can be a word, affix, punctuation mark, byte sequence, or arbitrary fragment.
Why can two visually similar strings tokenize differently?
Whitespace, casing, Unicode normalization, combining characters, script, and preceding context can change the underlying sequence presented to the tokenizer. Inspect code points and tokenizer output rather than relying only on visual appearance.
How should I estimate production context use?
Use the exact tokenizer, chat template, special tokens, tool schema serialization, and model limits supplied by the provider or model repository. Count both the assembled input and the output capacity you need.
Primary references and implementations
Neural Machine Translation of Rare Words with Subword UnitsSennrich, Haddow, and Birch introduce BPE-based subword segmentation for open-vocabulary translation.
SentencePieceKudo and Richardson describe language-independent subword tokenization directly from raw sentences.
tiktokenOpen-source implementation of a fast BPE tokenizer with educational documentation and model encodings.
Language Models are Unsupervised Multitask LearnersThe GPT-2 report discusses a byte-level BPE vocabulary designed for broad input coverage.
Unicode Text SegmentationUnicode Standard Annex 29 defines grapheme, word, and sentence boundary algorithms relevant to text handling.
Three.js DocumentationThe WebGL library used for the live, spatial token-block visualization.
Count the sequence the model receives.
Tokenizer awareness turns an invisible preprocessing detail into an explicit engineering constraint for context, latency, retrieval, and reliable agent behavior.
Explore Super