Tokenization in LLMs: From Text Humans Can Read to Numbers a Model Can Process

The first part opened up a Transformer from the inside. The second part went from the outside in, through pipeline() and the engines that serve inference. The third part was about changing what a model knows.

All three of those posts start from the same place: the text has already become numbers. This post goes back to exactly the step they skipped.

When we type into a chatbot:

I'm learning how Large Language Models work.

what goes into the neural network is not the sentence as a human sees it. A Transformer has no input port that accepts a string and then “reads” English on its own. Before any attention head does anything, the text has to be turned into a sequence of discrete symbols, and then into a sequence of integers.

The diagram at the top of the post (Figure 1) is that whole path. If you only think of a tokenizer as a function text -> list[int], you miss almost every important engineering decision that sits behind it.

The tokenizer decides:

• how finely a unit of language gets split;
• what the model’s vocabulary contains;
• how many positions a passage takes up in the context window;
• whether one language pays more “token tax” than another;
• how source code, numbers, emoji, proper names and rare words are represented;
• how many rows the embedding table needs;
• and, in many commercial systems, how much money it costs to process the same amount of information.

Tokenization is not a side detail living outside the model. It is the discrete interface between language and the neural network.

About the code: the Python snippets in this post are pseudocode and illustrative fragments, not runnable scripts. The cl100k_base token IDs quoted in section 1 were checked with Python 3.12 and tiktoken 0.14.0 in October 2026.

01What does a model actually see when we type text?

Say we have the sentence:

I love language models.

A tokenizer might turn it into:

["I", " love", " language", " models", "."]

Each token is then mapped to an integer in the vocabulary:

[40, 3021, 4221, 4211, 13]

The model does not treat 40 as semantically “bigger” than 13. These numbers are just indices.

The embedding layer uses the ID as an index into a matrix. If E is the embedding matrix of size |V| × d, where |V| is the vocabulary size and d is the hidden dimension, then the input vector of the i-th token is simply row token_id[i] of E.

This is a foundational distinction, and one that is very often glossed over:

Token       = a discrete symbol
Token ID    = the address of that symbol in the vocabulary
Embedding   = the real-valued vector the model learned for that token

A token ID is not an embedding. If the token "cat" has ID 4281, the number 4281 does not by itself mean “a cat”. It only tells the model: go fetch row 4281 of the embedding table.

1.1 A token is not the same thing as a word

This is the most common misconception. A token can be:

• a whole word: hello;
• part of a word: ing, ization;
• a space attached to a word: " world";
• punctuation: .;
• a byte or a sequence of bytes;
• a special control symbol;
• or a string that does not line up with any word boundary a human would intuitively draw.

With OpenAI’s tiktoken, the cl100k_base encoding gives:

"hello world"  →  [15339, 1917]

and those two tokens decode to bytes close to:

b'hello'
b' world'

The space in front of world has become part of the second token. That points to a principle worth remembering:

A tokenizer optimizes for a discrete representation that is useful to the model. It has no obligation to reproduce the notion of a “word” as human linguistic intuition sees it.

02Inside a tokenizer: more than just the BPE algorithm

A production tokenizer is usually a pipeline, not a single function. Figure 1 at the top of the post already draws all four stages; this section explains why each of them matters.

2.1 Normalization: two strings that look the same can differ at the code-point level

Unicode allows characters that display identically to be encoded as different code-point sequences: one precomposed accented character, or a base character plus a combining mark.

Unicode defines four main normalization forms:

Why does a tokenizer care? Because if two strings display as equivalent but have different bytes, they will be tokenized differently unless the pipeline normalizes them consistently.

But stronger normalization is not always better. Unicode warns that NFKC and NFKD can erase compatibility distinctions that carry formatting meaning, so they should not be applied blindly to every kind of text. Lowercasing is the same: Apple → apple shrinks the vocabulary but loses the uppercase/lowercase distinction.

Normalization is a modeling decision, not just “data cleanup”.

2.2 Pre-tokenization: setting boundaries before the subword algorithm runs

A tokenizer usually does a coarse split first:

Hello, world!
→ "Hello"  ","  " world"  "!"

and only then runs BPE or WordPiece inside each piece.

Pre-tokenization rules have a bigger impact than they appear to. If we do not allow merges to cross a certain boundary, the vocabulary learned later is also limited by that same boundary. tiktoken uses a regex pattern to split the input into pieces before applying byte-pair encoding to the bytes of each piece. BERT-style WordPiece likewise pre-tokenizes into words and only then runs maximum matching inside each word.

2.3 Post-processing: adding structure that raw text does not have

After segmentation, the pipeline can insert special tokens:

<BOS>  <EOS>  [CLS]  [SEP]  [MASK]  <|user|>  <|assistant|>

These tokens usually do not appear verbatim in user input. They exist to convey structure to the model, and we will come back to them in section 6.

03The core problem: how finely should text be split?

At its heart, tokenization is a problem of choosing a granularity. There are four levels that are easy to picture, and none of them is absolutely right: each choice trades one kind of cost for another kind of benefit.

3.1 Word-level: intuitive, but the vocabulary explodes fast

The most classic approach is to split by word. Sequences are short, each token is close to a lexical unit, and it is easy to interpret. But the vocabulary balloons very quickly - a single word family already has many variants:

run  runs  running  runner  runners

Proper names, typos, compounds, scientific terms, URLs, source code and morphologically rich languages make things even worse. If a word is not in the vocabulary, a traditional word-level system has to use <UNK> - that is out-of-vocabulary (OOV). Two completely different words can both get squashed into the same <UNK>, which means surface information is lost before the model even starts computing.

3.2 Character-level: almost no OOV, but long sequences

The vocabulary is much smaller and the model sees the spelling of each character directly. But a 100-character sentence produces roughly 100 units instead of a few dozen subwords.

With standard self-attention, the pairwise attention component has time complexity that is quadratic in sequence length. That is one of the reasons character-level models pay a price for their more fine-grained representation.

3.3 Byte-level: a tiny base vocabulary, very strong coverage

UTF-8 represents text as bytes, and each byte has only 256 possible values. A truly byte-level architecture can start from a vocabulary of about 256 basic units instead of tens of thousands of tokens.

The strengths: almost no OOV at the byte level, it can represent Unicode, emoji, code and unusual strings, and there is no need to list every Unicode character in the base vocabulary. The weakness: multi-byte text can turn into very long sequences.

3.4 Subword: the pragmatic balance point

Subword sits between the two extremes:

unbelievable  →  ["un", "believ", "able"]
tokenization  →  ["token", "ization"]

The idea is very powerful: frequent strings are kept as large tokens, rare strings are broken down into smaller units, and a finite vocabulary can still represent a nearly open space of words. The work of Sennrich, Haddow and Birch did a lot to popularize subword BPE in neural machine translation, with the direct motivation of handling rare words and the open-vocabulary problem.

A useful way to look at it:

Word-level      : lots of memorization, short sequences
Subword-level   : balanced
Character/byte  : lots of composition, long sequences

Subword does not guarantee that the pieces are always linguistically “correct” morphemes. It learns from the data distribution. A meaningful-looking token like ing is a happy outcome, not a requirement.

04BPE: learn the vocabulary once, encode many times

BPE comes from a compression idea: find pairs of adjacent symbols that occur frequently and merge them.

Say the toy corpus is low, lower, lowest. We start at the character level:

l o w
l o w e r
l o w e s t

Count the adjacent pairs, find that (l, o) is the preferred pair, and merge:

lo w
lo w e r
lo w e s t

Then merge (lo, w):

low
low e r
low e s t

The process repeats until it reaches the desired number of merges or vocabulary size:

vocab = base_symbols

while len(vocab) < target_vocab_size:
    pair_counts = count_adjacent_pairs(corpus)
    best_pair = select_best_pair(pair_counts)
    corpus = merge(corpus, best_pair)
    vocab.add(concat(best_pair))

The most important point here: training BPE and encoding with BPE are two completely different phases.

After training, the tokenizer does not need to recount the whole corpus every time it encodes a sentence. It just applies the learned vocabulary and merge priorities to the new input. In tiktoken’s educational implementation, the input is converted to UTF-8 bytes, and BPE starts from individual bytes and keeps merging by the learned rank until no valid merge is left.

This is also why a tokenizer file stores the list of merges and not just the list of tokens: lose the merge ranks and you lose the ability to reproduce the tokenization.

4.1 BPE is compression, but compression is not the whole story

BPE tends to dedicate tokens to frequent strings, which makes sequences shorter. The string ing appears very often, so it has a good chance of becoming a token. But BPE does not know that ing is an English suffix in the linguistic sense - it only knows that the string is useful according to the statistics of the tokenizer’s training data.

So BPE can produce tokens that line up neatly with morphemes, tokens that are whole words, fragments that are hard to interpret, tokens that are very efficient for domains it saw a lot of, and tokens that are inefficient for rarely seen domains.

4.2 Byte-level BPE: solving coverage at the byte level

Classic BPE starts from characters or symbols. Byte-level BPE replaces that foundation with bytes, so any input can be represented without putting every emoji or every ideograph into the base vocabulary. tiktoken describes its BPE as reversible and lossless at the byte-representation level, able to handle arbitrary text, and compressing byte strings into fewer tokens.

This is a deep difference between two situations:

"this character is not in the vocabulary"          ← word/char-level
"no larger token matches, fall back to bytes"      ← byte-level

Byte-level BPE makes OOV in the traditional sense all but disappear at the raw-text level. However, full coverage does not mean optimal segmentation: a language can always be encoded and still be split into far too many tokens. We will come back to this in section 8.

05WordPiece, Unigram, and where SentencePiece really fits

When comparing subword algorithms, it helps to separate two questions clearly:

1. How does the tokenizer learn its vocabulary?
2. When it meets a new string, how does the tokenizer choose a segmentation?

These are not the same question, and most of the confusion comes from lumping them together.

5.1 WordPiece: longest match against a fixed vocabulary

WordPiece is also subword tokenization, but when encoding a word, the typical implementation uses greedy longest-match-first, also called MaxMatch.

Say the vocabulary contains play, ##ing, ##er, ##s and the input is playing. The tokenizer tries to take the longest prefix that is in the vocabulary:

playing   ✗
playin    ✗
play      ✓

Then it continues with the rest: ing → ##ing ✓. The result is ["play", "##ing"]. The ## marker denotes a continuation token inside a word, not a string that stands at the start of a word.

An important nuance: modern documentation describes WordPiece’s inference algorithm very clearly, while the history and variants of WordPiece training have not been standardized into a simple recipe like BPE’s “merge the most frequent pair”. When explaining WordPiece, the safest thing is to separate vocabulary learning from MaxMatch tokenization, rather than saying it is just “BPE with a different scoring formula”.

5.2 Unigram: start big, then prune

Unigram follows an intuition that is almost the reverse of BPE:

BPE      : small units → merge step by step → larger vocabulary
Unigram  : large candidate vocabulary → estimate probabilities → drop the least useful tokens

For the string tokenizer, many segmentations are valid:

["token", "izer"]
["tok", "en", "izer"]
["tokenize", "r"]

Unigram assigns a probability to each token, and the probability of a segmentation is the product of the probabilities of its tokens - or the sum of log-probabilities if you work in log space. Every path from start to end through a lattice is one way to tokenize, and Viterbi-style dynamic programming finds the path with the best score.

Being able to produce many segmentations also opens the door to subword regularization: during training, instead of always using exactly one split, we sample several segmentations so the model depends less on one fixed subword boundary. Kudo’s Unigram work studies exactly this idea.

5.3 SentencePiece is not “the fourth algorithm”

SentencePiece is often listed alongside BPE, WordPiece and Unigram as if the four were peers. That framing is misleading.

SentencePiece is a framework/toolkit that can implement BPE or Unigram, and it supports training directly from raw sentences. One notable design choice is how it keeps whitespace as a representable symbol, using the ▁ marker:

I saw a girl
→ ▁I ▁saw ▁a ▁girl

That way, whitespace boundary information is kept in the token sequence instead of being thrown away as a delimiter. This is especially useful for writing systems where whitespace is not a reliable word boundary - something that will matter a lot for Vietnamese in section 9.

5.4 Quick comparison

06Vocabulary, special tokens, and “words” that are not natural language

6.1 Vocabulary size is an architectural hyperparameter

The input embedding alone has |V| × d parameters. As an illustrative example:

|V| = 50,000
d   = 4,096
→ 204,800,000 elements

If each element is stored in 2 bytes in a 16-bit representation, this matrix alone is about 409.6 MB of raw data - before counting optimizer states, gradients and other implementation details. Raise the vocabulary to 200,000 with the same d and the number of rows goes up 4 times.

But a large vocabulary reduces sequence length, because more strings are kept whole as large tokens. This is a real trade-off, with no direction that is “the bigger the better”:

Recent research keeps showing that vocabulary size affects both efficiency and quality, especially in multilingual and low-resource settings.

6.2 Special tokens are a protocol between the data and the model

Not every model has all of these tokens.

EOS is especially important for generation. An autoregressive model generates one token at a time, and EOS is exactly the signal for generation to stop.

PAD is only a placeholder, not content. When batching two sequences of different lengths:

A = [21, 52, 91, 17]
B = [81, 42]
→ B = [81, 42, 0, 0]        (0 is the PAD ID)

mask A = [1, 1, 1, 1]
mask B = [1, 1, 0, 0]

6.3 A chat template turns a conversation into a token protocol

The chat interface shows:

User: Explain BPE
Assistant: ...

but the model receives a representation closer to:

<start_of_message><user>Explain BPE<end_of_message>
<start_of_message><assistant>

The actual token names depend on the model. The important point is that roles have to be encoded too: the model does not know which part is the user and which is the assistant just from the color of the chat bubbles in the UI. A modern example is OpenAI’s o200k_harmony tokenizer, which extends a BPE vocabulary with many control tokens reserved for the conversation format and tooling.

Special tokens are not accessories. They are part of a protocol the model was trained to understand - and they take up real space in the context window.

07The contract between a tokenizer and a pretrained model

This is the most important point if you want to understand why you cannot just “switch to a better tokenizer” for a pretrained model.

Suppose tokenizer A has "cat" → 4281, and the embedding table has learned row 4281 for the token "cat". Now we replace it with tokenizer B, where "house" → 4281. The model still fetches row 4281 - but that vector was learned during pretraining for the old token. The contract has been broken.

7.1 Adding a new token is not just adding a line to a JSON file

Say we add the token "ChatGPTVietnam" to the vocabulary. We need a new embedding for it, but that vector has never been learned by the model. Possible strategies include:

• random initialization, then continued training;
• initializing from a composition of existing tokens;
• embedding transfer or cross-lingual mapping;
• continual pretraining;
• dedicated vocabulary adaptation methods.

Research on vocabulary expansion shows that initializing the new embeddings is a real problem, not a minor detail.

7.2 The output vocabulary is involved too

An autoregressive LM ultimately produces logits over the vocabulary - each dimension corresponds to one token that could come next. So changing the vocabulary can affect not only the input embedding but also the output projection. Some models tie weights between the input embedding and the output projection and some do not, so changing the tokenizer means looking at the specific architecture; there is no single rule.

7.3 Token IDs have no semantic order

If cat → 4281 and dog → 912, we cannot conclude that cat is “greater than” dog. An ID is only a categorical index. Meaning shows up in the embeddings and the hidden representations after the model learns from data.

The practical consequence: comparing token IDs across two models is meaningless, and a tokenizer has to be versioned together with its checkpoint, never separately.

08Context window, compute and cost

This is why tokenization cannot be treated as harmless preprocessing.

8.1 The context window is measured in tokens, not words

If a model supports a context length of N tokens, that does not mean N words. A 1,000-word passage might become 1,200, 1,600 or 2,500 tokens depending on the tokenizer, the language, the script, the domain, the ratio of code to prose, numbers and punctuation, rare characters, and how much that domain appeared in the tokenizer’s training corpus.

In other words, the amount of useful information you can fit into the context equals the token budget divided by the average number of tokens per unit of information. The more tokens a tokenizer produces for the same amount of text, the less real information gets in.

8.2 Longer sequences make a Transformer work harder

In vanilla self-attention, the attention matrix over n tokens has size n × n. Going from n to 2n tokens multiplies the number of attention pairs by four.

But it needs to be stated precisely: the total cost of a Transformer does not always grow by exactly four times. The MLP, projections, KV caching, kernel implementations, sparse or local attention and modern architectures all scale differently. What is true at the foundational level is that standard pairwise attention has time complexity quadratic in sequence length; modern implementations can cut memory significantly without necessarily reducing the asymptotic compute.

8.3 Token tax: the same content, two different bills

When a service charges by the number of input and output tokens, the same content in a tokenizer or a language that produces more tokens costs more, in direct proportion. An EMNLP 2023 study across 22 languages showed that tokenization creates significant cost differences between languages in commercial language model services. Another 2023 study reported that the same content translated into different languages can have very different token lengths, with large gaps even in multilingual tokenizers.

For readers outside Vietnam, it helps to see why Vietnamese sits on the expensive side, as part 1 already touched on: a BPE vocabulary spends its merges on the patterns that are most frequent in its training corpus, which is usually dominated by English, and Vietnamese adds diacritics that are multi-byte in UTF-8 plus short, space-separated syllables, so fewer merges ever cover a whole Vietnamese word. The same effect, to varying degrees, hits many languages written outside the Latin-ASCII range.

The point to stress is not that “tokenizer X is deliberately biased”, but this:

Frequency-based vocabulary learning reflects the distribution of the training data. Languages that appear less get fewer useful long tokens and are fragmented more.

Work from 2025 - 2026 keeps studying tokenizer fairness, fertility, and how to design BPE that is fairer across languages.

8.4 Fertility and the metrics that go with it

A commonly used definition is fertility: the number of tokens divided by the number of words. High fertility means each word is split into many tokens on average.

But for languages where the notion of a word boundary is complicated, this metric needs an appropriate reference word segmentation. You can add characters per token, bytes per token, compression ratio, normalized sequence length, single-token retention rate, OOV rate and cross-lingual parity. No single metric perfectly describes tokenizer quality on its own.

09Vietnamese case study: whitespace does not solve the problem

Vietnamese is a great example for seeing the difference between layers of boundary: syllable, word and subword, all sitting on top of Unicode characters.

9.1 Vietnamese whitespace usually separates syllables, not necessarily words

In English, the whitespace in artificial intelligence is fairly close to a word boundary. In Vietnamese, trí tuệ nhân tạo (“artificial intelligence”, written as four syllables) does not mean those four units have to be treated as four independent lexical words.

Work on Vietnamese word segmentation points out that many Vietnamese words are made of several syllables, while whitespace is also used to separate the very syllables that make up a word. So finding word boundaries is not as simple as split(" "). For example, sinh viên (“student”) has one space but is usually treated as a single multi-syllable compound word in many NLP pipelines.

9.2 Word segmentation and subword tokenization are two different layers

PhoBERT is a well-known example: its entire pretraining corpus is run through a Vietnamese word segmenter (VnCoreNLP’s RDRSegmenter) first, and only then applies BPE to build subwords from the word tokens. Another system might drop the word segmentation layer entirely and train SentencePiece or BPE straight from raw text.

The two designs create different inductive biases. Neither is wrong, but you have to know which one you are choosing - and stay consistent between training and serving.

9.3 Vietnamese diacritics make Unicode normalization worth caring about

Vietnamese uses a lot of diacritics. A character that displays the same way can have different Unicode representations depending on whether it is built from a precomposed character or a base character plus combining marks.

If the tokenizer was trained on one form but production input usually arrives in the other, fragmentation can increase or the token mapping can change. For Vietnamese, the pipeline should be consistent about Unicode normalization, and the team needs to know the tokenizer’s specific normalizer instead of just calling .encode() and assuming that all equivalent Unicode text gives the same result.

9.4 A multilingual vocabulary is a budget allocation problem

If a vocabulary has 100,000 slots and the model has to serve dozens of languages, then every long token dedicated to an English pattern is a slot not given to some other pattern.

You should not take this too mechanically, as if each language had its own “share of the vocabulary”, because tokens can be shared across languages. But statistically, the training corpus and the merge objective decide which patterns are common enough to earn a token of their own. Research on multilingual tokenization shows that the vocabulary’s coverage and overlap across languages affect downstream performance, and that tokenization efficiency is uneven across language groups.

9.5 What to check when evaluating a tokenizer for Vietnamese

 1. Tokens per sentence
 2. Tokens per word after standard word segmentation
 3. Characters or bytes per token
 4. Share of common words split into several pieces
 5. Person names, place names, compound words, abbreviations
 6. Unicode in composed and decomposed form
 7. Text with and without diacritics
 8. Vietnamese ↔ English code-switching
 9. URLs, phone numbers, product codes
10. Social media text with typos and teencode

(Teencode is the playful, abbreviated spelling Vietnamese teenagers use online.)

A tokenizer that does well on a news corpus is not necessarily good on chat, source code, or e-commerce product names.

10The hard areas: numbers, code, typos and counting characters

A model can be very good at semantic reasoning and still stumble on a few operations that are “easy” for humans. Tokenization is part of the reason.

10.1 Numbers are not necessarily represented digit by digit

The string 123456789 might be tokenized as ["123", "456", "789"], or ["12", "345", "678", "9"], or some other way depending on the vocabulary. The model does not get a digit-by-digit decimal representation by default, so there is a mismatch between the mathematical structure of a number and the token units the network sees.

10.2 Source code has different patterns from prose

Code contains snake_case, camelCase, ::, ->, ==, !=, indentation and newlines. A tokenizer trained on a lot of code can dedicate tokens to frequent patterns, reducing sequence length significantly.

def calculate_total(price, quantity):
    return price * quantity

The snippet above might be split around keywords, identifier fragments, parentheses, commas, whitespace and operators. Whether _, indentation or newlines are kept or split affects how well the model can learn syntax patterns.

10.3 Typos change boundaries abruptly

A common word can be a single token:

language   → 1 token
langauge   → several rare pieces

A single small character swap is enough to change the discrete representation dramatically. Research on the “curse of tokenization” shows that LLMs are still sensitive to typos, surface variations and internal token structure; increasing model size can reduce the problem but does not eliminate it entirely.

10.4 Why LLMs are sometimes bad at counting characters

When asked how many rs are in a word, a human operates directly on the character string. An LLM first sees subword token IDs:

human : character → operation
LLM   : text → subwords/bytes → embeddings → hidden states → reasoning

The path is longer, and token boundaries sometimes hide character-level structure. This does not mean “LLMs cannot count characters”; it only explains why character-exact tasks are different in nature from semantic completion.

11Encode, decode, and the question “is a tokenizer lossless?”

We often write decode(encode(text)) and expect to get exactly text back. The answer depends on the whole pipeline, not just the token-splitting step.

11.1 Byte-level BPE can be reversible at the byte level

tiktoken designs BPE over bytes and describes the encoding as reversible for text represented as bytes. The core API even distinguishes decoding to bytes from decoding to text, because a single token on its own can contain bytes that do not form valid UTF-8 when separated from its neighboring tokens.

This is a really interesting nuance: token boundaries do not necessarily line up with UTF-8 character boundaries.

11.2 Normalization can make the round trip no longer byte-identical

If, before tokenization, we deliberately lowercase, run NFKC, strip whitespace or replace characters, then information has been transformed before segmentation even happens. Compatibility normalization can erase presentation distinctions, exactly as Unicode warns.

So it is worth separating two questions clearly:

1. Is the segmentation reversible?
2. Is the whole preprocessing pipeline reversible with respect to the original raw input?

The answer to question 1 is usually yes. The answer to question 2 is usually no.

11.3 A tokenizer’s decode is not a Transformer’s decoder

A tokenizer’s decoder only maps IDs → token pieces/bytes → text. It does not use a Transformer and does not “think” about anything. Do not confuse tokenizer decoding with the neural decoder block in an encoder-decoder architecture: the two uses of “decode” live at completely different layers.

12Evaluating a tokenizer, and a practical checklist

A good tokenizer cannot be judged by looking at a few sentences and saying “the tokens look nice”. We need metrics and workloads.

12.1 Seven dimensions worth measuring

Two caveats come with this. First, the tokenizer with the shortest sequences does not necessarily win: the units it learns can make it harder for the model to generalize. Second, “linguistically pretty tokens” are not the only goal - a tokenizer trained on clean prose can fragment logs, code or product names very heavily.

12.2 Fifteen questions to ask when using a pretrained LLM

When I pick up a new model, I do not just ask “what is the tokenizer called”, but:

 1. What is the base unit - character, byte, or some other symbol?
 2. What is the segmentation algorithm - BPE, WordPiece, Unigram, or a variant?
 3. Which regex/rules does the pre-tokenizer use? Does it allow merges across whitespace?
 4. What does the normalizer do - NFC/NFKC? lowercase? strip accents?
 5. How large is the vocabulary, and how many of those are special/control tokens?
 6. How does the unknown fallback work - UNK or byte fallback?
 7. How is whitespace represented - a delimiter, a prefix space, or its own symbol?
 8. Which special tokens carry protocol semantics - BOS/EOS/role/tool tokens?
 9. What is the chat template - how does the UI's raw string become a token sequence?
10. How is the context budget counted - do prompt, system, tools and output all eat the same budget?
11. Is the tokenizer efficient for the target language - measure, don't guess?
12. Do code, numbers and URLs get heavily fragmented?
13. Does the tokenizer exactly match the model checkpoint?
14. If tokens are added, how are the new embeddings initialized and trained?
15. Does encode/decode preserve the raw text - including normalization?

If you are building a new tokenizer, add five more questions: is the training corpus balanced across languages and domains; is the vocabulary allocation dominated by high-resource languages; does the compression metric trade off downstream quality; how fair is it across languages; and is the tokenizer versioned together with the model checkpoint?

12.3 Ten misconceptions worth clearing up

A tokenizer is a discrete compression layer in front of the Transformer

There is an intuition I find very useful: a tokenizer turns a very long byte sequence into a shorter sequence of symbol IDs, by dedicating vocabulary entries to frequent patterns.

But this is not pure ZIP-style compression. A tokenizer has to balance many goals at once: compression, coverage, vocabulary budget, language coverage, domain coverage, compositionality, learnability for the model, and inference efficiency.

If you optimize only for maximum compression, you create very large tokens that are too rare, which makes it hard for the model to share statistics. If you optimize only for the smallest compositional units, you are back to bytes and sequences get long. That is why subword tokenization succeeded: it creates a structured bottleneck between memorization and composition.

One more thing worth remembering: a tokenizer freezes a view of the world at pretraining time. A static vocabulary learned from this year’s data does not know the lexical trends of years to come. New product names, new slang, new abbreviations and new domains can all be heavily fragmented. Recent research has started to treat tokenizer adaptation and dynamic tokenization as one way to loosen this rigidity. Whether the static tokenizer is the final discrete interface of LLMs, or whether the future moves to byte/character latent models - that question is still open.

In short, what to take away:

• a token is not a word, and a token ID is not a semantic value;
• subword tokenization is a balancing act between vocabulary and sequence length;
• BPE, WordPiece and Unigram have different mechanisms, while SentencePiece is a framework;
• Unicode normalization and pre-tokenization can change the result dramatically;
• a tokenizer and a pretrained model are bound together through the embeddings and the output vocabulary;
• tokenization directly affects context, latency and sometimes the bill;
• multilingual tokenization is not automatically fair across languages;
• for Vietnamese, syllables, words and subwords are three layers of boundary that need to be kept clearly apart.

And finally, tokenization only gets us as far as a sequence of integers:

[921, 44, 7211, 3512, ...]

Those numbers are still not something a Transformer “understands” in the mathematical sense. Turning each token ID into a continuous vector in a high-dimensional space is the job of embeddings, a topic big enough to get its own part later in the series. But first there is a question further upstream: where does a tokenizer learn its vocabulary from, and what text are the trillions of training tokens actually made of? The next part of the series walks that whole road, from a raw web page to a training sample: datasets for LLMs.

References

• Sennrich, Haddow, Birch — Neural Machine Translation of Rare Words with Subword Units
• Kudo, Richardson — SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing · repository
• Kudo — Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates
• Google Research — A Fast WordPiece Tokenization System
• OpenAI — tiktoken README · educational BPE · core encoding
• Unicode Consortium — UAX #15: Unicode Normalization Forms
• Vaswani et al. — Attention Is All You Need
• Rabe, Staats — Self-attention Does Not Need O(n²) Memory
• Google Research — Efficient Sequence Modeling for On-Device ML
• Đinh et al. — Word Segmentation of Vietnamese Texts: a Comparison of Approaches
• Vu et al. — VnCoreNLP: A Vietnamese Natural Language Processing Toolkit
• Nguyen, Nguyen — PhoBERT: Pre-trained language models for Vietnamese
• Ahia et al. — Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models
• Petrov et al. — Language Model Tokenizers Introduce Unfairness Between Languages
• Limisiewicz, Balhar, Mareček — Tokenization Impacts Multilingual Language Modeling: Assessing Vocabulary Allocation and Overlap Across Languages
• Foroutan et al. — Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization
• Chiu — TokLens: A Multilingual Lens on Tokenizer Quality for LLMs
• Chai et al. — Tokenization Falling Short: On Subword Robustness in Large Language Models
• He, Zeng, Jiang — Pre-trained Models Perform the Best When Token Distributions Follow Zipf’s Law
• Mundra et al. — An Empirical Comparison of Vocabulary Expansion and Initialization Approaches For Language Models
• Yamaguchi, Villavicencio, Aletras — How Can We Effectively Expand the Vocabulary of LLMs with 0.01GB of Target Language Text?
• Feher, Vulić, Minixhofer — Retrofitting Large Language Models with Dynamic Tokenization
• Purason et al. — Teaching Old Tokenizers New Words: Efficient Tokenizer Adaptation for Pretrained Models