Tokenizer pipeline: from a typed sentence to token IDs Diagram

A tokenizer is not a single function

Each of the four steps is configured independently
Changing the normalizer or pre-tokenizer regex changes the output
Same BPE algorithm, two pipelines can still yield different tokens

The bottom row shapes the top row

The pre-tokenizer sets boundaries that merges cannot cross
The vocabulary is learned once in training, then only looked up
Special tokens take up real space in the context window

Common misconceptions

A token ID is just a row address; it carries no meaning by itself
Normalization runs before segmentation, so it can lose original information
Token boundaries need not align with UTF-8 character boundaries