Evaluating a tokenizer: one number is not enough Diagram

No single metric captures quality

Best compression is not automatically the best tokenizer
Fertility needs a trustworthy reference word segmentation
Linguistically neat tokens are not the only goal either

Measure on the text you will actually run

A tokenizer good on news may shatter on logs and product names
Code, numbers, URLs, and identifiers differ greatly from prose
Social media text with typos and slang is a workload of its own

A minimum suite to run before you commit

Tokens per sentence, bytes per token, share of common words split
Proper nouns, compounds, abbreviations, text with and without diacritics
Both Unicode forms, plus Vietnamese ↔ English code-switching