One string, three ways to choose a segmentation

One string, three ways to choose a segmentation A data-flow diagram generated by Archify. 01 / String to split 02 / Deciding mechanism 03 / Typical result tokenizer · one string, many splits · 01 / String to split · no single right answer tokenizer one string, many splits no single right answer BPE · replay merges by rank · 02 / Deciding mechanism BPE replay merges by rank WordPiece · longest match, left to right · 02 / Deciding mechanism WordPiece longest match, left to right Unigram · Viterbi over a lattice · 02 / Deciding mechanism Unigram Viterbi over a lattice token + izer · merge order decides · 03 / Typical result token + izer merge order decides token + ##izer · ## marks a word-internal piece · 03 / Typical result token + ##izer ## marks a word-internal piece highest-probability path · other splits can be sampled · 03 / Typical result highest-probability path other splits can be sampled from bytes in-word whole lattice fixed fixed sample Legend primary data data store data flow