BPE: learn the vocabulary once, encode many times

BPE: learn the vocabulary once, encode many times A workflow diagram generated by Archify. 01 / Tokenizer training phase — runs once on the corpus 02 / Encode phase — runs for every new sentence Merge loop Freeze, then reuse Base symbols · bytes or characters · Tokenizer training phase — runs once on the corpus Base symbols bytes or characters Count adjacent pairs · across the whole corpus · Tokenizer training phase — runs once on the corpus › Merge loop Count adjacent pairs across the whole corpus Pick the best pair · by frequency or score · Tokenizer training phase — runs once on the corpus › Merge loop Pick the best pair by frequency or score Merge into a token · record the merge rank · Tokenizer training phase — runs once on the corpus › Merge loop Merge into a token record the merge rank Vocabulary + ranks · no more learning after this · Tokenizer training phase — runs once on the corpus › Freeze, then reuse · frozen Vocabulary + ranks no more learning after this frozen New sentence · never seen in the corpus · Encode phase — runs for every new sentence New sentence never seen in the corpus Back to base bytes · same symbol table as training · Encode phase — runs for every new sentence › Merge loop Back to base bytes same symbol table as training Apply merges by rank · no corpus recount · Encode phase — runs for every new sentence › Freeze, then reuse · very fast Apply merges by rank no corpus recount very fast Token IDs · deterministic result · Encode phase — runs for every new sentence › Freeze, then reuse Token IDs deterministic result reuse learned ranks normalize + split symbol sequence no valid merges left not yet, recount split corpus frequency table best pair vocab size reached Legend Agent logic Tool action Context / trace External system