Token tax: same content, two different bills

Token tax: same content, two different bills A data-flow diagram generated by Archify. 01 / Same content 02 / What the tokenizer has seen 03 / Actual token count 04 / The price you pay The same passage · about 1,000 words of content · 01 / Same content · not yet tokenized The same passage about 1,000 words of content not yet tokenized High-resource · frequent in the corpus · 02 / What the tokenizer has seen High-resource frequent in the corpus Low-resource · rare in the corpus · 02 / What the tokenizer has seen Low-resource rare in the corpus Fewer tokens · low fertility · 03 / Actual token count Fewer tokens low fertility More tokens · high fertility · 03 / Actual token count More tokens high fertility Context to spare · fewer decode steps, cheaper · 04 / The price you pay Context to spare fewer decode steps, cheaper Context runs out early · more decode steps, pricier · 04 / The price you pay Context runs out early more decode steps, pricier same content same content compact shredded small n large n Legend primary data policy / PII data store data flow