Choosing granularity: how far should you split?

Choosing granularity: how far should you split? A workflow diagram generated by Archify. 01 / Granularity, from large units to small units EX / The cost at both ends Large units Balanced zone Small units Word-level · split on whitespace · Granularity, from large units to small units › Large units · short sequences Word-level split on whitespace short sequences Subword · BPE · WordPiece · Unigram · Granularity, from large units to small units › Balanced zone · pragmatic choice Subword BPE · WordPiece · Unigram pragmatic choice Character-level · each Unicode character · Granularity, from large units to small units › Balanced zone Character-level each Unicode character Byte-level · 256 UTF-8 byte values · Granularity, from large units to small units › Small units · almost no OOV Byte-level 256 UTF-8 byte values almost no OOV Vocabulary blowup + OOV · unseen words become <UNK> · The cost at both ends › Large units Vocabulary blowup + OOV unseen words become <UNK> Longer sequences · pairwise attention costs n² · The cost at both ends › Small units Longer sequences pairwise attention costs n² split finer split finer split finer fall back to subword also fall back to subword unseen word multibyte text Legend Agent logic Context / trace External system