Vietnamese: syllables, words, and subwords are three different layers

Vietnamese: syllables, words, and subwords are three different layers A data-flow diagram generated by Archify. 01 / Text 02 / Unicode 03 / Word boundaries 04 / Subword 05 / Token IDs trí tuệ nhân tạo · (artificial intelligence) · 01 / Text · 4 syllables, not necessarily 4 words trí tuệ nhân tạo (artificial intelligence) 4 syllables, not necessarily 4 words Two Unicode forms · precomposed vs combining marks · 01 / Text Two Unicode forms precomposed vs combining marks NFC normalization · collapse to a single form · 02 / Unicode · must be consistent NFC normalization collapse to a single form must be consistent Word segmenter · trí_tuệ nhân_tạo · 03 / Word boundaries Word segmenter trí_tuệ nhân_tạo Skip the word layer · train directly on raw text · 03 / Word boundaries Skip the word layer train directly on raw text BPE on word tokens · PhoBERT-style design · 04 / Subword BPE on word tokens PhoBERT-style design SentencePiece · ▁trí ▁tuệ ▁nhân ▁tạo · 04 / Subword SentencePiece ▁trí ▁tuệ ▁nhân ▁tạo Subword IDs · two paths, two results · 05 / Token IDs Subword IDs two paths, two results syllables one form compounds IDs looks the same second design raw text also yields IDs Legend primary data data store data flow