Encode, then decode: what stays lossless and what doesn't?

Encode, then decode: what stays lossless and what doesn't? A sequence diagram generated by Archify. raw bytes, untouched normalized string — this step can be lossy input_ids — reversible at the byte level token IDs generated one step at a time readable text, reassembled from bytes Encode Generate tokens Decode Original text · bytes exactly as typed · Sequence participant Original text bytes exactly as typed Normalizer · NFC/NFKC, case, whitespace · Sequence participant Normalizer NFC/NFKC, case, whitespace Tokenizer · byte-level BPE · Sequence participant Tokenizer byte-level BPE Transformer · logits over |V| · Sequence participant Transformer logits over |V| Detokenizer · IDs → bytes → text · Sequence participant Detokenizer IDs → bytes → text Legend main request return security message