Tokenizer: from character string to model input tensor

Tokenizer: from character string to model input tensor A data-flow diagram generated by Archify. 01 / Text 02 / Token 03 / Token ID 04 / Batch tensor 05 / Model Raw text · character string · 01 / Text · human-readable Raw text character string human-readable Subword · BPE / WordPiece · 02 / Token · balanced Subword BPE / WordPiece balanced Special token · [CLS] [SEP] [PAD] · 02 / Token · per checkpoint Special token [CLS] [SEP] [PAD] per checkpoint Token IDs · [13809, 2545, ...] · 03 / Token ID · just indices Token IDs [13809, 2545, ...] just indices Truncation · max_length · 03 / Token ID · loses information Truncation max_length loses information Padding · same length · 04 / Batch tensor · right shape Padding same length right shape Attention mask · 1 real / 0 pad · 04 / Batch tensor · right meaning Attention mask 1 real / 0 pad right meaning Tensor · [batch, seq_len] · 05 / Model · return_tensors Tensor [batch, seq_len] return_tensors Embedding · [batch, seq, hidden] · 05 / Model · real-valued vectors Embedding [batch, seq, hidden] real-valued vectors normalize + split encoding wrap in format tokenizer rule vocabulary lookup lookup own IDs lookup group into a batch batching exceeds context limit truncate, then pad batching mark positions attention input_ids tensor attention_mask tensor embedding table lookup model input Legend primary data policy / PII async batch data store