How a sentence pair becomes encoder input

How a sentence pair becomes encoder input A data-flow diagram generated by Archify. 01 / Sentence pair 02 / Token 03 / Three number sequences 04 / Embedding 05 / Encoder Sentence A + B · two sentences to compare · 01 / Sentence pair · paraphrase detection Sentence A + B two sentences to compare paraphrase detection Token sequence · [CLS] A [SEP] B [SEP] · 02 / Token · subword + special token Token sequence [CLS] A [SEP] B [SEP] subword + special token input_ids · [101, 7592, 2088, 102] · 03 / Three number sequences · just indices input_ids [101, 7592, 2088, 102] just indices attention_mask · 1 real token / 0 padding · 03 / Three number sequences · used in attention attention_mask 1 real token / 0 padding used in attention token_type_ids · 0 for A / 1 for B · 03 / Three number sequences · some models only token_type_ids 0 for A / 1 for B some models only Embedding · sum of three embeddings · 04 / Embedding · one vector per token Embedding sum of three embeddings one vector per token Encoder · hidden states [n × d] · 05 / Encoder · bidirectional context Encoder hidden states [n × d] bidirectional context h_cls · whole-sequence vector · 05 / Encoder · into classification head h_cls whole-sequence vector into classification head subword split tokenization vocabulary lookup lookup mark real tokens masking mark sentence A or B segment embedding lookup lookup add segment embedding segment x_i into the first layer compute add -∞ before softmax attention take hidden state at position 0 pooling Legend primary data policy / PII async batch data store