Transformer: a high-level map

Transformer: a high-level map An architecture diagram generated by Archify. Attention · relations between tokens · Architecture component · the heart Attention relations between tokens the heart Pretraining · learn from massive data · Architecture component Pretraining learn from massive data Encoder-only · understand input - BERT, RoBERTa · Architecture component Encoder-only understand input - BERT, RoBERTa Decoder-only · generate tokens - GPT, Llama · Architecture component Decoder-only generate tokens - GPT, Llama Encoder-Decoder · seq-to-seq - T5, BART · Architecture component Encoder-Decoder seq-to-seq - T5, BART Classification, NER · extractive QA · Architecture component Classification, NER extractive QA Chat, writing, code · text generation · Architecture component Chat, writing, code text generation Translation, summary · generative QA · Architecture component Translation, summary generative QA contextual representations masked language modeling causal language modeling denoising / span corruption head on [CLS] / token LM head + sampling decoder + cross-attention Legend Architecture family Training stage Core mechanism Task group