Attention: how one token is computed

Attention: how one token is computed A data-flow diagram generated by Archify. 01 / Input token 02 / Q / K / V projection 03 / Affinity scores 04 / Attention weights 05 / Output representation Hidden state · token + position vector · 01 / Input token Hidden state token + position vector Query (Q) · what this token asks · 02 / Q / K / V projection Query (Q) what this token asks Key (K) · what gets matched · 02 / Q / K / V projection Key (K) what gets matched Value (V) · content carried · 02 / Q / K / V projection Value (V) content carried Affinity scores · QK^T / sqrt(d_k) · 03 / Affinity scores · decoder mask here Affinity scores QK^T / sqrt(d_k) decoder mask here Softmax · weights 0..1 · 04 / Attention weights Softmax weights 0..1 Multi-Head · concat n heads, times W_O · 05 / Output representation · one relation per head Multi-Head concat n heads, times W_O one relation per head times W_Q times W_K times W_V vs. every Key query matching key matching to distribution weights sum V by w aggregate content of V content Legend Main path Supporting branch