The residual stream: a shared bus every sub-layer reads from and writes to

The residual stream: a shared bus every sub-layer reads from and writes to A workflow diagram generated by Archify. 01 / Sub-layer 1: attention 02 / Residual stream, always d_model wide 03 / Sub-layer 2: FFN Norm → Attention · mixes across tokens · Sub-layer 1: attention Norm → Attention mixes across tokens Embedding x₀ · 4096 dims in Llama 3 8B · Residual stream, always d_model wide Embedding x₀ 4096 dims in Llama 3 8B x₁ = x₀ + Attn(…) · attention has written · Residual stream, always d_model wide x₁ = x₀ + Attn(…) attention has written x₂ = x₁ + FFN(…) · end of one block · Residual stream, always d_model wide x₂ = x₁ + FFN(…) end of one block Blocks 2 … 32 · then final norm, lm_head · Residual stream, always d_model wide Blocks 2 … 32 then final norm, lm_head Norm → FFN · one token at a time · Sub-layer 2: FFN Norm → FFN one token at a time read read straight through straight through write (add) write (add) Legend sub-layer that reads then writes residual stream state later blocks