The SwiGLU FFN in Llama 3 8B: three matrices, one gate

The SwiGLU FFN in Llama 3 8B: three matrices, one gate A workflow diagram generated by Archify. 01 / Gate branch: how much to keep 02 / Up and down: what gets written x · W_gate · 4096 → 14336 · Gate branch: how much to keep x · W_gate 4096 → 14336 SiLU · z · σ(z) · Gate branch: how much to keep SiLU z · σ(z) x after RMSNorm · 4096 dims · Up and down: what gets written x after RMSNorm 4096 dims x · W_up · 4096 → 14336 · Up and down: what gets written x · W_up 4096 → 14336 ⊙ element-wise · 14336 dims · Up and down: what gets written ⊙ element-wise 14336 dims · W_down · 14336 → 4096 · Up and down: what gets written · W_down 14336 → 4096 Add to stream · 4096 dims · Up and down: what gets written Add to stream 4096 dims gate content Legend parameter-free operation residual stream weight matrix