Three ways to place the norm around a sub-layer

Three ways to place the norm around a sub-layer A workflow diagram generated by Archify. 01 / Post-LN: Transformer 2017, BERT 02 / Pre-LN: GPT-2, Llama, Qwen 03 / Norm before and after: Gemma 2 x · into the sub-layer · Post-LN: Transformer 2017, BERT x into the sub-layer Sub-layer · Attn or FFN · Post-LN: Transformer 2017, BERT Sub-layer Attn or FFN x + Sub(x) · residual add · Post-LN: Transformer 2017, BERT x + Sub(x) residual add LayerNorm · sits on the stream · Post-LN: Transformer 2017, BERT LayerNorm sits on the stream x · into the sub-layer · Pre-LN: GPT-2, Llama, Qwen x into the sub-layer RMSNorm · on the branch only · Pre-LN: GPT-2, Llama, Qwen RMSNorm on the branch only Sub-layer · Attn or FFN · Pre-LN: GPT-2, Llama, Qwen Sub-layer Attn or FFN x + Sub(Norm(x)) · stream is not normalized · Pre-LN: GPT-2, Llama, Qwen x + Sub(Norm(x)) stream is not normalized x · into the sub-layer · Norm before and after: Gemma 2 x into the sub-layer RMSNorm · normalizes the input · Norm before and after: Gemma 2 RMSNorm normalizes the input Sub-layer · Attn or FFN · Norm before and after: Gemma 2 Sub-layer Attn or FFN RMSNorm · normalizes the output · Norm before and after: Gemma 2 RMSNorm normalizes the output x + Norm(Sub(…)) · residual add · Norm before and after: Gemma 2 x + Norm(Sub(…)) residual add residual straight through normalizes the stream residual residual straight through Legend sub-layer: attention or FFN norm residual stream residual add