Four ways to stack layers: what grows with context, what stays fixed

Four ways to stack layers: what grows with context, what stays fixed A workflow diagram generated by Archify. 01 / Dense Transformer (Llama 3.1 8B) 02 / MoE Transformer (Mixtral, DeepSeek-V3) 03 / Pure SSM (Mamba) 04 / Hybrid (Jamba, Qwen3-Next, Nemotron) KV cache · every layer, grows with n · Dense Transformer (Llama 3.1 8B) KV cache every layer, grows with n Attention · in every layer · Dense Transformer (Llama 3.1 8B) Attention in every layer Dense FFN · all parameters run · Dense Transformer (Llama 3.1 8B) Dense FFN all parameters run KV cache · same as dense · MoE Transformer (Mixtral, DeepSeek-V3) KV cache same as dense Attention · in every layer · MoE Transformer (Mixtral, DeepSeek-V3) Attention in every layer MoE FFN · k of E experts run · MoE Transformer (Mixtral, DeepSeek-V3) MoE FFN k of E experts run State · fixed, not tied to n · Pure SSM (Mamba) State fixed, not tied to n Mamba block · in every layer · Pure SSM (Mamba) Mamba block in every layer MLP (model-specific) · dense or MoE · Pure SSM (Mamba) MLP (model-specific) dense or MoE Small KV cache · attention layers only · Hybrid (Jamba, Qwen3-Next, Nemotron) Small KV cache attention layers only Attention · 1 in every 4–8 layers · Hybrid (Jamba, Qwen3-Next, Nemotron) Attention 1 in every 4–8 layers Mamba / DeltaNet · all other layers · Hybrid (Jamba, Qwen3-Next, Nemotron) Mamba / DeltaNet all other layers MoE FFN · usually MoE · Hybrid (Jamba, Qwen3-Next, Nemotron) MoE FFN usually MoE State · fixed per SSM layer · Hybrid (Jamba, Qwen3-Next, Nemotron) State fixed per SSM layer write K, V write K, V update write K, V update Legend FFN or MoE attention SSM or linear attention KV cache: grows with context length state: fixed size