One token through an MoE layer, and common misconceptions

One token through an MoE layer, and common misconceptions A workflow diagram generated by Archify. EX / Misconceptions about routers and experts 02 / One token through one layer (main) 03 / Supporting mechanisms EX / Misconceptions about memory, batching, and SSMs Mamba replaced Transformers · hybrids keep attention · Misconceptions about routers and experts Mamba replaced Transformers hybrids keep attention Routes per sentence · per token, per layer · Misconceptions about routers and experts Routes per sentence per token, per layer Speed = active params · only at small batch · Misconceptions about routers and experts Speed = active params only at small batch Experts by topic · follow syntax, position · Misconceptions about routers and experts Experts by topic follow syntax, position 47B as light as 13B · VRAM still holds 47B · Misconceptions about routers and experts 47B as light as 13B VRAM still holds 47B Attention · unchanged from dense · One token through one layer (main) Attention unchanged from dense Router · softmax(x·W_r), per token · One token through one layer (main) Router softmax(x·W_r), per token Top-k and gate · keep k experts, renormalize · One token through one layer (main) Top-k and gate keep k experts, renormalize Dispatch · all-to-all across GPUs · One token through one layer (main) Dispatch all-to-all across GPUs k expert FFNs · only k of E run · One token through one layer (main) k expert FFNs only k of E run Combine + residual · sum weighted by gates · One token through one layer (main) Combine + residual sum weighted by gates KV cache · grows with context length · Supporting mechanisms KV cache grows with context length Load balancing · aux loss or bias · Supporting mechanisms Load balancing aux loss or bias Expert full · token is dropped · Supporting mechanisms Expert full token is dropped SSMs recall all · fixed state, lossy · Misconceptions about memory, batching, and SSMs SSMs recall all fixed state, lossy MoE shrinks KV cache · MoE only replaces FFN · Misconceptions about memory, batching, and SSMs MoE shrinks KV cache MoE only replaces FFN Balance = fairness · avoids collapse, even GPUs · Misconceptions about memory, batching, and SSMs Balance = fairness avoids collapse, even GPUs Batch-independent · capacity can drop tokens · Misconceptions about memory, batching, and SSMs Batch-independent capacity can drop tokens write K, V count load residual only Legend compute step dropped-token branch router and token dispatch memory: KV cache common misconception