Three ways a router picks experts: collapse, aux loss, and loss-free bias

Three ways a router picks experts: collapse, aux loss, and loss-free bias A workflow diagram generated by Archify. 01 / No balancing: the router collapses 02 / Auxiliary loss (Switch Transformer) 03 / Loss-free bias (DeepSeek-V3) Router · LM-loss gradient only · No balancing: the router collapses Router LM-loss gradient only Top-k · pick the highest scores · No balancing: the router collapses Top-k pick the highest scores A few favorites · get most tokens · No balancing: the router collapses A few favorites get most tokens Learn faster · lower loss there · No balancing: the router collapses Learn faster lower loss there Router · LM + aux gradient · Auxiliary loss (Switch Transformer) Router LM + aux gradient Top-k · by probability p_i · Auxiliary loss (Switch Transformer) Top-k by probability p_i Count f_i and P_i · token share, mean prob · Auxiliary loss (Switch Transformer) Count f_i and P_i token share, mean prob Aux loss · α · N · Σ f_i · P_i · Auxiliary loss (Switch Transformer) Aux loss α · N · Σ f_i · P_i Score s_i · LM-loss gradient only · Loss-free bias (DeepSeek-V3) Score s_i LM-loss gradient only Pick by s_i + b_i · bias only for choosing · Loss-free bias (DeepSeek-V3) Pick by s_i + b_i bias only for choosing Gate from raw s_i · output ignores b_i · Loss-free bias (DeepSeek-V3) Gate from raw s_i output ignores b_i Count load c_i · after each batch · Loss-free bias (DeepSeek-V3) Count load c_i after each batch Update b_i · over −γ, under +γ · Loss-free bias (DeepSeek-V3) Update b_i over −γ, under +γ chosen even more extra gradient, may interfere outside the gradient Legend experts and load counting signal that enters the gradient router: scores and picks experts signal used only for selection, no gradient self-reinforcing loop without balancing