One request through an inference server: prefill, decode, and common misconceptions

One request through an inference server: prefill, decode, and common misconceptions A workflow diagram generated by Archify. EX / Misconceptions about prefill and decode 02 / Path of one request (main) 03 / Server-side mechanisms EX / Misconceptions about KV cache and speed 128k = uses 128k well · effective context is shorter · Misconceptions about prefill and decode 128k = uses 128k well effective context is shorter Bigger batch is better · TPOT and KV set limits · Misconceptions about prefill and decode Bigger batch is better TPOT and KV set limits Cost is mostly O(n²) · KV and bandwidth · Misconceptions about prefill and decode Cost is mostly O(n²) KV and bandwidth Decode lacks compute · it lacks bandwidth · Misconceptions about prefill and decode Decode lacks compute it lacks bandwidth Prompt → tokens · tokenize, may be very long · Path of one request (main) Prompt → tokens tokenize, may be very long Queue · wait for a batch slot · Path of one request (main) Queue wait for a batch slot Prefill · compute-bound · TTFT · Path of one request (main) Prefill compute-bound · TTFT Decode loop · memory-bound · TPOT · Path of one request (main) Decode loop memory-bound · TPOT Stream tokens · until EOS · Path of one request (main) Stream tokens until EOS Free blocks · or keep the prefix · Path of one request (main) Free blocks or keep the prefix Continuous batching · admit requests every step · Server-side mechanisms Continuous batching admit requests every step Paged KV cache · block table, shared prefix · Server-side mechanisms Paged KV cache block table, shared prefix Attention kernel · FlashAttention, tiling · Server-side mechanisms Attention kernel FlashAttention, tiling Speculative (optional) · draft k tokens, verify once · Server-side mechanisms Speculative (optional) draft k tokens, verify once GQA cuts FLOPs · only cuts KV cache · Misconceptions about KV cache and speed GQA cuts FLOPs only cuts KV cache Cache stores Q, scores · only K and V · Misconceptions about KV cache and speed Cache stores Q, scores only K and V FlashAttn is approximate · exact, fewer bytes · Misconceptions about KV cache and speed FlashAttn is approximate exact, fewer bytes Speculative alters output · same distribution · Misconceptions about KV cache and speed Speculative alters output same distribution Caches the answer · caches prefix K/V · Misconceptions about KV cache and speed Caches the answer caches prefix K/V write K, V read K, V many tokens/round Legend compute step on the GPU request in, tokens out scheduler KV cache in GPU memory common misconception