A generation request through an inference server

A generation request through an inference server A sequence diagram generated by Archify. prompt + sampling params enqueue prefill whole prompt write prompt's K, V first-step logits first token decode one token reuse cached K, V stream next token new request joins EOS hit, close stream Prefill — measured by TTFT Decode — one token per step Continuous batching Client · chat API · Sequence participant Client chat API Server · TGI / vLLM · Sequence participant Server TGI / vLLM Scheduler · batching · Sequence participant Scheduler batching GPU · attention · Sequence participant GPU attention KV cache · paged blocks · Sequence participant KV cache paged blocks Sampling · top-p / temp · Sequence participant Sampling top-p / temp Legend main request return async trace message