One round of speculative decoding: draft k tokens, verify in one forward

One round of speculative decoding: draft k tokens, verify in one forward A sequence diagram generated by Archify. current prefix k draft tokens, k small decode steps prefix + k tokens, one forward q at k positions p at k + 1 positions accept token i with prob. min(1, p/q) rejected: resample from max(0, p − q) all k accepted: 1 bonus token from p drop KV of rejected tokens, next round Draft Verify Accept Decode loop · holds prefix x₁…xₜ · Sequence participant Decode loop holds prefix x₁…xₜ Draft model · small, distribution q · Sequence participant Draft model small, distribution q Target model · large, distribution p · Sequence participant Target model large, distribution p Verifier · rejection sampling · Sequence participant Verifier rejection sampling Legend target model forward result returned side data, KV cleanup send data