From logits to the next token

From logits to the next token A workflow diagram generated by Archify. 01 / Shape distribution 02 / Prune candidates 03 / Pick token EX / Stop condition Adjust logits To probabilities Pick and repeat Logits · raw scores over vocab · Shape distribution › Adjust logits Logits raw scores over vocab Penalty · presence + frequency · Shape distribution › Adjust logits · less repetition Penalty presence + frequency less repetition Temperature · divide logits by T · Shape distribution › Adjust logits Temperature divide logits by T Softmax · to probabilities · Shape distribution › To probabilities Softmax to probabilities Top-k · keep top k tokens · Prune candidates › To probabilities Top-k keep top k tokens Top-p · keep mass p · Prune candidates › Pick and repeat Top-p keep mass p Sampling · draw 1 token · Pick token › Pick and repeat Sampling draw 1 token Next token · append to sequence · Pick token › Pick and repeat Next token append to sequence Stop generating · EOS / max / stop seq · Stop condition › Pick and repeat · stop sequence Stop generating EOS / max / stop seq stop sequence penalize repeats divide by T hit EOS? pruned distribution not done, loop again full distribution k candidates weighted draw adjusted logits Legend Logit transform Filtering and stop condition Probability step Output token