From logits to one token: the pipeline and common pitfalls

From logits to one token: the pipeline and common pitfalls A workflow diagram generated by Archify. 01 / What runs at every generation step EX / Common misconceptions Logits · one score per token · What runs at every generation step Logits one score per token Penalties · push down seen tokens · What runs at every generation step Penalties push down seen tokens Grammar mask · invalid tokens to -inf · What runs at every generation step Grammar mask invalid tokens to -inf Temperature · divide logits by T · What runs at every generation step Temperature divide logits by T Top-k, top-p, min-p · cut the tail · What runs at every generation step Top-k, top-p, min-p cut the tail Softmax → pick · argmax or sample · What runs at every generation step Softmax → pick argmax or sample Penalty = divide · negative logits multiply · Common misconceptions Penalty = divide negative logits multiply JSON mode is enough · only valid JSON · Common misconceptions JSON mode is enough only valid JSON T = 0 is repeatable · logits shift with batch · Common misconceptions T = 0 is repeatable logits shift with batch Top-p keeps fixed N · N follows the shape · Common misconceptions Top-p keeps fixed N N follows the shape Fixed seed is enough · batch, backend still vary · Common misconceptions Fixed seed is enough batch, backend still vary append the token, run forward Legend logit transform step hard mask: grammar or schema data: logits and the chosen token common misconception