From Logits to Answers: Decoding and Sampling

Part 8 ended where the residual stream goes through the final norm and into lm_head to become logits: one number for every token in the vocabulary. For Qwen2.5, that is 151,936 numbers per generation step. The model does nothing after that. Turning 151,936 numbers into one specific token is the job of a separate step called decoding.

That sounds like a technical detail, but it explains most of what users actually see:

• Why does the same model sometimes answer cleanly and sometimes repeat the same sentence over and over?
• Why does “return JSON” in the prompt work most of the time, but every so often the parser throws an error?
• Why does setting temperature = 0 still give two different answers when you call it twice?

None of these questions is about the model’s weights. They are all about how we pick a token from the logits.

Part 1, sections 8.5 to 8.11 and part 2, section 09 already introduced greedy decoding, temperature, top-k, top-p and the penalties at the level of definitions. This post does not repeat that. It goes deeper in five places:

1. Decoding is a pipeline, and the order of the steps changes the result. We read the source of Hugging Face Transformers and vLLM to see the real order.
2. Why beam search suits machine translation but makes open-ended text repeat itself.
3. Top-k, top-p and min-p cut the tail of the distribution in three different ways, run on a real model so the differences show.
4. How constrained decoding forces a model to return JSON that matches a schema, and what that costs.
5. Three layers of causes that make the same prompt return two answers, even with sampling turned off.

01From logits to one token

Recall the loop that part 1 described: the model reads the whole context, returns logits for the next position, we pick one token, append it to the context, and run again. Each round produces exactly one token. Decoding is the function that sits between two model calls: it takes the logits vector of length vocab_size that the model just returned, and gives back exactly one token to append to the context for the next round.

This function has no learned parameters. It is just a selection rule, and there are two families of rules:

• Search: look for a high-probability token sequence. Greedy and beam search belong here. There is no randomness, so in theory the same input gives the same output.
• Sampling: reshape the distribution (temperature, truncation), then draw a token at random from it. This is what every commercial chatbot uses by default.

One example for the whole post

To keep things computable by hand, we use a small distribution with six candidates. Suppose the model is writing the Vietnamese sentence “Món tôi muốn ăn tối nay là” (“What I want for dinner tonight is”) and the logits of the six strongest tokens are as follows. These are made-up logits chosen to be easy to compute, not taken from a real model; the real-model run is in section 04.

The code below uses only the Python standard library. It reimplements the four transformations this post walks through (temperature, top-k, top-p, min-p) with the same convention the libraries use: a removed token gets a logit of -inf, then softmax runs over what is left.

import math

tokens = ["phở", "bún chả", "cơm tấm", "pizza", "sushi", "kem"]
logits = [3.0, 2.4, 2.0, 1.0, 0.4, -0.5]

def softmax(xs, T=1.0):
    m = max(x / T for x in xs if x != -math.inf)
    e = [math.exp(x / T - m) if x != -math.inf else 0.0 for x in xs]
    return [v / sum(e) for v in e]

def mask(xs, keep):                       # a removed token -> logit = -inf
    return [x if k else -math.inf for x, k in zip(xs, keep)]

def top_k(xs, k):
    cut = sorted(xs, reverse=True)[k - 1]
    return mask(xs, [x >= cut for x in xs])

def top_p(xs, p):
    probs = softmax(xs)
    keep, total = [False] * len(xs), 0.0
    for i in sorted(range(len(xs)), key=lambda i: -probs[i]):
        keep[i], total = True, total + probs[i]
        if total >= p:
            break
    return mask(xs, keep)

def min_p(xs, p):
    probs = softmax(xs)
    return mask(xs, [q >= p * max(probs) for q in probs])

def temp(xs, T):
    return [x / T for x in xs]

def show(name, xs):
    cells = "  ".join(f"{t}={q:.2f}" if q else f"{t}=--" for t, q in zip(tokens, softmax(xs)))
    print(f"{name:<40}{cells}")

for T in (0.5, 1.0, 1.5):
    show(f"temperature {T}", temp(logits, T))
show("top-k 3", top_k(logits, 3))
show("top-p 0.9", top_p(logits, 0.9))
show("min-p 0.1", min_p(logits, 0.1))
print()
show("top-p 0.9 -> temperature 1.5", temp(top_p(logits, 0.9), 1.5))
show("temperature 1.5 -> top-p 0.9", top_p(temp(logits, 1.5), 0.9))
show("HF:   T 1.5 -> top-p 0.9 -> min-p 0.1", min_p(top_p(temp(logits, 1.5), 0.9), 0.1))
show("vLLM: T 1.5 -> min-p 0.1 -> top-p 0.9", top_p(min_p(temp(logits, 1.5), 0.1), 0.9))

Output:

temperature 0.5                         phở=0.68  bún chả=0.21  cơm tấm=0.09  pizza=0.01  sushi=0.00  kem=0.00
temperature 1.0                         phở=0.46  bún chả=0.25  cơm tấm=0.17  pizza=0.06  sushi=0.03  kem=0.01
temperature 1.5                         phở=0.37  bún chả=0.25  cơm tấm=0.19  pizza=0.10  sushi=0.06  kem=0.04
top-k 3                                 phở=0.52  bún chả=0.29  cơm tấm=0.19  pizza=--  sushi=--  kem=--
top-p 0.9                               phở=0.49  bún chả=0.27  cơm tấm=0.18  pizza=0.07  sushi=--  kem=--
min-p 0.1                               phở=0.49  bún chả=0.27  cơm tấm=0.18  pizza=0.07  sushi=--  kem=--

top-p 0.9 -> temperature 1.5            phở=0.41  bún chả=0.27  cơm tấm=0.21  pizza=0.11  sushi=--  kem=--
temperature 1.5 -> top-p 0.9            phở=0.38  bún chả=0.26  cơm tấm=0.20  pizza=0.10  sushi=0.07  kem=--
HF:   T 1.5 -> top-p 0.9 -> min-p 0.1   phở=0.38  bún chả=0.26  cơm tấm=0.20  pizza=0.10  sushi=0.07  kem=--
vLLM: T 1.5 -> min-p 0.1 -> top-p 0.9   phở=0.41  bún chả=0.27  cơm tấm=0.21  pizza=0.11  sushi=--  kem=--

The top half is what parts 1 and 2 already covered: temperature changes how peaked the distribution is, and the three truncation methods drop candidates. The bottom half is what this post is about. Same parameters, only the order changes, and the set of surviving tokens is different: sometimes sushi can still be picked, sometimes it cannot. The last two lines even carry the names HF and vLLM. The next section explains why.

02Decoding is a pipeline, and the order is part of the algorithm

When you call generate(temperature=0.7, top_p=0.9, repetition_penalty=1.1), the library does not have a single “sampling” function that takes those three arguments. It builds a list of steps, each of which takes logits and returns modified logits. Hugging Face calls each step a LogitsProcessor. vLLM gathers them into its Sampler class. Figure 1 at the top of this post draws the overall shape of that list.

The order of the steps is hard-coded in source, not in the documentation. So the only reliable way to know the order is to read the source.

Hugging Face Transformers 5.18.0

The list is built in _get_logits_processor in generation/utils.py. Trimmed to the common steps, in the exact order of the code:

Steps 9 to 12 sit inside the if generation_config.do_sample: block at L1574, so they only run when sampling is on. Steps 1 to 8 run under greedy decoding too. The generation loop in _sample then applies the whole list at L3219, takes the softmax at L3243, and draws a token with torch.multinomial at L3245. Without sampling it uses torch.argmax at L3247.

Two details worth remembering:

• Your own processors (step 8) are appended after the built-in steps but before temperature. The merge function at L1674–L1709 has one more rule: if your processor has the same type as a built-in step, yours replaces the built-in one in that position.
• The source has a comment at L1570: “TODO (joao): find a strategy to specify the order of the processors”. In other words, the authors themselves consider this ordering unfinished business.

vLLM 0.30.0

vLLM splits the work into two places. Before the sampler runs, the model runner applies the structured-output grammar bitmask to the logits in gpu_model_runner.py L4595–L4599, and only then calls the sampler at L4602. Inside sampler.py, the order is:

The docstring at the top of the Sampler class describes exactly this order. The “argmax-invariant” detail deserves a sentence: vLLM splits processors into two kinds. The kind that can change the argmax (such as logit_bias) must run before the greedy step. The kind that never changes the argmax (such as min-p, which always keeps the strongest token) only needs to run for requests that sample.

llama.cpp: temperature goes last

To see that the order is not obvious, look at llama.cpp. Its default sampler chain in common/common.h is: penalties → DRY → top-n-sigma → top-k → typical-p → top-p → min-p → XTC → temperature. Temperature comes after every truncation step. The consequence: in llama.cpp, raising the temperature never makes top-p or min-p keep more tokens. It only flattens the distribution inside the set that was already kept.

How the order changes the result

Back to the bottom half of the output in section 01:

• Temperature before or after top-p. Apply top-p to the original distribution (T = 1) and then divide by 1.5, and 4 tokens survive. Divide by 1.5 first and the distribution is flatter, so more tokens are needed to reach 90% of the probability mass, and 5 tokens survive. HF and vLLM do the second, llama.cpp does the first.
• Min-p before or after top-p. Both libraries apply temperature first. But HF runs top-p and then min-p, while vLLM runs min-p and then top-k and top-p. When min-p runs first, it drops kem, the distribution is renormalized over the remaining five tokens, and top-p 0.9 computed on that new distribution only needs four tokens. With the same T = 1.5, top_p = 0.9, min_p = 0.1, HF keeps 5 tokens and vLLM keeps 4.

The difference is small with six tokens, but it means this: the same parameter set does not necessarily give the same distribution on two different engines. If you tune parameters on HF and then deploy with vLLM or llama.cpp, check again.

The mask must come before truncation

There is one ordering constraint that both HF and vLLM respect: every step that bans tokens (grammar mask, bad_words, allowed_token_ids, your own processors) runs before temperature and truncation.

The reason is easy to see if you try it the other way round. Say the model is generating JSON and the next position must be a ". If top-k 20 runs first, it keeps the 20 strongest tokens, and quite possibly none of them is ". When the mask runs afterwards, it removes all 20, and there is no valid token left to pick. With the mask first, top-k only chooses among tokens that are already valid.

Penalties and the mask, on the other hand, commute: a penalty multiplies or divides a logit, and -inf multiplied or divided by a positive number is still -inf. That is why HF can put penalties before your processors and vLLM can put the grammar mask before penalties, and both give the same result.

Default parameters do not come from the library

A common trap: you pass no parameters and assume you are running greedy. In fact, both HF and vLLM read the generation_config.json that ships with the model. For Qwen2.5-0.5B-Instruct, that file sets:

{
  "do_sample": true,
  "temperature": 0.7,
  "top_p": 0.8,
  "top_k": 20,
  "repetition_penalty": 1.1
}

In vLLM 0.30.0, the generation_config option defaults to "auto", which means this config is also loaded as the default for every request. In HF, if you call generate(do_sample=False), the sampling steps are switched off but repetition_penalty = 1.1 still applies, because it is step 3, outside the do_sample block. Every greedy code sample in this post runs with that 1.1 penalty unless it says it was turned off.

03Search: greedy and beam search

Part 2, section 9.1 defined greedy decoding: at each step, take the token with the highest logit. Part 1, section 8.11 introduced beam search. Here we look at what they actually optimize, and why that target is a problem.

Greedy does not find the most likely sequence

The probability of a whole sequence is the product of the per-step probabilities. Greedy optimizes one step at a time, so it can miss a better sequence. A two-step example:

Greedy picks A at step 1 and ends with a sequence of probability 0.15. The sequence B → Y has probability 0.36, but greedy never looks at it.

Beam search: keep B candidates in parallel

Beam search fixes exactly that. Instead of one sequence, it keeps B sequences (the beam width). At each step it extends every sequence with every possible token, computes the summed log-probability of each new sequence, and keeps only the best B. With B = 2, the example above keeps both A and B at step 1 and finds B → Y at step 2.

Beam search has a built-in bias: log-probabilities are always negative, so the longer the sequence, the lower its total. Left alone, beam search always prefers short outputs. Libraries compensate with a length penalty. HF divides a sequence’s score by length ^ length_penalty, with length_penalty = 1.0 by default, i.e. the average log-probability per token (the compute_transition_scores docstring spells out this formula). Google’s GNMT translation system used a different form, ((5 + |Y|) / 6) ^ α (Wu et al. 2016).

How strong is that bias towards short outputs? Stahlberg and Byrne 2019 wrote an algorithm that finds the exact most likely sequence for a Transformer translation model on the WMT15 English–German test set. The result: for more than 50% of sentences, the most likely sequence is the empty translation. Beam search only gives good translations because it fails to find that optimum.

Why beam search still suits machine translation

If exact search gives bad results, why is beam search still the standard in translation and summarization? Meister et al. 2020 asked exactly that in the title of their paper, If beam search is the answer, what was the question?. Their answer: beam search implicitly optimizes a different objective on top of probability, an even spread of information across tokens (uniform information density), and how closely the output follows that property correlates strongly with BLEU in their translation experiments.

On top of that, tasks like translation have three properties that make search reasonable: the correct answer is fairly narrow, the distribution is usually very peaked, and the output length is roughly predictable from the input length.

Why search makes open-ended text repeat

Open-ended text (chat, articles, stories) is the opposite: there are many good answers, and the single most likely answer is usually bland and repetitive. Holtzman et al. 2019 showed two things:

• Human-written text does not sit in the model’s highest-probability region. People regularly pick words the model considers unlikely.
• Repetition reinforces itself: each time a phrase repeats, the probability that the model repeats it once more goes up. Greedy and beam search follow the highest probability, so they walk straight into that loop.

Zhang et al. 2021 confirm the existence of the likelihood trap: high-likelihood sequences are often of low quality.

Here is a run on Qwen2.5-0.5B-Instruct, letting the model continue an open-ended sentence without the chat template, and turning off the default repetition penalty to see the raw behavior:

# transformers==5.18.0, torch==2.14.1 (CPU)
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

name = "Qwen/Qwen2.5-0.5B-Instruct"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name, dtype=torch.float32).eval()

inputs = tok("The best thing about living in Hanoi is", return_tensors="pt")
n = inputs["input_ids"].shape[1]
common = dict(max_new_tokens=60, repetition_penalty=1.0)   # turn off the model's default 1.1 penalty

runs = {
    "greedy": dict(do_sample=False),
    "beam 5": dict(do_sample=False, num_beams=5),
    "sample T=0.7, top-p 0.8": dict(do_sample=True, temperature=0.7, top_p=0.8, top_k=0),
}
for label, kw in runs.items():
    torch.manual_seed(0)
    out = model.generate(**inputs, **common, **kw)[0, n:]
    text = tok.decode(out, skip_special_tokens=True)
    words = text.split()
    print(f"[{label}] distinct words / total: {len(set(words))}/{len(words)}\n{text!r}\n")

Output:

[greedy] distinct words / total: 21/51
" the city's beautiful architecture. The city is full of interesting buildings, including the National Museum of Vietnam, the National Museum of China, and the National Museum of the United States. The city is also full of interesting museums, including the National Museum of Vietnam, the National Museum of China, and the"

[beam 5] distinct words / total: 32/51
' that it is the capital city of Vietnam. It is the largest city in Vietnam, and it is also the largest city in Southeast Asia. It is located on the banks of the Mekong River, and it has a population of over 10 million people. The city is known for its'

[sample T=0.7, top-p 0.8] distinct words / total: 39/54
' the culture. The people are very friendly and welcoming, and the streets are bustling with activity. The city is also home to many different types of restaurants, from traditional Vietnamese dishes to modern fusion cuisine. There are also many parks and outdoor spaces to enjoy the beautiful scenery. The weather is pleasant and the'

The three outputs illustrate the three points of this section:

• Greedy falls into a loop after about 30 tokens: the whole phrase “the National Museum of Vietnam, the National Museum of China” comes back word for word.
• Beam 5 reads fluently and does not repeat within these 60 tokens, but it is wrong again and again: Hanoi is not on the Mekong, and it is not the largest city in Vietnam or in Southeast Asia. A sequence the model finds likely is not the same as a sequence that is true.
• Sampling is the most varied (39 distinct words out of 54) and sounds the most natural, even if the content is generic.

This is one run on a small model, not a benchmark. But it shows the same tendency the papers above describe.

Beam search is also expensive: B beams are B sequences running in parallel, with B copies of the KV cache. Commercial chat APIs such as OpenAI Chat Completions have no beam search parameter; their n parameter returns n independent samples, not n beams.

04Sampling: temperature, top-k, top-p, min-p

Part 2, section 9.2 and section 9.3 defined temperature, top-k and top-p. This section puts the four techniques side by side through one lens: temperature changes the shape of the distribution, and the other three cut off its tail.

Temperature changes the shape, not the ranking

p_i = softmax(z / T)_i

Dividing every logit by the same positive number does not change their order. The strongest token is still the strongest at any T. That is why:

• greedy decoding does not care about temperature (the argmax does not move), and vLLM computes the greedy result before the temperature step, as the table in section 02 shows;
• T → 0 pushes all the probability onto the strongest token, i.e. towards greedy. No library actually divides by zero. vLLM treats T below 1e-5 as greedy. HF raises an error if you set temperature = 0 while sampling is on, with a hint to use do_sample=False (logits_process.py L289–L296).

The problem with high temperature lies in the tail of the distribution. The vocabulary has 151,936 tokens. Most of them make no sense at any given position, each with a tiny probability, but together they add up. High temperature lifts that whole tail. The three techniques below are three ways to cut it off.

Three ways to cut the tail

Top-k is the oldest. Fan et al. 2018 used k = 10 for story generation. Its weakness is in the definition: k has no idea how confident the model is.

Top-p fixes that: when the distribution is peaked, a few tokens already reach p; when it is flat, the kept set grows by itself. But that self-growing is exactly its weak spot at high temperature, as the experiment below shows.

Min-p sets a threshold relative to the strongest token. With p_base = 0.1: if the top token has probability 0.9, every token below 0.09 is dropped; if the top token only has 0.2, the threshold drops to 0.02. The more confident the model, the higher the bar.

On a real model: a peaked and a flat distribution

The code below takes real Qwen2.5-0.5B-Instruct logits for two questions: one with an almost unique answer, one with many reasonable answers. It then passes the logits through HF’s actual LogitsProcessor classes and counts how many tokens can still be picked.

# transformers==5.18.0, torch==2.14.1 (CPU)
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers.generation.logits_process import (
    LogitsProcessorList, MinPLogitsWarper, TemperatureLogitsWarper,
    TopKLogitsWarper, TopPLogitsWarper,
)

name = "Qwen/Qwen2.5-0.5B-Instruct"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name, dtype=torch.float32).eval()

def next_token_logits(question):
    msgs = [{"role": "user", "content": question}]
    inputs = tok.apply_chat_template(msgs, add_generation_prompt=True,
                                     return_tensors="pt", return_dict=True)
    with torch.no_grad():
        return model(**inputs).logits[:, -1, :]          # shape [1, vocab]

def survivors(logits, *warpers):
    out = LogitsProcessorList(warpers)(None, logits.clone())
    return int(torch.isfinite(out).sum())

prompts = {
    "peaked": "What is the capital of France? Answer with one word.",
    "flat": "Name a food. Answer with one word.",
}
print(f"vocab = {model.config.vocab_size:,}")
for label, q in prompts.items():
    logits = next_token_logits(q)
    probs = logits.softmax(-1)[0]
    top = probs.topk(4)
    print(f"\n[{label}] top-4:", ", ".join(f"{tok.decode([int(i)])!r}={p:.3f}"
                                         for p, i in zip(top.values, top.indices)))
    for T in (0.7, 1.0, 1.5):
        t = TemperatureLogitsWarper(T)
        print(f"  T={T}: top-k 50 -> {survivors(logits, t, TopKLogitsWarper(50)):>4}"
              f" | top-p 0.9 -> {survivors(logits, t, TopPLogitsWarper(0.9)):>4}"
              f" | min-p 0.1 -> {survivors(logits, t, MinPLogitsWarper(0.1)):>4}")

Output:

vocab = 151,936

[peaked] top-4: 'Paris'=0.987, ' Paris'=0.004, 'L'=0.003, '巴黎'=0.002
  T=0.7: top-k 50 ->   50 | top-p 0.9 ->    1 | min-p 0.1 ->    1
  T=1.0: top-k 50 ->   50 | top-p 0.9 ->    1 | min-p 0.1 ->    1
  T=1.5: top-k 50 ->   50 | top-p 0.9 ->   12 | min-p 0.1 ->    1

[flat] top-4: 'N'=0.164, 'Pizza'=0.126, 'Chinese'=0.088, 'P'=0.076
  T=0.7: top-k 50 ->   50 | top-p 0.9 ->   13 | min-p 0.1 ->    7
  T=1.0: top-k 50 ->   50 | top-p 0.9 ->   41 | min-p 0.1 ->   11
  T=1.5: top-k 50 ->   50 | top-p 0.9 -> 3439 | min-p 0.1 ->   28

Read the table column by column:

• Top-k 50 always keeps exactly 50 tokens, even when the model is 98.7% sure the answer is Paris. At T = 1.5, the other 49 tokens (including L, 巴黎 and London) get a real chance of being picked.
• Top-p 0.9 is fine at T = 1, but it breaks on the flat distribution at high temperature: from 41 tokens it jumps to 3,439 tokens. Temperature 1.5 thickens the tail, so a huge number of low-probability tokens is needed to reach 90%.
• Min-p 0.1 grows slowly and under control: 7, 11, then 28 tokens. On the capital-city question it keeps exactly one token at every temperature.

One small detail the table reveals: the tokens N and P at the top of the flat distribution are not whole words. The model is about to write Noodles or Pasta piece by piece, as subwords, as the tokenization post explained. Decoding picks pieces, not words.

Is min-p actually better?

Nguyen et al. 2024 proposed min-p and concluded that it improves both quality and diversity over top-p, especially at high temperature. The paper was selected for an oral presentation at ICLR 2025, and min-p is available in HF, vLLM and llama.cpp.

However, Schaeffer et al. 2025 re-examined the evidence and reached a different conclusion: the original paper’s human evaluation omitted data and ran statistical tests incorrectly, and when every method gets a full hyperparameter sweep, min-p does not beat the baselines on the benchmarks the original paper used. The original authors ran a new human evaluation, and according to Schaeffer et al., that new evaluation does not show min-p being better either.

The safer conclusion: min-p’s advantage is its mechanism, not a proven claim about quality. Its threshold scales with the model’s confidence, so it is less sensitive to temperature than top-p, as the table above shows. If you need a high temperature for varied text, min-p is a sensible choice. If you keep the temperature around 0.7, the difference between min-p and top-p is much smaller.

Starting configurations by task

This table is a starting point, not a standard. If the model developer publishes a recommended configuration (such as Qwen’s generation_config.json in section 02), start from that.

One rule from part 1, section 8.6 still holds: temperature and top-p act on the same distribution, so change one parameter at a time.

05Fighting repetition: repetition, frequency and presence penalty

Part 2, section 9.4 listed the three kinds of penalty. This section looks at their actual formulas in code, because there is one detail that is easy to get wrong and one detail about scope that few people notice.

Repetition penalty: divide or multiply?

The repetition penalty comes from the CTRL paper (Keskar et al. 2019). The original formula divides the logit of every token that has already appeared by a factor θ, and the paper recommends θ ≈ 1.2:

p_i = exp( x_i / (T · I(i ∈ g)) ) / Σ_j exp( x_j / (T · I(j ∈ g)) )
I(c) = θ if c is true, otherwise 1          g = tokens generated so far

Dividing by θ > 1 shrinks a positive logit, i.e. it lowers the probability. But for a negative logit, dividing by θ pulls it towards 0, which raises the probability. A token the model already did not want to repeat (negative logit) gets rewarded instead of penalized.

HF and vLLM fix this in the same way: divide positive logits, multiply negative logits by θ. See HF’s logits_process.py L390 and vLLM’s _custom_ops.py L373–L375. Checking with numbers:

# transformers==5.18.0, torch==2.14.1
import torch
from transformers.generation.logits_process import RepetitionPenaltyLogitsProcessor

#               token 0 (seen)   token 1 (seen)   token 2 (not seen)
logits = torch.tensor([[2.0, -2.0, 0.5]])
seen = torch.tensor([[0, 1]])            # input_ids: tokens 0 and 1 are already in the context
theta = 1.3

naive = logits.clone()
naive[0, :2] = naive[0, :2] / theta      # divide only, as in the original CTRL formula
hf = RepetitionPenaltyLogitsProcessor(penalty=theta)(seen, logits.clone())

print("original   :", logits.tolist()[0])
print("divide only:", [round(x, 3) for x in naive.tolist()[0]])
print("HF         :", [round(x, 3) for x in hf.tolist()[0]])

Output:

original   : [2.0, -2.0, 0.5]
divide only: [1.538, -1.538, 0.5]
HF         : [1.538, -2.6, 0.5]

With “divide only”, token 1 goes from −2.0 up to −1.54, so it becomes more likely after being “penalized”. With HF it drops to −2.6 as intended. If you ever implement a repetition penalty yourself, this is the first bug to avoid.

Frequency and presence penalty: additive subtraction

Frequency and presence penalties come from the OpenAI API and work quite differently: they subtract an amount from the logit instead of multiplying or dividing. vLLM implements exactly that formula in layers/utils.py L79–L80:

logit_i -= frequency_penalty · number_of_times_token_i_appeared
logit_i -= presence_penalty  · (1 if token i has appeared, else 0)

The frequency penalty grows with the repetition count, so a word repeated ten times is penalized ten times as hard as a word repeated once. The presence penalty only cares whether the token has appeared at all. OpenAI allows both in the range −2.0 to 2.0.

Penalize the prompt, or only the output?

This is the detail few people notice, and it differs between penalty types:

The most visible practical consequence is in RAG. The prompt contains the document passage the answer needs to quote: a product name, an error code, a number. The repetition penalty penalizes exactly those tokens, pushing the model towards synonyms or misspelled names. HF lets you exclude the prompt from the penalty with the prompt_ignore_length parameter of RepetitionPenaltyLogitsProcessor, passed in through logits_processor.

When penalties hurt

A penalty is a blunt tool: it cannot tell which repetition is a bug and which is required.

• Proper nouns and technical terms must repeat exactly. Penalize them and the model changes the spelling.
• Code repeats tokens constantly: self, (, ), indentation, variable names. Penalizing them produces broken syntax.
• JSON repeats ", :, , and key names. This is why you should not turn on a strong penalty when you need structured output.
• no_repeat_ngram_size is even harsher: it forbids any n-gram that has already appeared, so a two-token proper noun showing up a second time forces the model to misspell it.

Model developers know this, so they set the penalty very lightly. Qwen2.5-0.5B-Instruct sets repetition_penalty = 1.1 in its generation_config.json, below the 1.2 the CTRL paper suggests.

06Constrained decoding and structured output

Real applications often do not need prose. They need a class label, a JSON object, a function call. The simplest approach is to write “return only JSON” in the prompt. That works most of the time, and “most of the time” is the problem: a system handling a million requests a day with a 1% failure rate still hits ten thousand parser errors.

The idea: mask the logits according to a grammar

Constrained decoding solves the problem at the decoding layer, not the prompt layer. At each step, before a token is picked, we set the logit of every token that could make the output syntactically invalid to -inf. That token has probability 0 and can never be chosen. The output is therefore always grammatically valid, by construction rather than by hope.

The simplest version: force the output to be one of a few given labels. The code below writes a custom LogitsProcessor that does exactly that (batch size 1 only, to keep it short):

# transformers==5.18.0, torch==2.14.1 (CPU)
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, LogitsProcessor, LogitsProcessorList

name = "Qwen/Qwen2.5-0.5B-Instruct"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name, dtype=torch.float32).eval()

class ChoiceProcessor(LogitsProcessor):
    """Only allow the output to be exactly one of `choices`, then EOS."""
    def __init__(self, choices, prompt_len):
        self.seqs = [tok.encode(c, add_special_tokens=False) for c in choices]
        self.prompt_len = prompt_len

    def __call__(self, input_ids, scores):
        done = input_ids[0, self.prompt_len:].tolist()
        allowed = set()
        for seq in self.seqs:
            if seq[:len(done)] == done:
                allowed.add(seq[len(done)] if len(done) < len(seq) else tok.eos_token_id)
        mask = torch.full_like(scores, float("-inf"))
        mask[:, list(allowed)] = 0.0
        return scores + mask                      # invalid tokens -> -inf

review = "The pho was cold, but the staff were lovely and fixed it fast."
msgs = [{"role": "user", "content": f"Classify the sentiment of this review: {review}"}]
inputs = tok.apply_chat_template(msgs, add_generation_prompt=True,
                                 return_tensors="pt", return_dict=True)
n = inputs["input_ids"].shape[1]

free = model.generate(**inputs, max_new_tokens=30, do_sample=False)
choices = ["positive", "negative", "mixed"]
forced = model.generate(**inputs, max_new_tokens=5, do_sample=False,
                        logits_processor=LogitsProcessorList([ChoiceProcessor(choices, n)]))
print("Unconstrained:", repr(tok.decode(free[0, n:], skip_special_tokens=True)))
print("Constrained  :", repr(tok.decode(forced[0, n:], skip_special_tokens=True)))
print("Label token ids:", {c: tok.encode(c, add_special_tokens=False) for c in choices})

Output:

Unconstrained: 'The sentiment of this review can be classified as positive. The reviewer expresses satisfaction with both the food quality (the pho) and the service provided by'
Constrained  : 'positive'
Label token ids: {'positive': [30487], 'negative': [42224], 'mixed': [56685]}

Unconstrained, the model writes a whole paragraph and the downstream code has to dig the label out of it. Constrained, the output is exactly one label and then it stops. This processor sits at step 8 of the HF table in section 02, so it runs before temperature and truncation, exactly as section 02 argued it should.

Notice one more thing: the review both complains (cold pho) and praises (lovely staff), so mixed is the more reasonable label. The model still picks positive. Constrained decoding guarantees the shape of the output, not its content.

From a list of labels to a JSON Schema

For a list of labels, checking prefixes is enough. A JSON Schema is much harder: required keys, data types, enums, arrays, nested objects. Two steps forward matter here:

• Outlines (Willard and Louf 2023) turns a regex into a finite-state machine (FSM). Reducing a JSON Schema to a regex before building the FSM is a feature of the Outlines library; the paper does not describe that step. The key idea of the paper is to precompute a lookup index: for each FSM state, which vocabulary tokens are allowed. At generation time, each step only needs a lookup instead of checking every token.
• XGrammar (Dong et al. 2024) handles context-free grammars (CFGs), which are needed for arbitrarily nested structure, with a stack-based automaton (a pushdown automaton, PDA). It splits the vocabulary into two groups: context-independent tokens, which can be prechecked, and context-dependent tokens, which have to be checked at runtime. The paper reports up to a 100x speedup over earlier solutions, and near-zero overhead when it runs inside an inference engine.

llama.cpp has its own grammar format, GBNF, built on the same idea.

Token boundaries do not match grammar boundaries

Grammars are written in characters, but models generate tokens. One token can contain several grammar symbols at once: the Qwen tokenizer has tokens for ",, for "} and for ":". To know whether a token is valid in the current state, you have to run the token’s whole character string through the automaton. That is why precomputing the index matters: doing this for 151,936 tokens at every step would be far too expensive.

It goes the other way too. The example above forces the output to start with the token positive (id 30487, no leading space). The tokenizer also has positive with a leading space (id 6785). If the model “wanted” to write the version with a space, the constraint pushes it into a token split it rarely saw in training. Small mismatches like that can lower quality in ways that are hard to spot.

JSON mode is not schema-based structured output

These two are often confused:

• JSON mode only guarantees that the output is valid JSON. Which keys, which types and which values are up to the model.
• Schema-based structured output guarantees the output matches a specific JSON Schema.

OpenAI keeps the two modes separate. When Structured Outputs launched in August 2024, they reported that on their eval of complex JSON Schema adherence, gpt-4o-2024-08-06 with Structured Outputs scored 100%, while gpt-4-0613 relying on the prompt alone scored under 40%.

Here is the difference on CPU with XGrammar and HF. The schema requires sentiment to be one of three lowercase values:

# transformers==5.18.0, torch==2.14.1 (CPU), xgrammar==0.2.8, pydantic==2.13.5
import json
import torch
import xgrammar as xgr
from pydantic import BaseModel
from typing import Literal
from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer

class Review(BaseModel):
    dish: str
    sentiment: Literal["positive", "negative", "mixed"]
    score: int

name = "Qwen/Qwen2.5-0.5B-Instruct"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name, dtype=torch.float32).eval()

# Compile the JSON Schema into a grammar once; each decode step only looks up a bitmask
vocab_size = AutoConfig.from_pretrained(name).vocab_size      # 151,936, larger than len(tok)
info = xgr.TokenizerInfo.from_huggingface(tok, vocab_size=vocab_size)
grammar = xgr.GrammarCompiler(info).compile_json_schema(Review.model_json_schema())

review = "The pho was cold, but the staff were lovely and fixed it fast."
msgs = [{"role": "user", "content": f"Extract dish, sentiment and a 1-5 score as JSON: {review}"}]
inputs = tok.apply_chat_template(msgs, add_generation_prompt=True,
                                 return_tensors="pt", return_dict=True)
n = inputs["input_ids"].shape[1]

free = model.generate(**inputs, max_new_tokens=80, do_sample=False)
forced = model.generate(**inputs, max_new_tokens=80, do_sample=False,
                        logits_processor=[xgr.contrib.hf.LogitsProcessor(grammar)])

for label, out in [("Unconstrained", free), ("With schema", forced)]:
    text = tok.decode(out[0, n:], skip_special_tokens=True)
    print(f"--- {label} ---\n{text}")
    try:
        print("=> valid against the schema:", Review.model_validate(json.loads(text)))
    except Exception as e:
        print("=> fails the schema:", type(e).__name__)

Output:

--- Unconstrained ---
{
  "dish": "Pho",
  "sentiment": "Positive",
  "score": 4
}
=> fails the schema: ValidationError
--- With schema ---
{
  "dish": "Pho",
  "sentiment": "positive",
  "score": 4
}
=> valid against the schema: dish='Pho' sentiment='positive' score=4

Unconstrained, the model still returns valid JSON: json.loads works. But it writes "Positive" with a capital letter, so schema validation fails. That is exactly the gap between “JSON mode” and “matches the schema”. Constrained, sentiment can only be one of the three declared values, and the output passes validation.

And once again: the valid output still says positive, even though the review has both praise and complaints. The schema forces the shape; the model still picks the values.

In vLLM

In vLLM 0.30.0, structured output is configured through structured_outputs. The old guided_json, guided_regex and guided_choice parameters were removed in v0.12.0 (vLLM structured outputs docs). The default backend is "auto", which picks between xgrammar, guidance, outlines and lm-format-enforcer depending on the request (config/structured_outputs.py L12–L25).

# vllm==0.30.0 — NOT run for this post: needs a vLLM install, usually with an NVIDIA GPU.
# Written against the API in vllm/sampling_params.py at tag v0.30.0.
from vllm import LLM, SamplingParams
from vllm.sampling_params import StructuredOutputsParams

# Review is the Pydantic class defined in the XGrammar snippet above.

llm = LLM(model="Qwen/Qwen2.5-0.5B-Instruct")
params = SamplingParams(
    temperature=0.0,
    structured_outputs=StructuredOutputsParams(json=Review.model_json_schema()),
)
out = llm.generate("Extract dish, sentiment and a 1-5 score as JSON: ...", params)
print(out[0].outputs[0].text)

Through the OpenAI-compatible server, you can use a json_schema response_format just as with the OpenAI API, or pass extra_body={"structured_outputs": {"json": schema}}.

The price of forcing a format

Constrained decoding is not free:

• Compile cost. The schema has to be compiled into an automaton before the first request. Engines usually cache the result, so this cost mostly lands on the first request for each new schema.
• Per-step cost. A bitmask over the whole vocabulary at every step. XGrammar brings this close to negligible, but older backends can be noticeably slower.
• Reasoning quality. Tam et al. 2024 (Let Me Speak Freely?) measured a significant drop in LLM reasoning ability under format restrictions, and the stricter the format, the larger the drop. That result is disputed: the team behind Outlines (.txt) replied in Say What You Mean that the drop came mostly from how the prompts in the experiment were written, and that with fixed prompts structured generation did better than free-form generation.

A practical way to reduce the third risk: give the model a free-text field to reason in before the answer field (for example a reasoning key placed before an answer key in the schema), or split the job into two steps, free reasoning first and extraction only in the second step.

07Why the same prompt gives two different answers

This question has three layers of causes, from the obvious to the subtle.

Layer 1: sampling is random, and a seed only fixes the randomness

With temperature > 0, each call draws a token at random from the distribution. Two calls differing is the design working as intended. A seed fixes the random number generator: torch.manual_seed in HF, the per-request seed parameter in vLLM. vLLM does not draw with torch.multinomial like HF; it divides the probabilities by exponential noise and takes the argmax (topk_topp_sampler.py L464–L486), a different way of drawing from the same distribution. That is the native path. On NVIDIA GPUs, when a request uses top-k or top-p, vLLM defaults to FlashInfer’s sampler (VLLM_USE_FLASHINFER_SAMPLER is on by default, envs.py L49). FlashInfer does not support per-request random generators, though, so as soon as any request in the batch sets a seed, the whole batch falls back to the native path (topk_topp_sampler.py L185–L192). What we just described is the path of model runner V1. Model runner V2, the default in vLLM v0.30.0, differs in two ways: the fallback path is a Triton Gumbel-max kernel whose random numbers are derived from (seed, pos), and FlashInfer is disabled for the whole batch when the batch contains any request with a seed or any greedy request (gpu/sample/sampler.py L300–L308). So the same seed on two different engines does not give the same token.

More importantly: a seed only fixes the random numbers. It does not fix the logits. If the logits change even slightly, the same random number can still land on a different token. That is the second layer.

Layer 2: greedy is not deterministic either, because of floating-point addition

Set temperature = 0 and the randomness is gone. But the logits can still change between two calls. The root cause is a property of floating-point numbers: addition is not associative.

>>> (0.1 + 1e20) - 1e20
0.0
>>> 0.1 + (1e20 - 1e20)
0.1

The same three numbers, added in a different order, give a different result. A matrix multiply on a GPU is millions of additions like this, and the order of the additions depends on how the kernel splits the work.

The popular hypothesis is: GPU threads run in parallel, whichever finishes first adds first, so the order of additions is random. Defeating Nondeterminism in LLM Inference by Thinking Machines (Horace He and colleagues, September 2025) shows that this hypothesis does not hold for an LLM forward pass. The kernels used in the forward pass usually give identical results from run to run when the input, batch included, is identical.

The real cause is that those kernels are not batch-invariant. A kernel chooses how to split the computation (tiling, how to split the reduction) based on the batch size, so the result for your request depends on how many other requests are running alongside it. And on a server, the batch size changes with load from other users. From your point of view, it is a random variable.

The Thinking Machines experiment, quoted from the post:

Checking it yourself on CPU

The experiment below tests the same effect at a small scale: run one prompt through the model on its own, then run the same prompt in a batch with other prompts, and compare the next-token logits. The prompt under test is the longest one, so it never gets padding. Any difference can only come from arithmetic, not from the input.

# transformers==5.18.0, torch==2.14.1 (CPU)
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

name = "Qwen/Qwen2.5-0.5B-Instruct"
tok = AutoTokenizer.from_pretrained(name, padding_side="left")

def chat(q):
    return tok.apply_chat_template([{"role": "user", "content": q}],
                                   add_generation_prompt=True, tokenize=False)

target = chat("Tell me about Richard Feynman, the physicist, in a few sentences.")
others = [chat(f"Haiku about {i}.") for i in range(15)]
n = len(tok(target).input_ids)
# target is the longest, so it is never padded: any difference comes from arithmetic, not input
assert all(len(tok(o).input_ids) < n for o in others)

for dtype in (torch.float32, torch.bfloat16):
    model = AutoModelForCausalLM.from_pretrained(name, dtype=dtype).eval()
    enc = lambda batch: tok(batch, return_tensors="pt", padding=True)
    with torch.no_grad():
        alone = model(**enc([target])).logits[0, -1].float()
        diffs = [(alone - model(**enc([target] + others[: s - 1])).logits[0, -1].float())
                 .abs().max().item() for s in (2, 4, 8, 16)]
        a = model.generate(**enc([target]), max_new_tokens=200, do_sample=False)[0, n:]
        b = model.generate(**enc([target] + others[:7]), max_new_tokens=200, do_sample=False)[0, n:]
    first = next((i for i, (x, y) in enumerate(zip(a.tolist(), b.tolist())) if x != y), None)
    print(f"{str(dtype):<15} max|Δlogit| batch 2/4/8/16 = "
          + " / ".join(f"{d:.1e}" for d in diffs)
          + f" | greedy batch 1 vs 8, first differing token: {first}")

Output with torch 2.14.1:

torch.float32   max|Δlogit| batch 2/4/8/16 = 1.5e-05 / 2.6e-05 / 2.8e-05 / 2.6e-05 | greedy batch 1 vs 8, first differing token: None
torch.bfloat16  max|Δlogit| batch 2/4/8/16 = 0.0e+00 / 0.0e+00 / 0.0e+00 / 0.0e+00 | greedy batch 1 vs 8, first differing token: None

Before upgrading to torch 2.14.1, I ran exactly the same code with torch 2.8.0 on the same machine, and the bf16 result was completely different:

torch.float32   max|Δlogit| batch 2/4/8/16 = 2.3e-05 / 2.7e-05 / 2.8e-05 / 3.3e-05 | greedy batch 1 vs 8, first differing token: None
torch.bfloat16  max|Δlogit| batch 2/4/8/16 = 3.8e-01 / 3.8e-01 / 3.8e-01 / 3.8e-01 | greedy batch 1 vs 8, first differing token: 61

Every configuration was run twice and gave the same output both times, so the results are deterministic as long as the batch stays the same. What changes the result is the batch size. Three takeaways:

1. The logits really do change with batch size, even on CPU. In float32 the difference is around 1e-5. That is not enough to flip any of the 200 greedy tokens in this experiment, because no step had its top two tokens that close together.
2. In bf16 with torch 2.8.0, the difference reaches 0.38, and the greedy answer takes a different turn from token 61. This is the same effect Thinking Machines observed, at a smaller scale. bf16 has far fewer mantissa bits than float32, so rounding errors are much larger.
3. Same code, only the torch version changed, and bf16 went from not batch-invariant to batch-invariant in this experiment. Batch invariance is a property of specific kernels, not of the model or of the decoding algorithm. Upgrading a library can make it appear or disappear.

Layer 3: the API provider’s side

When you call an API, there are also things underneath you cannot control:

• The model and the infrastructure change. OpenAI returns a system_fingerprint field that identifies the backend configuration. According to their cookbook, the seed parameter only makes outputs “mostly” consistent on a best effort basis, determinism is not guaranteed, and when system_fingerprint changes the output may change with it.
• Requests are served on different hardware and in different batches, so everything in layer 2 applies.
• For Mixture of Experts models, a commonly cited hypothesis is that when experts have a capacity limit, which expert your token gets routed to can depend on the other tokens in the batch. This is an inference from how MoE works; no major provider has publicly confirmed it as a cause in their system.

What you can do

Thinking Machines also gives another reason determinism matters: in reinforcement learning, if the numerics at sampling time differ from the numerics at training time, training effectively becomes off-policy. When they made the two sides bit-identical, the KL divergence between sampler and trainer was 0, and training was truly on-policy again.

08Common misconceptions

The first five rows match the bottom row of Figure 1.

Summary: the model gives a distribution, decoding picks the answer

The model stops at logits. Everything the user sees after that, a clean answer or a repetitive one, JSON that parses or not, two calls that match or differ, is decided at the decoding step.

What to take away:

• decoding is a pipeline of logit transformations, and the order of the steps is part of the algorithm: HF 5.18.0 runs temperature → top-k → top-p → min-p, vLLM 0.30.0 runs temperature → min-p → top-k/top-p, and llama.cpp puts temperature last;
• every token-banning step (grammar mask, bad words) must run before truncation, or there may be no valid token left;
• greedy and beam search look for high-probability sequences, which suits tasks with narrow answers like translation, but makes open-ended text repetitive and bland; high probability does not mean correct;
• temperature changes the shape of the distribution without changing the ranking; top-k, top-p and min-p cut the tail by a fixed count, by cumulative probability mass, and by a relative threshold; at high temperature top-p can keep thousands of tokens, min-p does not;
• the repetition penalty must multiply negative logits, applies to the prompt too, and hurts proper nouns, code and JSON; frequency and presence penalties are subtractions counted only over the output;
• constrained decoding masks logits using an automaton built from the schema, guaranteeing syntactically valid output but not correct content, and JSON mode is not schema-based structured output;
• the same prompt gives two answers because of three layers: random sampling, kernels that are not batch-invariant (even at temperature = 0), and provider-side changes that a seed cannot control.

This post treated the model as a given function that already returns good logits. Part 10 goes back to the question that comes before all of this: pretraining, how the model learns to produce those logits in the first place, through the next-token prediction objective and scaling laws. The two techniques that make decoding faster, the KV cache and speculative decoding, get their own post in part 13.

References

Papers

• Holtzman et al. 2019 — The Curious Case of Neural Text Degeneration
• Fan et al. 2018 — Hierarchical Neural Story Generation
• Nguyen et al. 2024 — Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs
• Schaeffer et al. 2025 — Min-p, Max Exaggeration: A Critical Analysis of Min-p Sampling in Language Models
• Keskar et al. 2019 — CTRL: A Conditional Transformer Language Model for Controllable Generation
• Meister et al. 2020 — If beam search is the answer, what was the question?
• Stahlberg and Byrne 2019 — On NMT Search Errors and Model Errors: Cat Got Your Tongue?
• Wu et al. 2016 — Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Zhang et al. 2021 — Trading Off Diversity and Quality in Natural Language Generation
• Willard and Louf 2023 — Efficient Guided Generation for Large Language Models
• Dong et al. 2024 — XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models
• Tam et al. 2024 — Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models

Blog posts and documentation

• Thinking Machines (Horace He and colleagues) 2025 — Defeating Nondeterminism in LLM Inference
• .txt 2024 — Say What You Mean: A Response to ‘Let Me Speak Freely’
• OpenAI 2024 — Introducing Structured Outputs in the API
• OpenAI — Structured Outputs guide
• OpenAI Cookbook — How to make your completions outputs consistent with the new seed parameter
• Hugging Face — Generation strategies
• vLLM — Structured Outputs (v0.30.0)
• vLLM — Batch Invariance (v0.30.0)
• llama.cpp — GBNF grammars

Source code

• Hugging Face Transformers v5.18.0 — generation/utils.py, generation/logits_process.py
• vLLM v0.30.0 — v1/sample/sampler.py, v1/sample/ops/topk_topp_sampler.py, v1/worker/gpu_model_runner.py, model_executor/layers/utils.py, _custom_ops.py
• llama.cpp — common/common.h (commit 207bdab)
• Hugging Face — generation_config.json for Qwen2.5-0.5B-Instruct