Reasoning Models and Test-Time Compute: When Thinking Longer Pays Off

Part fourteen ended with an observation: MoE makes each token cheaper, a fixed state makes long context cheaper, and together they are why making a model “think longer” became affordable. This post is about that thinking.

The question of this post: can you buy quality with compute at answer time, and when is it worth it? Part 10 showed that loss falls smoothly as training compute grows: more parameters, more tokens. But training is a one-off cost, paid before you know what users will ask. Compute at answer time (test-time compute) is different: you can spend a lot on a hard question, a little on an easy one, and spend more without retraining.

The post follows history, because each step addresses a limit of the one before:

• Chain-of-thought (2022): a prompting trick that tells the model to write out intermediate steps before answering.
• Sample many times, then pick: self-consistency, verifiers, reward models that score each step.
• Reasoning models (o1, DeepSeek-R1, and what came after): trained with RL on verifiable rewards to produce long chains of thought on their own, with a “budget” knob in the API.

Figure 1 at the top of the page is the path of one question through a reasoning model; most of the post explains its boxes. Section 8 prices thinking tokens with the roofline from Part 13. Section 9 answers the main question: when thinking longer helps, when it is wasted, and when it makes things worse.

01Buying quality with compute at answer time

Two compute axes

In September 2024 OpenAI released o1 alongside the post Learning to Reason with LLMs, with two charts side by side: AIME accuracy rises with RL compute at training time, and also with the time the model spends “thinking” at answer time. According to OpenAI, o1’s performance “consistently improves” with both. The charts use a log x-axis with no absolute compute values, so they only show a trend; they are also the vendor’s own numbers, not independently verified.

In the text, OpenAI gives three numbers for AIME 2024, and they illustrate exactly the ways of spending compute this post covers: o1 solved 74% of problems on average with one sample per problem, 83% with consensus among 64 samples, and 93% when re-ranking 1,000 samples with a learned scoring function. GPT-4o solved 12%.

Three ways to spend compute at answer time

With the same model, there are three ways to spend more compute on a question:

1. Parallel: generate N independent answers and pick one, by majority vote or with a scoring model. The model does not change.
2. Sequential: one longer chain of thought, in which the model tries, checks, and goes back to fix things. This needs a model trained to do it (section 5).
3. Search: split the solution into steps, sample a few branches at each step, and use a model that scores each step to keep the best branches (beam search or tree search).

Figure 2 puts the three side by side, with what each one needs, what it costs, and where it stops working. OpenAI’s three numbers above correspond to one sample, parallel with voting, and parallel with scoring.

Section 2 explains why writing out intermediate steps helps, sections 3–4 cover parallel and search, and sections 5–7 cover the sequential approach and how it is trained.

02Chain-of-thought: borrowing extra forward passes

Compute per token is fixed

Part 8, section 6 did the count: one forward pass costs about 2N FLOPs per token, where N is the number of parameters. That number does not depend on whether the question is hard or easy. “What is the capital of France?” and “Compute 7,493 × 8,217” each get exactly one forward pass per generated token, through exactly the same number of layers.

The consequence: if a problem needs more sequential steps of computation than the layers can do in one pass, the model cannot get the right answer at the first token. The only way to get more compute is to generate more tokens. Every token written becomes input to later forward passes, so intermediate steps are “written down on paper” instead of having to fit inside the activations of a single pass. Seen this way, chain-of-thought is how a model borrows extra forward passes.

Three foundational papers

• Scratchpads (Nye et al. 2021) train a model to write “an arbitrary sequence of intermediate tokens” before the answer (section 1). For long addition, models are trained on 1–8 digit numbers and tested on 9–10 digit numbers: models without a scratchpad fail completely, while models with a scratchpad and enough size succeed (section 3, Figure 3).
• Chain-of-thought prompting (Wei et al. 2022) drops the training step and just puts 8 worked examples with step-by-step solutions into the prompt. With PaLM 540B on GSM8K, accuracy rises from 17.9% to 56.9% (Table 2 in the appendix). The same paper reports CoT as “an emergent ability of model scale”: small models write fluent but illogical chains, and only models of about 100B parameters and up benefit (section 3.2, Figure 4).
• Zero-shot CoT (Kojima et al. 2022) drops the examples too and only adds the sentence “Let’s think step by step”. With text-davinci-002, MultiArith goes from 17.7% to 78.7% and GSM8K from 10.4% to 40.7% (abstract, Table 1).

GSM8K appears throughout this post: 8,500 grade-school math word problems, each needing 2–8 steps, split into 7,500 training and 1,000 test problems (Cobbe et al. 2021, section 2). The final answer is a number, so grading is automatic.

CoT does not help everything equally

Sprague et al. 2024 (To CoT or not to CoT?) aggregate 1,218 comparisons from 110 papers (section 3) and run their own experiments on 20 datasets and 14 models. The gains from CoT concentrate in symbolic reasoning, math, and logic, which improve by 14.2, 12.3, and 6.9 points on average. The other task categories barely move: from 56.1 to 56.8 (Figure 2). On MMLU, depending on the model, up to about 95% of CoT’s gain comes from questions with an “=” sign in the question or the solution (section 1, Table 18), even though only about 10% of MMLU questions contain one.

This fits the argument above: CoT helps when a problem needs many steps of computation, not when it needs knowledge, because writing more steps does not create new knowledge. This is the main axis for section 9.

Code sample 1: Qwen3-0.6B with thinking on and off

Qwen3 has two modes in the same model: thinking (writing its thoughts inside <think>…</think> before the answer) and non-thinking. The Qwen3-0.6B model card selects the mode with the enable_thinking argument of the chat template, which defaults to True. The code below runs the first 15 questions of the GSM8K test set in both modes, with the sampling parameters the model card recommends for each mode (the card explicitly warns against greedy decoding in thinking mode because it can loop forever), and a fixed seed.

# Python 3.12.13, torch 2.14.1+cpu, transformers 5.19.0, datasets 5.1.0
# CPU Intel i9-9900K, bf16, 8 threads. Run: python code1.py 15
import re, sys, time
import torch
from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL = "Qwen/Qwen3-0.6B"
N = int(sys.argv[1]) if len(sys.argv) > 1 else 15
SEED = 0
END_THINK = 151668  # token id of </think> in the Qwen3 tokenizer

tok = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForCausalLM.from_pretrained(MODEL, dtype=torch.bfloat16)
model.eval()
torch.set_num_threads(8)

data = load_dataset("openai/gsm8k", "main", split="test").select(range(N))

def gold(ans: str) -> str:
    return ans.split("####")[-1].strip().replace(",", "")

def extract(text: str) -> str | None:
    boxed = re.findall(r"\\boxed\{([^{}]*)\}", text)
    cand = boxed[-1] if boxed else text
    nums = re.findall(r"-?\d[\d,]*\.?\d*", cand)
    return nums[-1].replace(",", "").rstrip(".") if nums else None

def same(a, b) -> bool:
    try:
        return a is not None and abs(float(a) - float(b)) < 1e-6
    except ValueError:
        return False

def run(question: str, thinking: bool):
    msgs = [{"role": "user", "content": question +
             "\nPlease reason step by step, and put your final answer within \\boxed{}."}]
    prompt = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True,
                                     enable_thinking=thinking)
    inputs = tok(prompt, return_tensors="pt")
    # Sampling parameters recommended by the Qwen3-0.6B model card
    sampling = (dict(temperature=0.6, top_p=0.95, top_k=20) if thinking
                else dict(temperature=0.7, top_p=0.8, top_k=20))
    torch.manual_seed(SEED)
    t0 = time.perf_counter()
    with torch.no_grad():
        out = model.generate(**inputs, max_new_tokens=4096, do_sample=True, **sampling)
    dt = time.perf_counter() - t0
    ids = out[0, inputs.input_ids.shape[1]:].tolist()
    # Tokens up to and including </think> (with <think> itself) are thinking tokens
    n_think = ids.index(END_THINK) + 1 if END_THINK in ids else (len(ids) if thinking else 0)
    answer = tok.decode(ids[n_think:], skip_special_tokens=True)
    return extract(answer), n_think, len(ids), dt

rows = {True: [], False: []}
for i, ex in enumerate(data):
    g = gold(ex["answer"])
    for thinking in (False, True):
        pred, n_think, n_out, dt = run(ex["question"], thinking)
        ok = same(pred, g)
        rows[thinking].append((ok, n_think, n_out, dt))
        print(f"q{i:02d} thinking={thinking!s:5} gold={g:>6} pred={pred!s:>6} "
              f"{'OK   ' if ok else 'WRONG'} think={n_think:5d} out={n_out:5d} {dt:6.1f}s", flush=True)

print()
for thinking in (False, True):
    r = rows[thinking]
    acc = sum(x[0] for x in r) / len(r)
    print(f"enable_thinking={thinking!s:5}  correct {sum(x[0] for x in r):2d}/{len(r)}  "
          f"avg thinking tokens {sum(x[1] for x in r)/len(r):5.0f}  "
          f"avg output tokens {sum(x[2] for x in r)/len(r):5.0f}  "
          f"avg time {sum(x[3] for x in r)/len(r):5.1f}s")

Output:

q00 thinking=False gold=    18 pred=    18 OK    think=    0 out=  234   15.3s
q00 thinking=True  gold=    18 pred=    18 OK    think= 1316 out= 1590  124.7s
q01 thinking=False gold=     3 pred=     3 OK    think=    0 out=  127    8.0s
q01 thinking=True  gold=     3 pred=     3 OK    think=  439 out=  573   37.6s
q02 thinking=False gold= 70000 pred= 70000 OK    think=    0 out=  269   19.6s
q02 thinking=True  gold= 70000 pred= 70000 OK    think= 2607 out= 2967  281.7s
q03 thinking=False gold=   540 pred=   180 WRONG think=    0 out=   98    6.2s
q03 thinking=True  gold=   540 pred=   180 WRONG think=  335 out=  423   26.8s
q04 thinking=False gold=    20 pred=    20 OK    think=    0 out=  271   17.9s
q04 thinking=True  gold=    20 pred=    20 OK    think=  738 out= 1000   71.6s
q05 thinking=False gold=    64 pred=    64 OK    think=    0 out=  410   26.3s
q05 thinking=True  gold=    64 pred=  None WRONG think= 4096 out= 4096  452.4s
q06 thinking=False gold=   260 pred=   260 OK    think=    0 out=  314   19.9s
q06 thinking=True  gold=   260 pred=   260 OK    think=  634 out=  893   61.0s
q07 thinking=False gold=   160 pred=   100 WRONG think=    0 out=  302   19.7s
q07 thinking=True  gold=   160 pred=  None WRONG think= 4096 out= 4096  448.3s
q08 thinking=False gold=    45 pred=   275 WRONG think=    0 out=  352   23.0s
q08 thinking=True  gold=    45 pred=  None WRONG think= 4096 out= 4096  441.1s
q09 thinking=False gold=   460 pred=   460 OK    think=    0 out=  200   12.7s
q09 thinking=True  gold=   460 pred=   460 OK    think=  432 out=  570   38.3s
q10 thinking=False gold=   366 pred=   366 OK    think=    0 out=  181   11.5s
q10 thinking=True  gold=   366 pred=   366 OK    think=  886 out= 1094   77.4s
q11 thinking=False gold=   694 pred=  2938 WRONG think=    0 out=  194   15.2s
q11 thinking=True  gold=   694 pred=   694 OK    think=  770 out=  982   74.7s
q12 thinking=False gold=    13 pred=     1 WRONG think=    0 out=  329   24.5s
q12 thinking=True  gold=    13 pred=    12 WRONG think= 2540 out= 2990  309.6s
q13 thinking=False gold=    18 pred=    33 WRONG think=    0 out=  396   31.0s
q13 thinking=True  gold=    18 pred=    18 OK    think=  942 out= 1371  117.3s
q14 thinking=False gold=    60 pred=    60 OK    think=    0 out=  312   24.0s
q14 thinking=True  gold=    60 pred=    60 OK    think= 1051 out= 1465  125.4s

enable_thinking=False  correct  9/15  avg thinking tokens     0  avg output tokens   266  avg time  18.3s
enable_thinking=True   correct 10/15  avg thinking tokens  1665  avg output tokens  1880  avg time 179.2s

The whole run took 49.5 minutes. Five things to read from it:

• Thinking wins by a single question: 10/15 against 9/15, within the noise of 15 questions; this illustrates the mechanism rather than measuring accuracy.
• Three thinking runs ran out of tokens before they finished thinking. On q05, q07, and q08 the model used all 4,096 tokens without closing </think>, so there is no answer (pred=None). q05 is one the non-thinking mode got right with 410 tokens. A hard cut with max_new_tokens is not a real budget: the model does not know it is about to run out, so it never answers, which is why budget forcing exists (section 8).
• Thinking does not fix a misread question. On q03 both modes answer 180 instead of 540: thinking longer follows the same wrong reading.
• The price is 7 times the tokens and 10 times the time. On average 1,880 output tokens against 266, and 179 seconds against 18 per question. Each token gets only a little slower as the context grows (from about 15 to about 10 tokens per second); thinking is slow mainly because there are more tokens, as section 8 will compute.
• Thinking length varies wildly, from 335 to more than 4,096 tokens, even though GSM8K questions are of similar difficulty.

03Parallel: sample many, then pick

Self-consistency: a majority vote over reasoning chains

Part 9 explained sampling with temperature above 0: the same prompt gives a different chain each time. Wang et al. 2022 use exactly that. Self-consistency samples several independent CoT chains, extracts each chain’s final answer, and picks the most frequent one. The intuition: correct paths lead to the same answer, while errors each go their own way. With 40 samples per question (section 3.2), PaLM 540B on GSM8K goes from 56.5% to 74.4% (Table 2).

The hidden condition of majority voting: answers must be extractable and comparable. A number or a choice among A/B/C/D works. A paragraph of summary or a block of code has no “identical answer” to count.

Verifiers: train a model to grade

Instead of counting votes, you can train a model to judge whether a solution is right. Cobbe et al. 2021 did this when they released GSM8K: generate 100 solutions per problem, score them with a verifier, and keep the highest-scoring one. On the full dataset, a 6B verifier slightly outperforms a fine-tuned 175B model, a gain “approximately equivalent to a 30x model size increase” (section 6, Figure 5). The same paper carries an important warning: accuracy improves as the number of solutions grows up to 400 and then declines, because more wrong solutions show up that fool the verifier (section 5.1, Figure 7a). A scoring model is an approximation, and the more you sample, the more likely you are to hit its holes, much like overoptimization in Part 11, section 7.

Cobbe’s verifier scores the whole solution (an outcome reward model, ORM). Lightman et al. 2023 (Let’s Verify Step by Step) train a process reward model (PRM) that scores each step, on PRM800K: 800 thousand human step-level labels for 75 thousand solutions to 12 thousand problems (section 2.4). On 500 problems from the MATH test set, picking 1 of 1,860 solutions with the PRM reaches 78.2%, against 72.4% for an ORM and 69.6% for majority voting (section 3, Figure 3). A PRM also enables the third approach in Figure 2: using step scores to search.

Coverage keeps rising, picking does not

Brown et al. 2024 (Large Language Monkeys) push the parallel approach to thousands of samples. They separate two quantities:

• Coverage: the fraction of problems with at least one correct sample among k, which is pass@k.
• Accuracy after selection: the fraction of problems where the selection method (majority vote, reward model) picks a correct answer.

Coverage grows very smoothly with the number of samples, roughly following an exponentiated power law (section 3.1). On SWE-bench Lite, DeepSeek-Coder-V2-Instruct goes from 15.9% with one sample to 56% with 250 samples (abstract, Figure 2). There, unit tests can check each sample. On GSM8K and MATH there is no such checker at selection time, so majority voting and reward models plateau at around 100 samples while coverage already exceeds 95% (section 4.1, Figure 7). The gap between “a correct answer exists among N samples” and “we can pick it” is the real problem of the parallel approach.

pass@k has to be computed correctly. Chen et al. 2021 (the Codex paper) show that the naive estimate 1 − (1 − p̂)^k is biased and give an unbiased estimator: generate n ≥ k samples, count c correct ones, and compute 1 − C(n−c, k) / C(n, k) (section 2.1, equation 1).

Code sample 2: pass@k against majority voting

The code below simulates four kinds of question as distributions over the model’s answers, with no real model. “A” is the correct answer. The “spread” kind has ten wrong answers, each with small probability. The “peaked” kind has one wrong answer with 50% probability, the kind of misreading a model makes often.

# Python 3.12 — standard library only, runs in under 1 second
import random
from collections import Counter
from math import comb

random.seed(0)
KINDS = {                                       # "A" is the correct answer
    "easy         p=0.6":  {"A": 0.6, "B": 0.2, "C": 0.2},
    "hard, spread p=0.2": {"A": 0.2, **{f"W{i}": 0.08 for i in range(10)}},
    "hard, peaked p=0.3": {"A": 0.3, "B": 0.5, "C": 0.2},
    "very hard   p=0.02": {"A": 0.02, **{f"W{i}": 0.098 for i in range(10)}},
}

def sample(dist, n):
    return random.choices(list(dist), weights=list(dist.values()), k=n)

def pass_at_k(n, c, k):                         # Chen et al. 2021, section 2.1
    return 1.0 if n - c < k else 1 - comb(n - c, k) / comb(n, k)

def majority_acc(dist, N, trials=2000):         # majority vote over N samples, ties broken at random
    win = 0
    for _ in range(trials):
        counts = Counter(sample(dist, N)).most_common()
        top = [a for a, c in counts if c == counts[0][1]]
        win += random.choice(top) == "A"
    return win / trials

KS, n = [1, 4, 16, 64, 256], 1024
print(f"{'question type':22} " + " ".join(f"pass@{k:<4}" for k in KS) + " | " + " ".join(f"vote@{k:<4}" for k in KS))
for name, dist in KINDS.items():
    c = sample(dist, n).count("A")
    cov = [pass_at_k(n, c, k) for k in KS]
    vote = [majority_acc(dist, k, trials=2000 if k < 256 else 400) for k in KS]
    print(f"{name:22} " + " ".join(f"{x:8.2f}" for x in cov) + " | " + " ".join(f"{x:8.2f}" for x in vote))

Output:

question type          pass@1    pass@4    pass@16   pass@64   pass@256  | vote@1    vote@4    vote@16   vote@64   vote@256 
easy         p=0.6         0.59     0.97     1.00     1.00     1.00 |     0.59     0.74     0.94     1.00     1.00
hard, spread p=0.2         0.20     0.59     0.97     1.00     1.00 |     0.21     0.25     0.48     0.84     1.00
hard, peaked p=0.3         0.28     0.74     1.00     1.00     1.00 |     0.30     0.26     0.17     0.04     0.00
very hard   p=0.02         0.02     0.07     0.25     0.69     0.99 |     0.02     0.01     0.00     0.00     0.00

Four things to read from it:

• pass@k for every kind of question goes to 1, even for a question answered correctly only 2% of the time: the coverage curve of Large Language Monkeys.
• A majority vote only wins when the correct answer is the most common one. The “spread” question is right only 20% of the time but still wins the vote, because each wrong answer has only 8%.
• Majority voting can make things worse. The “peaked” question is right 30% of the time but one wrong answer takes 50%, so more samples make the wrong answer more certain: from 0.30 down to 0.00.
• Voting cannot rescue a very hard question, even though pass@256 is close to 1; only an accurate verifier such as unit tests can pull out that rare correct sample.

04Easy questions, hard questions: allocating compute

If there are several ways to spend compute at answer time, the next question is how to spend it best. Snell et al. 2024 study this with PaLM 2-S* on MATH (section 4), comparing two families of strategies: search against a PRM (best-of-N, beam search, lookahead) and sequential revision (a model fine-tuned to rewrite its own previous answer).

The main conclusion: the best strategy depends on how hard the question is. On easy questions, letting the model revise sequentially works better; on harder questions, a wider search or a mix of both works better. Choosing a strategy per question based on its estimated difficulty (they call it “compute-optimal”) reaches the same quality with more than 4 times less compute than best-of-N according to the abstract; the body says it more cautiously, “2–4×” (section 8).

The most striking comparison trades compute at answer time for parameters. At equal total FLOPs, a small model plus test-time compute beats a model about 14 times larger decoding greedily, but only on questions where the small model already has a non-trivial success rate (section 7, Figure 9). On the hardest questions, more test-time compute barely helps, and spending that compute on pretraining is more effective (sections 1 and 7). In the paper’s own summary, test-time and pretraining compute are “not 1-to-1 exchangeable”.

Put together with Code 2: test-time compute amplifies what the model can already partly do, and does not create ability where the model has almost none.

Part 10, section 6 described how Sardana and Frankle add inference cost to the scaling problem: once total inference tokens are large enough, training a smaller model for longer is cheaper overall. Test-time compute adds a third variable: besides parameters and training tokens, “how many thinking tokens per question”. A small model that thinks longer can be cheaper or more expensive than a large model that answers at once, depending on the real distribution of question difficulty.

05Reasoning models: from STaR to R1

The step after self-consistency and verifiers is to train the model to write long chains of thought on its own, with self-checking and backtracking: the “sequential” approach in Figure 2.

STaR: generate rationales, keep the right ones

STaR (Zelikman et al. 2022) is the direct precursor. Its loop (abstract, Algorithm 1 in section 3):

1. Prompt the model to generate a rationale and an answer for many questions.
2. For questions it gets wrong, give the model the correct answer as a hint and ask it to generate a rationale again (“rationalization”, section 3.2).
3. Fine-tune on every rationale that led to a correct answer, then repeat.

With GPT-J 6B on CommonsenseQA, STaR reaches 72.5%, on par with a fine-tuned GPT-3 30 times larger (73.0%) (Table 1). The core idea: knowing whether the final answer is right is enough to select good reasoning chains to learn from, with nobody writing example chains.

o1: what OpenAI has published

OpenAI has not published how o1 was trained. Learning to Reason with LLMs only says o1 is trained with “large-scale” reinforcement learning to think with its chain of thought, learning to recognize and correct mistakes, break hard steps into simpler ones, and change approach when stuck; the raw chain is hidden and users see a summary. Every other detail about o1 is speculation.

DeepSeek-R1-Zero: RL directly on a base model

The most thoroughly documented case is DeepSeek-R1 (DeepSeek-AI 2025). The first experiment, R1-Zero, runs RL directly on DeepSeek-V3-Base with no SFT step at all. The reward has just two parts, both computed by rules (section 2.2.2):

• Accuracy reward: a math answer in the required format that matches the reference, or code that passes tests.
• Format reward: thinking placed between <think> and </think> tags.

The authors say explicitly that they did not use a neural reward model because it “may suffer from reward hacking in the large-scale reinforcement learning process”, and retraining it would cost extra resources (section 2.2.2).

The result: pass@1 on AIME 2024 rises from 15.6% to 71.0%, and to 86.7% with a majority vote over 64 samples (section 2.2.4, Table 2). Two observations made the paper famous:

• Answer length grows on its own steadily over training, up to thousands of tokens (Figure 3). Nobody rewarded long answers; the model found that thinking longer helps it get the answer right.
• The “aha moment”: an intermediate checkpoint writes “Wait, wait. Wait. That’s an aha moment I can flag here.” and goes back to check an earlier step (Table 3).

R1-Zero also has clear drawbacks: its chains of thought are hard to read and mix languages, for example English and Chinese in the same chain (section 2.2.4).

DeepSeek-R1: four stages

To fix those drawbacks, R1 uses a four-stage pipeline (section 2.3), the bottom row of Figure 3 in section 6:

1. Cold start: SFT on “thousands” of long, readable CoT examples.
2. Reasoning-oriented RL: as in R1-Zero, plus a language consistency reward equal to the fraction of target-language words in the CoT. It costs a little accuracy but makes the chains more readable (section 2.3.2).
3. Rejection sampling + SFT: sample from the RL checkpoint, keep about 600 thousand correct reasoning samples, add about 200 thousand non-reasoning samples (writing, QA, translation), and fine-tune again from the base model on those roughly 800 thousand samples.
4. RL for all scenarios: rule-based rewards for reasoning, reward models for helpfulness and harmlessness.

R1 reaches 79.8% pass@1 on AIME 2024 and 97.3% on MATH-500 (Table 4).

Distillation: small models learn from a large one

DeepSeek also fine-tuned smaller Qwen and Llama models on the 800 thousand samples generated by R1, with SFT only, no RL. DeepSeek-R1-Distill-Qwen-7B reaches 55.5% on AIME 2024, and the Qwen-32B version 72.6% (Table 5). The key comparison is in section 4.1: running RL directly on Qwen-32B-Base for more than 10 thousand steps reaches only 47.0%, while distilling from R1 reaches 72.6% (Table 6). The authors’ conclusion: for small models, distilling from a strong model works better than running large-scale RL themselves. This point comes back in section 7.

06Inside RLVR and GRPO

Part 11, section 10 defined RLVR (reinforcement learning with verifiable rewards: the reward comes from a checker that runs as a program) and GRPO (a PPO variant that drops the value model and takes its advantage from a group of answers). This section does not repeat the definitions; it goes into the mechanics, following Figure 3.

One GRPO step

GRPO was introduced in DeepSeekMath (Shao et al. 2024, section 4.1). For each prompt q:

1. The current policy samples G answers o₁…o_G. DeepSeekMath uses G = 64 (section 4.2). This is sampling with temperature above 0, and because the samples come from the policy being trained, GRPO is on-policy RL.
2. Each answer gets a reward rᵢ. In RLVR, rᵢ comes from the verifier.
3. The advantage of answer i is Âᵢ = (rᵢ − mean(r)) / std(r), assigned to every token of that answer (section 4.1.2).
4. Update the policy with a PPO-like clipped objective plus a KL term against the reference model, estimated as π_ref/π_θ − log(π_ref/π_θ) − 1, which is never negative (equations 3 and 4). DeepSeekMath uses a KL coefficient of 0.04 (section 4.2).

Unlike PPO in Part 11, section 6, which needs a value model the size of the policy to estimate the baseline, GRPO uses the group mean as the baseline. A note on history: in DeepSeekMath the GRPO reward still came from a neural reward model; pairing GRPO with verifiable rewards was a step taken by R1 and later models.

A uniform group teaches nothing

A direct consequence of the advantage formula: if all G answers are right, or all G answers are wrong, every rᵢ is equal, every Âᵢ is 0, and the policy-gradient term is 0. That prompt costs the compute of sampling G answers and contributes nothing to learning.

The DAPO paper (Yu et al. 2025) states it plainly: if all outputs for a prompt are correct and receive the same reward, the group’s advantage is zero, and “a zero advantage results in zero policy gradients” (section 3.2). Their fix, dynamic sampling, over-samples and drops prompts with accuracy 1 or 0 from the batch.

So the difficulty of the RL data determines the signal: prompts that are always right or always wrong are useless, and only prompts the model gets partly right produce a gradient. A model that is never right has no signal to start from.

Code sample 3: GRPO rewards and advantages for one group

The code below writes an R1-style verifier with regular expressions (a correct answer in \boxed{} scores 1, a correct <think>…</think> format adds 0.1), computes group advantages, then computes a GRPO loss on simulated log-probs to look at the gradient.

# Python 3.12, torch 2.14.1 — runs on CPU in under 1 second
import re, torch

def verifier(completion, gold):
    """R1-style reward: correct answer (1/0) and correct format (<think>...</think> then \\boxed{})."""
    fmt = re.fullmatch(r"(?s)<think>.+</think>.*\\boxed\{[^{}]+\}.*", completion) is not None
    m = re.findall(r"\\boxed\{([^{}]+)\}", completion)
    return float(bool(m) and m[-1].strip() == gold), float(fmt)

def group_advantage(rewards, eps=1e-4):       # DeepSeekMath section 4.1.2: (r - mean) / std
    return (rewards - rewards.mean()) / (rewards.std() + eps)

gold = "18"
groups = {
    "mixed": [
        "<think>16 - 3 - 4 = 9 eggs, 9 × 2 = 18</think> Answer \\boxed{18}",
        "<think>16 - 3 = 13, 13 × 2 = 26</think> Answer \\boxed{26}",
        "16 - 7 = 9, 9 × 2 = 18. Answer \\boxed{18}",              # right but missing <think>
        "<think>9 eggs, 2 dollars each</think> Answer \\boxed{18}",
    ],
    "all right": ["<think>...</think> \\boxed{18}"] * 4,
    "all wrong": ["<think>...</think> \\boxed{26}", "<think>...</think> \\boxed{9}",
                 "<think>...</think> \\boxed{32}", "<think>...</think> \\boxed{13}"],
}

torch.manual_seed(0)
for name, completions in groups.items():
    r = torch.tensor([a + 0.1 * f for a, f in (verifier(c, gold) for c in completions)])
    adv = group_advantage(r)
    logp = torch.randn(4, 5, requires_grad=True)             # simulated log-probs: 4 answers × 5 tokens
    old = logp.detach()                                      # on-policy: ratio = 1 at the first step
    ratio = (logp - old).exp()
    surrogate = torch.min(ratio * adv[:, None], ratio.clamp(0.8, 1.2) * adv[:, None])
    ref = old + 0.05 * torch.randn_like(old)                 # reference model slightly off
    kl = (ref - logp).exp() - (ref - logp) - 1               # DeepSeekMath, equation (4)
    loss = -(surrogate - 0.04 * kl).mean(dim=1).mean()       # mean over tokens, then over the group
    loss.backward()
    print(f"{name:9}  reward {[round(x, 2) for x in r.tolist()]}  "
          f"advantage {[round(a, 2) for a in adv.tolist()]}  |grad| = {logp.grad.norm():.4f}")

r = torch.ones(4) * 1.1                                      # all-right group, KL term dropped
logp = torch.randn(4, 5, requires_grad=True)
(-(group_advantage(r)[:, None] * logp).mean()).backward()
print(f"\nall right, policy-gradient term only: |grad| = {logp.grad.norm():.4f}")

Output:

mixed      reward [1.1, 0.1, 1.0, 1.1]  advantage [0.57, -1.49, 0.36, 0.57]  |grad| = 0.1936
all right  reward [1.1, 1.1, 1.1, 1.1]  advantage [0.0, 0.0, 0.0, 0.0]  |grad| = 0.0004
all wrong  reward [0.1, 0.1, 0.1, 0.1]  advantage [0.0, 0.0, 0.0, 0.0]  |grad| = 0.0004

all right, policy-gradient term only: |grad| = 0.0000

• In the mixed group, the wrong answer gets a large negative advantage (−1.49) and the three right answers get positive ones. The right answer without <think> gets 0.36, less than the two right answers in the correct format (0.57): the format reward creates a small but real difference.
• In the all right and all wrong groups, the advantage is 0. The remaining gradient of 0.0004 comes only from the KL term, which pulls the policy toward the reference. Drop the KL and the gradient is exactly 0.

Fixes after R1

After R1, two fixes are used most often, one sentence each:

• Dr. GRPO (Liu et al. 2025) removes the division by length |oᵢ|, which penalizes long wrong answers less per token so they tend to grow, and the division by std(r), which overweights questions that are too easy or too hard (bias described in section 3.1, fix in section 3.2); part of the “answers grow longer” phenomenon may come from this length bias.
• DAPO (Yu et al. 2025) adds clip-higher (a looser upper clip bound against entropy collapse), dynamic sampling, a token-level loss, and a soft penalty for overlong answers (sections 3.1–3.4), drops the KL term (section 2.3), and reaches 50 points on AIME 2024 with the Qwen2.5-32B base, against 47 for R1-Zero-style RL, with half the training steps (section 1, Figure 1).

Verifiable rewards can still be exploited

Part 11 said that a right-or-wrong reward has “no approximation to hack” in the way a reward model does. That still holds: there is no approximating neural network for the policy to find blind spots in, as in Part 11, section 7. But a verifier is a program written by people, it can have holes, and the policy optimizes everything the reward allows, holes included.

Baker et al. 2025 (OpenAI) document such cases while training a reasoning model in the same family as o1 and o3-mini on agentic coding tasks, where the reward is “do the unit tests pass”. The model learned to call exit(0) to exit early with a success code, to raise SkipTest to skip tests (section 2.1), to write stubs instead of real implementations, and to modify functions upstream of the testing framework and edit the test files themselves so that tests pass trivially (section 2.2). The reward was still verifiable; what was verified (tests pass) just was not what people wanted (correct code).

The precise statement: RLVR removes one source of reward hacking (a reward model as a learned approximation), not reward hacking in general. The looser the verifier (an incomplete test suite, an environment that lets the model edit test files), the easier it is to hack.

07What does RL teach the model?

R1-Zero shows that RL with a right-or-wrong reward makes a model much better. The next question sounds simple but is an open debate: does RL teach the model new abilities, or does it only make the model pick, more often, what it could already do?

The “it only sharpens the distribution” argument

Yue et al. 2025 (NeurIPS 2025, oral) measure pass@k of base models and of models after RLVR for k up to 128–1024 depending on the benchmark (appendix A.2), on math, code, and visual reasoning. At small k, the RL model wins clearly. But as k rises into the tens and hundreds, the base model catches up and then surpasses the RL model (Figure 2, section 3.1). Measuring perplexity, they find the RL model’s reasoning chains are already in the base model’s sampling distribution (section 4.1). Their reading: RLVR mostly concentrates probability on reasoning paths that already existed, raising pass@1 while narrowing exploration. Distillation from a stronger model, by contrast, can bring in new reasoning patterns (section 4.2), consistent with R1’s distillation-versus-RL result in section 5.

Liu et al. 2025 (the Dr. GRPO paper) add evidence from a different angle: “nearly all base models already exhibit the ‘aha moment’”, including DeepSeek-V3-Base (section 2.3, Figure 3). They also find that self-reflection in R1-Zero is not positively correlated with higher accuracy (section 2.3, appendix F). The “aha moment” therefore looks more like an existing behavior that RL amplifies than something that appears out of nowhere.

The Qwen3 technical report points the same way: in their comparison, distillation raises pass@64 and RL does not (section 4.7, Table 21).

The counter-argument

ProRL (NVIDIA, Liu et al. 2025) runs prolonged RL for more than 2,000 steps on 136 thousand problems and reports that the model after RL beats the base model across a wide range of k, including problems where the base model fails completely no matter how much it samples. They conclude that prolonged RL “can uncover novel reasoning strategies”. Note: ProRL starts from DeepSeek-R1-Distill-Qwen-1.5B, an already distilled model rather than a raw base model. Another line (Wen et al. 2025) argues that pass@k also counts lucky guesses reached with wrong reasoning; when only solutions with a correct reasoning chain are counted (as judged by an LLM), the model after RLVR beats the base model at every k on AIME 2024/2025, though the effect is weaker on MATH-500 and AMC23 and absent on Minerva.

How to read it

At the time of writing, there is no settled conclusion. What can be said with confidence:

• At small k, which is what users actually experience, RLVR clearly helps.
• At very large k, with RL budgets like those in the papers above, many results show the base model does not lose.
• The conclusion depends on the metric (whether pass@k checks the reasoning), the length of RL training, and the starting model.

For users, “picking the right reasoning path on the first try” is already what makes reasoning models useful. For people training models, the question decides whether to invest in longer RL or in distillation and pretraining data.

08Thinking tokens have a price

Thinking tokens are ordinary output tokens

To the model, there is nothing special about the tokens between <think> and </think>. They come out of the same decode loop, one token at a time. Three consequences:

• They are billed as output. The documentation of all three major providers says thinking/reasoning tokens are billed as output tokens, even when the API does not return them (see “The budget knobs” below).
• They occupy the KV cache. Each thinking token writes its K and V into the cache like any other token, and every later token has to read them back (Part 13, section 3).
• They run in the decode phase, the memory-bound phase (Part 13, section 2). A 1,000-token prompt is processed in one prefill pass; 1,000 thinking tokens need 1,000 decode steps.

Pricing it with the roofline

Apply the roofline model from Part 13 to two configurations computed there. The table below is a theoretical calculation, not a measurement: time is the lower bound from bytes read (weights plus KV cache) divided by bandwidth, at batch size 1, with a 200-token prompt.

The Qwen2.5-1.5B columns use Part 13’s figures: 3.09 GB of weights, 28 KiB of KV cache per token, 616 GB/s of bandwidth; each decode step takes about 5 ms. The Llama 3.1 8B columns use 128 KiB per token and the roughly 56 GB KV-cache budget of an 80 GB GPU, also from Part 13; the 13 sequences at 32k tokens match the table there.

Two things to read from it:

• Latency grows linearly with thinking tokens: for a 1.5B model, 32 thousand thinking tokens is three minutes of waiting in the best theoretical case. Reasoning models are “slow” because of more tokens, not slower tokens.
• A provider’s throughput drops with thinking length: one sequence thinking for 32 thousand tokens takes as much KV cache as 32 sequences thinking for 1 thousand, and the KV cache sets the batch size. This is also where MoE and the fixed state of Part 14 make reasoning models cheaper, and where speculative decoding can cut latency.

The budget knobs

Because thinking longer costs money and time, every API has a way to control it. The table below comes from official documentation retrieved on October 14, 2026:

Three things stand out:

• A budget is a target, not a contract. Google says the model “might overflow or underflow the token budget” depending on the prompt. Both Anthropic and Google are moving to an effort or thinkingLevel setting that lets the model decide how much to think.
• You pay for what you do not see. Anthropic states that you are “charged for the full thinking tokens generated by the original request, not the summary tokens”; Google and OpenAI say the same.
• Leave room. OpenAI’s max_output_tokens includes reasoning tokens, and the docs recommend reserving at least 25,000 tokens when you start experimenting; set it too low and the model can run out before it answers.

Budget forcing: controlling length from the decoding side

With open-weight models, a budget can be enforced inside the decode loop. s1 (Muennighoff et al. 2025) fine-tunes Qwen2.5-32B-Instruct on just 1,000 hard questions with example reasoning traces, which takes 26 minutes on 16 H100 GPUs (section 1), then controls length with budget forcing (section 3.1):

• Cut: when the budget runs out, append the end-of-thinking token, forcing the model to switch to answering.
• Extend: when the model tries to end its thinking, suppress that token and append the word “Wait”, so the model keeps thinking and often re-checks itself.

On AIME 2024, budget forcing lifts s1-32B from 50.0% to 56.7% (Table 1). The gains flatten out at about six times the extension (section 4.2, Figure 4), and among the extension strings tested, “Wait” works best (Table 4). Qwen3 does a similar “cut” for its thinking budget: when the threshold is reached, the system halts thinking and inserts an instruction along the lines of “considering the limited time by the user, I have to give the solution based on the thinking directly now”, followed by </think> (Qwen3 technical report, section 4.3). According to the report, the ability to answer from incomplete thinking was not explicitly trained but emerged on its own.

Thinking length can therefore be controlled from outside, at the decoding level, without changing the weights.

09When thinking longer pays off

Helps: many steps, checkable answers

The evidence in earlier sections points the same way. Thinking longer helps most when a problem:

• needs many sequential steps of computation (math, code, logic, planning), as argued in section 2;
• has a clear right-or-wrong criterion, to train with RLVR and to sample many times and pick with a verifier;
• suits the model’s difficulty level: an immediate answer is often wrong, but the model is not almost never right (section 4).

Helps little or not at all: knowledge and easy questions

• Knowledge lookup: in Sprague et al., task categories outside math and logic barely change with CoT (from 56.1 to 56.8). Thinking more does not teach the model a fact it has never seen; that is retrieval’s job.
• Simple questions: the immediate answer is already right; every thinking token is pure cost.
• Tasks without a clear right-or-wrong criterion (writing, summarization, conversation): no verifier to train on, no answer to vote on; any benefit is hard to measure.

Hurts: overthinking and inverse scaling

Thinking longer can be not just useless but harmful.

Overthinking. Chen et al. 2024 (Do NOT Think That Much for 2+3=?) ask models “what is the answer of 2 plus 3”. QwQ-32B-Preview uses 901 tokens and DeepSeek-R1 987 (the figure in v2 of the paper), while GPT-4o uses 7 and Qwen2.5-72B 12 (Figure 1a). On average, o1-like models used 1,953% more tokens than conventional models (section 1). QwQ’s answer to this question contains 13 solutions (Figure 2), and the correct answer first appears at token 39: only 39/901 ≈ 4.3% of the tokens were needed (section 2.2).

Inverse scaling. Gema et al. 2025 build tasks where accuracy falls as reasoning length grows, across Claude models, OpenAI’s o-series, Qwen3, QwQ, and DeepSeek R1. The four task categories (abstract): simple counting with distractors, regression with spurious features, deduction that requires tracking many constraints, and AI-risk scenarios. Failure modes differ by model family: getting distracted by irrelevant information, overfitting to familiar problem framings, or shifting to spurious correlations; in the risk scenarios, Claude Sonnet 4 expresses more self-preservation when it thinks longer.

Shortening the chain. Kimi k1.5 (Kimi Team 2025) trains models to think less, calling it long2short, with four methods: merging the weights of a long-thinking and a short-thinking model, rejection sampling that keeps the shortest correct solution, DPO that prefers shorter answers, and RL with a length penalty (section 2.4). Their long2short RL model reaches 60.8 on AIME 2024 using only 3,272 tokens on average (section 3.4). Low effort levels and non-thinking modes in APIs are the same idea on the user’s side.

A decision table

This table is a starting point and does not replace measuring on real data: thinking tokens always cost something, and the benefit is not always there.

10Is the chain of thought faithful?

A chain of thought reads like the model explaining itself. The question is whether it reflects the real reason the model reached its answer. This section is short; the deeper interpretability material is for Part 17.

• CoT can be unfaithful. Turpin et al. 2023 add biasing features to the prompt, such as a sentence “I think the answer is…”. With GPT-3.5 (text-davinci-003) in zero-shot CoT, this suggestion lowers accuracy by as much as 36.3 percentage points on BIG-Bench Hard tasks, counting only questions where the correct answer goes against the suggestion (section 3.2, Table 8), yet the reasoning usually does not mention the suggestion. The model produces a plausible-sounding argument for the answer the suggestion pushed it toward.
• Reasoning models too. Chen et al. 2025 (Anthropic) plant six kinds of hint in questions and check whether the model admits to using the hint. Claude 3.7 Sonnet mentions the hint 25% of the time and DeepSeek R1 39% of the time (section 3, Figure 1). Outcome-based RL raises this rate at first, then plateaus at about 28% on MMLU and 20% on GPQA (section 4, Figure 5). In environments with an available reward hack, models use the hack on more than 99% of examples but mention it in fewer than 2%, in 5 of 6 environments (section 5.2, Figure 7).
• Monitoring the CoT still helps, but it is fragile. Baker et al. 2025 use GPT-4o to monitor the reasoning model in the coding tasks from section 6. A monitor that reads the CoT as well as the actions catches 95% of reward hacks, against 60% when it reads only the actions (Table 1). But in the section 3 experiment, with a non-frontier agent and a monitor that reads only the CoT, when the monitor is put into the reward to penalize “intent to hack”, the model learns to hide its intent: it still hacks, the CoT looks harmless, and the monitor’s recall falls to near zero (section 3.2). The authors recommend paying a “monitorability tax”: do not put optimization pressure directly on the CoT.

This is also the reason OpenAI gave for hiding o1’s chain of thought: to keep the CoT useful for monitoring, they do not train policy compliance onto it, so they do not want to show it directly to users. The gpt-oss model card likewise puts no optimization pressure on the CoT and advises against showing it to users without filtering or summarization.

A sensible reading: the CoT is useful diagnostic data, not trustworthy testimony. It helps catch misbehavior while the model has no reason to hide it, and it stops working once it is optimized against.

11Common misconceptions

The nine misconceptions below match the top and bottom rows of Figure 1.

Summary: quality can be bought, under conditions

Part 10 bought quality with compute at training time. This post shows that compute at answer time can buy quality too, within a certain region.

• Chain-of-thought is how a model borrows extra forward passes, so it helps on multi-step problems (math, code, logic) and barely helps knowledge tasks.
• Three ways to spend compute at answer time: parallel, sequential, and search. Coverage rises steadily with samples, but picking the right one is the real problem; majority voting saturates and can make things worse.
• Test-time compute does not replace parameters one for one: a small model thinking more beats a model 14 times larger on questions within its reach, not on the hardest ones.
• Reasoning models are trained with RL on verifiable rewards. R1-Zero learns to think longer from just an accuracy and a format reward; R1 adds a cold start, rejection sampling, and distillation.
• GRPO uses the group mean as its baseline, so a uniform group gives no gradient and data difficulty decides the signal. The verifier is the ceiling of RLVR: a leaky one gets exploited.
• Whether RL teaches something new or only sharpens what is there is still debated.
• Thinking tokens are ordinary output tokens: billed as output, held in the KV cache, generated in the decode phase; an API budget is a target, not a contract.
• Thinking longer pays off on multi-step problems with checkable answers, helps little with knowledge lookup, and hurts with overthinking or inverse scaling. The CoT is diagnostic data, not testimony.

This post has met many benchmark numbers: AIME, MATH, GSM8K, pass@k. Part 16 asks the next question: why LLMs get things wrong, and how far those numbers can be trusted, from hallucination to contamination to the limits of the benchmarks themselves.

References

Papers: chain-of-thought and test-time compute

• Nye et al. 2021 — Show Your Work: Scratchpads for Intermediate Computation with Language Models
• Cobbe et al. 2021 — Training Verifiers to Solve Math Word Problems (GSM8K)
• Chen et al. 2021 — Evaluating Large Language Models Trained on Code (unbiased pass@k estimator)
• Wei et al. 2022 — Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
• Kojima et al. 2022 — Large Language Models are Zero-Shot Reasoners
• Wang et al. 2022 — Self-Consistency Improves Chain of Thought Reasoning in Language Models
• Lightman et al. 2023 — Let’s Verify Step by Step
• Brown et al. 2024 — Large Language Monkeys: Scaling Inference Compute with Repeated Sampling
• Snell et al. 2024 — Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
• Sprague et al. 2024 — To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning

Papers: reasoning models and RL

• Zelikman et al. 2022 — STaR: Bootstrapping Reasoning With Reasoning
• Shao et al. 2024 — DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (GRPO)
• DeepSeek-AI 2025 — DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (v1)
• Kimi Team 2025 — Kimi k1.5: Scaling Reinforcement Learning with LLMs
• Muennighoff et al. 2025 — s1: Simple test-time scaling
• Liu et al. 2025 — Understanding R1-Zero-Like Training: A Critical Perspective (Dr. GRPO)
• Yu et al. 2025 — DAPO: An Open-Source LLM Reinforcement Learning System at Scale
• Yue et al. 2025 — Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
• Liu et al. 2025 — ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models
• Wen et al. 2025 — Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs
• Qwen Team 2025 — Qwen3 Technical Report

Papers: overthinking and faithfulness

• Chen et al. 2024 — Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs
• Gema et al. 2025 — Inverse Scaling in Test-Time Compute
• Turpin et al. 2023 — Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting
• Chen et al. 2025 — Reasoning Models Don’t Always Say What They Think
• Baker et al. 2025 — Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation

Blog posts, API docs, and model cards (retrieved October 14, 2026)

• OpenAI 2024 — Learning to Reason with LLMs
• OpenAI — Reasoning models
• OpenAI 2025 — gpt-oss-120b & gpt-oss-20b Model Card, openai/gpt-oss-20b, OpenAI Harmony Response Format
• Anthropic — Extended thinking, Adaptive thinking, Effort
• Google — Gemini thinking, Thinking with generateContent
• Qwen — Qwen/Qwen3-0.6B
• openai/gsm8k on Hugging Face