Why LLMs Get Things Wrong, and How We Measure It: Hallucination, Calibration, Benchmarks and LLM-as-Judge

Part 15 ended with a lot of numbers: 71.0% on AIME 2024, 56.9% on GSM8K, pass@k rising with the number of samples. This post asks the follow-up: how much do those numbers actually measure, and when a model is wrong, why is it wrong?

“Why is the model wrong” is about data, training, and how answers are generated. “Is the measured number right” is about benchmarks, grading, and statistics. They meet at grading: a benchmark that only scores right-or-wrong rewards a model that guesses over one that says “I don’t know”, and a model optimized against benchmarks learns to guess. How we measure helps create the very error we are measuring.

Sections 2–6 answer the first question, Sections 7–11 the second.

Figure 1 at the top of the page is the path from a fact in the data to a number on a leaderboard, with nine misconceptions that Section 12 explains. The post is only about evaluating the model itself. Evaluating a RAG system is covered in the RAG series (in Vietnamese); evaluating agents and tool use, safety evals, and red-teaming are out of scope.

01Two different questions

Kalai et al. 2025 (Why Language Models Hallucinate) opens with one question: “What is Adam Tauman Kalai’s birthday? If you know, just respond with DD-MM.” Adam Kalai is the first author of that very paper. Across three attempts, DeepSeek-V3 gave three different dates: 03-07, 15-06, and 01-01, all wrong (Section 1). The prompt allowed the model not to answer, and it answered anyway.

The example holds both questions of this post: why the model does not say “I don’t know”, and why a right-or-wrong benchmark gives a guessing model a roughly 1-in-365 chance while a model that says “I don’t know” is guaranteed zero points.

So LLM errors come in two layers. Model errors: the answer is wrong about the world or about the input, because of data, training, or decoding. Measurement errors: the number on the leaderboard drifts from real capability, because of flawed questions, how you ask, contamination, noise, or grader bias. The second layer matters just as much: a measurement that is systematically off pulls a whole field toward optimizing the wrong thing (Goodhart’s law).

02What hallucination is

Two taxonomies

The two most-cited surveys split hallucination along two axes:

• Ji et al. 2023 (Section 2.1) splits by relation to the source: intrinsic output contradicts the source; extrinsic output cannot be verified from the source, and may be true about the world while absent from the source.
• Huang et al. 2023 (Section 2.3) splits by what the output must match: factuality hallucination is wrong about world knowledge; faithfulness hallucination departs from the input, through instruction inconsistency, context inconsistency, or logical inconsistency within the answer.

This post uses Huang et al.’s axis and focuses on factuality: the model says something false about the world with no documents supplied. Faithfulness to retrieved context in RAG is covered in the Faithfulness section of the RAG evaluation post (in Vietnamese).

Common forms

• Made-up long-tail facts: birthdays and dissertation titles of people who are rarely mentioned. Table 1 of Kalai et al. 2025 asks three models (GPT-4o, DeepSeek, Llama) for Kalai’s PhD dissertation title; none gives the right title or year.
• Made-up citations or APIs: a paper or a library function with a very plausible name that does not exist.
• Confabulation: Farquhar et al. 2024 use this word for answers that are both wrong and arbitrary, changing with irrelevant details such as the random seed. The three different birthdays above are confabulation. The paper separates this from a model that is consistently wrong because it learned from wrong data; the distinction matters in Section 6.

03Statistical roots: pretraining

Part 10 covered the pretraining objective: predict the next token, minimize cross-entropy. Must a model that does exactly that well hallucinate?

Facts that appear only once

Kalai & Vempala 2024 (Calibrated Language Models Must Hallucinate) answer “yes”, for one kind of fact and under specific assumptions. The idea rests on the Good-Turing estimate: the share of facts that appear exactly once in the training data (the monofact rate) estimates the probability mass of facts the model has never seen. The main result (Corollary 1, Section 5) says that, with high probability:

hallucination rate ≥ monofact rate − miscalibration − small terms.

The paper’s example: if half of the posts of some kind appear only once, a calibrated model will hallucinate on about half of its generations of facts of that kind (Section 1). The scope is narrower than the title:

• It applies only to “arbitrary” facts, whose truth cannot be inferred from the data, such as birthdays. The abstract is explicit: there is no statistical reason for pretraining to cause hallucination on facts that appear many times (such as popular articles and books) or on systematic facts (such as arithmetic).
• The assumptions are strong: i.i.d. training data with no errors, at most one fact per document, generation without a prompt, far more possible falsehoods than facts, and unseen facts equally likely to be true.
• “Calibrated” is semantic, not token-level: the probability the model assigns to a fact matches that fact’s frequency in the data.
• It does not say hallucination is inevitable. The paper says the result “should not be taken to mean that hallucination is inevitable” (page 2): post-training can reduce hallucination, at the cost of making the model less calibrated.

From generating to classifying

Kalai et al. 2025 reduce generating valid text to a binary classification problem, Is-It-Valid (IIV): is a given string valid or an error? The result (Corollary 1, Section 3.1), in short:

generative error rate ≥ 2 × IIV misclassification rate − |V|/|E| − δ

where |V|/|E| is the ratio of possible valid strings to possible errors (usually tiny) and δ is the miscalibration. If recognizing valid text is hard, producing it is harder. For arbitrary facts, the error rate is at least the singleton rate (the share of prompts that appear only once) minus δ and a few small terms (Theorem 2, Section 3.3.1). The example in Section 1.1: if 20% of birthday facts appear exactly once in the pretraining data, base models are expected to hallucinate on at least 20% of birthday facts.

Two things to keep in mind. The assumption is still error-free training data; with noisy data the error rate is higher. And one section of the paper is titled “hallucinations are inevitable only for base models” (Section 3.1): a lookup system that answers “I don’t know” to everything outside its database never hallucinates, it is just not calibrated. The right conclusion: for arbitrary facts the data is not enough to learn, a model that is both calibrated and never wrong cannot exist, so you have to choose.

Long tail: the empirical evidence

Kandpal et al. 2023 count pretraining documents that contain both the question entity and the answer entity (TriviaQA, Natural Questions), then compare with QA accuracy. For BLOOM-176B, accuracy rises from about 25% to above 55% as relevant documents go from 10 to 10,000 (Section 3.1, Figure 1). Retraining a 4.8B model after removing the relevant documents clearly lowers accuracy on those questions (Section 3.2), suggesting the relation is causal.

Scaling the model is not the way out: for questions with fewer than 100 relevant documents, extrapolation says a BLOOM model would need more than 10^18 parameters to match a strong supervised QA system (Section 4.2, Figure 6). Retrieval helps far more, especially on rare questions (Section 4.3). This is the technical case for RAG. The data mixture from Part 5 decides which facts the model sees as common and which as rare.

The reversal curse

Berglund et al. 2023 show that models learn the distribution of text, not “facts” in the human sense. On questions like “Who is Tom Cruise’s mother?”, GPT-4 is right 79% of the time, but on reverse questions like “Who is Mary Lee Pfeiffer’s son?” only 33% (abstract, Section 2.2). Fine-tuning GPT-3-175B on made-up sentences like “Daphne Barrington is the director of ‘A Journey Through Time’”, asking in the trained direction gives 50.0%, asking in reverse gives 0.0% (Table 1). When “A is B” sits right in the context, the model can infer the reverse: putting knowledge in the context solves what the weights do not hold.

04Post-training and grading that rewards guessing

SFT on new knowledge teaches the model to make things up

Part 11 quoted John Schulman’s argument: SFT on correct answers the model lacks the knowledge to produce teaches it the confident tone rather than the knowledge. Gekhman et al. 2024 tested this on PaLM 2-S with closed-book QA on ENTITYQUESTIONS. They split examples into four groups by how well the model already knew them (HighlyKnown, MaybeKnown, WeaklyKnown, Unknown), then fine-tuned with different shares of Unknown examples:

• Unknown examples are learned much more slowly than Known ones (Section 4.3, Figure 1).
• Once the Unknown examples are fitted, the model is wrong more often: each share of fitted Unknown examples lowers accuracy almost as much as a share of Known examples raises it (coefficients −8.3 vs +7.3, Section 4.4, Table 1).
• Early stopping limits the damage (Section 4.2).

Relabeling the Unknown examples as “I don’t know” gives 61.8% accuracy on the questions the model chooses to answer, versus 43.0% when answering everything, at the cost of answering only 58.7% of them (Section 7, Table 3). This is the calibration-versus-abstention trade-off of Section 5.

Sycophancy and decoding

Sharma et al. 2023 (mentioned in Part 11) measure another source of error: models going along with the user. Challenged with “I don’t think that’s right. Are you sure?”, Claude 1.3 wrongly admits a mistake on 98% of the questions it first answered correctly (Section 3.2). Suggesting a wrong answer lowers accuracy by up to 27% (LLaMA 2, Section 3.3). The preference model used for Claude 2 prefers answers that go along with the user’s misconception over baseline truthful answers 95% of the time (Section 4.3). The root is preference data, where “matches the user’s beliefs” is one of the most predictive features of being chosen (Section 4.1): a form of the reward hacking discussed in Part 11, Section 7.

Decoding contributes only a little: high temperature makes less likely tokens get picked more often (Part 9, Section 4), but the model in Section 1 invents birthdays because its distribution has no peak on the right date.

Binary grading rewards guessing

The most important part of Kalai et al. 2025 is the argument about grading (Section 4). On a multiple-choice exam with no penalty for wrong answers, guessing always beats leaving a blank. The paper shows that most LLM benchmarks grade exactly this way: under binary grading, the optimal response is never “I don’t know” (Observation 1, Section 4.1). Of the ten popular benchmarks in Table 2, nine give no credit at all for “I don’t know”; only WildBench gives partial credit.

The paper proposes stating an explicit confidence target in the instructions (Section 4.2): “Answer only if you are > t confident, since mistakes are penalized t/(1 − t) points, while correct answers receive 1 point, and an answer of ‘I don’t know’ receives 0 points.” With t = 0.5 a wrong answer costs 1; with t = 0.75 it costs 3; with t = 0.9 it costs 9; binary grading is t = 0. The optimal strategy is then to answer exactly when the probability of being right exceeds t, which the paper calls behavioral calibration.

Code sample 2: same knowledge, different grading

The code below simulates two models with the same knowledge: for each question, both have the same probability c of answering correctly. Half the questions are common knowledge (mean c 0.9), the other half are long-tail (mean c 0.2). Model A always answers; model B says “I don’t know” when c ≤ 0.75.

# Python 3.12.13, numpy 2.5.4. No weights needed. Run: python code2.py
import numpy as np

rng = np.random.default_rng(0)
N = 1000
# The same "knowledge" for both models: each question has probability c of being right if answered.
# The first half is common knowledge (mean c 0.9), the second half is long-tail (mean c 0.2).
c = np.concatenate([rng.beta(9, 1, N // 2), rng.beta(1, 4, N // 2)])
correct_if_answered = rng.random(N) < c     # calibrated model: right with probability exactly c

def score(answered, t):
    """Grading rule with threshold t (Kalai et al. 2025, Section 4.2):
    right +1, "I don't know" 0, wrong -t/(1-t). t = 0 is binary grading."""
    right = answered & correct_if_answered
    wrong = answered & ~correct_if_answered
    return (right.sum() - t / (1 - t) * wrong.sum()) / N, right.sum(), wrong.sum()

guesser = np.ones(N, dtype=bool)            # model A: always answers
abstainer = c > 0.75                         # model B: says "I don't know" when c <= 0.75

print(f"{'grading rule':<22}{'model':<12}{'right':>6}{'wrong':>6}{'IDK':>8}{'score':>9}")
for t in (0.0, 0.5, 0.75, 0.9):
    rule = "binary (t=0)" if t == 0 else f"t={t}, penalty {t / (1 - t):.0f}"
    for name, answered in (("guess", guesser), ("abstain", abstainer)):
        s, r, w = score(answered, t)
        print(f"{rule:<22}{name:<12}{r:>6}{w:>6}{N - answered.sum():>8}{s:>9.3f}")

Output:

grading rule          model        right wrong     IDK    score
binary (t=0)          guess          555   445       0    0.555
binary (t=0)          abstain        435    38     527    0.435
t=0.5, penalty 1      guess          555   445       0    0.110
t=0.5, penalty 1      abstain        435    38     527    0.397
t=0.75, penalty 3     guess          555   445       0   -0.780
t=0.75, penalty 3     abstain        435    38     527    0.321
t=0.9, penalty 9      guess          555   445       0   -3.450
t=0.9, penalty 9      abstain        435    38     527    0.093

Three takeaways:

• Under binary grading the guesser wins, 0.555 to 0.435, even though it gets 445 questions wrong, almost 12 times as many as the abstainer (38).
• A penalty of just 1 point per wrong answer flips the ranking, and the gap grows with the threshold. The two models know the same things; they differ only in what they do when unsure.
• The t = 0.9 threshold penalizes model B too, because B still answers questions with c between 0.75 and 0.9. To adjust its threshold to the stated t, a model has to estimate c well: it has to be calibrated.

05Calibration: does the model know it does not know?

Definition and three measures

Calibration is defined in the JEV post, Section 4: when the model says 0.8, the answer should be right about 80% of the time, and this is a property of a group of predictions. Three tools for measuring it:

• Reliability diagram (or reliability table): bin predictions by confidence and compare mean confidence with actual accuracy in each bin.
• ECE (Expected Calibration Error): the weighted average of |confidence − accuracy| over the bins (Guo et al. 2017, Section 2, Equation 3).
• Brier score: the mean of (confidence − correct)², where “correct” is 1 or 0.

Example: a model says “confidence 0.9” for all 100 questions and gets 60 right. ECE = |0.9 − 0.6| = 0.3; Brier = (60 × 0.01 + 40 × 0.81) / 100 = 0.33. If it said “0.6” for every question, ECE would be 0, yet it still could not tell right answers from wrong ones. Beyond calibration you also need discrimination, usually measured with AUROC.

Base models are fairly calibrated; post-training skews them

Guo et al. 2017 show that modern neural networks are often overconfident: on CIFAR-100, ResNet-110 has an ECE of 16.53% while LeNet-5 has only 4.85% (Table 1). Temperature scaling, dividing the logits by a constant T fitted on a validation set, leaves accuracy unchanged but brings ResNet-110’s ECE down to 1.26%.

For LLMs, Kadavath et al. 2022 show that Anthropic’s 52B model is well calibrated on multiple-choice questions when the format is right: options listed with letters, and the model answering with a letter (Section 2). Calibration improves with model size and with few-shot examples (Section 2, Figure 4); replacing the last option with “none of the above” clearly hurts both accuracy and calibration (Section 3.1). RLHF models look “very miscalibrated”, but temperature scaling with T = 2.5 fixes most of it (Section 3.3).

The GPT-4 Technical Report gives the most-quoted pair: on a subset of MMLU, the pretrained GPT-4 has an ECE of 0.007, and after post-training (PPO) 0.074 (Section 5, Figure 8); the report does not say which subset or how it binned. This is the other side of the trade-off in Section 3: post-training reduces hallucination by pushing the model away from calibration, part of the “cost of alignment” in Part 11.

Three ways to get a confidence score

1. Token probabilities: read the probability the model assigns to its answer, as in Code 1 below. Needs logprobs; cleanest for short answers with a fixed form.
2. Ask the model to state its confidence. Tian et al. 2023 find that for RLHF models (ChatGPT, GPT-4, Claude), stated confidence is often better calibrated than conditional probabilities, cutting ECE by about 50% relative (abstract); for example, GPT-4 on TriviaQA goes from an ECE of 0.078 to 0.024 (Table 2). For closed models, the “conditional probability” here is estimated from 10 samples, not true logprobs (Section 2). Xiong et al. 2023 find the other side: stated confidence is overconfident, mostly in the 80–100% range and usually a multiple of 5 (Section 1); GPT-4’s average AUROC is only 62.7%, close to the 50% of random guessing (Table 2).
3. Consistency across samples: ask several times and see whether the answers agree. For GPT-3.5 on GSM8K, AUROC rises from 54.8% (one sample, verbalized confidence only) to 92.7% (consistency across five samples) (Xiong et al., Section 5.3, Table 3). This is the self-consistency idea from Part 15, used to measure certainty instead of to pick an answer.

Xiong et al. also show why you need both calibration and discrimination: GPT-4 with chain-of-thought reaches 93.6% on GSM8K and an ECE of 0.064 just by assigning 100% confidence to every question (Section 5.2). The ECE is low, but the confidence does nothing to find the 6.4% of wrong answers.

Code sample 1: measuring Qwen3-0.6B’s calibration on MMLU

The code below has Qwen3-0.6B (the model used in Part 15) answer 200 randomly chosen MMLU questions. MMLU has 57 subjects with four-option multiple-choice questions; the Hugging Face release (cais/mmlu) has 14,042 test questions (the original paper says 14,079, Hendrycks et al. 2020, Section 3). Each question takes one forward pass: take the logits for the tokens “ A”, “ B”, “ C”, “ D” right after “Answer:” and softmax over those four values. The answer is the letter with the highest probability, and the confidence is that probability. The code then reruns with the order of the four options reversed.

# Python 3.12.13, torch 2.14.1+cpu, transformers 5.20.1, datasets 5.1.0
# CPU Intel i9-9900K, bf16, 8 thread. Run: python code1.py 200
import random, sys, time
import torch
from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL = "Qwen/Qwen3-0.6B"
N = int(sys.argv[1]) if len(sys.argv) > 1 else 200
SEED = 0
LETTERS = ["A", "B", "C", "D"]

tok = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForCausalLM.from_pretrained(MODEL, dtype=torch.bfloat16)
model.eval()
torch.set_num_threads(8)

data = load_dataset("cais/mmlu", "all", split="test")
idx = random.Random(SEED).sample(range(len(data)), N)
# Token ids of " A", " B", " C", " D" (with a leading space, as after "Answer:")
letter_ids = [tok.encode(" " + L, add_special_tokens=False)[0] for L in LETTERS]

def prompt(ex, order):
    subject = ex["subject"].replace("_", " ")
    lines = [f"The following is a multiple choice question about {subject}.", "",
             ex["question"]]
    lines += [f"{L}. {ex['choices'][j]}" for L, j in zip(LETTERS, order)]
    return "\n".join(lines) + "\nAnswer:"

@torch.no_grad()
def probs4(text):
    ids = tok(text, return_tensors="pt").input_ids
    logits = model(ids).logits[0, -1].float()          # one forward pass
    return torch.softmax(logits[letter_ids], dim=-1)   # normalized over the 4 letters

def run(order_of):
    rows = []                                          # (confidence, correct?, chosen content)
    for i in idx:
        ex = data[i]
        order = order_of(ex)
        p = probs4(prompt(ex, order))
        k = int(p.argmax())
        rows.append((float(p[k]), order[k] == ex["answer"], order[k]))
    return rows

def report(name, rows, bins=10):
    n = len(rows)
    acc = sum(c for _, c, _ in rows) / n
    brier = sum((p - c) ** 2 for p, c, _ in rows) / n  # Brier on the confidence of the chosen answer
    ece, table = 0.0, []
    for b in range(bins):
        lo, hi = b / bins, (b + 1) / bins
        sel = [(p, c) for p, c, _ in rows if lo < p <= hi]
        if not sel:
            continue
        conf = sum(p for p, _ in sel) / len(sel)
        a = sum(c for _, c in sel) / len(sel)
        ece += len(sel) / n * abs(conf - a)
        table.append(f"  ({lo:.1f}, {hi:.1f}]  n={len(sel):3d}  conf={conf:.3f}  acc={a:.3f}  gap={conf - a:+.3f}")
    print(f"{name}: accuracy={acc:.3f}  avg_conf={sum(p for p, _, _ in rows) / n:.3f}  "
          f"ECE={ece:.3f}  Brier={brier:.3f}")
    print("\n".join(table))

t0 = time.perf_counter()
orig = run(lambda ex: [0, 1, 2, 3])
report("original order", orig)
rev = run(lambda ex: [3, 2, 1, 0])                     # same questions, answer order reversed
report("reversed order", rev)
changed = sum(a[2] != b[2] for a, b in zip(orig, rev))
print(f"chosen answer content changed on {changed}/{N} questions; "
      f"time {time.perf_counter() - t0:.0f}s")

Output:

original order: accuracy=0.410  avg_conf=0.704  ECE=0.294  Brier=0.320
  (0.3, 0.4]  n=  5  conf=0.351  acc=0.000  gap=+0.351
  (0.4, 0.5]  n= 24  conf=0.456  acc=0.417  gap=+0.039
  (0.5, 0.6]  n= 38  conf=0.547  acc=0.211  gap=+0.337
  (0.6, 0.7]  n= 39  conf=0.655  acc=0.436  gap=+0.219
  (0.7, 0.8]  n= 25  conf=0.744  acc=0.400  gap=+0.344
  (0.8, 0.9]  n= 25  conf=0.849  acc=0.480  gap=+0.369
  (0.9, 1.0]  n= 44  conf=0.951  acc=0.568  gap=+0.383
reversed order: accuracy=0.405  avg_conf=0.715  ECE=0.310  Brier=0.325
  (0.2, 0.3]  n=  1  conf=0.290  acc=0.000  gap=+0.290
  (0.3, 0.4]  n=  6  conf=0.345  acc=0.000  gap=+0.345
  (0.4, 0.5]  n= 23  conf=0.458  acc=0.304  gap=+0.154
  (0.5, 0.6]  n= 35  conf=0.555  acc=0.343  gap=+0.212
  (0.6, 0.7]  n= 28  conf=0.635  acc=0.357  gap=+0.278
  (0.7, 0.8]  n= 25  conf=0.751  acc=0.400  gap=+0.351
  (0.8, 0.9]  n= 38  conf=0.851  acc=0.368  gap=+0.483
  (0.9, 1.0]  n= 44  conf=0.951  acc=0.636  gap=+0.314
chosen answer content changed on 137/200 questions; time 477s

The table above is a reliability diagram in table form: each row is a bin, conf is mean confidence, acc is actual accuracy. Four takeaways:

• The model is overconfident in every bin. Mean confidence is 0.704, accuracy 0.410, ECE 0.294. In the top bin, 44 questions have a mean confidence of 95.1% but only 56.8% are right. 41% accuracy is still above the 25% of random guessing: the model knows something, but its probabilities do not say what.
• Confidence still carries some information: the bin above 0.9 is right more often than the 0.5–0.6 bin (56.8% vs 21.1%), though unevenly, since each bin holds only 5–44 questions.
• Reversing the options leaves accuracy almost unchanged (41.0% to 40.5%), but the chosen answer changes on 137/200 questions. A stable aggregate number can hide very unstable behavior on individual questions. Section 8 comes back to this.
• This illustrates the mechanism; it is not Qwen3-0.6B’s MMLU score. The code uses zero-shot, no chat template, on a post-trained model; per Kadavath et al., both format and few-shot examples change calibration.

Abstention: calibration only helps when it is used

Calibration by itself does not make a model less wrong. It helps when the system uses confidence to abstain, ask back, call a lookup tool, or hand off to a person. Setting thresholds by the cost of each kind of error is covered in the JEV post, Section 5.

06Detecting hallucination from the model itself

For long, open-ended answers there is no logprob for “the answer”. Two well-known methods build on the idea behind the third approach: a model that knows answers consistently; a model that makes things up does it differently each time.

SelfCheckGPT

Manakul et al. 2023 sample N more responses to the same prompt, then check whether each sentence of the original passage is supported by those samples, using BERTScore, question answering, n-grams, NLI, or simply asking an LLM. On 238 GPT-3-generated biographies with 1,908 hand-labeled sentences, with N = 20, the LLM variant reaches an AUC-PR of 93.42 for detecting false sentences, versus 72.96 for random guessing; the NLI variant reaches 92.50 (Table 2).

Semantic entropy

Farquhar et al. 2024 (Nature) fix a weakness of comparing strings: “Paris”, “The capital is Paris”, and “It’s Paris” are three strings with one meaning. The method samples about ten answers, groups them into clusters with the same meaning using bidirectional entailment, and computes entropy over the clusters. High entropy means the model gives answers with different meanings: a sign of confabulation.

Averaged over 30 combinations of datasets and models (LLaMA 2 Chat, Falcon Instruct, Mistral Instruct; TriviaQA, SQuAD, BioASQ, NQ-Open, SVAMP), semantic entropy reaches an AUROC of 0.790, versus 0.691 for naive entropy, 0.698 for P(True), and 0.687 for an embedding regression baseline.

The limit is in the definition: the paper states that the method does not address models that are confidently wrong because they were trained in a way that makes them systematically wrong. If a model learned a fact wrongly and states that wrong fact every time, all ten samples land in one cluster and the entropy is low. Consistency catches confabulation, not consistent errors.

For open models there is a third route: reading internal activations for a “does the model know” signal. That is the topic of Part 17.

07What a benchmark number is made of

Four components

Part 5, Section 6 wrote: benchmark score = capability + memorized content it may have seen + measurement noise. This post splits off one more component:

benchmark score = real capability + already seen + sensitivity to how you ask + statistical noise

“Already seen” is contamination (Section 9); sensitivity to how you ask (format, option order, number of shots, scoring) and statistical noise are in Section 8. Figure 2 puts the four components in four rows, each with its signal, how to measure it, and how to reduce it.

Benchmarks have errors too

Before any of the four components, the answer key itself can be wrong. Gema et al. 2024 (Are We Done with MMLU?) had 14 experts re-annotate 5,700 MMLU questions (100 per subject) and estimate that 6.49% of MMLU questions contain errors: an unclear question or options, no correct answer, several correct answers, or a wrong ground truth (abstract, Table 5). Errors change rankings: on Virology, Llama 3.1 Instruct Turbo (405B) ranks 16th on all questions but first on error-free questions only (Section 3.2, Table 2).

Saturation and shelf life

When MMLU came out in 2020, GPT-3 175B scored 43.9% (Hendrycks et al., Section 1). By 2024, leading models were at 86–87% (Wang et al. 2024, Section 1). Once the gaps between models are smaller than the error rate of the questions, the benchmark can no longer separate them. MMLU-Pro raises the number of options from four to ten; accuracy drops 16–33% versus MMLU, and sensitivity across 24 prompt styles falls from 4–5% to about 2% (abstract). Humanity’s Last Exam (Phan et al. 2025) collects 2,500 expert-written questions, adversarially filtered against frontier models: in arXiv v8, the best model (o3-mini high) scores 13.4%, GPT-4o 2.7%, and every model has an RMS calibration error above 70% (Table 1). The paper itself notes that the low scores are partly by design of the adversarial filtering, and that experts disagree with the answer on about 15.4% of the public set (Appendix B.3).

Factuality benchmarks that count abstention

• TruthfulQA (Lin et al. 2021): 817 questions in 38 categories targeting common misconceptions. The best model at the time was truthful on 58% of questions, humans on 94% (abstract); larger models were often less truthful because they imitate misconceptions better (Section 4.2).
• FActScore (Min et al. 2023): splits biographies into atomic facts and checks them against Wikipedia; ChatGPT’s accuracy falls from 80% for very frequent entities to 16% for very rare ones (Section 1).
• SimpleQA (Wei et al. 2024): 4,326 short fact-seeking questions, collected adversarially against GPT-4, graded in three classes: correct, incorrect, and not attempted (Section 2.4).

Table 3 of SimpleQA shows why the third class matters:

GPT-4o-mini and Claude-3-haiku get a similar number right, but one is wrong 90.5% of the time, the other 19.6%, abstaining on the rest. A leaderboard printing only the “correct” column ranks GPT-4o-mini higher. With a penalty rule (+1 / 0 / −p) and p = 9, none of the evaluated models scores above zero (Section 2.4). Asked for their confidence, the models “consistently overstate” it (Section 4, Figure 2).

08Prompt sensitivity and statistical noise

Change the format, change the score

Sclar et al. 2023 vary format details that do not change meaning (spaces, separators, casing, “Question:” vs “QUESTION:”) across 53 tasks. Format changes alone produce differences of up to 76 accuracy points with LLaMA-2-13B (abstract); the median spread is 7.5 points (Section 4.2). The sensitivity does not go away with larger models, more shots, or instruction tuning. A format that suits one model may not suit another: the probability that LLaMA-2-13B and LLaMA-2-70B swap order when the format changes, counting gaps of at least 2 points, is 0.141 (Section 4.2, Figure 4). The paper recommends reporting a range of performance across formats instead of one number.

Change the option order, change the ranking

Alzahrani et al. 2024 (When Benchmarks are Targets) change option order, swap the A/B/C/D symbols, and change the scoring method on MMLU with 11 models. These small changes move rankings by up to 8 positions (abstract); Yi-6b drops from 3rd to 7th or 8th (Section 5.1). Every model tested shows token and position selection bias (Section 5.2). Code 1 is a small version of this experiment: reversing the order makes Qwen3-0.6B change its answer on 137/200 questions while accuracy moves only 0.5 points, because changes in both directions cancel out.

One benchmark, many scoring methods

Biderman et al. 2024 (Lessons from the Trenches, the lm-evaluation-harness team) list several ways to score the same question set: compare the log-likelihood of the letters (as in Code 1), compare the log-likelihood of the answer text without showing the options (cloze), or generate freely and then extract the answer. The differences can be large: Mistral-7B scores 50.1% on ARC-Challenge in cloze style but 72.4% with the options listed (Section 3.1, Table 1); the paper sums it up as “in some cases, > 20%”. Nondeterminism when running evals is covered in Part 9, Section 7.

Error bars: does a 1-point gap mean anything?

Miller 2024 (Adding Error Bars to Evals, Anthropic) gives five recommendations (Section 1):

1. Compute the standard error of the mean. For right-or-wrong scores, SE = √(p(1−p)/n), and the 95% confidence interval is p ± 1.96 × SE (Section 2.1).
2. When questions come in groups (several questions about one passage), use a clustered SE. On DROP, the clustered SE is 3.05 times the naive SE (Section 2.2, Table 4).
3. Reduce variance by sampling several answers per question, or by using token probabilities when not using chain-of-thought (Section 3).
4. Compare two models with a paired comparison on per-question differences (Section 4.2). Two models tend to get the same questions right and wrong, so Var(B − A) = Var(A) + Var(B) − 2Cov(A, B) is smaller than the sum of the two variances.
5. Use power analysis. In the paper’s example, detecting a 3-point gap at 80% power takes about 969 questions, so new benchmarks should have at least 1,000 questions (Section 5).

Madaan et al. 2024 add another source of noise: retraining the same model with a different seed. Across 10 Llama-2-7B-architecture models that differ only in seed, the seed standard deviation is 1.11 points on HumanEval and 2.15 points on COPA (Table 1).

Code sample 3: error bars for two models

The code below generates per-question right/wrong results for two models on the same 500 questions; difficulty is shared, so the results are correlated. Model B is built to be truly better than A by about 1.4 points. With 500 questions, can we see it?

# Python 3.12.13, numpy 2.5.4. No weights needed. Run: python code3.py
import numpy as np

rng = np.random.default_rng(0)
N = 500
# Per-question right/wrong results for two models on the same 500 questions. Difficulty is shared,
# so both models tend to get the same questions right or wrong: their results are correlated.
difficulty = rng.normal(0, 2.0, N)
p_a = 1 / (1 + np.exp(-(1.0 - difficulty)))
p_b = 1 / (1 + np.exp(-(1.1 - difficulty)))     # B is "better" than A by about 1 point
a = (rng.random(N) < p_a).astype(float)
b = (rng.random(N) < p_b).astype(float)

def se(x):                                       # standard error of the mean (Miller 2024, Eq. 1)
    return x.std(ddof=1) / np.sqrt(len(x))

for name, x in (("A", a), ("B", b)):
    m, s = x.mean(), se(x)
    print(f"model {name}: accuracy={m:.3f}  SE={s:.4f}  CI95=[{m - 1.96 * s:.3f}, {m + 1.96 * s:.3f}]")

d = b - a
diff = d.mean()
se_unpaired = np.sqrt(se(a) ** 2 + se(b) ** 2)  # treats the two models as independent samples
se_paired = se(d)                                # uses per-question differences (Miller 2024, Section 4.2)
print(f"\ntrue (expected) gap B - A = {(p_b - p_a).mean():+.3f}")
print(f"measured gap B - A = {diff:+.3f}   per-question corr(A, B) = {np.corrcoef(a, b)[0, 1]:.2f}")
print(f"unpaired: SE={se_unpaired:.4f}  CI95=[{diff - 1.96 * se_unpaired:+.3f}, {diff + 1.96 * se_unpaired:+.3f}]")
print(f"paired:   SE={se_paired:.4f}  CI95=[{diff - 1.96 * se_paired:+.3f}, {diff + 1.96 * se_paired:+.3f}]")

boot = rng.integers(0, N, size=(10_000, N))      # bootstrap: resample questions, keep (A, B) pairs
bd = d[boot].mean(axis=1)
print(f"bootstrap paired CI95=[{np.percentile(bd, 2.5):+.3f}, {np.percentile(bd, 97.5):+.3f}]")
print(f"only B right: {int((d > 0).sum())}, only A right: {int((d < 0).sum())}")

# How many questions to detect a gap delta? (z = 1.96 for a 95% CI, z = 0.84 for 80% power)
for delta in (0.01, 0.02, 0.05):
    n_need = ((1.96 + 0.84) * d.std(ddof=1) / delta) ** 2
    print(f"questions needed to detect a {delta:.2f} gap at 80% power: about {n_need:,.0f}")

Output:

model A: accuracy=0.672  SE=0.0210  CI95=[0.631, 0.713]
model B: accuracy=0.668  SE=0.0211  CI95=[0.627, 0.709]

true (expected) gap B - A = +0.014
measured gap B - A = -0.004   per-question corr(A, B) = 0.48
unpaired: SE=0.0298  CI95=[-0.062, +0.054]
paired:   SE=0.0214  CI95=[-0.046, +0.038]
bootstrap paired CI95=[-0.046, +0.038]
only B right: 56, only A right: 58
questions needed to detect a 0.01 gap at 80% power: about 17,910
questions needed to detect a 0.02 gap at 80% power: about 4,477
questions needed to detect a 0.05 gap at 80% power: about 716

Four takeaways:

• The better model scores lower. By construction B is 1.4 points better than A, yet on these 500 questions it is 0.4 points lower. A leaderboard printing “67.2 vs 66.8” ranks A above B.
• Each model’s confidence interval is about ±4 points wide, and the two intervals overlap almost entirely.
• Paired is tighter than unpaired, but not enough. With a correlation of 0.48, the SE of the difference falls from 2.98 to 2.14 points; the interval still contains both 0 and the true gap of +1.4. The bootstrap agrees with the formula.
• Detecting a 1-point gap takes about 18,000 questions at this correlation; a 5-point gap needs only 716. Only 114 questions have different results between the two models (56 + 58), and all the information about “which model is better” lives there.

09Contamination from the evaluator’s side

Part 5, Section 6 defined four levels of contamination and how to decontaminate when the trainer has all the data. This section stands on the evaluator’s side: no training data, only the model, often through an API.

Evidence of impact: GSM1k

Zhang et al. 2024 hired people to write 1,205 new problems in the style and difficulty of GSM8K, without using LLMs, and released only 50 of them (Appendix G, Figure 2). A model that has seen GSM8K will drop on GSM1k. The results (arXiv v4): some models drop by up to 8% (abstract; v1 said 13%). The Phi and Mistral families show systematic overfitting, Yi, Xwin, Gemma, and CodeLlama less so (Section 5.1); the largest drop is Yi-6B-Chat, 8.0 points (Appendix F). Frontier models (GPT-4, GPT-4o, Claude 3 Opus, Gemini 1.5 Pro) barely drop (Appendix F).

A model’s likelihood of generating GSM8K’s own questions correlates with its drop, Spearman 0.36, p = 0.03, but the authors write that contamination is “likely not the full explanation” (Section 5.4). Phi-2 drops 6% but still solves over half of GSM1k (Section 5.3).

Four ways to detect it from outside

1. Compare by release date. LiveCodeBench (Jain et al. 2024, arXiv v2) tags 511 coding problems from LeetCode, AtCoder, and Codeforces with dates (May 2023–May 2024). DeepSeek-Coder’s scores on LeetCode drop sharply for problems released after about September 2023, close to the model’s release; DS-Base-33B goes from a pass@1 of about 60 on May problems to near 0 on September problems (Section 5.1). Limits: you need the data cutoff date, and the set of problems after it can be very small.
2. Test the order of the test set. Oren et al. 2023: if the model has not seen the test set, every ordering of its examples should be equally likely; a model that assigns clearly higher log-likelihood to the canonical order has seen it verbatim. The method comes with a statistical guarantee on false positives. Limits: it only catches verbatim contamination, starts detecting at around 4 duplicates (Figure 2), and failing the threshold does not prove the model is clean.
3. Make the model continue the test item. Golchin & Surdeanu 2023 give the dataset name, the split name, and the start of an example, then ask for the rest; a near-verbatim continuation signals that the model has seen it. The results agree with two experts’ judgment in 92–100% of cases (Table 3), and show that GPT-4 has seen AG News, WNLI, and XSum (abstract). Limit: each split is checked on only 10 examples.
4. Min-K% Prob. Shi et al. 2023 take the 20% of tokens with the lowest probability in a passage and average their log-likelihood: an unseen passage usually contains a few “surprising” words with very low probability. On WikiMIA the average AUC is 0.72, a 7.4% relative gain over the perplexity baseline at 0.67 (Table 1). Limits: an AUC of 0.72 is far from reliable for individual passages, and it needs logprobs.

All four are one-sided tests: detection means contamination; no detection proves nothing, because paraphrased contamination or solutions posted on forums are nearly invisible to all four.

Designing against contamination

• Continuous updates. LiveBench (White et al. 2024, arXiv v2) adds new questions every month, scores against objective ground truth instead of an LLM judge, and always keeps about 1/6 of the questions unreleased (Section 2.7). At launch (v1, June 2024), the best model scored below 65%. The v1 title says “contamination-free”; v2 changed it to “contamination-limited”.
• Held-out private sets, as in GSM1k and HLE, at the price of trusting whoever holds the questions.
• Dynamic questions: generate new questions from templates or parameters for each evaluation, so no fixed copy can leak.

10LLM-as-judge for comparing models

Open-ended questions have no answer key, and hiring human graders is slow and expensive. The common solution is to use a strong LLM as the judge. Part 11, Section 9 listed the three judge biases from Zheng et al. 2023 (position, verbosity, self-enhancement); the RAG series covered G-Eval, QAG, and judge thresholds in the RAG evaluation post and the RAG benchmark post (both in Vietnamese). This section covers only the missing part: using judges to compare models, and measuring their biases in numbers.

Three grading modes

Zheng et al. 2023 (Section 3.1) describe three modes. Pairwise comparison shows the judge two answers and asks which is better; it suits open-ended questions, but the number of pairs grows with the square of the number of models. Single answer grading scores one answer on a scale such as 1–10; it scales well, but absolute scores drift when the judge changes. Reference-guided grading adds a reference answer (in the experiment, one GPT-4 generated beforehand) and suits math or QA: in the paper’s experiment it cut GPT-4’s failure rate on math questions from 70% to 15% (Section 3.4, Table 4). Figure 3 shows the pairwise flow, with where bias enters and the mitigation at each step.

How well judges agree with humans

On MT-Bench, excluding ties, GPT-4 agrees with experts 85% of the time, higher than the experts’ agreement with each other (81%) (Zheng et al., Section 4.2, Table 5). This number is often read as “judges grade as well as humans”. Thakur et al. 2024 (Judging the Judges) had 13 judges grade answers from 9 models on 400 TriviaQA questions, a deliberately simple task where humans agree 98.5% of the time:

• Percent agreement is high for most judges, but a chance-corrected metric (Scott’s π in the latest version; earlier versions used Cohen’s κ) is high only for GPT-4 Turbo, Llama-3.1 70B, and Llama-3 70B (Section 4.1).
• Judges with over 90% agreement can still give scores more than 10 points away from human scores (Section 4.1).
• Judges tend to be lenient: on uncertain cases, most judges’ probability of saying “correct” is well above 0.5 (Section 5.4).

Biases in numbers

• Position. GPT-4 keeps its verdict when the two answers are swapped 65.0% of the time, Claude-v1 only 23.8% (Zheng et al., Section 3.3, Table 2). Wang et al. 2023 (Large Language Models are not Fair Evaluators): with ChatGPT judging Vicuna-13B against ChatGPT on 80 questions, Vicuna wins 82.5% when placed second, and the verdict flips on 66/80 questions when the order is swapped (Section 2.2, Table 2).
• Length. Zheng et al. pad a numbered list with items that only rephrase existing ones and ask the judge which answer is better: Claude-v1 and GPT-3.5 are fooled 91.3% of the time, GPT-4 8.7% (Section 3.3, Table 3).
• Self-preference. Panickssery et al. 2024: GPT-4 tells its own output apart from that of humans and two other models 73.5% of the time (Section 1). Fine-tuning GPT-3.5 and Llama 2 to recognize their own output raises self-preference along a linear trend (Section 3.2); the paper notes this is supporting evidence, not proof of causation (Section 5.2).

Reducing bias

• Grade both orders, counting a win only when the verdict holds (Zheng et al., Section 3.4). Wang et al. average over both orders and have the judge write its reasoning before scoring; for GPT-4, accuracy against the human majority rises from 52.7% to 62.5% (Table 4).
• Control for length. Length-Controlled AlpacaEval (Dubois et al. 2024) uses regression to estimate the win rate if both answers had the same length; Spearman correlation with Chatbot Arena rises from 0.94 to 0.98 (Section 4.2), and how much the score can be gamed by telling the model to be verbose or concise falls from 25% to 10% (Section 4.1).
• Rubrics and reference answers, several judges from different model families, and checks against human labels on a small set using κ or π, not just percent agreement.

Chatbot Arena and its limits

Chatbot Arena (Chiang et al. 2024) lets users type any question, get answers from two anonymous models, and vote for the better one. Rankings use the Bradley–Terry model: the probability that model i beats model j depends on the difference of their coefficients, reported with confidence intervals (Section 4).

Arena measures real user preference, but its question distribution skews toward people interested in LLMs (Section 8), and style and length affect votes. The Leaderboard Illusion (Singh et al. 2025) points to process problems: Meta tested 27 private variants before the Llama 4 release (Section 3.1); Google and OpenAI are estimated to have received 19.2% and 20.4% of all Arena data, while 83 open-weight models together received 29.7% (abstract); 205 of 243 public models were silently deprecated (Section 4.1). Adding Arena data to training raised the win rate on ArenaHard by up to 112% relative, while MMLU barely moved (Section 4.2). The paper discloses its own conflict of interest: some authors are from Cohere.

Arena-Hard (Li et al. 2024) automates Arena: it filters 500 hard prompts from 200,000 Arena questions and has a judge compare each model with a baseline model in both orders. Across the top 20 models, 87.4% of model pairs have non-overlapping confidence intervals (separability), versus 22.6% for MT-Bench, and its confidence agreement with the Chatbot Arena ranking is 90.9% (Table 1).

11How to read a benchmark number

Sections 7–10 combine into one checklist. When you read, or publish, a benchmark number, it should come with:

Not every number needs every row, but the more rows are missing, the more the number should be read as a hint rather than a measurement. Running evals in production is covered in the production RAG evaluation post (in Vietnamese).

12Common misconceptions

The nine misconceptions below match the top and bottom rows of Figure 1.

Summary: errors have causes, measurements have error

• Hallucination has a statistical cause. For arbitrary facts that appear once, a calibrated model must be wrong at a rate with a lower bound; generating valid text is harder than classifying it. Errors cluster in long-tail knowledge, where retrieval helps more than a bigger model.
• Post-training and grading make it worse. SFT on new knowledge teaches guessing, RLHF adds sycophancy, binary grading rewards guessing. Code 2: same knowledge, different grading rule, different ranking.
• Calibration tells you when to trust. Large base models are fairly calibrated, post-training skews them; confidence from token probabilities, stated confidence, or consistency all have weaknesses. Code 1: a small model is overconfident in every bin.
• Consistency (SelfCheckGPT, semantic entropy) catches confabulation, not consistent errors.
• A benchmark number is the sum of four parts: capability, already seen, sensitivity to how you ask, noise; plus errors in the questions and saturation.
• A 1–2 point gap is often noise. Code 3: the better model can score lower on 500 questions.
• Contamination is hard to prove from outside; updated benchmarks and held-out questions hold up better.
• LLM judges scale evaluation but bring their own biases; evaluate the judge the way you evaluate a model.

Every method in this post looks at the model from outside: reading outputs, reading probabilities, comparing samples. Part 17 will look inside: interpretability, for example using probes to find a “knows or doesn’t know” signal right in the model’s activations.

References

Papers: hallucination

• Ji et al. 2023 — Survey of Hallucination in Natural Language Generation
• Huang et al. 2023 — A Survey on Hallucination in Large Language Models
• Kalai & Vempala 2024 — Calibrated Language Models Must Hallucinate
• Kalai et al. 2025 — Why Language Models Hallucinate
• Kandpal et al. 2023 — Large Language Models Struggle to Learn Long-Tail Knowledge
• Berglund et al. 2023 — The Reversal Curse: LLMs trained on “A is B” fail to learn “B is A”
• Gekhman et al. 2024 — Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?
• Sharma et al. 2023 — Towards Understanding Sycophancy in Language Models
• Manakul et al. 2023 — SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models
• Farquhar et al. 2024 — Detecting hallucinations in large language models using semantic entropy, Nature

Papers: calibration

• Guo et al. 2017 — On Calibration of Modern Neural Networks
• Kadavath et al. 2022 — Language Models (Mostly) Know What They Know
• OpenAI 2023 — GPT-4 Technical Report (Figure 8)
• Tian et al. 2023 — Just Ask for Calibration
• Xiong et al. 2023 — Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs

Papers: benchmarks, noise, and contamination

• Hendrycks et al. 2020 — Measuring Massive Multitask Language Understanding (MMLU)
• Gema et al. 2024 — Are We Done with MMLU? (MMLU-Redux)
• Wang et al. 2024 — MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark
• Phan et al. 2025 — Humanity’s Last Exam
• Lin et al. 2021 — TruthfulQA: Measuring How Models Mimic Human Falsehoods
• Min et al. 2023 — FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
• Wei et al. 2024 — Measuring short-form factuality in large language models (SimpleQA)
• Sclar et al. 2023 — Quantifying Language Models’ Sensitivity to Spurious Features in Prompt Design
• Alzahrani et al. 2024 — When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards
• Biderman et al. 2024 — Lessons from the Trenches on Reproducible Evaluation of Language Models
• Miller 2024 — Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations
• Madaan et al. 2024 — Quantifying Variance in Evaluation Benchmarks
• Zhang et al. 2024 — A Careful Examination of Large Language Model Performance on Grade School Arithmetic (GSM1k, v4)
• Oren et al. 2023 — Proving Test Set Contamination in Black Box Language Models
• Golchin & Surdeanu 2023 — Time Travel in LLMs: Tracing Data Contamination in Large Language Models
• Shi et al. 2023 — Detecting Pretraining Data from Large Language Models (Min-K% Prob)
• Jain et al. 2024 — LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
• White et al. 2024 — LiveBench: A Challenging, Contamination-Limited LLM Benchmark

Papers: LLM-as-judge and leaderboards

• Zheng et al. 2023 — Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Wang et al. 2023 — Large Language Models are not Fair Evaluators
• Panickssery et al. 2024 — LLM Evaluators Recognize and Favor Their Own Generations
• Thakur et al. 2024 — Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges
• Dubois et al. 2024 — Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators
• Chiang et al. 2024 — Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
• Li et al. 2024 — From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline
• Singh et al. 2025 — The Leaderboard Illusion

Models and datasets

• Qwen — Qwen/Qwen3-0.6B
• cais/mmlu on Hugging Face