Pretraining: Next-Token Prediction and Scaling Laws

Part 9 treated the model as a function that already exists: feed in a context, get good logits back, and the rest is a matter of picking a token. This post goes back to the question that comes before all of that: how does the model learn to produce those logits?

The short answer is simple enough to underestimate. Pretraining optimizes exactly one thing: the cross-entropy of the next token, over a few trillion tokens. No human labels, no reward, no meaningful auxiliary objective. The difficulty is not in the objective but in the scale. When one training run costs millions of GPU-hours, three engineering questions decide whether it succeeds:

• How do you split the budget? With a fixed amount of compute, should you train a large model on few tokens or a small model on many? That is the question scaling laws answer.
• How do you fit it onto hardware? An 8B model already does not fit on one 80 GB GPU during training. You need mixed precision, and you need to split the model and the data across thousands of GPUs.
• How do you know it is breaking? Loss jumps, gradients explode, GPUs die mid-run. At large scale these are not accidents; they happen every day.

This post covers all three, in the order of Figure 1 at the top of the page.

01From logits to loss

Part 1, section 5.2 already made the most important point: pretraining is self-supervised learning. Text produces its own labels, because the next token is always the answer for the tokens before it. This post does not repeat that.

What needs adding is the role of the logits. In part 9, logits were the input to decoding. Here, logits are the input to the loss. Same forward pass, same [batch, seq_len, vocab_size] tensor; only the consumer changes:

• at inference time, we only need the logits at the last position to choose the next token;
• at training time, we use the logits at every position, compare each one with the real token that follows it, and average.

Everything else in pretraining, from scaling laws to parallelism, is engineering to run that comparison on as many tokens as possible, as cheaply as possible.

02The objective: next-token prediction

The formula

For a sequence x₁, x₂, …, x_T, the model outputs a probability p(x_t | x_<t) at each position. The pretraining loss is the average negative log-likelihood:

L = −(1/T) · Σ_t log p(x_t | x₁, …, x_{t−1})

This is exactly the cross-entropy between the model’s predicted distribution and the “one-hot” distribution of the real token. The unit is nats when using the natural logarithm, as every deep learning library does.

Three details make this objective surprisingly cheap:

1. Teacher forcing. At position t, the model always sees the real tokens x_<t, never its own guesses from earlier steps. That makes every position independent in terms of its input.
2. The causal mask. The triangular mask in attention guarantees that position t cannot see x_t or anything after it. Combined with teacher forcing, we get the loss for all T positions in a single forward pass instead of running the model T times. One sequence of 8,192 tokens yields 8,191 prediction problems in one run.
3. No labels needed. Shifting the sequence by one position gives the targets.

Shifting labels by one step, why labels = input_ids does not mean the model learns to copy the current token, and the three kinds of mask in a sample are covered in part 5, section 11. This post builds on that.

Loss, perplexity and bits-per-byte

The same quantity can be read three ways:

A perplexity of 8 means that, on average, the model is no more certain than picking uniformly among 8 candidates. That is intuitive, but loss and perplexity are both per token, so they depend on the tokenizer. A tokenizer that splits text into many small tokens creates many easy positions (the second half of a word is usually easy to guess), which pulls the average loss down without the model being any better.

Bits-per-byte fixes this by dividing the total information by the number of bytes in the original text, which does not depend on the tokenizer. That is why comparisons between models with different tokenizers, such as tables comparing Llama, Qwen and Gemma, use bits-per-byte or bits-per-character rather than perplexity. Part 4 showed that the same Vietnamese sentence can produce very different token counts under different tokenizers, which is exactly where comparing perplexities stops making sense.

Computing it on a real model

The code below runs the base version of Qwen2.5-0.5B (no post-training) on CPU. It computes the loss in two ways, the library’s and a hand-written one from the logits, then converts it to perplexity and bits-per-byte for an English passage and a Vietnamese passage with equivalent content.

# Python 3.12, torch 2.14.1 (CPU), transformers 5.18.0
import math
import torch
import torch.nn.functional as F
from transformers import AutoModelForCausalLM, AutoTokenizer

name = "Qwen/Qwen2.5-0.5B"                     # base model, no post-training
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name, dtype=torch.float32).eval()

texts = {
    "en": "The capital of France is Paris. It is known for the Eiffel Tower, "
          "the Louvre museum and its cafés along the river Seine.",
    "vi": "Thủ đô của Pháp là Paris. Thành phố nổi tiếng với tháp Eiffel, "
          "bảo tàng Louvre và những quán cà phê dọc sông Seine.",
}

for lang, text in texts.items():
    # Prepend <|endoftext|> so the first token of the text is predicted too.
    ids = tok(tok.eos_token + text, return_tensors="pt").input_ids
    with torch.no_grad():
        out = model(ids, labels=ids)           # HF shifts the labels internally

    # Recompute by hand: logits at position t predict the token at t+1.
    logits = out.logits[0, :-1]                # [T-1, vocab]
    target = ids[0, 1:]                        # [T-1]
    nll = F.cross_entropy(logits, target, reduction="none")   # nats, per token

    n_tok, n_byte = target.numel(), len(text.encode("utf-8"))
    loss = nll.mean().item()
    print(f"[{lang}] logits {tuple(out.logits.shape)}  tokens={n_tok}  bytes={n_byte}")
    print(f"     loss HF     = {out.loss.item():.4f} nats/token")
    print(f"     loss manual = {loss:.4f} nats/token")
    print(f"     perplexity  = {math.exp(loss):.2f}")
    print(f"     bits/byte   = {nll.sum().item() / math.log(2) / n_byte:.4f}")
    worst = nll.topk(3)
    print("     hardest:", [(tok.decode(target[i]), round(v.item(), 2))
                                   for v, i in zip(worst.values, worst.indices)])

Output:

[en] logits (1, 32, 151936)  tokens=31  bytes=121
     loss HF     = 2.0763 nats/token
     loss manual = 2.0763 nats/token
     perplexity  = 7.97
     bits/byte   = 0.7674
     hardest: [(' capital', 9.09), (' caf', 9.04), (' along', 6.94)]
[vi] logits (1, 36, 151936)  tokens=35  bytes=145
     loss HF     = 2.5614 nats/token
     loss manual = 2.5614 nats/token
     perplexity  = 12.95
     bits/byte   = 0.8920
     hardest: [('Th', 10.22), ('ủ', 7.34), (' th', 6.64)]

Four things worth noticing:

• There are logits at every position. The (1, 32, 151936) tensor means one forward pass produces a distribution over all 151,936 tokens of the vocabulary at each of the 32 positions. We use the first 31 positions to predict the 31 tokens that follow, exactly as the previous section described.
• The library’s loss is just the average cross-entropy. loss HF and loss manual agree to the fourth decimal, because model(ids, labels=ids) performs exactly the shift and average above.
• The hardest tokens are the ones that start new information. capital at the start of the English sentence and Th at the start of the Vietnamese one have the highest loss, because there is nothing before them to go on. Once the model has read “The capital of France is”, “Paris” costs almost nothing.
• Comparing two languages requires bits-per-byte. The two passages have different token and byte counts, so perplexities of 7.97 and 12.95 are not on the same scale. Bits-per-byte (0.77 versus 0.89) is what tells you the model compresses Vietnamese text slightly worse than English per byte, which is unsurprising for a model trained mostly on English and Chinese. Two short passages only illustrate the calculation; they are not a reliable measurement of the model’s Vietnamese ability.

03What one training step looks like

This section gives just enough to read the rest of the post; it does not go deep into optimizers. One training step, following the top row of Figure 1, does four things:

1. Forward: take a batch of sequences, run them through the model, get logits at every position.
2. Loss: cross-entropy between the logits and the next token, as in section 02.
3. Backward: compute the gradient of the loss with respect to every parameter. This costs roughly twice the forward pass.
4. Optimizer step: use the gradients to update the parameters. Almost every LLM today uses AdamW, together with gradient clipping by global norm (PaLM uses a threshold of 1.0, PaLM paper section 5) to cap overly large updates.

The two hyperparameters that drive most of step 4’s behavior are the learning rate and the batch size.

The learning rate schedule

The learning rate is not fixed; it follows a schedule:

• Warmup: increase linearly from near zero to the peak value over the first few thousand steps. Early in training the weights are random and Adam’s moment estimates are not yet stable; large steps at this point easily make the run diverge.
• Cosine decay: after warmup, decrease along half a cosine cycle to a small value. Llama 3 405B uses a peak of 8 × 10⁻⁵, a linear warmup of 8,000 steps, then cosine decay to 8 × 10⁻⁷ over 1.2 million steps (Llama 3 paper, section 3.4.1).
• WSD (Warmup-Stable-Decay): warm up, keep the learning rate constant for most of the run, then decay quickly at the end. MiniCPM (Hu et al. 2024) proposed this schedule because it lets you branch off the “stable” phase to try several training lengths without retraining from scratch. With cosine, the schedule length has to be fixed in advance, a detail that comes back in section 05.

Batch size is measured in tokens

At LLM scale, batch size is counted in tokens, not sequences. Llama 3 405B starts at 4M tokens per batch (sequences of 4,096), moves to 8M tokens (sequences of 8,192) after the first 252M tokens, then to 16M tokens after 2.87T tokens (section 3.4.1). PaLM 540B also ramps from 1M to 2M to 4M tokens (PaLM paper, section 5). Both papers give the same reason: smaller batches are more stable and more sample-efficient early in training, while larger batches use the hardware better later on.

A 16M-token batch does not live on one GPU. It is split into micro-batches for each GPU, and how that split works is the subject of section 09.

04Scaling laws: Kaplan 2020

The question scaling laws answer is very practical: if I get 10 times more compute, should I grow the model or the number of tokens, and how much will the loss drop?

Three power laws

Kaplan et al. 2020 trained a large number of Transformers with non-embedding parameter counts from 768 to 1.5 billion, on datasets from 22 million to 23 billion tokens (WebText2), and found that the loss falls along a fairly clean power law in each quantity whenever that quantity is the bottleneck:

L(N) = (N_c / N)^α_N        α_N ≈ 0.076    N_c ≈ 8.8 × 10¹³ parameters (non-embedding)
L(D) = (D_c / D)^α_D        α_D ≈ 0.095    D_c ≈ 5.4 × 10¹³ tokens
L(C_min) = (C_c / C_min)^α_C   α_C ≈ 0.050    C_c ≈ 3.1 × 10⁸ PF-days

(Kaplan et al. 2020, equations 1.1–1.3 and Table 5. The third line is L(C_min) from equation 1.3: C_min is the minimum compute needed to reach a given loss, adjusted to the case of a batch size much smaller than the critical batch size, not the raw compute of each run. One PF-day is 10¹⁵ FLOP/s sustained for a day, about 8.64 × 10¹⁹ FLOPs.)

Small exponents mean improvement is very slow but very steady. With α_N = 0.076, doubling N multiplies the loss by 2^(−0.076) ≈ 0.95, a drop of about 5%. Halving the loss through N alone would take roughly 2^(1/0.076) ≈ 9,000 times more parameters. That is why each model generation needs an order of magnitude more compute than the last.

The constants N_c and D_c have no physical meaning; the paper states that they depend on the tokenizer and the data. What transfers between setups is the form of the law and the relative size of the exponents.

The “grow N first” conclusion

From these laws, Kaplan derives how to split a compute budget C to minimize loss (Table 6):

N_opt ∝ C^0.73        D_opt ∝ C^0.27

With 10 times more compute, the model should grow by 10^0.73 ≈ 5.4 times while the token count grows only 10^0.27 ≈ 1.9 times. The accompanying conclusion: large models are more sample-efficient, so the optimum is to train very large models and stop early, well before convergence.

This conclusion shaped a whole generation of models. GPT-3 175B, Jurassic 178B and Gopher 280B were all trained on about 300 billion tokens (Chinchilla paper, Table 1), fewer than 2 tokens per parameter.

The assumptions to remember

Three methodological details from Kaplan become the center of the debate in the next section:

• N is counted without embeddings, and compute also excludes the embedding part.
• Every run uses the same learning rate schedule with a fixed length, regardless of how many tokens that run is meant to see. Losses at intermediate points are then used as if they were the final loss of a shorter run.
• Most models are small, many below 100M parameters.

05Chinchilla and the debate that followed

Three fits, one conclusion

Hoffmann et al. 2022 (DeepMind) asked Kaplan’s question again, this time with over 400 models from 70M to more than 16B parameters, trained on 5B to over 400B tokens (section 1; the abstract rounds this to 5 to 500 billion tokens). They used three independent methods:

1. Fix model sizes, vary the number of tokens (section 3.1): for each FLOP count, take the lowest loss across all training curves.
2. IsoFLOP profiles (section 3.2): fix the FLOPs, sweep model size, find the bottom of the U-shaped loss curve.
3. Fit a parametric loss function (section 3.3):

L(N, D) = E + A / N^α + B / D^β

Each term has its own meaning: E is the irreducible loss (the entropy of language itself), A / N^α is the shortfall from a model that is too small, and B / D^β is the shortfall from training on too little data. The reported fit is E = 1.69, A = 406.4, B = 410.7, α = 0.34, β = 0.28 (Appendix D.2, equation 10).

Results of all three methods (Table 2):

All three say N and D should grow equally. Table 3 of the paper turns this into the number everyone remembers: a 1B model needs about 20.2B tokens, a 10B model 205.1B tokens, roughly 20 tokens per parameter. By that ratio, GPT-3 and Gopher were badly undertrained.

The example of Chinchilla 70B trained on 1.4T tokens beating Gopher 280B at similar compute is already in part 5, section 8.2, so this post does not retell it.

Why Kaplan and Chinchilla disagree

Both groups used Transformers and measured cross-entropy, so how did they get 0.73 and 0.50? The answer became clearer over three papers.

Chinchilla offered the first hypothesis itself (section 2): Kaplan used one fixed cosine schedule for every run. The loss in the middle of a long schedule is higher than the final loss of a run whose schedule matches that token count. Using those intermediate points as estimates underrates the benefit of training a small model on fewer tokens and pushes the conclusion toward “grow N faster than D”. The paper’s own appendix (Figure A1) shows that when the cosine cycle overshoots the actual number of training steps by more than 25%, the final loss gets noticeably worse.

Porian et al. 2024 reproduced Kaplan and fixed one factor at a time. They found three causes: Kaplan did not count the cost of the last layer (lm_head), used a fixed warmup that was too long for small models, and did not tune the optimizer for each scale. Fixing all three makes the result match Chinchilla. The surprise: they found that matching the learning rate schedule to the token count, Chinchilla’s own hypothesis, was not the deciding factor.

Pearce and Song 2024 came from the other direction: start from Chinchilla-style data, count N the way Kaplan did (excluding embeddings), and restrict to Kaplan’s small scale. Most of the discrepancy appears on its own. In small models, embeddings are a large share of the parameters, so leaving them out distorts the curve exactly where Kaplan measured most.

The two papers come from opposite sides and both reaffirm Chinchilla’s coefficients. The trend Kaplan measured can still be reproduced in Kaplan’s setup, but the 0.73 exponent is a biased estimate caused by how parameters are counted and how the experiments are designed. Correct those factors and the result matches Chinchilla’s ~0.5. Pearce and Song write that their paper “reaffirms Chinchilla’s scaling coefficients” and explains “much of” the discrepancy, that is, most of it rather than all of it. The two papers also point to different causes: Pearce and Song to embeddings and small scale; Porian to the last layer, warmup and optimizer tuning, noting explicitly that Pearce and Song did not identify the warmup issue. One detail that is easy to mix up: 0.73 is the adjusted exponent with respect to C_min; Kaplan’s raw exponent with respect to actual compute is 0.88.

Chinchilla’s own fit was corrected too

Besiroglu et al. 2024 extracted the data from Chinchilla’s figures, refit method 3, and found three problems. The rounded coefficients in the paper fit the data poorly. The confidence intervals are implausibly narrow for the number of data points. Most importantly, the method-3 coefficients imply about 70 tokens per parameter, contradicting the 20-to-1 ratio DeepMind actually used to train Chinchilla. Their refit (Table 1):

E = 1.8172    A = 482.01    B = 2085.43    α = 0.3478    β = 0.3658

With these coefficients, method 3 agrees with methods 1 and 2 again, around 20 tokens per parameter.

A scaling-law calculator

The code below uses both sets of coefficients to compute N_opt and D_opt for a budget C. The closed form is Chinchilla’s equation 4: minimize L(N, D) subject to C = 6ND. The factor 6 is the 6N FLOPs per training token counted in part 8, section 6.

# Scaling-law calculator: Python 3.12, standard library only.
FITS = {
    # L(N, D) = E + A / N**alpha + B / D**beta
    "Hoffmann 2022 (Approach 3)": dict(E=1.69, A=406.4, B=410.7, alpha=0.34, beta=0.28),
    "Besiroglu 2024 (refit)":     dict(E=1.8172, A=482.01, B=2085.43, alpha=0.3478, beta=0.3658),
}

def loss(f, N, D):
    return f["E"] + f["A"] / N ** f["alpha"] + f["B"] / D ** f["beta"]

def optimal(f, C):
    """Minimize L(N, D) subject to C = 6ND (Hoffmann et al., equation 4)."""
    a, b = f["alpha"], f["beta"]
    G = (a * f["A"] / (b * f["B"])) ** (1 / (a + b))
    N = G * (C / 6) ** (b / (a + b))
    return N, C / (6 * N)

for C in (5.76e23, 7.2e23, 3.8e25):
    print(f"C = {C:.2e} FLOPs")
    N20 = (C / 120) ** 0.5                     # 20 tokens/param rule: C = 6 * N * 20N
    print(f"  {'20 tokens/param':<28} N = {N20/1e9:6.1f}B  D = {20*N20/1e12:5.2f}T  D/N = 20")
    for name, f in FITS.items():
        N, D = optimal(f, C)
        print(f"  {name:<28} N = {N/1e9:6.1f}B  D = {D/1e12:5.2f}T  D/N = {D/N:4.0f}")

f = FITS["Besiroglu 2024 (refit)"]
C = 6 * 8e9 * 15e12
N, D = optimal(f, C)
print(f"\nSame C = {C:.2e} FLOPs, Besiroglu fit:")
print(f"  compute-optimal  {N/1e9:5.1f}B x {D/1e12:5.2f}T  loss = {loss(f, N, D):.4f}")
print(f"  Llama 3 8B-style   8.0B x 15.00T  loss = {loss(f, 8e9, 15e12):.4f}")

Output:

C = 5.76e+23 FLOPs
  20 tokens/param              N =   69.3B  D =  1.39T  D/N = 20
  Hoffmann 2022 (Approach 3)   N =   32.2B  D =  2.98T  D/N =   93
  Besiroglu 2024 (refit)       N =   72.2B  D =  1.33T  D/N =   18
C = 7.20e+23 FLOPs
  20 tokens/param              N =   77.5B  D =  1.55T  D/N = 20
  Hoffmann 2022 (Approach 3)   N =   35.6B  D =  3.37T  D/N =   95
  Besiroglu 2024 (refit)       N =   81.0B  D =  1.48T  D/N =   18
C = 3.80e+25 FLOPs
  20 tokens/param              N =  562.7B  D = 11.25T  D/N = 20
  Hoffmann 2022 (Approach 3)   N =  213.5B  D = 29.67T  D/N =  139
  Besiroglu 2024 (refit)       N =  618.7B  D = 10.24T  D/N =   17

Same C = 7.20e+23 FLOPs, Besiroglu fit:
  compute-optimal   81.0B x  1.48T  loss = 1.9683
  Llama 3 8B-style   8.0B x 15.00T  loss = 2.0221

Reading the output:

• The first budget is Gopher’s (5.76 × 10²³ FLOPs, Chinchilla Table 3). The 20-tokens-per-parameter rule gives about 69B on 1.4T tokens, close to the Chinchilla 70B configuration. The Besiroglu fit gives 72B, also close.
• Hoffmann’s rounded coefficients give something quite different: only 32B but 93 tokens per parameter. The Chinchilla paper itself, using unrounded coefficients, reports 40B for this budget (Figure 4 caption). This is a direct illustration of Besiroglu’s point: method 3 with the published coefficients does not agree with the other two methods.
• The optimal tokens-per-parameter ratio is not a constant. With the Besiroglu coefficients, α and β are almost equal, so the ratio barely moves with C (18, 18, 17). With Hoffmann’s, the ratio grows with C (93, 95, 139). “20” is one point on the curve for one particular dataset, not a law of nature.
• Change the data and the answer changes. Meta fit a scaling law on its own data and, for a budget of 3.8 × 10²⁵ FLOPs, got 402B parameters on 16.55T tokens, about 41 tokens per parameter (Llama 3 paper, section 3.2.1). The calculator above with Besiroglu’s coefficients gives 619B on 10.2T. Scaling-law coefficients depend on the data, the tokenizer and the architecture; to use them seriously you have to fit them on your own setup.
• The last two lines bridge to the next section. At the same compute as Llama 3 8B, the compute-optimal model under this fit is 81B on 1.5T tokens, with a predicted loss about 0.054 nats lower. Meta knew this and still chose 8B. The reason is what happens after training.

06Going past Chinchilla-optimal

Compute-optimal only optimizes training cost

“Compute-optimal” in Chinchilla has a very narrow meaning: the lowest loss for a fixed amount of training FLOPs. It says nothing about how many times the model will be run afterwards.

Sardana and Frankle 2023 added inference cost to the problem. A model may be called billions of times a day, and the cost per inference token scales with N (2N FLOPs, also from part 8). Once total inference tokens are large enough, the calculation flips: spending more compute to train a smaller model on more tokens is cheaper overall, because every call is cheaper. By their analysis, with inference demand around one billion requests you should train smaller and longer than Chinchilla-optimal. They also trained 47 models from 150M to 6B parameters at ratios from 10 to 10,000 tokens per parameter and found no saturation point: quality kept improving as the ratio grew.

Today’s open models train far past the Chinchilla point

That is why popular open models are deliberately “overtrained”:

For Llama 3 8B: 15 × 10¹² / 8.03 × 10⁹ ≈ 1,870, close to 1,900 tokens per parameter, about 90 times the Chinchilla ratio. The Qwen family goes further: Qwen2.5 raised its pretraining data to 18T tokens, and Qwen3 states that every model in the family, including the 0.6B one, goes through a first pretraining stage on over 30T tokens. For the smallest model that is more than 50,000 tokens per parameter.

In exchange, we accept a slightly higher loss than the compute-optimal model at the same budget, like the 0.054 nats in the last example of section 05, to get a model that is ten times cheaper to run.

When data runs out: repeating epochs

Overtraining needs a lot of tokens, and high-quality data is finite (part 5 covered how filtering and deduplication shrink a dataset). When fresh data runs out, the natural move is to read old data again.

Muennighoff et al. 2023 measured this with runs of up to 900B tokens and models up to 9B parameters. Result: at a fixed compute budget, repeating data for up to 4 epochs gives a loss almost indistinguishable from using all-new data. After that, the value of repeated tokens drops quickly. In their fit, the “half-life” of repeated data is about 15 repetitions (16 epochs); beyond that, extra compute barely helps.

Lower loss is not necessarily better

Scaling laws predict loss on a validation set, and that is the only thing they predict. What users care about is scores on specific tasks, and the relationship between the two is not simple:

• Two models with different tokenizers have per-token losses that cannot be compared (section 02).
• Two models trained on different data mixes have validation losses that reflect how “similar” the mix is to that validation set, not just capability.
• Even within one model family, the relationship between loss and benchmark accuracy has to be fit separately. Meta does it in two steps: fit the negative log-likelihood on downstream tasks against FLOPs, then fit the relationship between that negative log-likelihood and accuracy (Llama 3 paper, section 3.2.1). They state the reason directly: standard scaling laws only predict next-token prediction loss, not benchmark scores.

07The compute budget

This part can be done entirely by hand. The C ≈ 6ND formula is already in part 8, section 6; here we only add the hardware.

The three numbers you need

1. C: total training FLOPs, approximately 6 · N · D.
2. Peak FLOPs of one GPU at the training precision. NVIDIA’s H100 product page lists the H100 SXM at 1,979 teraFLOPS BF16 with sparsity. LLM training uses dense matrices, not sparsity, so the usable figure is half: about 989 TFLOP/s.
3. MFU (Model FLOPs Utilization): the ratio between the “useful” FLOPs the model actually needs (counted as 6N per token plus the attention part) and the hardware’s peak FLOPs. The concept was introduced by PaLM (Appendix B). PaLM 540B reached 46.2%. Llama 3 405B reached 38–43% BF16 MFU depending on the configuration (Llama 3 paper, Table 4). The rest is lost to communication between GPUs, waiting for data, non-optimal kernels and activation recomputation.

From that:

GPU-seconds = C / (peak FLOPs × MFU)

Worked example: Llama 3 8B on 15T tokens

Step 1, compute C. Take N = 8.03 × 10⁹ (the count from part 8, section 6) and D = 15 × 10¹² (the model card says “15T+”):

C = 6 × 8.03 × 10⁹ × 15 × 10¹² = 7.227 × 10²³ FLOPs

Step 2, real throughput per GPU. Assume 40% MFU, within the 38–43% range Meta reports for 405B:

989 × 10¹² × 0.40 = 3.958 × 10¹⁴ FLOP/s

Step 3, convert to GPU-hours.

7.227 × 10²³ / 3.958 × 10¹⁴ = 1.826 × 10⁹ GPU-seconds
1.826 × 10⁹ / 3,600         ≈ 507,000 GPU-hours ≈ 0.51M GPU-hours

Step 4, compare. The Llama 3 model card lists 1.3M GPU-hours on H100-80GB for Llama 3 8B. The real number is about 2.6 times the estimate. Working backwards, 1.3M GPU-hours corresponds to an effective MFU of only about 15.6%.

Meta does not publish a separate MFU for the 8B model, so we cannot say exactly where the gap comes from. Plausible candidates: small models usually reach lower MFU than large ones because the matrices are smaller and communication takes a larger share of the time; the real token count may be above 15T (“15T+”); 6ND ignores the context-dependent attention cost; and the model card’s GPU-hours may include time lost to interruptions and the annealing and long-context stages (still part of pretraining, since the model card explicitly reports pretraining GPU-hours). Attention, on the other hand, can be computed and is not the main cause: with 32 layers, d_model = 4,096 and an 8K context, the 6 × 32 × 8,192 × 4,096 ≈ 6.4 × 10⁹ FLOPs per token add only about 13% on top of 6N ≈ 4.8 × 10¹⁰, and about 7% once halved for the causal mask. Also do not confuse this with the Llama 3.1 model card, where the 3.1 8B model is listed at 1.46M GPU-hours; the 1.3M figure above is for Llama 3 8B. The practical lesson: 6ND gives you a lower bound, not the bill.

Checking against 405B

The large model matches much better, because Meta publishes enough data:

C        = 6 × 405 × 10⁹ × 15.6 × 10¹²            = 3.79 × 10²⁵ FLOPs   (paper: 3.8 × 10²⁵)
MFU 43%  : 3.79 × 10²⁵ / (989 × 10¹² × 0.43) / 3,600 ≈ 24.7M GPU-hours
MFU 38%  : 3.79 × 10²⁵ / (989 × 10¹² × 0.38) / 3,600 ≈ 28.0M GPU-hours

The Llama 3.1 model card lists 30.84M GPU-hours for 405B, about 10% above the 38%-MFU estimate and about 25% above the 43%-MFU one. The real MFU over the whole run lies between 43% and 38% (Table 4), so the excess above the estimate has to come from elsewhere: effective training time of only “higher than 90%” (section 3.3.4), meaning up to about 10% lost to interruptions, plus the attention cost that 6ND ignores. Run continuously on 16,384 GPUs, 30.84M GPU-hours is about 78 days.

08Mixed precision

Four number formats

Training in 16-bit is much faster because Tensor Cores run many times faster at lower precision, and it uses half the memory. The problem is that training entirely in 16-bit breaks.

fp16 needs loss scaling, bf16 does not

Micikevicius et al. 2017 identified two problems with fp16. First, many gradients are smaller than 2⁻²⁴, the smallest value fp16 can represent, so they round to 0. Second, the weight update w ← w − lr · g is usually far smaller than w, and at fp16 precision the addition is swallowed. Two fixes became standard:

• Loss scaling: multiply the loss by a large factor before backward to push gradients into fp16’s representable range, then divide it back out before the update. The factor is usually adjusted dynamically: reduced on overflow, increased gradually while things are stable.
• fp32 master weights: forward and backward use the fp16 copy, but the update runs on an fp32 copy of the weights.

bf16 solves the first problem at the root. It keeps fp32’s 8 exponent bits, so its representable range matches fp32 and small gradients do not round to 0. Kalamkar et al. 2019 showed that bf16 trains many kinds of networks without loss scaling. The price is only 7 mantissa bits, less precise than fp16, so master weights and optimizer state stay in fp32. This is the default configuration for most LLM pretraining today, including Llama 3 (whose MFU is reported in BF16).

A counterexample shows how fiddly fp16 is: OPT-175B kept its weights in fp16 with dynamic loss scaling, and the authors noted a correlation between loss divergences and the loss scale collapsing toward 0 (OPT paper: fp16 and loss scaling in section 2.4, the divergences in section 2.5).

FP8 is starting to be used for training

DeepSeek-V3 is one of the first large models trained with FP8 mixed precision at scale: 14.8T tokens and 2.788M H800 GPU-hours for the full training. The main matrix multiplies run in FP8, with fine-grained tile-wise and block-wise quantization to bound the error. Master weights, weight gradients and optimizer state stay at higher precision (DeepSeek-V3 report, section 3.3). They report a relative loss error versus BF16 that stays below 0.25%.

16 bytes per parameter

This is the count that drives section 09. With AdamW and mixed precision, each parameter needs (ZeRO paper, section 3.1):

An 8.03B-parameter model needs 8.03 × 10⁹ × 16 ≈ 128 GB for model states alone, before activations, while an H100 has 80 GB. The ZeRO paper gives a similar example with GPT-2 1.5B: 3 GB for fp16 weights, but at least 24 GB during training. Activations add another large chunk that grows with batch size and sequence length.

So even a model that counts as “small” today cannot be trained on one GPU. Not for lack of FLOPs, but for lack of memory.

09Parallelism: DP, FSDP, TP, PP

There are two reasons to use many GPUs: to fit in memory and to finish in acceptable time. Each kind of parallelism cuts the problem along a different axis and pays with a different kind of communication.

Data parallelism (DP): split the batch

Each GPU holds a full copy of the model and optimizer state and processes a different slice of the batch. After backward, the GPUs all-reduce the gradients so every copy has the same averaged gradient, then they apply identical updates.

DP is easy to implement and communicates once per step, but it saves no memory at all: each GPU still needs the full 16 bytes per parameter. If the model does not fit on one GPU, plain DP does not help.

FSDP / ZeRO: still data parallel, but the state is sharded

Rajbhandari et al. 2019 (ZeRO) noticed that in DP, N GPUs hold N identical copies of optimizer state, gradients and weights. They split those up so each GPU holds 1/N, in three levels:

• shard the optimizer state: about 4x less memory, same communication as DP;
• also shard gradients: about 8x less memory, still the same communication;
• also shard the weights: memory falls linearly with N, communication rises by about 50%.

PyTorch FSDP (Zhao et al. 2023) is PyTorch’s implementation of the highest level. Before computing a layer, the GPUs all-gather to temporarily hold that layer’s full weights, then discard them afterwards. After backward, gradients are reduce-scattered: summed and split in one go, so each GPU receives only the gradient for its own shard.

The key point: FSDP is still data parallelism. Each GPU still processes its own slice of the batch and still runs all of the computation for every layer. It just does not store the whole model permanently. Llama 3 uses FSDP as its DP dimension, sharding optimizer state and gradients but not resharding weights after the forward pass, to avoid an extra all-gather during backward (Llama 3 paper, section 3.3.2).

Tensor parallelism (TP): split inside a layer

Shoeybi et al. 2019 (Megatron-LM) split each matrix in a layer across several GPUs. In the FFN, the first matrix is split by columns and the second by rows, so each GPU computes its share with no communication in between. Only one all-reduce is needed at the end of the block in the forward pass and one in the backward pass. Attention is split by heads in the same way.

The upside: it reduces memory for both weights and activations, and all GPUs work on the same batch. The downside: communication happens several times per layer, on the critical path. TP is only efficient when the GPUs are connected by NVLink inside one server. Narayanan et al. 2021 state the rule: use TP up to the number of GPUs in a server (usually 8), and switch to PP beyond the server (Takeaway #1).

Pipeline parallelism (PP): split by layer

PP splits the layers into stages, one stage per GPU. Activations flow from one stage to the next in the forward pass and gradients flow back in the backward pass. Communication is just sending activations across stage boundaries, small and point-to-point, so it tolerates the network between servers.

The price is the bubble. With a single batch, stage 2 waits for stage 1 to finish, and stage 1 sits idle while stage 2 computes. GPipe (Huang et al. 2018) splits the batch into m micro-batches that run back to back to fill the gaps. With p stages, the idle fraction is:

bubble = (p − 1) / m          (Narayanan et al. 2021, section 2.2.1)

With 16 stages, as in Llama 3 405B, keeping the bubble under 10% takes more than 150 micro-batches per step. GPipe reports the bubble becomes negligible when m ≥ 4p. More micro-batches means holding activations for more micro-batches, so a PP schedule is always a trade-off between bubble and memory.

Comparison

Combining them into multi-dimensional parallelism

Large models use all four at once, each placed on the network tier that suits it. Narayanan et al. 2021 combined TP, PP and DP to train a 1-trillion-parameter model at 502 petaFLOP/s on 3,072 GPUs, or 52% of peak throughput per GPU.

Llama 3 405B uses 4D parallelism in the order [TP, CP, PP, DP], where DP is FSDP (section 3.3.2). The first configuration in Table 4: TP 8 · CP 1 · PP 16 · DP 64 on 8,192 GPUs, 430 TFLOPs per GPU, 43% MFU. The layout follows bandwidth: TP 8 fits inside one 8-GPU server connected by NVLink; PP and DP span the network between servers.

The remaining two dimensions only need a name. Context parallelism (CP) splits along the sequence length and is needed for the long-context stage: Llama 3 turns on CP 16 when training with 131,072-token sequences, and MFU drops to 38% (Table 4). Expert parallelism spreads the experts of a Mixture-of-Experts model across GPUs and is covered in part 14.

10When a training run is going wrong

What to monitor

The loss alone is too slow a signal. Teams training large models watch at least:

• Loss, both overall and per domain (web, code, math, multilingual). One domain splitting away from the rest is often an early sign of a problem in the data mix or the data pipeline.
• Global grad norm. A spike in grad norm often precedes a loss spike by a few steps.
• Logit magnitudes, both attention logits and output logits. Wortsman et al. 2023 identify these as the two main sources of instability; more below.
• Throughput and MFU. MFU dropping while the loss looks normal is usually a hardware issue: a slow GPU, a bad network link.

Loss spikes and divergence

A loss spike is when a steadily falling loss suddenly jumps; sometimes it recovers, sometimes it drags the whole run into divergence. Three documented examples:

• PaLM 540B (section 5.1): the loss spiked about 20 times during training even with gradient clipping on, at very irregular moments, some late in training. Smaller models did not show it. The fix: restart from a checkpoint about 100 steps before the spike and skip roughly 200–500 batches of data around it. The important detail: they tried training on those same batches from a different, earlier checkpoint and saw no spike. Their conclusion: spikes are not caused by “bad data” per se, but by the combination of specific batches with a specific parameter state.
• OPT-175B (section 2.5): when the loss diverged, the authors lowered the learning rate and restarted from an earlier checkpoint. They also lowered gradient clipping from 1.0 to 0.3 early in training. The whole process is recorded in a public logbook that reads like a shift diary.
• Llama 3 405B (section 3.4.1): with a gradually increasing batch size, Meta reports a “very stable” run: few loss spikes and no interventions needed to correct divergence.

Causes and the common fixes

Wortsman et al. 2023 reproduced the instabilities of large models in small models by using high learning rates, which makes them cheap to study. They isolate two main sources:

• Attention logit growth. q · k grows until the attention softmax is nearly one-hot and the gradients become erratic. Dehghani et al. 2023 hit exactly this when scaling ViT to 22B and fixed it with QK-norm: apply LayerNorm to queries and keys before the dot product. Wortsman confirms QK-norm also helps LLMs.
• Output logit divergence. The logits at lm_head drift away from 0 and the final softmax becomes unstable. The fix is z-loss: add a small penalty that pushes log Z (the log of the softmax normalizer) toward 0. PaLM uses 10⁻⁴ · log² Z (section 5).

They also find that longer warmup makes the model less sensitive to the learning rate, and that scaling weight decay independently of the learning rate, as Loshchilov and Hutter recommend, has a similar effect. Summary:

Hardware failures are a daily event

A synchronous training run across thousands of GPUs behaves like a string of lights wired in series: one failed GPU can stop the whole job. The numbers Meta published for Llama 3 405B (section 3.3.4 and Table 5):

• over a 54-day period, the job was interrupted 466 times;
• 47 were planned (firmware upgrades, configuration changes, dataset updates), 419 were unexpected;
• about 78% of the unexpected ones were confirmed or suspected hardware issues; GPUs alone accounted for 58.7%, led by faulty GPUs (148, 30.1%) and HBM3 memory (72, 17.2%);
• only 3 needed significant manual intervention; automation handled the rest;
• effective training time still stayed above 90%.

Some failures are harder to see: silent data corruption (6 cases in Table 5), where a GPU computes wrong values without reporting an error, and GPUs that are “slow but still working” and drag thousands of other GPUs down with them. Meta even observed a 1–2% throughput swing by time of day, caused by midday temperatures affecting GPU clock speeds.

OPT-175B, run on 992 A100 80GB GPUs, shows the same picture at a smaller scale: at least 35 manual restarts and more than 100 hosts cycled out over about 2 months; the authors estimate more than 70 additional automatic restarts due to hardware failures (OPT section 2.5; the 992-GPU setup is in section 2.4).

The consequence: frequent checkpoints and fast restarts are not side features. They are part of the training algorithm, on par with the optimizer. And as part 5 pointed out, the data loader must know exactly how many tokens have been consumed for a restart to resume at the right place.

11Common misconceptions

The six rows below match the bottom row of Figure 1.

Summary: one objective, three engineering problems

Pretraining is simple in its objective and hard in everything else. The model only learns to predict the next token, but doing that over more than ten trillion tokens means solving three problems: splitting the budget, fitting onto hardware, and keeping the run alive.

What to take away:

• the objective is next-token cross-entropy, computed in parallel at every position thanks to teacher forcing and the causal mask; loss, perplexity and bits-per-byte are the same quantity, and only bits-per-byte compares across tokenizers;
• scaling laws are empirical regularities: loss falls as a power law in N, D and C ≈ 6ND; Kaplan advised growing N faster than D, Chinchilla showed they should grow equally, at about 20 tokens per parameter; most of the gap comes from how things are counted and how experiments are designed, and Chinchilla’s own fit has since been corrected;
• compute-optimal only optimizes training cost; once inference counts, a small model trained for very long wins, which is why Llama 3 8B used close to 1,900 tokens per parameter; repeating data for up to about 4 epochs costs almost nothing;
• the compute budget can be done by hand: 6ND / (peak FLOPs × MFU) gives GPU-seconds; for Llama 3 405B the calculation comes in about 10–25% below the published figure depending on the assumed MFU, while for 8B the real cost was about 2.6 times higher, so treat 6ND as a lower bound;
• memory is why we parallelize: AdamW with mixed precision costs 16 bytes per parameter, so an 8B model already needs 128 GB; DP splits the batch, FSDP splits the batch and the state, TP splits matrices inside a node, PP splits layers across nodes, and large models combine all four according to network bandwidth;
• bf16 keeps fp32’s exponent range and so needs no loss scaling, master weights stay in fp32, and FP8 is now usable for training;
• a run going wrong shows up in grad norm, logit magnitude, per-domain loss and MFU; spikes are handled by rolling back and skipping batches, logit-driven instability is contained with QK-norm and z-loss; at 16,000 GPUs, hardware fails several times a day.

A model fresh out of pretraining is a base model. It predicts the next token extremely well, but as part 1 noted, its only job is to continue text: ask it a question and it may continue with five more questions. Part 11 covers post-training, the stage that turns a base model into an assistant that follows instructions.

References

Papers

• Kaplan et al. 2020 — Scaling Laws for Neural Language Models
• Hoffmann et al. 2022 — Training Compute-Optimal Large Language Models
• Besiroglu et al. 2024 — Chinchilla Scaling: A replication attempt
• Porian et al. 2024 — Resolving Discrepancies in Compute-Optimal Scaling of Language Models
• Pearce and Song 2024 — Reconciling Kaplan and Chinchilla Scaling Laws
• Sardana and Frankle 2023 — Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws
• Muennighoff et al. 2023 — Scaling Data-Constrained Language Models
• Hu et al. 2024 — MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
• Micikevicius et al. 2017 — Mixed Precision Training
• Kalamkar et al. 2019 — A Study of BFLOAT16 for Deep Learning Training
• Rajbhandari et al. 2019 — ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
• Zhao et al. 2023 — PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel
• Shoeybi et al. 2019 — Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• Narayanan et al. 2021 — Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM
• Huang et al. 2018 — GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
• Chowdhery et al. 2022 — PaLM: Scaling Language Modeling with Pathways
• Zhang et al. 2022 — OPT: Open Pre-trained Transformer Language Models
• Wortsman et al. 2023 — Small-scale proxies for large-scale Transformer training instabilities
• Dehghani et al. 2023 — Scaling Vision Transformers to 22 Billion Parameters
• Llama Team, Meta 2024 — The Llama 3 Herd of Models
• DeepSeek-AI 2024 — DeepSeek-V3 Technical Report
• Qwen Team 2024 — Qwen2.5 Technical Report
• Qwen Team 2025 — Qwen3 Technical Report

Documentation and hardware figures

• Meta — Llama 3 model card (GPU-hours for Llama 3 8B)
• Meta — Llama 3.1 model card (GPU-hours for Llama 3.1 405B)
• Meta — OPT-175B logbook
• NVIDIA — H100 Tensor Core GPU, specifications
• Hugging Face — The Ultra-Scale Playbook: Training LLMs on GPU Clusters
• Hugging Face — Qwen/Qwen2.5-0.5B