How LLMs Work

A systematic mental model of the Transformer: from tokens and why some languages cost more of them, through Attention, residuals, the FFN and RoPE, to the three architecture families, MoE and the full path of inference.

pipeline() is only a thin layer of abstraction. This post opens it up: the tokenizer, the model head, logits, padding and attention masks, then on to device, dtype, the KV cache, continuous batching, quantization, and how to choose between vLLM, SGLang and llama.cpp now that TGI is archived.

The first two parts were about a model that already knows things. This one is about changing what it knows: splits and data leakage, padding and the real cost of a tensor, cross-entropy, backpropagation, AdamW, learning-rate schedules, and reading a learning curve to tell overfitting from underfitting.

The first three parts all started from a point where text had already become numbers. This one goes back to that step: normalization, pre-tokenization, subword segmentation, vocabulary and special tokens - the discrete interface that decides your context window, your costs, and even whether a language pays a token tax.

A web page is not a training sample. Neither is a book. This post walks the whole distance in between - collection, extraction, normalization, filtering, dedup, decontamination, domain mixing, tokenization, chunking - to show why the dataset is not side work outside the model, but part of the very problem of building an LLM.

Attention is only half of a Transformer block, and not even the heavier half. This post covers the other half: the residual stream as a shared bus, the FFN as a key–value memory, GELU and SwiGLU, LayerNorm, RMSNorm and where the norm goes, then counts the parameters and FLOPs of Llama 3.1 8B to show why the FFN ≈ 2/3 rule of thumb is out of date.

The model only produces logits; choosing the token is the job of decoding. This post reads the source of Hugging Face Transformers and vLLM to see which steps logits go through and in what order, runs temperature, top-k, top-p and min-p on a real model, explains repetition penalty and constrained decoding for JSON, then answers why temperature 0 still does not guarantee the same answer twice.

All of pretraining optimizes one thing: the cross-entropy of the next token. This post computes loss, perplexity and bits-per-byte on a real model, rereads the Kaplan and Chinchilla scaling laws and the corrections that came later, works out the compute budget of Llama 3 by hand and checks it against Meta's numbers, explains why memory forces us to split a model across many GPUs, and shows how to tell that a training run is going wrong.

A base model predicts tokens very well but does not know how to answer a question. This post runs the same question through a base model and an instruct model, prints the chat template and the SFT loss mask, derives step by step how Bradley–Terry leads to the reward-minus-KL objective and then to the DPO loss, computes the DPO loss on a real model, and puts the Llama 2, Llama 3 and Tülu 3 pipelines side by side. The thread running through it: post-training adds almost no knowledge; it shapes how the model uses the knowledge it already has.

Full fine-tuning an 8B model needs about 128 GB for the model states alone. This post breaks that number down and shows which part LoRA cuts and which part quantization cuts. It measures the SVD of ΔW between Qwen2.5 base and Instruct, attaches LoRA with PEFT, trains real QLoRA on an 11 GB RTX 2080 Ti, rebuilds the NF4 table by hand, and compares INT8, INT4 and NF4 on English and Vietnamese text. The thread running through it: two knobs, each cutting a different part of memory, and a cost you have to measure on your own task.

Decoding one token at batch size 1 does about 1 FLOP for every byte it reads, while an RTX 2080 Ti needs almost 90 FLOPs per byte to keep its compute busy. This post uses that number as its axis: every LLM serving technique, from batching, GQA, and PagedAttention to FlashAttention and speculative decoding, answers one question: read fewer bytes per token, or do more work per byte read. It computes the KV cache from the config.json of five real models, rewrites online softmax in plain PyTorch, and shows that long context is expensive mostly because of memory, not because of O(n²).

DeepSeek-V3 has 671B parameters but each token runs only about 37B, and Qwen3-Next stacks three Gated DeltaNet layers on every attention layer. This post opens up the router: how it picks experts, why left alone it piles everything onto a few of them, and three ways to balance the load, from an auxiliary loss to a loss-free bias. It then redoes the roofline for MoE and moves to the other direction: replacing attention with a fixed state, why pure SSMs are weak at exact recall, and why hybrids win. The code counts the parameters of nine real models from safetensors headers, writes an MoE layer in plain PyTorch, and shows that three ways of computing an SSM give the same result.

DeepSeek-R1-Zero went from 15.6% to 71.0% on AIME 2024 with nothing but RL on a right-or-wrong reward, and its answers grew longer on their own during training. This post explains why writing out intermediate steps lets a model compute more, three ways to spend compute at answer time (sample many times, think in one long chain, search with step-level scoring), what happens inside GRPO and why a group of all-correct answers teaches nothing, the debate over whether RL teaches new skills or only sharpens existing ones, and the real price of thinking tokens. The code runs Qwen3-0.6B with thinking on and off on GSM8K, simulates pass@k against majority voting, and computes GRPO advantages in plain PyTorch.

On SimpleQA, GPT-4o-mini answers 90.5% of questions wrongly and abstains on only 0.9%. This post explains why that is not random noise: facts that appear only once in the data are where a calibrated model must make things up, and most benchmarks grade right-or-wrong, which rewards guessing. Then come calibration and ways to get a confidence score, the four components of a benchmark number (capability, what was already seen, sensitivity to how you ask, noise), contamination from the evaluator's side, and the biases of LLM-as-judge. The code measures Qwen3-0.6B's calibration on 200 MMLU questions, simulates two grading rules, and computes error bars for comparing two models.