Notes on gen ai in production
Posts are grouped into topics, each one a thread you can read straight through. Pick a topic in the left column or search by keyword.
Why LLMs Get Things Wrong, and How We Measure It: Hallucination, Calibration, Benchmarks and LLM-as-Judge
On SimpleQA, GPT-4o-mini answers 90.5% of questions wrongly and abstains on only 0.9%. This post explains why that is not random noise: facts that appear only once in the data are where a calibrated model must make things up, and most benchmarks grade right-or-wrong, which rewards guessing. Then come calibration and ways to get a confidence score, the four components of a benchmark number (capability, what was already seen, sensitivity to how you ask, noise), contamination from the evaluator's side, and the biases of LLM-as-judge. The code measures Qwen3-0.6B's calibration on 200 MMLU questions, simulates two grading rules, and computes error bars for comparing two models.
Why LLMs Get Things Wrong, and How We Measure It: Hallucination, Calibration, Benchmarks and LLM-as-Judge
On SimpleQA, GPT-4o-mini answers 90.5% of questions wrongly and abstains on only 0.9%. This post explains why that is not random noise: facts that appear only once in the data are where a calibrated model must make things up, and most benchmarks grade right-or-wrong, which rewards guessing. Then come calibration and ways to get a confidence score, the four components of a benchmark number (capability, what was already seen, sensitivity to how you ask, noise), contamination from the evaluator's side, and the biases of LLM-as-judge. The code measures Qwen3-0.6B's calibration on 200 MMLU questions, simulates two grading rules, and computes error bars for comparing two models.
Reasoning Models and Test-Time Compute: When Thinking Longer Pays Off
DeepSeek-R1-Zero went from 15.6% to 71.0% on AIME 2024 with nothing but RL on a right-or-wrong reward, and its answers grew longer on their own during training. This post explains why writing out intermediate steps lets a model compute more, three ways to spend compute at answer time (sample many times, think in one long chain, search with step-level scoring), what happens inside GRPO and why a group of all-correct answers teaches nothing, the debate over whether RL teaches new skills or only sharpens existing ones, and the real price of thinking tokens. The code runs Qwen3-0.6B with thinking on and off on GSM8K, simulates pass@k against majority voting, and computes GRPO advantages in plain PyTorch.
AI Weekly, 12 Oct 2026: GPT-6 for Everyone, Text Watermarks in the EU and On-Device Multimodal RAG
GPT-6 reaches free ChatGPT users and builds an interface for each answer. OpenAI starts watermarking text in the EU and publishes the watermark's weak spots. Google brings multimodal embeddings onto the device, and RTX Spark laptops promise to run 100B+ parameter models locally.
Mixture of Experts and Alternative Architectures: Routers, Experts, and Mamba
DeepSeek-V3 has 671B parameters but each token runs only about 37B, and Qwen3-Next stacks three Gated DeltaNet layers on every attention layer. This post opens up the router: how it picks experts, why left alone it piles everything onto a few of them, and three ways to balance the load, from an auxiliary loss to a loss-free bias. It then redoes the roofline for MoE and moves to the other direction: replacing attention with a fixed state, why pure SSMs are weak at exact recall, and why hybrids win. The code counts the parameters of nine real models from safetensors headers, writes an MoE layer in plain PyTorch, and shows that three ways of computing an SSM give the same result.
Inference at Scale: KV Cache, Prefill, and Decode
Decoding one token at batch size 1 does about 1 FLOP for every byte it reads, while an RTX 2080 Ti needs almost 90 FLOPs per byte to keep its compute busy. This post uses that number as its axis: every LLM serving technique, from batching, GQA, and PagedAttention to FlashAttention and speculative decoding, answers one question: read fewer bytes per token, or do more work per byte read. It computes the KV cache from the config.json of five real models, rewrites online softmax in plain PyTorch, and shows that long context is expensive mostly because of memory, not because of O(n²).
Efficient Fine-Tuning: LoRA, QLoRA, and Quantization
Full fine-tuning an 8B model needs about 128 GB for the model states alone. This post breaks that number down and shows which part LoRA cuts and which part quantization cuts. It measures the SVD of ΔW between Qwen2.5 base and Instruct, attaches LoRA with PEFT, trains real QLoRA on an 11 GB RTX 2080 Ti, rebuilds the NF4 table by hand, and compares INT8, INT4 and NF4 on English and Vietnamese text. The thread running through it: two knobs, each cutting a different part of memory, and a cost you have to measure on your own task.
AI Weekly, 28 Sep – 5 Oct 2026: Cheaper Frontier Models, Controlled Releases and New Rules of the Game
This week three frontier models landed at roughly $2/$10 per 1M tokens, Google opened Gemini 4 Argon to cyber defenders first, and both the White House and California added new rules. Developers got new ways to hook into agents and RAG: Claude Code mods, Clef and Index-Translate.
Post-training: From Base Model to Assistant
A base model predicts tokens very well but does not know how to answer a question. This post runs the same question through a base model and an instruct model, prints the chat template and the SFT loss mask, derives step by step how Bradley–Terry leads to the reward-minus-KL objective and then to the DPO loss, computes the DPO loss on a real model, and puts the Llama 2, Llama 3 and Tülu 3 pipelines side by side. The thread running through it: post-training adds almost no knowledge; it shapes how the model uses the knowledge it already has.
Pretraining: Next-Token Prediction and Scaling Laws
All of pretraining optimizes one thing: the cross-entropy of the next token. This post computes loss, perplexity and bits-per-byte on a real model, rereads the Kaplan and Chinchilla scaling laws and the corrections that came later, works out the compute budget of Llama 3 by hand and checks it against Meta's numbers, explains why memory forces us to split a model across many GPUs, and shows how to tell that a training run is going wrong.
From Logits to Answers: Decoding and Sampling
The model only produces logits; choosing the token is the job of decoding. This post reads the source of Hugging Face Transformers and vLLM to see which steps logits go through and in what order, runs temperature, top-k, top-p and min-p on a real model, explains repetition penalty and constrained decoding for JSON, then answers why temperature 0 still does not guarantee the same answer twice.
Inside a Transformer Block: FFN, the Residual Stream, and Normalization
Attention is only half of a Transformer block, and not even the heavier half. This post covers the other half: the residual stream as a shared bus, the FFN as a key–value memory, GELU and SwiGLU, LayerNorm, RMSNorm and where the norm goes, then counts the parameters and FLOPs of Llama 3.1 8B to show why the FFN ≈ 2/3 rule of thumb is out of date.
How JEV Works: Choice, Score, Noul and RLCD
One JEV call takes roughly 70–500 ms. In that window, a request made of state, a model name and typed questions goes in, and a set of probabilities comes out. This post opens each layer of that loop: the schema, the three question types, runnable code, the parallel sampler, RLCD, calibration, how to set thresholds, and the line between what TypeSafe has published and what nobody knows yet.
No posts match the current filters.
Once a month, new posts only. No ads, unsubscribe anytime.