Transformer from A to Z: Understanding the Foundation of LLMs

If you are starting to learn Generative AI, sooner or later you will run into one word that shows up almost everywhere: Transformer.

GPT uses a Transformer.
BERT uses a Transformer.
T5, BART, Llama, Gemma, Mistral, Whisper, Vision Transformer… all have a Transformer at their core.

But what is a Transformer, really? Why did it change NLP? How does Attention work? How do the Encoder and the Decoder differ? Why can an LLM generate text? What is the KV Cache? How does the context window affect inference?

This post lays out the most fundamental concepts behind the Transformer in a systematic way, and deliberately goes beyond “introduction” level exactly where a good mental model makes a real difference.

Concretely, besides Attention and the three architecture families, the post also answers the questions introductory material usually skips:

• Why does Vietnamese (and many other non-English languages) cost more tokens than English, and how does that affect your bill?
• Why is there a sqrt(d_k) in the Attention formula?
• What do the residual connection and the FFN do - and why is the FFN where most of the parameters live?
• Why is Attention completely blind to order, and how does RoPE fix that?
• Why do prefill and decode bottleneck in two different places, and why does that drive almost every infrastructure decision?
• Exactly how many GB does the KV Cache take - and how much does GQA cut?
• Why do models make things up, and why does telling them “don’t make things up” help so little?

The goal is that by the end you have enough of a foundation to move on to RAG, Agents or LLMOps without feeling like you are operating a black box.

01What is a Transformer, really?

In short:

A Transformer is a neural network architecture that processes sequences of data using the Attention mechanism, which lets each element in the sequence decide which other elements matter to it.

The Transformer was first introduced in 2017 for machine translation. Since then, the architecture has become the foundation of a huge number of modern models.

A simple mental model:

INPUT
"Transformers are powerful"

        ↓

TOKENIZATION
["Transform", "ers", "are", "powerful"]

        ↓

TOKEN + POSITION REPRESENTATION
token → numeric vector

        ↓

TRANSFORMER BLOCKS
Attention
+
Feed-Forward layers

        ↓

CONTEXTUAL REPRESENTATIONS

        ↓

TASK HEAD / LANGUAGE MODEL HEAD

        ↓

OUTPUT
classification / token / translation / embedding / ...

The most important idea sits in the phrase contextual representation.

A word is no longer understood in isolation. Its representation depends on the tokens around it.

Take the word:

bank

It can mean:

• a financial institution;
• the side of a river.

A Transformer can use context to tell these two meanings apart.

That is one of the reasons Attention matters so much.

02Tokens: what the model actually sees

Before talking about Attention, we need to clear up something many learners skip over: the model does not read letters, and it does not read words. It reads tokens.

This is not a minor technical detail. The token is the unit everything downstream is measured in: the context window is counted in tokens, API pricing is per token, generation speed is tokens/second, and KV Cache memory grows per token. Get this layer wrong and every later estimate of cost and limits is wrong with it.

2.1 Why not just use words or characters?

There are three obvious options, and all three have problems.

Word-level splitting

"Tôi thích học máy" → ["Tôi", "thích", "học", "máy"]

(Vietnamese for “I like machine learning”; Vietnamese writes every syllable as a separate space-delimited word.)

The problem: the vocabulary explodes. English has millions of word forms once you count variants (run, runs, running, ran, runner…). Worse, any word not seen during training becomes [UNK] — the model loses that information entirely. Proper names, new terms and typos all fall into this hole.

Character-level splitting

"máy" → ["m", "á", "y"]

The vocabulary is small and you never get [UNK]. But sequences become 4–5 times longer, and the cost of Attention grows with the square of the length — so this approach is brutally expensive. On top of that, each character carries too little meaning, and the model has to relearn how letters combine into words from scratch.

Subword — the practical choice

The compromise: common words stay as a single token, rare words get split into meaningful pieces.

"học"          → ["học"]              ← common, 1 token
"tokenization" → ["token", "ization"] ← rare, split
"Nguyễn"       → ["Ngu", "yễn"]       ← rare for an English tokenizer

(học means “learn/study”; Nguyễn is the most common Vietnamese family name.)

This keeps the vocabulary at a few tens of thousands, never hits [UNK], and words that share a root still share tokens.

2.2 Three tokenizer algorithms you will meet

The important common thread: a tokenizer is learned from data, not hand-written with rules. It is a trained artifact that ships with the checkpoint and cannot be swapped for another tokenizer — using the wrong tokenizer for a checkpoint produces a meaningless sequence of ids.

One variant worth knowing is byte-level BPE (GPT-2 onwards): the tokenizer works on UTF-8 bytes instead of Unicode characters. That lets it represent any string, including emoji or characters it has never seen — the price is that non-English characters take more bytes, and therefore more tokens.

2.3 The token tax on Vietnamese

This is something almost no Vietnamese-language article mentions, even though it hits your bill directly. Call it the token tax: the extra tokens a language pays to say the same thing as English.

Most popular tokenizers are trained mostly on English data. The result: Vietnamese gets chopped up much more finely. Accented characters take 2–3 bytes in UTF-8, and with byte-level BPE a single Vietnamese syllable often costs 2–3 tokens instead of 1.

For readers outside Vietnam, the reason is worth spelling out. A BPE vocabulary is built by merging the most frequent byte pairs in its training corpus, so when that corpus is overwhelmingly English, most merges go to English fragments. Vietnamese adds two handicaps on top: its diacritics (ă, ơ, ễ, ự…) are multi-byte UTF-8 characters, and its words are short, space-separated syllables, so the tokenizer learns far fewer merges that cover a whole Vietnamese word or phrase. The same effect, to varying degrees, applies to many languages written outside the Latin-ASCII range.

A practical rule of thumb:

English     ~ 1 token   ≈ 4 characters  ≈ 0.75 words
Vietnamese  ~ 1.5 – 2.5x the tokens of the same content in English

What that means in practice:

• Higher API cost for the same amount of information. A Vietnamese document and its English translation can differ by almost 2x in token count.
• The context window shrinks accordingly. A 128K-token model holds noticeably less Vietnamese content than English.
• RAG chunk sizes need adjusting. If you copy the chunk_size from an English tutorial as-is, your Vietnamese chunks will carry less meaning than you expect.
• Don’t estimate tokens by counting words. Always count with the tokenizer of the model you are actually using.

Models with better multilingual tokenizers (Gemma, Qwen, models trained on lots of Asian-language data) narrow this gap considerably — a model-selection criterion few people pay attention to.

2.4 Why LLMs get seemingly easy things wrong

A lot of “silly” LLM behavior is rooted at the token layer, not at the reasoning layer.

Counting letters. Ask “how many r’s are in strawberry” and the answer is often wrong, because the model sees ["str", "aw", "berry"], not individual characters. It has to infer the characters indirectly.

Arithmetic with long numbers. Numbers are split inconsistently: 1234 may become ["123", "4"] while 12345 becomes ["12", "345"]. The ones/tens/hundreds structure is broken up.

Reversing strings, wordplay, counting syllables. Same root cause.

A mental model worth remembering:

The model does not see letters. It sees pre-compressed chunks of text — and every character-level operation is indirect inference through that compression layer.

2.5 Special tokens

Besides content tokens, the tokenizer also inserts control tokens:

[CLS] [SEP]      ← BERT: start of sequence / separator
<s>   </s>       ← Llama, Mistral: begin / end
<|endoftext|>    ← GPT: end of text
[PAD]            ← padding to equal length in a batch
[MASK]           ← masked position in MLM

Chat models add a chat template on top — a set of tokens marking who is speaking:

<|im_start|>system
You are a helpful assistant.
<|im_end|>
<|im_start|>user
Hello
<|im_end|>
<|im_start|>assistant

This is a very common source of bugs in practice: if you assemble the prompt by hand without following the checkpoint’s exact template, output quality can drop noticeably even though the model itself is perfectly fine.

2.6 From token to vector

After tokenization, each token is an integer — just an index into the vocabulary, with no meaning yet.

"học"  →  id 15234  →  embedding vector [0.21, -0.83, ..., 0.05]
                        (d_model dimensions, e.g. 4096)

The embedding matrix has shape vocab_size × d_model and is learned during training. This is where meaning starts to appear: tokens used in similar contexts gradually end up with nearby vectors.

But one point needs stressing, because it leads straight into the next part:

Embeddings at this layer are still static. The token bank has exactly one vector, whether we are talking about money or rivers.

Turning that static vector into a contextual representation is precisely the job of the Transformer blocks — and specifically of Attention.

03NLP and Large Language Models

3.1 What is NLP?

Natural Language Processing – NLP is the field that studies how computers process and work with human language.

NLP covers many groups of problems:

The key point:

NLP is a field, while LLMs are a family of models used to solve many problems within that field.

3.2 What is an LLM?

A Large Language Model is a large-scale language model trained on a very large amount of data to learn the patterns of language.

A modern LLM usually stands out in four ways:

Scale

A model can have anywhere from millions to billions, even hundreds of billions, of parameters.

General capability

The same model can perform many kinds of tasks.

In-context learning

The model can learn how to do a task purely from the prompt and the examples in its context.

Emergent capabilities

Once a model is scaled up enough, some capabilities can appear even though nothing was programmed directly for each task.

NLP used to look like this:

Model A → sentiment
Model B → translation
Model C → QA
Model D → summarization

In the LLM era:

             ┌→ summarize
             ├→ translate
One LLM  ────┼→ classify
             ├→ answer
             ├→ write code
             └→ generate

That is a huge shift.

3.3 But an LLM is not an “all-knowing machine”

LLMs still have plenty of limitations:

• hallucination;
• bias;
• a finite context window;
• high compute cost;
• no guarantee of understanding the world the way humans do;
• probabilistic output.

So:

A strong LLM is not the same thing as a trustworthy AI system.

When building a production system, we need to add RAG, evaluation, guardrails, monitoring, human-in-the-loop and governance.

04What can a Transformer do?

An easy way into Transformers in Python is the pipeline-style API:

from transformers import pipeline

pipeline() provides a very convenient abstraction:

RAW INPUT
    ↓
PREPROCESSING
    ↓
MODEL
    ↓
POSTPROCESSING
    ↓
HUMAN-READABLE OUTPUT

For example:

from transformers import pipeline

classifier = pipeline("sentiment-analysis")

classifier("This course is very useful.")

Instead of writing the tokenizer, model inference and output handling yourself, pipeline() wraps it all up.

4.1 Zero-shot classification

One really interesting application is zero-shot classification.

Instead of:

collect labeled data
↓
train classifier
↓
predict

we can do:

TEXT:
"Tesla announced a new factory."

LABELS:
business
sports
education

The model judges which label fits best without being trained specifically for this label set.

The key concept:

Zero-shot = the model performs a task without any task-specific training examples in the current fine-tuning step.

An easy place to get confused

The phrase “zero-shot” is used for two completely different mechanisms, and mixing them up is a common mistake.

In other words, zero-shot classification is not the model “just knowing”. It leverages a capability it has already learned - entailment - and maps your classification problem onto exactly that capability.

4.2 Text generation

Prompt:

Artificial intelligence will...

A decoder model works in a loop:

predict token 1
↓
append token
↓
predict token 2
↓
append token
↓
...

This is autoregressive generation.

Almost every modern LLM chatbot works this way.

4.3 Fill-mask

Example:

Paris is the [MASK] of France.

A BERT-like model can predict the masked token.

This is directly tied to:

Masked Language Modeling – MLM

4.4 Named Entity Recognition

Example:

Tim Cook works at Apple in California.

The model can output:

Tim Cook   → PERSON
Apple      → ORGANIZATION
California → LOCATION

This is Token Classification.

NER is used in:

• document processing;
• CRM;
• legal;
• financial documents;
• information extraction;
• search.

4.5 Question Answering

With extractive QA:

Context:
"John works at Microsoft in Seattle."

Question:
"Where does John work?"

The model can find the span:

Microsoft

The distinction to keep in mind:

Extractive QA

Finds an answer that sits right inside the context.

Generative QA

Generates a new answer with a decoder.

4.6 Summarization and Translation

These are sequence-to-sequence problems. For example:

long document
      ↓
   Transformer
      ↓
short summary

or:

English
  ↓
Transformer
  ↓
Vietnamese

Encoder-Decoder architectures such as T5/BART are especially well suited to this kind of problem.

4.7 Transformers are not just for text

Transformers are used across many modalities today.

Text

• text generation;
• classification;
• summarization;
• translation;
• QA;
• embeddings.

Audio

• speech recognition;
• audio classification;
• text-to-speech.

Vision

• image classification;
• object detection;
• image understanding.

Multimodal

• image + text;
• audio + text;
• video + text;
• document understanding.

What started as an NLP architecture has become the common foundation of a huge number of foundation models.

05How does a Transformer work?

This is the most important part.

5.1 A bit of history

A simplified timeline:

2017  Transformer          the original architecture, for machine translation
2018  GPT                  decoder-only, generative pretraining
2018  BERT                 encoder-only, bidirectional, MLM
2019  GPT-2                scale starts to unlock new capabilities
2019  T5                   every task cast as text-to-text
2020  GPT-3                few-shot learning right in the prompt
2022  InstructGPT          RLHF - from text completer to assistant
2023  Llama / Mistral      open models, GQA, sliding-window attention
2023  Mixtral              MoE goes mainstream
2024  Llama 3 / Gemma 2    long context, better multilingual tokenizers
2025+ Reasoning models     trained to reason, more compute at inference time

Read top to bottom, there are three fairly clear waves:

Notice that the focus has shifted from architecture to training and serving. The Transformer block itself has barely changed since 2017 - what changed is the normalization, the positional encoding, how attention is organized, and everything that comes after pretraining (the post-training stage).

The years are not the thing to remember.

The thing to remember is that the Transformer grew into three big families:

Encoder-only
Decoder-only
Encoder-Decoder

5.2 Pretraining

Transformers are usually not trained from scratch for each individual task such as sentiment or QA.

Instead:

Massive raw data
      ↓
PRETRAINING
      ↓
General language representation

Then:

Pretrained model
      ↓
Fine-tuning / Instruction tuning
      ↓
Task-specific behavior

A critically important point:

Language model pretraining is usually self-supervised learning.

The data creates its own supervision.

For example:

The capital of France is Paris

we can create a next-token prediction task from it automatically, without anyone labeling anything by hand.

5.3 Masked Language Modeling

BERT is usually explained through MLM.

Input:

The dog [MASK] across the road.

The model predicts:

ran

BERT can see both:

LEFT CONTEXT
+
MASK
+
RIGHT CONTEXT

So its representations are bidirectional.

5.4 Causal Language Modeling

GPT works differently.

Input:

The capital of France is

The model predicts:

Paris

When predicting token t:

token t
depends on
token 1 ... token t-1

The model is not allowed to see future tokens.

This is:

Causal Language Modeling – CLM

And it is the foundation of almost every modern generative LLM.

5.5 Pretraining vs Fine-tuning

Keep these two clearly apart:

Pretraining

random weights
↓
massive dataset
↓
large compute
↓
general pretrained model

Fine-tuning

pretrained weights
↓
smaller task/domain dataset
↓
additional training
↓
specialized model

Fine-tuning usually:

• needs less data;
• needs less compute;
• is faster;
• reuses the representations learned during pretraining.

This is a form of Transfer Learning.

An easy mental model:

Pretraining = learning general foundations.
Fine-tuning = learning a specialty.

5.6 General Transformer Architecture

The original Transformer has two parts:

        INPUT
          ↓
     ┌─────────┐
     │ ENCODER │
     └─────────┘
          ↓
   Representation
          ↓
     ┌─────────┐
     │ DECODER │
     └─────────┘
          ↓
        OUTPUT

Encoder

Focuses on:

understanding and representing the input.

Decoder

Focuses on:

generating the output.

From these two blocks we get three common architectures:

ENCODER ONLY

DECODER ONLY

ENCODER + DECODER

5.7 Attention – the heart of the Transformer

Example:

The animal didn't cross the street because it was tired.

What does the word:

it

refer to?

To understand it, the model has to judge which tokens are most relevant.

One way to picture it:

it
↓ attention
animal     ████████
street     ██
because    █
tired      █████

Attention lets a token’s representation change with context.

That is the key difference from using a fixed embedding for each word.

5.8 Query, Key, Value

To understand Attention more deeply, you need three concepts:

Query – Q

What information is the current token looking for?

Key – K

What kind of information does another token hold, to be matched against?

Value – V

The content actually retrieved once Attention has decided how relevant each token is.

Mental model:

Query of the current token
          ↓
compared with
          ↓
Keys of the other tokens
          ↓
attention scores
          ↓
weighted combination
          ↓
Values

The core formula:

Attention(Q,K,V) = softmax((QK^T) / sqrt(d_k))V

Reading the formula out loud

Every piece of the formula has a very concrete meaning:

Why divide by sqrt(d_k)?

This detail is often skimmed over, but the reason is well worth knowing.

Q and K are d_k-dimensional vectors. Their dot product is a sum of d_k terms, so the variance of the result grows with d_k. With d_k = 128, raw scores can swing by tens.

Feeding numbers that large into softmax makes the distribution almost one-hot: one token gets a weight of ~1, the rest ~0. At that point the softmax gradient is close to 0 and the model can barely learn anymore.

Dividing by sqrt(d_k) pulls the variance back to around 1, keeping softmax in the region where gradients still flow.

This is a textbook example of a kind of detail found all over deep learning: a constant that looks arbitrary but actually exists to keep training from collapsing.

A concrete numeric example

Take the sentence "the cat sleeps" and suppose we are computing a new representation for the token sleeps.

Raw scores (Q_sleeps · K_i):

  the      1.2
  cat      4.8
  sleeps   2.1

Divide by sqrt(d_k), say d_k = 4 → divide by 2:

  the      0.60
  cat      2.40
  sleeps   1.05

After softmax:

  the      0.12
  cat      0.70
  sleeps   0.18

New representation of "sleeps":

  0.12 × V_the  +  0.70 × V_cat  +  0.18 × V_sleeps

The token sleeps now carries most of the information from cat — it knows who is sleeping. That is exactly what “contextual representation” means.

You don’t need to memorize the formula yet.

But you do need to understand the logic:

Q × K
↓
relevance score
↓
softmax
↓
attention weights
↓
weighted V
↓
new contextual representation

5.9 Multi-Head Attention

A Transformer does not have just one Attention head.

Instead:

Head 1 → learns one kind of relationship
Head 2 → learns another kind
Head 3 → learns another kind
...

The results are then combined.

Some heads may be sensitive to:

• syntax;
• subject-object relationships;
• long-distance dependencies;
• positional patterns;
• semantic similarity.

That is why it is called:

Multi-Head Attention

5.10 From Multi-Head to MQA and GQA

Multi-Head Attention as described above is the original 2017 version. Modern LLMs almost never use it as-is, for a very practical reason: each head has to store its own Keys and Values in the cache during generation, and that is the biggest VRAM consumer at long context.

Three variants, distinguished by “how many K/V sets are shared”:

Picture it:

MHA        Q1 Q2 Q3 Q4          GQA        Q1 Q2 Q3 Q4
           |  |  |  |                       \  /  \  /
           K1 K2 K3 K4                        KV1   KV2


MQA        Q1 Q2 Q3 Q4
            \  \  /  /
                KV

The number of Query heads stays the same in all three variants - only the number of Key/Value sets changes. So expressiveness on the Query side is not cut; only the “information store” is shared.

A concrete number: Llama-3-70B uses 64 Query heads but only 8 Key/Value heads. That 8:1 ratio cuts the KV Cache by 8x compared with MHA, with almost no loss in quality.

Remember GQA: it is today’s most common answer to the KV Cache problem we will run into again in the inference section.

5.11 Masked Attention

The decoder is not allowed to see future tokens.

Say we are predicting the 4th token:

token1 token2 token3 [token4] token5
  ✓      ✓      ✓        ?       ✗

The model uses a causal attention mask.

Besides the causal mask, a system can also use attention masks to ignore:

• padding tokens;
• invalid positions;
• restricted regions.

5.12 Inside a Transformer block

So far we have only talked about Attention. But Attention is not the whole Transformer block - in fact it is not even the part with the most parameters.

A full block looks like this:

        x  (incoming hidden state)
        |
        +--------------+
        |              |  residual
   +----v----+         |
   |  Norm   |         |
   +----+----+         |
        |              |
   +----v--------+     |
   |  Attention  |     |
   +----+--------+     |
        |              |
        v              |
       (+) <-----------+
        |
        +--------------+
        |              |  residual
   +----v----+         |
   |  Norm   |         |
   +----+----+         |
        |              |
   +----v--------+     |
   |     FFN     |     |
   +----+--------+     |
        |              |
        v              |
       (+) <-----------+
        |
        v
       x2  (outgoing hidden state)

A real model is just this block stacked many times: 12 times in BERT-base, 32 in Llama-3-8B, 80 in 70B-class models.

The three components besides Attention all matter.

Residual connection

That is the + path that loops around each sub-block. Instead of forcing each layer to produce an entirely new representation, the residual lets the layer learn only the difference it needs to add:

output = x + f(x)      instead of      output = f(x)

Why this is vital: when backpropagation runs back through 80 layers, the gradient is multiplied through every layer and tends to vanish toward 0. The residual path gives the gradient a straight road through the full depth of the model.

Without residuals, deep Transformers simply cannot be trained. This is why the architecture scales to tens of billions of parameters, and it is also the piece most often left out when people describe the Transformer.

A useful way to see it: the residual turns the hidden state into a shared flow of information (the residual stream) that each layer reads from, computes on, and writes back into. Layers communicate with each other through that stream.

Normalization

After many additions, the magnitude of the hidden state tends to drift. Normalization pulls it back to a stable scale.

LayerNorm normalizes by the mean and standard deviation of each vector:

LayerNorm(x) = gamma * (x - mean) / std + beta

RMSNorm - what most modern LLMs (Llama, Mistral, Gemma) use - drops the mean subtraction entirely:

RMSNorm(x) = gamma * x / sqrt(mean(x^2))

Simpler, faster, and empirically just as good.

A small detail with a big effect is where the Norm goes:

Post-LN (original 2017)         Pre-LN (every modern LLM)

x -> Attn -> (+) -> Norm        x -> Norm -> Attn -> (+)

Post-LN needs very careful warmup and diverges easily in deep models. Pre-LN trains far more stably - which is why the whole field moved to it.

Feed-Forward Network

This is the most underrated part, even though it holds about two thirds of the model’s parameters.

The FFN processes each token independently, with no information exchanged between positions:

d_model  ->  expand ~4x   ->  non-linearity  ->  shrink back to d_model

  4096   ->    16384      ->  GELU/SwiGLU    ->     4096

The division of labor between the two blocks is very clear:

Interpretability research suggests the FFN works much like a key-value memory: it is where most of the model’s factual knowledge is stored. In short, Attention decides where to look, while the FFN decides what it knows.

Recent LLMs replace GELU with SwiGLU, a gated variant that gives better quality for the same parameter budget.

5.13 Positional encoding: how the model knows order

There is one property of Attention that many people miss: it does not care about order at all.

Attention only computes dot products between pairs of vectors. If you shuffle the input tokens, the set of outputs is shuffled in exactly the same way - nothing in the math tells the model which token comes before which.

That means without positional information, these two sentences are identical in the model’s eyes:

"Dog bites man"
"Man bites dog"

So positional information has to be injected explicitly.

Sinusoidal - the 2017 original

Add a sine/cosine wave pattern directly to the embedding, with a different frequency per dimension. Nothing to learn, and in theory it extends to lengths never seen before.

Learned positional embedding - BERT, GPT-2

Treat each position like a token and learn its own vector. Simple and effective, but rigid: the model cannot handle positions beyond the length it was trained on.

RoPE - Rotary Position Embedding

This is what Llama, Mistral, Gemma and Qwen all use, and the idea is well worth understanding.

Instead of adding position to the vector, RoPE rotates the Q and K vectors by an angle proportional to the position:

token at position 1  ->  rotate by 1 theta
token at position 2  ->  rotate by 2 theta
token at position 5  ->  rotate by 5 theta

The elegant part: when you take the dot product between Q at position m and K at position n, the result depends only on the difference m - n. In other words, RoPE naturally encodes relative position - which matters far more in language than absolute position.

The practical consequence: RoPE makes it possible to extend the context after training is done by interpolating the rotation angles (position interpolation, NTK-aware scaling, YaRN). That is exactly how many models have been stretched from 4K to 32K or 128K context without pretraining again from scratch.

ALiBi

Drops positional embeddings altogether and instead subtracts an amount proportional to distance directly from the attention score - the farther the token, the heavier the penalty. Simple, and it extends to longer lengths very well.

If you remember only one idea from this section: Attention is blind to order, and positional encoding is the only thing that tells the model the order of a sentence. RoPE is the default choice today, and it is the technical reason context windows can be stretched.

5.14 Architecture vs Checkpoint vs Model

These three words are often used interchangeably.

Architecture

The model’s blueprint.

Example:

BERT

Checkpoint

A specific set of trained weights.

Example:

bert-base-cased

Model

A loose term that can mean either the architecture or the checkpoint depending on context.

When learning AI, get used to this distinction from the start.

06How does a Transformer solve tasks?

Now let’s connect the architecture to applications.

6.1 Text Generation – GPT

GPT is decoder-only.

A simplified pipeline:

text
↓
tokenizer
↓
tokens
↓
embedding + position
↓
decoder blocks
↓
masked self-attention
↓
hidden states
↓
LM Head
↓
logits
↓
sampling
↓
next token

The key point:

A Transformer needs positional information to know the order of tokens.

Attention on its own does not understand which token comes before or after.

6.2 Text Classification – BERT

Input:

[CLS] I love this movie [SEP]

The representation of [CLS] can be used to stand for the whole sequence.

But an important warning is needed here, because this is a very common mistake:

[CLS] only carries meaning after it has been fine-tuned for a specific task. Taking [CLS] from a raw BERT checkpoint and using it as a sentence embedding gives results worse than simply averaging the tokens - and far worse than dedicated models such as Sentence-BERT or E5.

During pretraining, [CLS] is only trained through the Next Sentence Prediction task, which is too weak to produce a good sentence representation. If you need embeddings for similarity comparison or RAG, use a model trained specifically for that.

Pipeline:

BERT
 ↓
[CLS] hidden state
 ↓
classification head
 ↓
logits
 ↓
softmax
 ↓
positive / negative

BERT is a great fit for:

• sentiment;
• intent classification;
• spam detection;
• topic classification.

6.3 Token Classification

Unlike sentence classification:

Sentence Classification
ONE LABEL / SENTENCE

Token Classification:

John    → PERSON
works   → O
at      → O
Google  → ORG

Each token’s hidden state goes through the classification head.

Applications:

• NER;
• POS tagging;
• entity extraction.

6.4 Extractive Question Answering

A BERT-like model can predict:

start position
+
end position

For example:

John works at [Google] in New York
               ↑    ↑
             start end

This approach is especially useful when the answer sits inside a document.

6.5 Summarization – BART/T5

Encoder-Decoder:

LONG TEXT
   ↓
ENCODER
   ↓
representation
   ↓
DECODER
   ↓
SUMMARY

BART is pretrained by corrupting the input and learning to reconstruct it.

One such strategy is text infilling:

Original:
The cat is sleeping on the sofa.

Corrupted:
The cat <mask> on the sofa.

Target:
The cat is sleeping on the sofa.

6.6 Translation

Same pattern:

Sequence A
↓
Encoder
↓
meaning representation
↓
Decoder
↓
Sequence B

For example:

English
↓
Encoder
↓
Decoder
↓
Vietnamese

6.7 Audio – Whisper

Transformers are also used for audio.

A general pipeline:

AUDIO
↓
log-Mel spectrogram
↓
ENCODER
↓
audio representation
↓
DECODER
↓
TEXT TOKENS

Whisper can support:

• transcription;
• translation;
• language identification.

6.8 Vision Transformer – ViT

ViT’s really neat insight:

If text is a sequence of tokens, an image can also be turned into a sequence of patches.

For example, with a 224 × 224 pixel image and a patch size of 16 × 16 pixels:

Image:       224 × 224 pixels
Each patch:   16 × 16  pixels

Patches per side:  224 / 16 = 14

Total patches:  14 × 14 = 196

Note how to read this: 16 × 16 here is the size of one patch, not the number of patches. It is very easy to misread.

Each patch is flattened into a vector and linearly projected into an embedding - exactly the role the embedding table plays for text tokens. ViT also prepends a [CLS] token, so the encoder actually receives 196 + 1 = 197 tokens.

Pipeline:

IMAGE
↓
split patches
↓
[P1][P2][P3]...[P196]
↓
embeddings + positions
↓
Transformer Encoder
↓
classification / representation

This is how the Transformer expanded from NLP into Vision.

07The three key Transformer architectures

This is the table worth remembering.

7.1 When to use an Encoder?

When the task is mainly about:

understanding the input.

For example:

INPUT
↓
UNDERSTAND
↓
LABEL / EMBEDDING / SPAN

Good for:

• classification;
• NER;
• semantic representation;
• extractive QA.

7.2 When to use a Decoder?

When the task is mainly about:

generating output.

For example:

PROMPT
↓
GENERATE
↓
TOKEN
↓
TOKEN
↓
TOKEN
...

Used for:

• chatbots;
• content writing;
• code generation;
• reasoning;
• assistants.

Most modern LLMs today are decoder-only.

7.3 When to use an Encoder-Decoder?

When the task has the shape:

Input sequence → transformed output sequence.

For example:

English → Vietnamese
Document → Summary
Question + Context → Generated Answer

7.4 Modern LLM Training

A freshly pretrained model is not an assistant yet. It only knows how to complete text. Ask it “What is the capital of France?” and a merely pretrained model may well respond by writing a few more similar questions, because that is what tends to appear on the internet.

The gap between a “text-completion model” and ChatGPT lies in the later stages. This is the part most often skipped when people explain LLMs.

1. PRETRAINING
   next-token prediction on massive data
   -> knows language and facts, but not how to behave
              |
              v
2. SFT - Supervised Fine-Tuning
   learns from human-written (instruction, good answer) pairs
   -> knows how to answer in conversational format
              |
              v
3. ALIGNMENT - RLHF or DPO
   learns from comparisons: which answer humans prefer
   -> more helpful, safer, refuses at the right moments
              |
              v
4. (optional) REASONING TRAINING
   trained with rewards on reasoning steps
   -> solves math, writes code, multi-step reasoning

Why a separate alignment step?

The problem is very practical: for many questions, we cannot write a single “correct” answer to use as a label. What is a good summary? A polite refusal? An easy-to-follow explanation?

Humans struggle to write the perfect answer, but they are very good at comparing two answers. Alignment exploits exactly that strength.

RLHF - Reinforcement Learning from Human Feedback - has three steps:

a. Collect human comparisons:  A is better than B
b. Train a reward model to predict that choice
c. Use RL (usually PPO) to fine-tune the LLM to maximize reward

DPO - Direct Preference Optimization - reaches the same goal but drops the reward model and the RL loop entirely, optimizing directly on the comparison data. It is much simpler and more stable, and has therefore become a popular choice.

This explains quite a lot of behavior you see every day

A mental model worth remembering: pretraining decides what the model knows, alignment decides how the model behaves. Many complaints about an LLM’s “personality” are really complaints about the alignment stage, not about the Transformer architecture.

7.5 The O(n²) problem of Attention

For a sequence length of:

n

full self-attention builds a matrix of roughly:

n x n

Complexity:

O(n^2)

An intuitive example:

1,000 tokens   → ~1M pair relationships
10,000 tokens  → ~100M
100,000 tokens → ~10B

This is one of the reasons long context is expensive.

7.6 Local Attention

Instead of letting every token look at every token, Local Attention limits the range.

... T3 T4 [T5] T6 T7 ...
     \   \  |  /   /
       window of w tokens

If each token only looks at the nearest w tokens, the cost drops from O(n²) to O(n × w) - that is, linear in sequence length.

The obvious question: what about information that is far away?

The answer is quite subtle - stack many layers and the visible range widens, exactly like the receptive field in a CNN:

Layer 1: each token sees directly    w  tokens
Layer 2: sees indirectly            2w
Layer 3:                            3w
...
Layer L:                        L × w

Mistral 7B uses a 4096 window with 32 layers, so in theory the top layer can still reach about 131K tokens, even though each layer only looks locally.

Some architectures (Longformer, BigBird) also add:

• global tokens - a few special tokens that see everything and are seen by everything;
• random attention - a few random links to shorten the paths information has to travel.

The trade-off to know: local attention does save compute, but its ability to capture long-range relationships precisely is still worse than full attention. That is why many models interleave - some local layers, some full layers.

7.7 FlashAttention: optimizing without changing the result

The approaches above all change the computation - they drop some links in exchange for speed, so the result differs to some degree.

FlashAttention takes a completely different route, and it is the idea most worth learning in this group:

Keep the attention math exactly the same, without skipping a single link, but change how data moves through GPU memory.

The problem it solves: standard attention builds the n × n matrix and then writes the whole matrix out to HBM - the GPU’s large but slow memory - and reads it back. With n = 8192, that matrix is 64 million numbers for each head of each layer. The bottleneck is not the multiplication, but the memory read/write traffic.

FlashAttention processes attention in small tiles that fit in SRAM - the fast memory right next to the compute units - computes softmax incrementally, and never materializes the full n × n matrix.

Standard attention:  compute -> write n×n matrix to HBM -> read back -> multiply by V
FlashAttention:      tile -> compute inside SRAM -> write only the final result

The result:

• many times faster in practice;
• memory drops from O(n²) to O(n);
• mathematically identical output, not an approximation.

That is why FlashAttention is enabled by default almost everywhere today, while the approximate attention methods of the 2020 era - Reformer with LSH hashing, Linformer, Performer - are rarely used in production LLMs. They remain interesting as ideas, but most of them trade away quality, and FlashAttention does not.

The broader lesson: in modern deep learning, many bottlenecks are memory bandwidth, not compute. This idea comes right back in the inference section.

7.8 Mixture of Experts: decoupling size from cost

So far we have implicitly assumed every parameter in the model takes part in every token. Such a model is called dense.

Mixture of Experts breaks that assumption, and it is the architecture behind most of today’s frontier models.

The idea: replace the single FFN in each block with many parallel FFNs called experts, plus a small router that picks a few experts for each token.

Dense                          MoE

token                          token
  |                              |
  v                              v
 FFN   (always uses              router
        all of it)               |  picks 2 of 8
                        +--------+--------+
                        v        v        v
                      FFN1     FFN2 ...  FFN8
                        |        |
                        +---+----+
                            v
                      weighted sum

The crux is that parameter count and compute cost are decoupled:

The model holds knowledge equivalent to 47B parameters, but each token only pays the compute cost of roughly 13B.

This fits directly with the point in section 5.12 - that the FFN is where knowledge is stored. MoE essentially says: have lots of specialized knowledge stores, but only open a few at a time.

The price to pay, and this is the part often overlooked:

• VRAM still has to hold every expert, even though each pass only uses a few. MoE saves compute, not memory.
• Experts need load balancing, otherwise the router piles everything onto a few popular experts.
• Serving the model is much more complex, especially when it has to be split across many GPUs.

When you see a model advertised as “size A but as fast as a much smaller model”, it is almost certainly MoE. Read the two numbers carefully: total parameters determine the VRAM you need, active parameters determine speed.

As for positional encoding for long context - sinusoidal, learned, RoPE, ALiBi and the interpolation techniques used to stretch context - that was covered in section 5.13, because it is a core component of the architecture rather than an add-on optimization.

08Inference with LLMs

This part is critically important if your goal is to build AI applications.

8.1 Training vs Inference

Training:

data
↓
forward pass
↓
loss
↓
backpropagation
↓
update weights

Inference:

fixed weights
+
prompt
↓
prediction

Generative inference:

prompt
↓
predict next token
↓
append
↓
predict next token
↓
append
↓
...

8.2 Context Length

Context length is the number of tokens the model can process in one pass.

The context can include:

system prompt
+
conversation history
+
user input
+
retrieved documents
+
tool outputs
+
generated tokens

Mental model:

Context = the model’s temporary working memory for the current request.

This is the foundation for learning Context Engineering later.

8.3 What is prompting, really?

Prompting is not magic.

In more technical terms:

A prompt changes the input context, which changes the hidden states and the distribution of the next token.

Pipeline:

INPUT TOKENS
↓
Transformer
↓
LOGITS
↓
Probability distribution
↓
NEXT TOKEN

Change the prompt → the distribution changes → the output changes.

8.4 The two phases of inference

Phase 1 – Prefill

The model processes the entire prompt.

prompt
↓
tokenization
↓
tokens
↓
Transformer
↓
representations + KV cache

The whole prompt is processed in parallel in one pass, because every token is already available.

Phase 2 – Decode

Then the model generates one token at a time:

token 1
↓
token 2
↓
token 3
↓
...

Decode is autoregressive: each step processes only one token, and has to wait for the previous step to finish before it can run.

The two phases bottleneck in completely different places

This is the single most important idea in this section, and it explains almost every performance behavior you will observe.

Why is decode bottlenecked by memory? Because to generate one token, the GPU has to read all of the model’s weights from VRAM. For a 70B model in FP16, that is 140 GB of data moved across the memory bus - just to produce a single token. The actual amount of computation is tiny.

Prefill:  lots of compute   per  one read of the weights  ->  GPU busy
Decode:   little compute    per  that same read           ->  GPU waiting on memory

Three very practical consequences:

1. Buying a GPU with more compute often does not make decode much faster. The number to look at is memory bandwidth, not TFLOPS.
2. That is why batching is so effective. Once the weights are loaded, serving 1 or 32 requests at once costs almost the same - see section 8.14.
3. That is also why quantization speeds things up. Smaller weights mean fewer bytes to read at each step - see section 8.15.

If you remember just one sentence from the inference part: prefill is a compute problem, decode is a memory problem. Almost every serving optimization technique stems from this asymmetry.

8.5 Logits

A Transformer does not directly “pick a word”.

It produces a score for every item in the vocabulary.

For example:

Paris      4.20
London     2.30
France     1.50
Berlin     1.40
... (the rest of the vocabulary)

These raw scores are called:

Logits

A point often overlooked: logits are produced for the entire vocabulary, meaning tens of thousands to hundreds of thousands of numbers for every generation step. We only see the top few lines because the rest have very low scores.

Softmax turns them into probabilities:

Paris     72%
London    11%
France     5%
Berlin     4%
rest of the vocabulary    8%

The calculation, if you want to check it yourself:

exp(4.20) = 66.7      66.7 / 92.6 = 0.72
exp(2.30) = 10.0      10.0 / 92.6 = 0.11
exp(1.50) =  4.5       4.5 / 92.6 = 0.05
exp(1.40) =  4.1       4.1 / 92.6 = 0.04
tail      =  7.3       7.3 / 92.6 = 0.08
                     -------
total     = 92.6

A detail worth noticing: the gaps between logits matter more than their absolute values, because softmax exponentiates and then normalizes. Adding the same constant to every logit does not change the distribution at all.

That is exactly why temperature, in the next section, works: it divides the logits, stretching or compressing those very gaps.

The generation strategy decides which token is finally chosen.

8.6 Temperature

Temperature controls how spread out the distribution is.

Low temperature

Paris 90%
London 5%
...

→ more predictable.

High temperature

Paris 45%
London 20%
Lyon 12%
...

→ more diverse.

The actual mechanism

Temperature is not some mysterious knob. It simply divides the logits before they go into softmax:

softmax(logits / T)

Connecting back to section 8.5, where we saw that only the gaps between logits matter:

T < 1   divide by a small number  ->  gaps widen    ->  sharper distribution
T = 1   unchanged                 ->  exactly the distribution the model learned
T > 1   divide by a large number  ->  gaps shrink   ->  flatter distribution

Back to the example from section 8.5:

Logits:      Paris 4.20   London 2.30   France 1.50

T = 0.5      Paris  95%   London   1%   France   0%
T = 1.0      Paris  72%   London  11%   France   5%
T = 2.0      Paris  45%   London  17%   France  12%

The special case T = 0 does not actually divide by 0 - libraries treat it as greedy decoding: always pick the most probable token.

One point needs spelling out, because it is very often misunderstood:

T = 0 gives stable results, but it does not guarantee correct ones. The model can still make up false information with complete confidence - it just makes up the same sentence every time. Temperature controls diversity, not accuracy.

A practical rule of thumb:

A final note: temperature and top-p act on the same distribution. Tuning both at once makes the effect hard to predict. Common practice is to fix one and tune only the other.

8.7 Top-k

Keep:

the k tokens with the highest probability.

For example:

Vocabulary = 100,000

top_k = 5

The model only samples from the top 5 candidates.

8.8 Top-p / Nucleus Sampling

The number of candidates is not fixed.

Take the smallest group of tokens whose cumulative probability reaches the threshold p.

For example:

Paris    .50
London   .20
Berlin   .12
Madrid   .08
Tokyo    .03

With:

top_p = 0.9

the candidate group could be:

Paris + London + Berlin + Madrid

8.9 Presence and Frequency Penalty

Two ways to reduce repetition.

Presence penalty

A token that has already appeared → gets penalized.

Frequency penalty

The more often a token appears → the stronger the penalty.

8.10 EOS and generation limits

Generation can stop because of:

• max tokens;
• a stop sequence;
• the EOS token;
• a task-specific rule.

EOS:

End Of Sequence

8.11 Beam Search

Instead of keeping a single candidate:

A
↓
B
↓
C

beam search keeps several paths:

        ┌ B → C
A ──────┼ D → E
        └ F → G

It then keeps the sequences with the best scores.

Beam search is better suited to some structured generation problems than to creative chat.

8.12 Inference metrics you need to know

If you want to be an AI Engineer, these are the numbers you will be talking about every day.

TTFT – Time To First Token

The time from sending the request until the first token appears. It is determined by prefill, so it grows with prompt length.

This is the metric users feel most clearly: the silence before the text starts flowing. Long prompts, RAG stuffed with documents, conversation history that has not been trimmed - all of them hit TTFT directly.

TPOT – Time Per Output Token

The time to generate each token after the first. It is determined by decode, so it depends on memory bandwidth.

TPOT decides how fast the text streams out. Around 30 ms/token is about 33 tokens/second - faster than most users read, so it is usually enough.

Total latency

The two metrics above combine into the actual waiting time:

Total time = TTFT + (TPOT × number of generated tokens)

Example:  TTFT 400 ms,  TPOT 30 ms,  300 tokens generated

  400 ms + (30 ms × 300) = 9.4 seconds

A valuable observation: output length affects waiting time far more than input length. Asking the model to answer concisely is often the cheapest latency optimization, and the one everyone overlooks.

Throughput

How much the system can serve:

• requests per second;
• tokens per second;
• concurrent users.

Keep it clearly apart from latency, because the two trade off against each other: increasing the batch size raises throughput but also lengthens the latency of each request.

VRAM Usage

The GPU memory you need, in three parts:

Model weights      +     KV Cache       +     Activation buffers
   (fixed)            (per token)               (per batch)

Beginners often count only the weights and are then surprised by out-of-memory errors at runtime. The KV Cache is usually what decides how many concurrent users you can serve - the next section works it out concretely.

8.13 KV Cache

Autoregressive generation is very repetitive.

For example:

Token 1
Token 2
Token 3
Token 4
...

When generating a new token, the model still needs to attend to the old tokens.

Recomputing all the past Keys/Values every time would be very wasteful.

KV Cache:

Past Keys / Values
↓
CACHE
↓
reuse at next decode step

How much does it save?

Without a cache, every decode step has to recompute K and V for the entire context. Generating n tokens means total work proportional to n².

With a cache, each step only computes K/V for the new token, then appends it to what is stored:

No cache:     step t recomputes K,V for all t tokens   ->  total O(n²)
With cache:   step t computes K,V for just 1 token     ->  total O(n)

For sequences of a few thousand tokens, the difference is hundreds of times. Without a KV Cache, today’s LLM chatbots would simply be unusable.

The price: a formula worth memorizing

The KV Cache trades compute for memory, and that memory can be calculated exactly:

KV Cache size
  = 2  ×  n_layers  ×  n_kv_heads  ×  head_dim  ×  n_tokens  ×  bytes_per_value

    ^
    2 because both Key and Value are stored

Try it with Llama-3-8B in FP16, with an 8192-token context:

n_layers    = 32
n_kv_heads  = 8        (GQA - with MHA this number would be 32)
head_dim    = 128
n_tokens    = 8192
bytes       = 2        (FP16)

2 × 32 × 8 × 128 × 8192 × 2  =  1.07 GB

More than 1 GB of VRAM for a single user. Serving 50 users at once means over 50 GB for the cache alone - more than the memory reserved for the model weights.

And this is where section 5.10 pays off: if this model used MHA instead of GQA, n_kv_heads would be 32 instead of 8, and the number above would become 4.3 GB.

GQA is not an academic detail. It is the difference between serving 50 users or only 12 on the same GPU.

Why long context is expensive twice over

Now we can put the pieces together:

That is why a long conversation starts more slowly, generates more slowly, and uses more memory - three different kinds of cost stacked on top of each other.

Mitigation techniques in real serving systems:

• PagedAttention (vLLM) - manages the cache in pages like an operating system’s virtual memory, cutting fragmentation significantly;
• Prefix caching - conversations that share a system prompt also share that part of the cache;
• Quantizing the KV cache - storing the cache in INT8 instead of FP16, halving it;
• Window truncation - evicting old tokens from the cache when a conversation gets too long.

8.14 Batching: why serving many users is almost free

Back to the fact that decode bottlenecks on memory, not compute. The consequence is fairly counter-intuitive.

Once the GPU has loaded all of the model’s weights to generate one token, it can generate tokens for many requests at once at almost no extra cost - because the weights are already there.

Batch of 1 request:    read 16 GB of weights  ->  1 token
Batch of 32 requests:  read 16 GB of weights  ->  32 tokens

Same memory reads. 32x the throughput.

This is why API providers’ cost per token is so low: they pack many requests from many customers into the same batch.

Continuous batching pushes the idea further. Old-style batching has to wait for the whole batch to finish, so a short request gets stuck behind a long one. Continuous batching lets new requests jump in as soon as a slot frees up:

Static batching      [====][====][====]   waits for the slowest request
Continuous batching  [==][======][===]    fills free slots immediately

The limit of the approach: the bigger the batch, the more the KV Cache balloons, and at some point memory runs out before compute saturates. That is exactly when every number in section 8.13 becomes a real constraint.

8.15 Quantization: trading precision for speed

Model weights are usually trained in FP16 or BF16 - 2 bytes per number. Quantization stores them at lower precision.

Because decode bottlenecks on memory, halving the bytes of the weights nearly doubles decode speed. Quantization saves VRAM and runs faster at the same time - trade-offs are rarely this favorable.

This is what makes running models on a laptop feasible: an 8B model in INT4 fits in 4-5 GB.

A caveat: how much quality drops depends heavily on the method (GPTQ, AWQ, GGUF at various levels) and on the specific model. Always evaluate on your own task rather than trusting generic benchmarks.

8.16 Speculative decoding: guess ahead, then verify

An elegant technique, also born from the same compute/memory asymmetry.

The idea: use a small, fast draft model to guess a few tokens ahead, then let the large model verify them all in a single pass.

Draft model (1B) guesses:      "capital"  "of"  "France"  "was"
                                   |        |       |        |
Large model (70B) verifies:        v        v       v        v     in ONE pass
                                  OK       OK      OK      wrong
                                                             |
                                  keep 3 tokens, fix the 4th

Why it pays off: verifying 4 tokens at once costs about the same as generating 1 - again because the weights only need to be read once. If the draft model guesses right most of the time, we get 2-3 tokens for the price of one.

The crux: the output is distributionally identical to letting the large model generate on its own. This is a pure speed-up, not an approximation - only tokens confirmed by the large model are kept.

8.17 Try it yourself: from logits to token

Everything in sections 8.5 to 8.8 fits into about thirty lines of code. Running it once will make it stick far better than just reading.

Tested with: Python 3.12, transformers 5.18.0 and torch 2.14.1 (CPU build), October 2026. The script only needs the small gpt2 checkpoint, so it runs fine without a GPU.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

name = "gpt2"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name)

prompt = "The capital of France is"
ids = tok(prompt, return_tensors="pt").input_ids

with torch.no_grad():
    out = model(ids)

# Logits for the NEXT token: take the last position of the sequence
logits = out.logits[0, -1, :]          # shape: [vocab_size]
print("Vocabulary size:", logits.shape[0])

# --- Temperature: divide the logits (section 8.6) ---
T = 0.8
scaled = logits / T

# --- Softmax: turn into probabilities (section 8.5) ---
probs = torch.softmax(scaled, dim=-1)

# --- Look at the top 5 candidates ---
top = torch.topk(probs, 5)
for prob, idx in zip(top.values, top.indices):
    print("%-12s %.4f" % (tok.decode(idx), prob.item()))

# --- Top-p: keep the smallest group reaching the cumulative threshold (section 8.8) ---
sorted_probs, sorted_idx = torch.sort(probs, descending=True)
cumulative = torch.cumsum(sorted_probs, dim=-1)
keep = (cumulative - sorted_probs) < 0.9   # also keeps the token that crosses 0.9

# --- Sampling: draw one token ---
candidates = sorted_probs[keep]
chosen = torch.multinomial(candidates / candidates.sum(), 1)
next_id = sorted_idx[keep][chosen]

print("Chosen token:", repr(tok.decode(next_id)))

Two experiments worth doing right away:

1. Set T = 0.1, then T = 2.0, and watch how the probability table changes. You will see exactly the effect described in section 8.6.
2. Run the last part several times in a row. With a high T, each run gives a different token - which is exactly why LLMs don’t repeat the same answer word for word.

This is also the very loop every generate() function runs, except that it repeats until it hits EOS and adds a KV Cache.

09Bias and limitations

This is not an “optional ethics add-on”.

It is mandatory production knowledge.

Models learn from data.

If the data contains stereotypes, the model can internalize those patterns.

For example:

This man works as a [MASK]

and:

This woman works as a [MASK]

can produce different distributions of occupations.

This shows:

Internet / corpus bias
        ↓
Pretraining
        ↓
Pretrained model
        ↓
Fine-tuning
        ↓
Application
        ↓
User

Fine-tuning does not automatically remove bias inherited from the pretrained model.

9.1 Hallucination: why models make things up

Bias is not the only limitation, and hallucination is the one you meet every day. The notable thing is that it is not a bug - it is a direct consequence of how the model is trained.

Recall section 5.4: the model is trained to predict the most probable next token. It has never been trained to distinguish “I know this” from “I don’t know this”.

Prompt:  "The paper on X was written by ..."

Model:   This sentence structure is almost always followed
         by a name. So let's generate a name
         that sounds plausible.

The result is an answer that is fluent, grammatical, confident and wrong. There is no built-in mechanism telling the model to stop.

A few practical consequences:

• The model is most confident exactly where it is most often wrong - rare events, proper names, numbers, citations, dates.
• Telling the model “don’t make things up” has only limited effect. You are asking it to do something its architecture does not support.
• RAG works because it changes the nature of the problem - from “recall from parameters” to “read from context”. That is why RAG is the natural next topic after this post.

9.2 Other limitations worth knowing before going to production

9.3 Testing for bias concretely

Saying “we need to evaluate bias” is easy, but here are things you can do right away:

Counterfactual testing. Run the same prompt while swapping demographic attributes, then compare the outputs:

"Would you approve this loan for John Smith?"
"Would you approve this loan for Mary Smith?"

Everything else stays exactly the same.
Significantly different outputs = a bias signal.

Stability checks across variants. Rephrase the same question 5-10 ways and measure how much the results swing.

Sliced evaluation. Don’t just look at overall accuracy - break it down by user group, by topic, by language. A model at 90% overall can still be at only 60% on an important segment.

Regression test set. Keep a fixed set of examples that reflects your real scope of use, and rerun it every time you change the model, the prompt or the version.

9.4 Why this is an engineer’s job, not just the compliance team’s

A production system needs layers of protection the model itself does not provide:

Evaluation           measure quality on your own task, not generic benchmarks
Guardrails           filter input and output, limit the scope of actions
Citations            force answers to point to verifiable sources
Monitoring           track quality over time, not just latency
Human in the loop    manual review for high-risk decisions
Fallbacks            a plan for when the model refuses or answers wrongly

A mental model worth remembering: an LLM is a probabilistic component inside a deterministic system. All the engineering work lies in what surrounds it - constraints, verification, and handling it when it is wrong.

10The whole Transformer in one mental model

If the whole post had to fit into one diagram:

And for generative inference:

PROMPT
  ↓
TOKENIZER
  ↓
PREFILL
  ↓
ATTENTION
  ↓
LOGITS
  ↓
TEMPERATURE / TOP-P / TOP-K
  ↓
NEXT TOKEN
  ↓
KV CACHE
  ↓
NEXT TOKEN
  ↓
...

If those two diagrams now feel intuitive, you have a pretty solid Transformer foundation.

Conclusion

The Transformer is not just a neural network architecture to memorize.

It is the foundation that lets you understand almost every important concept in modern Generative AI:

• token;
• context;
• attention;
• inference;
• embeddings;
• generation;
• long context;
• KV Cache;
• RAG;
• Agents.

Once you understand the Transformer well, later topics such as RAG or Agents feel a lot less like a “black box”.

And that is the real goal of learning the Transformer: not memorizing formulas, but building a mental model good enough that, when you meet a new GenAI system, you can ask yourself:

How is the input represented? How does the model see the context? Is it understanding, transforming or generating? Where does inference cost come from? And at which layer does the real limitation sit?

If you can answer those questions, you have passed the most important foundational step into AI Engineering.

The next part of the series looks at the same model from the outside in: what actually happens when you call pipeline() in Hugging Face Transformers, from the tokenizer and the attention mask to dtype, the KV cache and serving a model to many users at once. As for the hallucination and knowledge-cutoff problems from sections 9.1 and 9.2, the usual fix is RAG, which has its own series on this blog (in Vietnamese for now).

References

Original papers

• Attention Is All You Need (2017) - the paper that gave birth to the Transformer. Shorter than you think, and worth reading at least once.
• BERT (2018) - encoder-only and masked language modeling.
• Language Models are Few-Shot Learners (2020) - the GPT-3 paper, where in-context learning is laid out clearly.
• Training language models to follow instructions (2022) - InstructGPT and RLHF.

Going deeper into each mechanism

• RoFormer - the paper that introduced RoPE.
• GQA: Training Generalized Multi-Query Transformer Models - the origin of Grouped-Query Attention.
• FlashAttention - fast, memory-efficient attention without approximation.
• Direct Preference Optimization - DPO, a simpler replacement for RLHF.
• Efficient Memory Management for LLM Serving - PagedAttention, the foundation of vLLM.
• The Curious Case of Neural Text Degeneration (2019) - the paper that introduced top-p (nucleus) sampling.

Learning resources

• The Illustrated Transformer - Jay Alammar’s classic visual explanation.
• Hugging Face LLM Course - free, hands-on, with runnable code.
• Let’s build GPT - Andrej Karpathy writes a Transformer from scratch. If you can pick only one resource, pick this one.
• Tokenizer Playground - paste in some Vietnamese (or any non-English) text alongside its English translation to see the token tax from section 2.3 for yourself.