Using Transformers: From pipeline() to Optimized LLM Deployment

In the previous post, we walked through the inside of a Transformer: tokens, attention, residuals, the FFN, RoPE, the three architecture families, and the text generation loop.

This post goes the other way: from the outside in. You open an editor, type three lines of code, and you immediately have a working model:

from transformers import pipeline

classifier = pipeline("sentiment-analysis")
classifier("I really like this course!")

Those three lines are very convenient. But they also hide almost everything that decides whether your system is fast or slow, right or wrong, and whether it needs 4 GB or 32 GB of VRAM.

A model does not read strings. It takes tensors of numbers. A tokenizer does not just “split words” - it determines the vocabulary, the token IDs, the special tokens, the padding and the attention mask. And the output of a Transformer is usually not the answer yet: it is a hidden representation or logits, which still need a task head or a decoding step after it.

The single most important mental model in this post:

Raw text
   ↓
Tokenizer
   ↓
Token IDs + attention mask + metadata
   ↓
Embedding
   ↓
Transformer layers
   ↓
Hidden representations
   ↓
Task head / LM head
   ↓
Logits
   ↓
Post-processing or decoding
   ↓
Prediction / generated text

The post starts at the beginning of that chain - how an English sentence becomes a tensor - and goes all the way to the end: how to serve a model to many users at once with vLLM, SGLang or llama.cpp, and what is still worth learning from TGI.

The questions this post answers:

• Why must the tokenizer and the model come from the same checkpoint, and what happens if they don’t?
• How does AutoModel differ from AutoModelForSequenceClassification, and why does the library ship a dozen AutoModelFor... classes?
• What are logits, and why is Softmax not just a cosmetic step?
• How do padding and the attention mask solve two different problems?
• Why does a 7B model with “only” 7 billion parameters need 28 GB of VRAM in FP32?
• What does the KV cache save you, and what does it cost you in return?
• What do PagedAttention, continuous batching and FlashAttention actually optimize?
• When should you pick vLLM or SGLang, what is TGI still worth learning for now that it is archived, and when is llama.cpp the right answer?

Tested with: Python 3.12, transformers 5.18.0 and torch 2.14.1 (CPU build), October 2026. The transformers API changes noticeably between major versions; for example, the torch_dtype argument is now called dtype. If a snippet does not run, check your version first.

01A model is more than one file

When we say “I downloaded the model”, we usually picture a single file. In reality, using a pretrained Transformer requires at least three things that travel together:

The first two get mentioned all the time. The third is the one most often overlooked, and it is the source of a particularly nasty class of bugs.

1.1 Why the tokenizer and the model must match

A token ID is not a number with any mathematical meaning. It is a row index into the embedding table.

Suppose tokenizer A assigns:

"bank" → 2924

while tokenizer B, with a different vocabulary, assigns:

2924 → "##ology"

If you tokenize with A and feed the IDs into model B, the model will fetch exactly row 2924 of its embedding table - but that is the vector for ##ology, not for bank. The model raises no error. It just returns a wrong result.

This is one of the worst kinds of bug: a silent failure. The shapes are right, the dtypes are right, the code runs smoothly, and the semantics are broken from the very first step.

That is why the standard pattern is to use the same checkpoint for both:

from transformers import AutoTokenizer, AutoModel

checkpoint = "bert-base-cased"

tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModel.from_pretrained(checkpoint)

1.2 Auto is a factory, not an architecture

AutoModel is not an architecture. It is a factory: it reads the checkpoint’s config file, looks up the model_type field, and instantiates the matching class.

config.json
   ↓
model_type: "bert"   →   BertModel
model_type: "llama"  →   LlamaModel
model_type: "gpt2"   →   GPT2Model

Thanks to that, hundreds of different architectures share one API:

AutoTokenizer
AutoModel
AutoModelForSequenceClassification
AutoModelForTokenClassification
AutoModelForQuestionAnswering
AutoModelForMaskedLM
AutoModelForCausalLM

This is about more than convenience. It means your code can switch from BERT to RoBERTa to DeBERTa with almost no changes - as long as the task head stays the same.

02Inside pipeline()

pipeline() is three stages wrapped into one call:

"This movie is amazing!"
          ↓
       Tokenizer          ← preprocessing
          ↓
input_ids + attention_mask
          ↓
        Model             ← inference
          ↓
        Logits
          ↓
       Softmax            ← post-processing
          ↓
   POSITIVE: 0.982

2.1 Why text has to become numbers

Inside a Transformer there are only four groups of operations: matrix multiplication, addition, normalization and activation functions - attention itself is a combination of them. None of them takes a string of characters as input.

So the first step is always mapping text to numbers:

"I love transformers"
          ↓
["I", "love", "transform", "##ers"]
          ↓
[1045, 2293, 10938, 2545]

It bears repeating: these IDs carry no arithmetic meaning. Token 2000 is not “twice as big” as token 1000. They are just indices.

2.2 Embedding: from index to vector

If the vocabulary has 30,522 tokens and the hidden size is 768, the embedding layer is a matrix:

30,522 × 768

Each token ID selects exactly one row - a 768-dimensional vector.

So the shape changes like this:

input_ids:   [batch_size, sequence_length]
                    ↓
embeddings:  [batch_size, sequence_length, hidden_size]

Concretely, for a batch of 4 sentences, 128 tokens each, hidden size 768:

input_ids:   [4, 128]
embeddings:  [4, 128, 768]

This is worth pausing on. The embedding table of a model with a 32,000-token vocabulary and a hidden size of 4,096 is already:

32,000 × 4,096 ≈ 131 million parameters

just for lookup. That is why vocabulary size is an architectural decision, not a minor detail.

2.3 Static embeddings versus contextual representations

After the embedding layer, the vectors pass through many Transformer layers. This is where self-attention makes each token’s representation depend on its context.

I deposited money at the bank.
The fisherman sat on the river bank.

Both sentences start from the same embedding vector for bank. But after a few attention layers, the two hidden states are completely different: one sits near the “finance” region, the other near the “geography” region.

That is exactly the difference between a static embedding - Word2Vec, GloVe: one fixed vector per word - and a contextual representation, where the vector depends on the whole sentence.

2.4 Hidden states are not the answer yet

The output of a base Transformer usually has the shape:

[batch_size, sequence_length, hidden_size]

and is called the hidden states. A plain AutoModel stops here - it produces a representation, not a prediction.

To solve a specific task, you need to attach a head:

Transformer body
       ↓
Hidden representation
       ↓
Classification head
       ↓
Class logits

2.5 Logits and Softmax

AutoModelForSequenceClassification puts a classification head on top of the body. For a batch of 2 sentences and a 2-label task, the logits have the shape:

[2, 2]

The values inside might be:

[-1.7, 2.3]

These are not probabilities yet. They can be negative, they can be greater than 1, and they don’t sum to 1. To read them as a distribution, we apply Softmax:

import torch

probabilities = torch.softmax(outputs.logits, dim=-1)

The formula:

              exp(z_i)
p_i = -------------------------
           Σ_j exp(z_j)

The result:

[-1.7, 2.3]  →  [0.018, 0.982]

If index 0 is NEGATIVE and index 1 is POSITIVE, the model concludes POSITIVE with probability 0.982.

An easy detail to miss: when training with CrossEntropyLoss in PyTorch, you pass in logits, not probabilities - the loss function already applies log-softmax internally. A manual Softmax is only for interpreting results.

Seen this way, pipeline() is not a black box. It is just three familiar steps packaged together.

03The model: config, weights and head

A pretrained model is the sum of a configuration and weights.

3.1 The configuration is the blueprint

The configuration holds everything that determines the shape of the network:

model_type
hidden_size
num_hidden_layers
num_attention_heads
vocab_size
intermediate_size
max_position_embeddings
num_labels

A BERT-base model has 12 layers, a hidden size of 768 and 12 attention heads. From the config alone you can rebuild exactly the right structure - but as a network with random weights that knows nothing yet.

3.2 The weights are the knowledge

The weights are what gets optimized during training: the embedding table, the Q/K/V projection matrices, the output projection matrix, the feed-forward weights, the normalization parameters, and the task head.

This distinction matters a lot:

Architecture ≠ learned knowledge

Two models with the same architecture but pretrained on different corpora behave very differently. Conversely, a model can keep its body unchanged and only swap the head to move to a new task - that is exactly the mechanism behind fine-tuning.

Saving and loading:

model.save_pretrained("./my_model")

The resulting directory contains the config and the weights, usually in the safetensors format - safer than pickle because it does not allow arbitrary code execution on load.

from transformers import AutoModel

model = AutoModel.from_pretrained("./my_model")

3.3 The head determines the output

This mapping table is worth memorizing:

For causal language modeling, the objective is:

P(x_t | x_1, x_2, ..., x_(t-1))

This is the objective of almost every modern generative LLM.

Notice the shape of the last two rows: vocab_size is in there. With a 32,000-token vocabulary and a 4,096-token context, the logits tensor alone in FP16 is:

4,096 × 32,000 × 2 bytes ≈ 262 MB

for one sequence. That is why inference engines only compute logits for the last position during decoding, instead of for the whole sequence.

3.4 Not every model takes the same input

Besides input_ids, some BERT-like architectures need token_type_ids to tell apart the two segments of a sentence pair:

[CLS] Sentence A [SEP] Sentence B [SEP]
  0      0 0 0    0      1 1 1     1

Decoder-only models usually don’t have this field. You don’t need to remember which model needs what - the tokenizer that ships with the checkpoint already knows, and it returns exactly the set of inputs the model expects. One more reason not to build inputs by hand.

04The tokenizer: the bridge between text and numbers

The tokenizer’s job is to pick the unit of representation: not so large that the vocabulary balloons, not so small that sequences drag on forever.

4.1 Word-based: simple but it falls apart

"Transformers are powerful"
          ↓
["Transformers", "are", "powerful"]

Easy to understand. But the vocabulary explodes immediately: run, runs, running, runner, runners become five separate entries, even though they share nearly all of their meaning. Add proper nouns, rare words, typos and hashtags, and the vocabulary can exceed millions.

Worse: words outside the vocabulary have to map to [UNK]. The information disappears entirely - the model sees [UNK], not antidisestablishmentarianism.

4.2 Character-level: the other end of the spectrum

"hello" → ["h", "e", "l", "l", "o"]

A tiny vocabulary, and almost no [UNK]. But sequences become four to five times longer.

And this is not just about RAM. Classic self-attention has roughly O(n²) complexity in sequence length:

20 tokens   → 20 × 20 = 400 attention cells
100 tokens  → 100 × 100 = 10,000 cells

A sequence 5 times longer makes the attention matrix 25 times larger.

4.3 Subword: the balance point

The idea: common words stay whole, rare words are split into meaningful pieces.

unbelievable
→ un + believ + able

That way believable, unbelievable and believer share the core believ, and the model learns how they relate.

Three main algorithm families:

WordPiece produces output like:

["Transform", "##ers"]

where ## signals “this token continues the previous one, with no space in between”.

4.4 Three concepts that are easy to confuse

String        "Transformers are great"
  ↓
Token         ["Transform", "##ers", "are", "great"]
  ↓
Integer ID    [13809, 2545, 2024, 2307]
  ↓
Embedding     4 real-valued vectors, each d-dimensional

A token is a piece of text. An ID is an index. An embedding is a vector. Three different things, at three different levels.

4.5 Encoding and decoding

Encoding is the forward direction: text to token IDs. Decoding is the way back: token IDs to text.

For text-generating models, decoding is not a side detail. The model only produces a sequence of numbers; it is the tokenizer that stitches them back into text, handles whitespace, joins ##ers onto Transform, and drops special tokens when needed.

tokenizer.decode(output_ids, skip_special_tokens=True)

4.6 Special tokens

Depending on the architecture, you will run into:

The rule: don’t guess which tokens the model needs. The tokenizer that ships with the checkpoint knows exactly the format the model was trained on, and adds the right tokens by default.

Chat models add one more layer - the chat template - which decides which tokens wrap the system, user and assistant roles. Feeding a raw prompt into a chat model while skipping the template is a very effective way to make output quality drop without understanding why.

05Batching, padding and the attention mask

Processing one sentence at a time is the fastest way to waste a GPU. A GPU’s strength is doing the same operation on lots of data in parallel.

Sequence A ┐
Sequence B ├──→ Batch → GPU
Sequence C ┘

The problem: tensors have to be rectangular, while sentences come in different lengths.

5.1 Padding fixes the shape

Suppose two sequences are 5 and 3 tokens long. Add [PAD] to the shorter one:

A: [12, 45, 89, 32, 18]
B: [55, 29, 19,  0,  0]

With pad_token_id = 0, the tensor now has shape [2, 5]. Valid.

5.2 But padding breaks the meaning

If the model treats [PAD] like a normal token, self-attention will still compute scores against the padding positions and mix them into the representation. The result: the same sentence, placed in two different batches, gives two different results.

attention_mask solves this:

input_ids:      [55, 29, 19, 0, 0]
attention_mask: [ 1,  1,  1, 0, 0]

Here 1 means a real token and 0 means a position to ignore.

The mechanism inside is surprisingly simple: before Softmax, the attention scores at masked positions get a very large negative number added, usually -inf or -1e9:

score       = [2.1, 0.8, 1.5, -1e9, -1e9]
             ↓ softmax
attention   = [0.55, 0.15, 0.30, 0.00, 0.00]

exp(-1e9) is approximately 0, so the attention weight on padding all but vanishes.

In short:

Padding gives the tensor the right shape. The attention mask gives the computation the right meaning.

That is also why you should almost never build input_ids by hand and forget the attention_mask.

5.3 The batch dimension

A single sequence:

[101, 2023, 2003, 102]

has shape [4]. But the model expects [batch_size, sequence_length]. So even when you only have one sentence, it has to be wrapped in an extra dimension:

[[101, 2023, 2003, 102]]     shape [1, 4]

Forgetting this step is the source of a great many shape mismatch errors when you are starting out.

5.4 Truncation and sequences that are too long

Every architecture has its own context limit. When you exceed it, the simplest fix is to cut:

tokenizer(
    text,
    truncation=True,
    max_length=512,
)

But hard-cutting the tail is a decision with a cost. If your document puts its conclusion at the end, you have just thrown away the most important part.

For long documents, more practical strategies are:

• Chunking: split into several pieces, process them separately, merge the results.
• Sliding window: overlapping pieces so you don’t lose context at the boundaries.
• Retrieval: only load the parts relevant to the question - this is exactly the idea behind RAG.
• Long-context model: pick an architecture with a larger context window.
• Hierarchical processing: summarize each piece, then summarize the summaries.

5.5 Left padding or right padding

right padding:
[token, token, token, PAD, PAD]

left padding:
[PAD, PAD, token, token, token]

For an encoder model used for classification, right padding is a sensible default.

For a decoder-only model generating text in batches, left padding is usually the right choice. The reason: the model generates each new token from the last position of the sequence. If padding sits on the right, the “last position” of the shorter sentences lands on a [PAD] cell, and generation starts from the wrong place. Pushing padding to the left makes every sequence in the batch end aligned on the right.

tokenizer.padding_side = "left"

06Putting it together in one script

Once you understand each piece, the real-world workflow turns out to be very short:

import torch
from transformers import (
    AutoTokenizer,
    AutoModelForSequenceClassification,
)

checkpoint = "distilbert-base-uncased-finetuned-sst-2-english"

tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForSequenceClassification.from_pretrained(checkpoint)
model.eval()

texts = [
    "I really enjoyed this movie.",
    "This was extremely disappointing.",
]

inputs = tokenizer(
    texts,
    padding=True,
    truncation=True,
    return_tensors="pt",
)

with torch.no_grad():
    outputs = model(**inputs)

probabilities = torch.softmax(outputs.logits, dim=-1)
print(probabilities)
print(model.config.id2label)

6.1 One call, a lot of work

The tokenizer call above runs the entire chain below:

1. Normalize the text: lowercasing, stripping accents, Unicode normalization - depending on the tokenizer.
2. Pre-tokenization: splitting on whitespace and punctuation.
3. Subword tokenization.
4. Mapping tokens to IDs.
5. Adding special tokens in the model’s exact format.
6. Truncation if the input exceeds max_length.
7. Padding so all sequences have equal length.
8. Generating the attention_mask, plus token_type_ids if the model needs them.
9. Converting everything into PyTorch tensors.

That is why you should use the high-level API instead of doing each step yourself - unless you are debugging the tokenizer itself.

6.2 The model(**inputs) syntax

inputs is a dictionary:

{
    "input_ids": ...,
    "attention_mask": ...
}

so model(**inputs) is equivalent to:

model(
    input_ids=inputs["input_ids"],
    attention_mask=inputs["attention_mask"],
)

This unpacking syntax is everywhere in the Transformers ecosystem, and it has a practical benefit: if the tokenizer also returns token_type_ids, your code doesn’t need to change at all.

6.3 no_grad and eval are two different things

These two are often lumped together:

• model.eval() switches the module into inference mode: it disables dropout and uses running statistics for batch norm. It affects the results.
• torch.no_grad() tells PyTorch not to build a computation graph. It affects memory and speed, not the results.

You should have both during inference. For inference-only workloads, torch.inference_mode() is a stricter version of no_grad() and is usually slightly faster:

with torch.inference_mode():
    outputs = model(**inputs)

6.4 Don’t forget id2label

The model returns an index, not a label name. The mapping lives in the config:

model.config.id2label
# {0: 'NEGATIVE', 1: 'POSITIVE'}

Hardcoding “0 is NEGATIVE” is an assumption that only holds for that exact checkpoint. Switch models and the order may flip.

07Device, dtype and the memory problem

7.1 The model and the inputs must be on the same device

device = "cuda" if torch.cuda.is_available() else "cpu"

model = model.to(device)

inputs = {
    key: value.to(device)
    for key, value in inputs.items()
}

If you forget, PyTorch throws an error along the lines of Expected all tensors to be on the same device. This is the easiest error to fix in this whole post.

7.2 Dtype decides which models you can run

For a 7-billion-parameter model, counting the weights alone:

FP32:   7B × 4 bytes    ≈ 28 GB
FP16:   7B × 2 bytes    ≈ 14 GB
INT8:   7B × 1 byte     ≈  7 GB
4-bit:  7B × 0.5 byte   ≈  3.5 GB

The real number is always higher, because there is also:

• The KV cache, which grows with context length and batch size - see section 8.
• Activations during the forward pass.
• Runtime buffers and memory fragmentation from the allocator.
• Some tensors, usually layer norms and embeddings, that are not quantized.

But this rough calculation is enough to answer the most practical question: “Will this model run on my card?”

A useful rule of thumb: take the parameter count in billions, multiply by 2 for FP16, then add 20-30% for everything else. A 7B model in FP16 needs around 16-18 GB to run comfortably with a moderate context.

model = AutoModelForCausalLM.from_pretrained(
    checkpoint,
    dtype=torch.float16,
    device_map="auto",
)

device_map="auto" lets the model spread across several GPUs, or push some layers down to the CPU when VRAM runs short.

08From notebook to production

A model that runs in a notebook is not necessarily ready for production. When serving many users at once, the system has to handle requests with prompts of different lengths, different output lengths, different sampling parameters, all arriving at different times.

The problem is no longer “how do I run the model” but “how do I keep the GPU continuously busy while latency, throughput and memory stay acceptable”.

8.1 Two metrics, and they often conflict

Increasing the batch size usually raises throughput and raises the latency of each request along with it. No configuration optimizes both; you pick the balance point for your product.

8.2 Why generation is harder to optimize than classification

Classification is a single forward pass. Generation is autoregressive: every new token depends on all the tokens before it.

Prompt
  ↓
Predict token 1
  ↓
Prompt + token 1
  ↓
Predict token 2
  ↓
Prompt + token 1 + token 2
  ↓
...

Generating 200 tokens means 200 sequential forward passes. You cannot parallelize along the time axis.

8.3 KV cache: trading VRAM for compute

In self-attention:

Q = XW_Q
K = XW_K
V = XW_V

The key point: when generating a new token, the K and V of the earlier tokens do not change. Recomputing them at every step is pure waste.

So we store K_previous and V_previous, and at the next step we only need to compute Q_new, K_new, V_new for exactly one token.

Without a cache, step t recomputes K and V for the whole prefix and a full t × t attention matrix, so generating n tokens adds up to roughly O(n³). With a cache, step t only computes one new row of attention, which is linear in the current length, for about O(n²) in total.

The price is memory. The size of the KV cache:

2 × num_layers × num_kv_heads × head_dim × seq_len × batch_size × bytes_per_value

The 2 is for K and V. Take a Llama-style 7B model with 32 layers, 32 KV heads, head_dim 128, in FP16, with a 4,096-token context and batch 1:

2 × 32 × 32 × 128 × 4096 × 1 × 2 bytes ≈ 2.1 GB

At batch 16, that number becomes roughly 34 GB - more than the model weights themselves. That is exactly why modern architectures moved to GQA, covered in the previous post: cutting num_kv_heads from 32 to 8 shrinks the KV cache by 4x.

8.4 PagedAttention: managing the cache like virtual memory

The naive allocation strategy reserves a contiguous region large enough for each sequence’s max_length. The problem: most requests never use all of it, and the excess stays locked up for nothing. This is internal fragmentation, and in practice it can waste more than half of the VRAM set aside for the cache.

vLLM solves this with PagedAttention: it splits the KV cache into small fixed-size blocks, like memory pages in virtual memory. The blocks of one sequence don’t need to sit next to each other:

GPU memory:

[A1] [B1] [A2] [C1] [B2] [A3]

A mapping table knows which block belongs to which sequence. The consequences:

• Fragmentation all but disappears - at most one final block is wasted.
• More sequences can be served concurrently on the same hardware.
• Requests that share a prefix, such as a system prompt or few-shot examples, can share blocks instead of duplicating them.

8.5 Continuous batching

Traditional batching, or “static batching”, gathers a group of requests, runs until all of them are done, and only then takes a new group. For generation this is a disaster: if request A finishes after 20 tokens while request B needs 200, A’s slot sits idle for 180 steps.

Continuous batching lets a new request slot in as soon as a sequence finishes:

Step 1: A B C
Step 2: A B C
Step 3: D B C     ← A done, D joins immediately
Step 4: D B E     ← C done, E joins immediately

In practice this is one of the most profitable throughput optimizations - often a multiple-fold gain over static batching, without changing anything inside the model.

8.6 FlashAttention: optimizing at the kernel level

Standard attention:

Attention(Q, K, V) = softmax(QKᵀ / √d)V

The problem is not the number of operations but the path the data takes. A GPU has several tiers of memory with very different speeds:

SRAM / shared memory   → very fast, very small
HBM / VRAM             → large, many times slower

A naive implementation materializes the full QKᵀ matrix of size seq_len × seq_len in HBM, reads it back to compute the softmax, writes it out, then reads it again to multiply by V. With an 8,192-token context, that matrix has 67 million elements - for every head, in every layer.

FlashAttention organizes the computation in blocks: it loads a block of Q, K and V into SRAM, computes the softmax piece by piece with an online algorithm, accumulates the result, and only then writes it out once. The full attention matrix never exists in HBM.

The important point: FlashAttention does not change the mathematical definition of attention. The result is the same, with negligible floating-point differences. It only changes how the computation is executed to fit the GPU’s memory hierarchy.

8.7 TGI: the inference server in the Hugging Face ecosystem

Update (October 2026): Hugging Face has put TGI into maintenance mode, and the text-generation-inference repository is now archived (read-only). For new deployments, Hugging Face itself recommends vLLM or SGLang, with llama.cpp or MLX for local inference. This section stays because TGI is still a clean picture of what an inference server does, and plenty of existing systems still run it. For a new project, pick vLLM or SGLang.

Text Generation Inference sits between the client and the GPU:

Client
  ↓
HTTP API
  ↓
Inference Server
  ├── queue requests
  ├── batch requests
  ├── manage KV cache
  ├── run generation
  └── stream tokens
  ↓
GPU

The capabilities that usually matter at this layer: continuous batching, token streaming, optimized attention kernels, multi-GPU via tensor parallelism, metrics for observability, request scheduling, and parameters for controlling generation.

8.8 llama.cpp: a completely different direction

llama.cpp does not compete on datacenter GPU turf. Its strengths are local inference, CPU inference, commodity hardware, portability - it runs on Mac, Windows, Linux, Android, even a Raspberry Pi - and quantized models.

It is the right choice when you want to run an LLM on a laptop, on a customer’s machine, or in an environment without internet access.

Quantization is what makes that feasible. Instead of storing weights in FP16, the model is represented in 8, 6, 5, 4 bits or fewer. For an 8B model:

FP32:  8B × 4 bytes    ≈ 32 GB
FP16:  8B × 2 bytes    ≈ 16 GB
8-bit: 8B × 1 byte     ≈  8 GB
4-bit: 8B × 0.5 byte   ≈  4 GB

At its core, quantization maps a range of float values onto a smaller set of discrete levels, together with a scale and sometimes a zero-point to approximate the original values at compute time. The price of the memory savings is some lost precision - it is always a trade-off between memory, speed and quality.

In the llama.cpp ecosystem, GGUF is the format that packs tensors, architecture metadata, tokenizer metadata and quantization information into a single file. Names like Q4_K_M, Q5_K_M and Q8_0 describe different schemes:

• Q4 / Q5 / Q8 - the average number of bits per weight.
• _K - uses k-quants, splitting into blocks and storing scales per group to reduce error.
• _M / _S / _L - medium, small and large variants within the same family.

Don’t take this to mean that fewer bits is always better. The choice depends on your RAM/VRAM, the model size, the speed you want, and how much quality loss you can accept. A common rule of thumb: a larger model at a lower quantization level usually beats a smaller model at a higher quantization level, as long as both fit in memory.

When a model doesn’t fully fit in VRAM, llama.cpp supports offloading: put the first n layers on the GPU and the rest on the CPU. The more layers on the GPU, the faster it runs, but the more VRAM it needs. This is a very practical knob when your hardware sits between two tiers.

8.9 OpenAI-compatible APIs

One trend has become a de facto standard: every inference server offers an OpenAI-compatible endpoint.

Application
    ↓
Generic chat/completions client
    ↓
Inference endpoint
    ├── hosted service
    ├── TGI
    ├── vLLM
    └── local server

The benefit is very concrete: you can prototype with a paid API, switch to self-hosted vLLM when you go to production, and fall back to llama.cpp for offline environments - with the application layer barely changing a single line.

8.10 Quick picker

09Controlling generation

Memory management is only half the story. The other half: how to pick the next token.

9.1 Greedy decoding

Always pick the token with the highest probability:

next_token = torch.argmax(logits, dim=-1)

The output is deterministic - the same input always gives the same output. Great for tasks that need stability, such as data extraction, classification and structured code generation. But the text easily becomes stiff and repetitive.

9.2 Temperature

Temperature reshapes how “sharp” the distribution is before sampling:

softmax(logits / T)

For example, with logits [3.0, 1.0, 0.5]:

T = 0.5  →  [0.98, 0.02, 0.01]
T = 1.0  →  [0.82, 0.11, 0.07]
T = 2.0  →  [0.60, 0.22, 0.17]

9.3 Top-k and top-p

Top-k keeps the k most probable tokens and only then samples. A 50,000-token vocabulary with top_k = 50 means 49,950 candidates are eliminated before the draw.

The drawback: a fixed k doesn’t suit every context. Sometimes the model is very confident and one candidate is enough; sometimes it is very unsure and even 50 candidates is too few.

Top-p, or nucleus sampling, is more flexible: it keeps the smallest set of tokens whose cumulative probability reaches the threshold p.

A 0.35   ← cumulative 0.35
B 0.25   ← cumulative 0.60
C 0.15   ← cumulative 0.75
D 0.10   ← cumulative 0.85  ✓ past 0.8, stop
E 0.05

With top_p = 0.8, the kept set is A + B + C + D. When the model is confident, the set shrinks on its own; when the model is unsure, it widens on its own.

In practice, top-p and temperature are usually used together, with top-k set on top as a safety limit.

9.4 Types of penalty

An LLM can fall into a loop - repeating a phrase, or even repeating the same sentence forever. Three common mechanisms:

Presence penalty pushes the model toward new topics. Frequency penalty fights word repetition. They solve two slightly different problems, and setting them too high makes the text sound unnatural.

9.5 Stopping conditions

Generation has to know when to stop. Three mechanisms:

1. The model generates an EOS token.
2. It reaches max_new_tokens.
3. It hits a stop sequence defined by the application:

stop = ["###", "</answer>", "\nUser:"]

Stop sequences matter especially when you build your own prompt format - without them, the model will happily go on to generate the user’s turn of the conversation too.

9.6 Context length and output length are not independent

This is a very commonly misunderstood point. If the model has an 8,192-token context window and the prompt already takes up 7,000 tokens, the room left for output is only:

8,192 - 7,000 = 1,192 tokens

Setting max_new_tokens = 4000 in that situation won’t give you 4,000 tokens. It will get cut off, or raise an error, depending on the engine.

In other words: the context window is a shared budget for both input and output, not a limit on the input alone.

10Three layers for debugging an LLM system

This whole post can be packed into three layers. It is not just a way of organizing knowledge - it is a diagnostic procedure.

When something goes wrong, the first question is not “which model should I switch to now”, but “which layer does this symptom belong to?”. Answer that, and the search space shrinks immediately from the whole system to a few specific places.

The table below is the quick-reference version of the “classify” step in the diagram:

This split works because it prevents a very common mistake: switching models when the problem is in the tokenizer, or tweaking the prompt when the real problem is running out of VRAM.

Conclusion

At first we tend to see a Transformer as one model. But a real LLM system is a combination of eight components, each of which can be a point of failure and each of which has its own knobs.

What this diagram adds over a plain list: the eight components do not sit at the same level of decision-making power. The three in the middle - architecture, weights, task head - are bound to the checkpoint; you take the whole package or none of it, and you cannot graft one model’s weights onto another model’s tokenizer. The three on the right - generation, memory, inference engine - are what you can actually configure once the model is fixed, and they are also where most of the optimization work happens.

The three cards under the diagram are the three most common debugging paths, matching the three most common symptoms: wrong output, slow inference, and a model that doesn’t fit on the GPU.

At that point pipeline() is no longer a black box. It is just a convenient abstraction on top of a processing chain you now understand deeply enough to open up, change and optimize when you need to.

Section 9.6 of this post touched on a constraint every serious LLM system has to live with: the context window is a finite budget, shared by input and output. Part 4 of the series, on tokenization, comes back to that budget from the other side: why the same content can cost a very different number of tokens depending on the language and the tokenizer.

But first, the next post goes in the opposite direction. Both this post and the first one are about a model that already knows something and how to use it well. Part three is about changing what it knows: fine-tuning a pretrained model - from the data pipeline and data leakage, through padding and the true cost of a tensor, to cross-entropy, gradients, optimizers and how to read a learning curve.

References

Hugging Face documentation

• LLM Course, chapter 2: Using 🤗 Transformers - the official walkthrough of pipeline(), the tokenizer and the model that this post goes deeper into.
• Pipelines - the pipeline() API and its default task heads.
• Auto Classes - AutoTokenizer, AutoModel and how the factory picks an architecture from the config.
• Summary of the tokenizers - word-level, character-level and subword tokenization (BPE, WordPiece, Unigram).
• Padding and truncation - the padding and truncation options and how they combine.
• Generation strategies - greedy decoding, sampling, temperature, top-k, top-p and penalties.
• Caching - how the KV cache works inside transformers.
• Big Model Inference - device_map="auto" and offloading layers to CPU or disk.
• Quantization - an overview of the quantization methods transformers supports.

Papers

• Attention Is All You Need (2017) - the original Transformer architecture.
• Efficient Memory Management for Large Language Model Serving with PagedAttention (2023) - PagedAttention and vLLM.
• Orca: A Distributed Serving System for Transformer-Based Generative Models (OSDI 2022) - iteration-level scheduling, the idea behind continuous batching.
• FlashAttention (2022) and FlashAttention-2 (2023) - IO-aware exact attention.
• GQA: Training Generalized Multi-Query Transformer Models (2023) - Grouped-Query Attention and its effect on KV cache size.
• SGLang: Efficient Execution of Structured Language Model Programs (2023) - the paper behind SGLang.
• Hierarchical Neural Story Generation (2018) - top-k sampling.
• The Curious Case of Neural Text Degeneration (2019) - top-p (nucleus) sampling.
• CTRL: A Conditional Transformer Language Model (2019) - the repetition penalty.

Inference engines

• vLLM - high-throughput serving with PagedAttention and an OpenAI-compatible API.
• SGLang - a fast serving framework that Hugging Face recommends alongside vLLM.
• Text Generation Inference - Hugging Face’s inference server, now archived and in maintenance mode.
• llama.cpp - local inference on CPU and consumer GPUs.
• GGUF specification - the model file format llama.cpp uses.