Efficient Fine-Tuning: LoRA, QLoRA, and Quantization

Part 11 ended in an uncomfortable place: every post-training step fine-tunes all the weights, and PPO even keeps four copies of the model in memory at once. This post answers the next question: how do you fine-tune and run a large model on a single GPU?

There are two ideas, and they act on two different things:

• LoRA freezes the original weights and learns only a low-rank update. The base needs no gradients, so the gradients, master weights and optimizer state almost disappear.
• Quantization stores the original weights themselves in fewer bits, from 2 bytes down to about half a byte per parameter.

QLoRA combines the two: a frozen 4-bit base and a LoRA adapter trained in 16 bits. The last part of the post is about the cost: an almost unchanged perplexity does not mean the model lost nothing, and Vietnamese usually loses more. The post follows the order of Figure 1 at the top of the page.

01Memory is the bottleneck

The previous posts already have all the pieces: part 3, section 1.2 stopped at full fine-tuning; part 2, section 7 computes memory for the weights only (7B in FP16 is about 14 GB, in 4-bit about 3.5 GB); part 10, section 8 counts 16 bytes per parameter when training with AdamW and mixed precision, so an 8B model cannot be trained on one GPU for lack of memory; part 11, section 6 shows that PPO is even worse.

The question for this post is: of those 16 bytes, which part do we actually need? Here is the breakdown by component for Llama 3.1 8B (8.03B parameters, counted in part 8, section 6):

The LoRA and QLoRA figures assume a rank r = 16 adapter on every linear layer in the blocks, i.e. 41,943,040 parameters (section 3 does the count). The QLoRA column stores the linear-layer weights at about 4.127 bits per parameter (section 7 explains that number), while the embedding and lm_head (1.05B parameters) stay in 16 bits, which is what bitsandbytes does by default.

Reading the table column by column:

• LoRA removes the three middle rows: gradients, master weights and Adam state of the base, 12 of the 16 bytes, because the base is not updated.
• Quantization cuts the first row: 2 bytes of weights down to about 0.5 bytes.
• QLoRA cuts both, from 128 GB to about 6 GB, plus activations.

The two knobs are independent: LoRA without quantization (sections 3–4), quantization for deployment (sections 5–6), or both (section 7).

02Parameter-efficient fine-tuning: the big picture

LoRA belongs to a family of methods called parameter-efficient fine-tuning (PEFT): keep almost the whole model frozen and train only a small number of new parameters. There were three main directions before LoRA.

Adapters (Houlsby et al. 2019) insert two small bottleneck adapter modules into each block (project down to a few dozen dimensions, then back up), one after attention and one after the FFN, and train only those modules. On GLUE, adapters come within 0.4% of full fine-tuning while adding only 3.6% parameters per task (abstract). The drawback is that the inserted modules sit in sequence in the computation: on GPT-2 medium at batch size 1, adapters make one forward pass 20.7–30.3% slower (LoRA paper, section 3, Table 1).

Prefix tuning (Li & Liang 2021) and prompt tuning (Lester et al. 2021) leave the weights alone and add trainable “virtual token” vectors at the start of the sequence, at every layer or only at the embedding layer. In exchange, the virtual tokens take up room in the context and are usually harder to optimize.

LoRA (Hu et al. 2021) won for three practical reasons:

1. No extra inference latency. The update B·A can be added straight into W₀, so after merging the model has exactly its original architecture (section 4.1 of the LoRA paper, paragraph “No Additional Inference Latency”).
2. Easy to swap adapters. An adapter is only tens of MB; one base can come with many adapters for many tasks.
3. No change to the architecture or the sequence length. Everything else in the pipeline, from the tokenizer to the chat template of part 11, stays the same.

03LoRA: the low-rank math

Matrix rank and low-rank approximation

The rank of a d × k matrix is the dimension of the space its columns span. A rank-r matrix can always be written as the product of two thin matrices: M = B·A with B ∈ ℝ^{d×r}, A ∈ ℝ^{r×k}. When r is much smaller than d and k, this costs r(d + k) numbers instead of d·k.

SVD (singular value decomposition) tells you how close to low-rank a matrix is. Every matrix can be written as M = U·Σ·Vᵀ, with singular values σ₁ ≥ σ₂ ≥ … on the diagonal of Σ. The Eckart–Young theorem says that the best rank-r approximation of M in the Frobenius norm keeps exactly the r largest singular values, and the remaining error is σ²_{r+1} + σ²_{r+2} + …. So the ratio (σ₁² + … + σ_r²) / Σ σ_i², the share of energy held by the first r values, is a direct measure: if the first 8 values already hold 90% of the energy, the matrix is nearly rank 8.

The hypothesis: the fine-tuning update is low-rank

Aghajanyan et al. 2020 measure the intrinsic dimension of fine-tuning: optimize only d parameters, project them randomly back into the model’s full parameter space, and ask how small d can be while still reaching 90% of full fine-tuning. For RoBERTa-large on MRPC, the answer is only about 200 parameters (abstract; 207 exactly in section 4.2, Table 1). The better the pretraining, the smaller this number: fine-tuning only needs to move within a very small subspace.

LoRA turns that observation into a concrete hypothesis: the update ΔW during fine-tuning has a low “intrinsic rank” (section 4.1 of the LoRA paper). Note the word “update”. LoRA does not say the model is low-rank. W₀ remains a full matrix; only ΔW is forced to be low-rank.

The formula

For a linear layer with weights W₀ ∈ ℝ^{d×k}, LoRA replaces the forward pass h = W₀x with:

h = W₀·x + (α/r)·B·A·x        B ∈ ℝ^{d×r},  A ∈ ℝ^{r×k},  r ≪ min(d, k)

• W₀ is frozen and has no gradient. Only A and B are trained.
• Initialization: A random (Gaussian), B = 0 (section 4.1, right after equation 3). Because B = 0, B·A = 0 at the start and the model produces exactly the original output. Training starts right at the base model, with no shock.
• The α/r factor: the paper divides by r to reduce the need to retune the learning rate when r changes (section 4.1). Kalajdzievski 2023 (rsLoRA) shows that for large r, dividing by r makes the gradients too small, and proposes dividing by √r instead (section 3, equation 4). PEFT turns this on with use_rslora=True.

Counting parameters

For Llama 3.1 8B, using the same configuration counted in part 8, section 6 (d = 4096, d_ff = 14336, 8 KV heads so W_K and W_V have only 1024 rows), each d × k matrix gets r(d + k) LoRA parameters. Shapes in the table follow PyTorch’s out × in convention. With r = 16:

Attaching it only to W_Q and W_V, as most experiments in the original paper did, drops the count to 6,815,744. Section 4 will show that the two choices differ in quality too.

At the largest scale, the LoRA paper reports that compared with fine-tuning GPT-3 175B with Adam, LoRA cuts the number of trainable parameters by about 10,000× and GPU memory by 3× (abstract). Training VRAM drops from 1.2 TB to 350 GB and the checkpoint from 350 GB to 35 MB (section 4.2, paragraph “Practical Benefits and Limitations”). Memory drops only 3× because the base’s 16-bit weights still sit on the GPU, as the LoRA column in section 1 shows.

Merge and unmerge

After training, W = W₀ + (α/r)·B·A is an ordinary d × k matrix. There are two ways to use it:

• Merge: add B·A into W₀ once and drop the LoRA branch. The model has exactly the architecture and speed of the base, and W₀ + B·A is in general full rank.
• Keep the adapter separate: save A and B as a small file and load it on top of the base when needed. To switch tasks, subtract B·A (unmerge) or load another adapter.

Code sample 1a: is a real ΔW low-rank?

LoRA’s hypothesis is about the fine-tuning update itself. So when you fully fine-tune, is the ΔW the optimizer finds actually low-rank? There is a ready-made pair to check: Qwen2.5-0.5B (base) and Qwen2.5-0.5B-Instruct, fully post-trained from that same base. The code below computes ΔW = W_instruct − W_base for a few matrices, measures the share of energy in the first 8 and 64 singular values, and the number of singular values needed to keep 90% of the energy. The last column is that number for a random Gaussian matrix of the same shape, as a reference point.

# Python 3.12, torch 2.14.1+cu130, transformers 5.18.0 — SVD runs on the GPU (RTX 2080 Ti), fp32
import torch
from transformers import AutoModelForCausalLM
from transformers.utils import logging
logging.disable_progress_bar()

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B", dtype=torch.float32)
inst = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct", dtype=torch.float32)
W0 = dict(base.named_parameters())
W1 = dict(inst.named_parameters())

def energy(s, r):                      # share of the "energy" ‖·‖²_F held by the top r singular values
    e = s.pow(2)
    return (e[:r].sum() / e.sum()).item()

def rank_for(s, frac):                # number of singular values needed to keep `frac` of the energy
    c = s.pow(2).cumsum(0) / s.pow(2).sum()
    return int((c < frac).sum().item()) + 1

print(f"{'matrix':<26}{'shape':>12}{'‖ΔW‖/‖W₀‖':>11}{'r=8':>8}{'r=64':>8}{'r for 90%':>11}{'random':>12}")
for layer in (0, 12, 23):
    for proj in ("self_attn.q_proj", "self_attn.v_proj", "mlp.down_proj"):
        name = f"model.layers.{layer}.{proj}.weight"
        w0 = W0[name].detach().cuda()
        dW = W1[name].detach().cuda() - w0
        s = torch.linalg.svdvals(dW)
        torch.manual_seed(0)
        s_rand = torch.linalg.svdvals(torch.randn_like(dW))   # baseline: Gaussian matrix of the same shape
        print(f"L{layer:<2} {proj:<22}{str(tuple(dW.shape)):>12}"
              f"{(dW.norm() / w0.norm()).item():>11.3f}"
              f"{energy(s, 8):>8.1%}{energy(s, 64):>8.1%}"
              f"{rank_for(s, 0.9):>11}{rank_for(s_rand, 0.9):>12}")

Output:

matrix                           shape  ‖ΔW‖/‖W₀‖     r=8    r=64  r for 90%      random
L0  self_attn.q_proj        (896, 896)      0.072   53.6%   83.7%        105         457
L0  self_attn.v_proj        (128, 896)      0.087   32.5%   81.6%         86         104
L0  mlp.down_proj          (896, 4864)      0.060    4.1%   15.7%        698         707
L12 self_attn.q_proj        (896, 896)      0.088    5.5%   31.3%        402         457
L12 self_attn.v_proj        (128, 896)      0.047   12.4%   68.1%        102         104
L12 mlp.down_proj          (896, 4864)      0.082    2.0%   14.1%        696         707
L23 self_attn.q_proj        (896, 896)      0.103    7.0%   36.2%        371         457
L23 self_attn.v_proj        (128, 896)      0.055   15.8%   78.2%         89         104
L23 mlp.down_proj          (896, 4864)      0.072    4.5%   17.0%        685         707

Reading the output:

• ΔW is small compared with W₀: its norm is only 5–10% of the original weights. Post-training really does only “nudge” the model, consistent with “post-training adds almost no knowledge” from part 11.
• But in the energy sense, most of ΔW is not low-rank. In mlp.down_proj, you need nearly 700 of 896 singular values to keep 90% of the energy, almost the same as a random matrix (707). q_proj in layers 12 and 23 also needs close to 400. Only q_proj in layer 0 has a clear low-rank structure: the first 8 values already hold 54% of the energy.
• This matches Biderman et al. 2024: full fine-tuning finds weight changes whose rank is 10–100× higher than typical LoRA configurations. Qwen2.5-Instruct was also post-trained on a very large amount of data: over 1 million SFT examples, then DPO on about 150 thousand pairs and online RL with GRPO (Qwen2.5 Technical Report, sections 4.1–4.3).

This result goes against the original hypothesis. The LoRA paper is explicit that it is about the real update: “the updates to the weights also have a low intrinsic rank during adaptation” (section 4.1). For a heavy post-training run like Qwen2.5-Instruct’s, that hypothesis does not hold in the energy sense. But two points need weighing. First, the “90% of the energy” yardstick is very strict: even W₀ of L0 mlp.down_proj needs 680 of 896 singular values (Biderman et al. 2024 note this too, section 4.6). Second, energy is not behavior: if we keep only the top few directions of ΔW, how much of Instruct’s behavior do we get back?

The code below runs right after the one above. It builds a hybrid: every parameter comes from Instruct, except that every linear layer in the blocks uses W₀ + ΔW_r, where ΔW_r is the best rank-r approximation of ΔW. Instruct answers 10 chat prompts, then the code measures how far the hybrid drifts from Instruct on exactly those answer tokens, using KL and the top-1 flip rate (the same two measures section 8 uses).

# Run right after the code above (reuses W0, W1, inst, energy, rank_for) — same GPU, fp32
import copy
import torch.nn.functional as F
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
inst = inst.cuda().eval()
hyb = copy.deepcopy(inst)          # hybrid: identical to Instruct, except linear layers use W₀ + (ΔW truncated to rank r)
names = [n for n, m in hyb.named_modules() if isinstance(m, torch.nn.Linear) and ".layers." in n]
svd = {}
for n in names:
    w0 = W0[n + ".weight"].detach().cuda()
    svd[n] = (w0, *torch.linalg.svd(W1[n + ".weight"].detach().cuda() - w0, full_matrices=False))

# 10 chat prompts (5 en, 5 vi); Instruct generates greedy answers, then we compare distributions on those tokens
prompts = ["What does a compiler do? Answer in two sentences.", "Write a short poem about the sea.",
           "List three uses of a paperclip.", "Why do leaves change color in autumn?",
           "Convert 5 kilometers to miles and explain.",
           "Giải thích ngắn gọn vì sao trời có mưa.", "Viết một lời cảm ơn gửi thầy giáo.",
           "Kể tên ba món ăn nổi tiếng của Hà Nội.", "Làm sao để ngủ ngon hơn? Trả lời ngắn.",
           "Tóm tắt truyện Tấm Cám trong hai câu."]
seqs = []
for p in prompts:
    ids = tok.apply_chat_template([{"role": "user", "content": p}], add_generation_prompt=True,
                                  return_tensors="pt", return_dict=True)["input_ids"].cuda()
    with torch.no_grad():
        seqs.append((inst.generate(ids, max_new_tokens=64, do_sample=False), ids.shape[1]))

@torch.no_grad()
def answer_logprobs(model):        # log-probs at every position of the answers
    return torch.cat([F.log_softmax(model(o).logits[0, L - 1:-1], -1) for o, L in seqs])

ref = answer_logprobs(inst)
print("W₀ of L0 mlp.down_proj needs", rank_for(torch.linalg.svdvals(svd["model.layers.0.mlp.down_proj"][0]), 0.9),
      "singular values to keep 90% of the energy")
print(f"{'ΔW kept rank':<14}{'energy (mean)':>16}{'KL(inst ‖ ·)':>14}{'flip top-1':>12}")
for r in (0, 8, 64, 256):
    kept = []
    for n in names:
        w0, U, S, Vh = svd[n]
        hyb.get_submodule(n).weight.data = w0 + (U[:, :r] * S[:r]) @ Vh[:r]
        kept.append(energy(S, r))
    lp = answer_logprobs(hyb)
    kl = (ref.exp() * (ref - lp)).sum(-1).mean().item()
    flip = (lp.argmax(-1) != ref.argmax(-1)).float().mean().item()
    print(f"{'r = ' + str(r):<14}{sum(kept) / len(kept):>16.1%}{kl:>14.3f}{flip:>12.1%}")

Output:

W₀ of L0 mlp.down_proj needs 680 singular values to keep 90% of the energy
ΔW kept rank     energy (mean)  KL(inst ‖ ·)  flip top-1
r = 0                     0.0%         0.407       26.2%
r = 8                     7.6%         0.198       20.8%
r = 64                   36.4%         0.120       17.2%
r = 256                  70.3%         0.054       11.0%

The exact KL and flip numbers change with the prompt set, while the relative reduction stays stable; the energy column depends only on the weights, so it does not change.

Reading the output:

• r = 0 is the base’s linear layers combined with the rest of Instruct. It drifts from Instruct by a KL of 0.407 nats per token, and 26% of the top-1 tokens differ.
• Rank 8 keeps only 7.6% of ΔW’s energy, yet already halves the KL (0.407 down to 0.198). Rank 64 keeps 36% of the energy and cuts the KL by 71%. The part of the update that matters for behavior is concentrated in the top few directions far more than their share of the energy suggests.
• Truncation is not free: at rank 8, 21% of the top-1 tokens still differ. But this is an SVD cut after training. A LoRA adapter trained directly makes better use of its r directions, so these numbers are only a lower bound.

That is why LoRA still works even though full fine-tuning’s ΔW is not low-rank in the energy sense: a low-rank update keeps most of the change in behavior. What it loses depends on the amount of data and the difficulty of the task, and section 4 measures that with controlled experiments.

Code sample 1b: attaching LoRA with PEFT

The next code uses PEFT to: (1) count trainable parameters when LoRA is applied to q, v versus every linear layer; (2) check that at initialization the model with LoRA gives exactly the original output; (3) train a few steps; (4) turn the adapter off and merge, then measure the logit differences. Model and adapter stay in fp32 so the comparisons are free of rounding error.

# Python 3.12, torch 2.14.1+cu130, transformers 5.18.0, peft 0.21.2
# GPU: RTX 2080 Ti 11 GB; model and adapter stay in fp32 so the logit comparisons carry no rounding error
import copy
import torch
from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers.utils import logging
logging.disable_progress_bar()
torch.manual_seed(0)

name = "Qwen/Qwen2.5-0.5B"
tok = AutoTokenizer.from_pretrained(name)
base = AutoModelForCausalLM.from_pretrained(name, dtype=torch.float32).cuda().eval()
n_total = sum(p.numel() for p in base.parameters())

# 1. Trainable parameters: q,v only (as in the LoRA paper) versus every linear layer
for targets in (["q_proj", "v_proj"], "all-linear"):
    cfg = LoraConfig(r=16, lora_alpha=32, target_modules=targets, task_type="CAUSAL_LM")
    m = get_peft_model(copy.deepcopy(base), cfg)
    n_train = sum(p.numel() for p in m.parameters() if p.requires_grad)
    print(f"target={str(targets):<22} trainable={n_train:>10,}  ({n_train / n_total:.3%} of {n_total:,})")
    del m

# 2. At init B = 0, so the model with LoRA gives exactly the original output
text = ("LoRA keeps the original weights frozen and only learns a low-rank update. "
        "The optimizer then stores state for a few million parameters instead of the whole model.")
batch = tok(text, return_tensors="pt").to("cuda")
with torch.no_grad():
    ref_logits = base(**batch).logits

cfg = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.0, target_modules="all-linear", task_type="CAUSAL_LM")
model = get_peft_model(base, cfg)
with torch.no_grad():
    print("init: max |Δlogits| =", (model(**batch).logits - ref_logits).abs().max().item())

# 3. Train a few steps on a single sentence, just to see the adapter change the model
opt = torch.optim.AdamW([p for p in model.parameters() if p.requires_grad], lr=1e-3)
model.train()
for step in range(30):
    loss = model(**batch, labels=batch["input_ids"]).loss
    loss.backward(); opt.step(); opt.zero_grad()
    if step in (0, 9, 19, 29):
        print(f"step {step + 1:>2}  loss {loss.item():.4f}")
model.eval()

with torch.no_grad():
    lora_logits = model(**batch).logits
    with model.disable_adapter():                 # adapter off = back to the base, no second copy needed
        off_logits = model(**batch).logits
print("after training: max |logits(LoRA) − logits(base)| =", (lora_logits - ref_logits).abs().max().item())
print("adapter off:    max |logits − logits(base)|       =", (off_logits - ref_logits).abs().max().item())

# 4. Merge BA into W₀: no side branch at inference, output almost unchanged
merged = model.merge_and_unload()
with torch.no_grad():
    merged_logits = merged(**batch).logits
print("merge:          max |logits(merged) − logits(LoRA)| =", (merged_logits - lora_logits).abs().max().item())
print("layer after merge:", type(merged.model.layers[0].self_attn.q_proj).__name__)

Output:

target=['q_proj', 'v_proj']   trainable= 1,081,344  (0.219% of 494,032,768)
target=all-linear             trainable= 8,798,208  (1.781% of 494,032,768)
init: max |Δlogits| = 0.0
step  1  loss 4.0773
step 10  loss 0.0017
step 20  loss 0.0004
step 30  loss 0.0001
after training: max |logits(LoRA) − logits(base)| = 30.087738037109375
adapter off:    max |logits − logits(base)|       = 0.0
merge:          max |logits(merged) − logits(LoRA)| = 0.0001659393310546875
layer after merge: Linear

Reading the output:

• The parameter counts match the r(d + k) formula. Qwen2.5-0.5B has d = 896, d_ff = 4864, and 2 KV heads, so W_K and W_V are 128 × 896. With r = 16, one block has 16·(1792 + 1024 + 1024 + 1792 + 3·5760) = 366,592 LoRA parameters; times 24 blocks gives exactly 8,798,208. Attaching only to q, v gives 1,081,344, 8× fewer. Note that PEFT’s "all-linear" does not attach to lm_head (per the PEFT documentation).
• At initialization the difference is exactly 0. This is a direct consequence of B = 0.
• Training 30 steps on one sentence makes the model memorize it (loss from 4.08 to 0.0001), and the logits move by up to 30 from the base. This is deliberate overfitting, only to show the adapter really changes the model.
• Turning the adapter off gives back exactly the base, a difference of 0. Section 4 uses this for DPO.
• Merging barely changes the output: the largest difference is 0.00017 on logits in the tens, i.e. fp32 rounding error from adding two matrices. After merging, q_proj is a plain torch.nn.Linear again.

04LoRA in practice

Which layers to apply it to

The original LoRA paper froze the entire MLP and experimented only with the attention matrices, using W_Q and W_V in most experiments (sections 4.2, 5.1). Later results point the other way:

• QLoRA (Dettmers et al. 2023) found that the most important hyperparameter is the total number of adapters, and that LoRA must be attached to every linear layer in the block to match 16-bit full fine-tuning (section 4, Figure 2).
• LoRA Without Regret (John Schulman and Thinking Machines Lab, 29 Sep 2025) reached the same conclusion: attention-only LoRA underperforms clearly, even with the same number of trainable parameters, and the MLP layers matter most (section “Layers Where LoRA Is Applied”). For example, attention-only LoRA at r = 256 (0.25B parameters) still loses to MLP-only LoRA at r = 128 (0.24B parameters).

The reason is intuitive: the FFN holds about 80% of each block’s parameters (part 8).

Choosing r, α and the learning rate

• A small r is often enough. For GPT-3 175B on WikiSQL and MultiNLI, the LoRA paper found that a rank as small as r = 1 suffices when applied to both W_Q and W_V (section 7.2, Table 6), though the paper itself warns not to expect this for every task (footnote 6). QLoRA also found that once LoRA is on every layer, r barely affects the final result (appendix A.1, Figure 4). In practice, r between 8 and 64 is common; raising r only helps with enough data.
• α: usually set α = r or α = 2r, then tune the learning rate rather than α.
• A higher learning rate than full fine-tuning: LoRA Without Regret finds the optimal LoRA learning rate is about 10× that of full fine-tuning on the same model (fitted multiplier 9.8, section “Optimal learning rates for LoRA vs. FullFT”; closer to 15× for short runs under about 100 steps, section “Learning rates in short and long runs”). The reason: B starts at 0, and a low-rank update needs bigger steps to cover the same distance.

Does LoRA match full fine-tuning?

The short answer: it depends on the amount of data.

• Biderman et al. 2024 (“LoRA Learns Less and Forgets Less”) compare on code and math, in both continued pretraining (billions of tokens) and instruction fine-tuning. At the low ranks people usually use, LoRA learns noticeably less than full fine-tuning (section 4.1, Figure 1). The gap does not close in continued pretraining; in instruction fine-tuning, high ranks can catch up (section 1). In exchange, LoRA forgets less: the model keeps its out-of-domain abilities better (section 4.2), and this regularizing effect is stronger than weight decay or dropout (section 4.5, Figure 4).
• LoRA Without Regret adds to the picture: on small-to-medium instruction-tuning and reasoning datasets, LoRA matches full fine-tuning; it falls behind only when the data exceeds the adapter’s “capacity” (section “What matters for LoRA”). For RL, LoRA matches full fine-tuning even at r = 1, because each RL episode provides very few bits of information (section “Reinforcement learning”).

Putting the two sources together: LoRA is a sensible default for SFT on a few thousand to a few hundred thousand examples and for RL; when you want to teach the model a new domain with billions of tokens, full fine-tuning still wins.

DPO with LoRA: turn off the adapter to get the reference

Part 11, section 8 showed that the DPO loss needs the log-probs of both the policy and a reference model, and the usual approach keeps two copies of the model. With LoRA, the reference is simply the base with the adapter turned off. Code sample 1b just checked this: disable_adapter() gives a logit difference of 0 from the base. So only one copy of the weights needs to be in memory: adapter on for the policy log-probs, adapter off for the reference log-probs. TRL uses exactly this trick when you pass a peft_config to DPOTrainer and no ref_model.

Serving many adapters, and variants

Because adapters are small and not merged, one server can hold a single base and thousands of adapters, pairing each request with its adapter during computation, as S-LoRA (Sheng et al. 2023) does. Variants such as rsLoRA (a different scaling factor), DoRA (splitting magnitude and direction), LoRA+ (a separate learning rate for B) and PiSSA (initializing from the SVD of W₀) all keep the core idea “low-rank update, frozen base”, with gains usually of a few percentage points depending on the task.

05Quantization: from a number to a tensor

Part 10, section 8 already has the table of floating-point formats fp32, fp16, bf16 and fp8. This section goes lower: 8-bit and 4-bit integers. An integer has no exponent, so to represent weights in INT8 or INT4 we have to choose a scale ourselves.

Absmax and zero-point

The simplest method is absmax (symmetric). For a group of values x and b bits:

s = max|x| / (2^(b−1) − 1)        # INT8: divide by 127, INT4: divide by 7
q = round(x / s)                   # an integer, stored together with s
x̂ = q · s                          # dequantize

The value with the largest magnitude maps exactly onto the top level, and every other value is rounded onto an evenly spaced grid between −max|x| and max|x|. If the distribution is skewed to one side, half the grid is wasted. Zero-point (asymmetric) quantization adds an offset z: q = round(x/s) + z, so the grid covers exactly [min x, max x]. The price is storing z as well.

Per tensor, per channel, per group

The most important question is not the formula but how many values share one scale:

• Per tensor: one scale for the whole matrix. Cheapest, but a single very large value stretches the whole grid, and most small values round to 0.
• Per channel: one scale per row (or column).
• Per group: one scale per g consecutive values, usually g = 64 or 128. Every modern 4-bit format does this.

Scales are not free. With groups of 128 and an fp16 scale, each weight costs an extra 16/128 = 0.125 bits; with a zero-point too, twice that. So a “4-bit” model actually costs about 4.1–4.9 bits per weight (table in section 6).

Outliers: the reason there are so many methods

If weights and activations were nicely distributed, per-group absmax would be enough. But they are not. LLM.int8() (Dettmers et al. 2022) found outlier features in the activations: a very small number of hidden-state dimensions whose magnitude is far larger than the rest. In small models they appear in about 25% of layers; around 6.7B parameters there is a phase shift, and they appear in every layer and in about 75% of sequence positions (section 1, section 4.2, Figure 3). Naive quantization lets these dimensions stretch the scale and wreck the model. LLM.int8() fixes this with mixed-precision decomposition: the outlier dimensions are computed in 16 bits, and the remaining 99.9%+ of values are multiplied in 8 bits (abstract).

This is the crux of section 6: GPTQ, AWQ, SmoothQuant and k-quants are all different ways to handle the fact that a few values matter far more than the rest.

Weight-only versus weight + activation

• Weight-only (W4A16, W8A16): only the weights are quantized; before multiplication they are dequantized back to 16 bits. This is the most common setup and the focus of this post.
• Weight + activation (W8A8): quantize both so the matrix multiplication runs entirely on 8-bit integers. SmoothQuant (Xiao et al. 2022) moves the “difficulty” from activations to weights by dividing the activations and multiplying the weights by the same per-channel factor (section 4), reaching up to 1.56× speedup and 2× memory reduction (abstract, section 5.3).

Why does weight-only speed things up when the multiplication still runs in 16 bits? Because during decode, each new token has to read all the weights from GPU memory while doing very little computation per byte read. Decode is bound by memory bandwidth, not compute, so weights that are 4× smaller can make it much faster. Part 13 will cover prefill and decode in detail.

PTQ and QAT

Every method in section 6 is post-training quantization (PTQ): quantizing a model that has finished training, sometimes with a little calibration data. Quantization-aware training (QAT, Jacob et al. 2018), by contrast, simulates rounding during training and passes gradients through round with the straight-through estimator (Bengio et al. 2013), so it does better at low bit widths but requires rerunning part of training.

06GPTQ, AWQ and GGUF

These three names are often listed side by side as three options of the same kind. They are actually three different kinds of thing: GPTQ and AWQ are algorithms for choosing quantized values, while GGUF is a file format.

GPTQ: compensating error with second-order information

GPTQ (Frantar et al. 2022) works one layer at a time. Given weights W and input activations X from a little calibration data, it looks for a quantized Ŵ that changes the layer’s output as little as possible:

argmin_Ŵ ‖W·X − Ŵ·X‖²₂

The key is that error is measured on the output, not on the weights: rounding one weight off does not matter if the others can compensate. GPTQ quantizes column by column, then uses the Hessian H = 2XXᵀ to update the not-yet-quantized columns, compensating for the error just introduced (section 3, equations 1–2, building on Optimal Brain Quantization); section 4 makes it fast enough for LLMs by processing columns in batches of 128 and using a Cholesky decomposition. The result: quantizing a 175B model to 3 or 4 bits in about 4 GPU hours, with very small degradation (abstract). For OPT-175B at 3 bits, inference is about 3.24× faster than fp16 on an A100 (section 5, Table 6).

AWQ: protect the 1% of channels that matter

AWQ (Lin et al. 2023) starts from an observation: protecting only about 1% of salient weights greatly reduces quantization error (abstract, section 3.1). On OPT-6.7B at INT3 with groups of 128, keeping that 1% of channels in fp16 brings perplexity from 43.2 down to 13.0 (Figure 2). Salient channels are chosen by the magnitude of the activations feeding them, because a channel that receives large activations amplifies its error more; choosing by the magnitude of the weights themselves is barely better than random (section 3.1, Table 1). Keeping that 1% in fp16 requires mixed-precision multiplication, which hardware does not like, so AWQ multiplies the salient channels by a factor before quantizing (and divides the corresponding activations) to shrink their relative error. It needs neither backpropagation nor GPTQ-style output reconstruction, so it overfits less to the calibration set (section 3.2).

GGUF and k-quants

GGUF is the file format of llama.cpp: one file holding the weights, tokenizer and metadata, runnable on CPU, Apple Silicon and GPU. Inside a GGUF file there can be many different quantization types. The most common is the k-quant family, introduced in PR #1684:

• Super-blocks: the weights are split into super-blocks of 256 values, each made of small blocks of 16 or 32 values. Each block has its own scale (and min), and those scales are themselves quantized to 6 or 8 bits. For example, Q4_K uses 8 blocks × 32 weights with 6-bit scales and mins, which comes to 4.5 bits per weight.
• The name Q4_K_M reads as: 4-bit k-quant, Medium variant. The S/M/L variants decide which layers get extra bits. According to the original description in PR #1684, Q4_K_M uses Q6_K for half of the attention.wv and feed_forward.w2 tensors and Q4_K for the rest. This logic has been tuned many times since, so take the real numbers from measurements.
• Importance matrix (imatrix): an optional step that runs the model on a little text to learn which weights matter, then favors them when rounding, in the same spirit as AWQ.

The table below comes from the llama-quantize README of llama.cpp at commit 5e03bdd on master (6 Oct 2026), measured on Llama-3.1-8B:

Three things worth noting:

• “4-bit” is not 4 bits. Q4_K_S uses Q4_K for almost every tensor and still averages 4.67 bits, above the 4.5 bits of Q4_K alone, because some tensors such as the embedding and output are kept in higher-bit types.
• Decode speeds up as section 5 predicted. Text generation rises about 2.5×, from 29 tokens/s at F16 to 72 tokens/s at Q4_K_M, though the file shrinks 3.3×, so it does not quite reach the size ratio.
• Prompt processing gets slightly slower (923 to 822 tokens/s). That phase is compute-bound, so smaller weights do not help, and dequantization costs extra. The README does not state the hardware, so read only the ratios.

What to pick for which need

Rule of thumb: serving on a GPU server, use GPTQ or AWQ through an inference engine; personal machine, CPU or Mac, use GGUF; fine-tuning, use bitsandbytes through QLoRA in the next section.

07QLoRA: putting the two together

QLoRA (Dettmers et al. 2023) quantizes the base model to 4 bits, freezes it, and trains a 16-bit LoRA adapter on top. Its best-known result: fine-tuning a 65B model on a single 48 GB GPU, whereas 16-bit fine-tuning needs more than 780 GB (abstract, section 1). The Guanaco family trained this way reaches 99.3% of ChatGPT’s score on the Vicuna benchmark after 24 hours on one GPU (abstract, Table 6). That figure should be read with the limits the paper itself states: the benchmark has only 80 prompts judged by GPT-4, GPT-4 favors whichever answer comes first (section 5.2), and agreement between GPT-4 and human raters is only Fleiss κ = 0.25 (section 6.2). Three techniques make up QLoRA.

NF4: placing the levels by the normal distribution

The weights of a trained model are roughly normally distributed, concentrated around 0. An evenly spaced INT4 grid wastes levels on the rarely used tails. NormalFloat4 (NF4) places its 16 levels at quantiles of the normal distribution, so each level “holds” roughly the same number of weights. The paper calls this an information-theoretically optimal data type for normally distributed weights (abstract; construction in section 3, equation 4). Each block of 64 weights is divided by the block’s absmax and rounded to the nearest NF4 level.

The grid must have a level exactly at 0, so NF4 is asymmetric: 8 positive levels, 7 negative levels and zero. The code below rebuilds that table the way bitsandbytes does (the create_normal_map function), then compares it with the table bitsandbytes itself uses:

# Python 3.12, torch 2.14.1, bitsandbytes 0.50.2
import torch
import bitsandbytes.functional as bnbF

def nf4_levels(offset=0.9677083):
    icdf = torch.distributions.Normal(0.0, 1.0).icdf
    pos = icdf(torch.linspace(offset, 0.5, 9)[:-1])        # 8 positive levels
    neg = -icdf(torch.linspace(offset, 0.5, 8)[:-1])       # 7 negative levels
    v = torch.cat([pos, torch.zeros(1), neg]).sort().values
    return v / v.abs().max()                                # normalize to [-1, 1]

mine = nf4_levels()
print(mine)
print("max difference from bitsandbytes:", (mine - bnbF.get_4bit_type("nf4", device="cpu")).abs().max().item())

tensor([-1.0000, -0.6962, -0.5251, -0.3949, -0.2844, -0.1848, -0.0911,  0.0000,
         0.0796,  0.1609,  0.2461,  0.3379,  0.4407,  0.5626,  0.7230,  1.0000])
max difference from bitsandbytes: 5.960464477539063e-08

The levels are dense around 0 (about 0.08–0.09 apart) and spread out toward the ends (from 0.72 to 1.0). That is exactly the shape you want for a distribution concentrated around 0.

Double quantization

Each block of 64 weights has one fp32 scale constant (absmax), i.e. 32/64 = 0.5 bits per weight. QLoRA quantizes those scale constants too: it groups 256 scales together and quantizes them to FP8 with one fp32 scale for the group. The cost drops to 8/64 + 32/(64·256) ≈ 0.127 bits per weight, a saving of about 0.37 bits per parameter, or about 3 GB for a 65B model (the calculation is in section 3, paragraph “Double Quantization”; the rounded figure is in section 1). That is where the 4.127 bits in the table of section 1 come from.

Paged optimizers

Gradient checkpointing (recomputing activations in the backward pass instead of keeping them) creates short memory spikes, especially for batches with long sequences. Paged optimizers use NVIDIA unified memory to automatically move optimizer state to CPU RAM when the GPU is about to run out, and move it back when it is needed for the update (section 3, paragraph “Paged Optimizers”). This does not reduce total memory; it only stops a whole training run from dying because of one short spike.

Forward and backward

Figure 2 draws one linear layer in QLoRA:

Three things to take from the figure:

• The NF4 weights are never used directly in a multiplication. They are only a storage format; on every forward pass, each layer is dequantized to the compute dtype before multiplying. This is why QLoRA is slower than LoRA on a 16-bit base.
• Gradients still flow through W₀. Gradients for the adapter in layer 3 have to flow back through layers 4, 5, …, including the frozen W₀. What is dropped is the gradient of W₀ (the derivative with respect to W₀ itself), not the gradient through W₀.
• The adapter trains in 16 or 32 bits, not 4 bits. The paper uses bf16 as the compute dtype (section 3, equations 5–6). PEFT keeps the adapter weights in fp32 by default.

Running it for real on an RTX 2080 Ti

The code below trains real QLoRA on Qwen2.5-1.5B: an NF4 base with double quantization, LoRA r = 16 on every linear layer, paged AdamW, 40 steps, batches of 4 × 512 tokens from wikitext-2. For comparison, the same script runs once more with an unquantized fp16 base (plain LoRA).

About the dtype: the RTX 2080 Ti is a Turing card, with no bf16 compute units in hardware (bf16 arrived with Ampere). Interestingly, torch.cuda.is_bf16_supported() still returns True on this card, because by default PyTorch counts emulated bf16; adding the argument including_emulation=False makes it return False. So the script uses fp16 and a GradScaler for loss scaling, the fp16 problem described in part 10, section 8. On Ampere or newer GPUs, change all three torch.float16 spots (the NF4 compute dtype, the base dtype in the lora-fp16 branch, and the autocast dtype) to torch.bfloat16, and drop the GradScaler.

# Python 3.12, torch 2.14.1+cu130, transformers 5.18.0, peft 0.21.2, bitsandbytes 0.50.2,
# accelerate 1.15.0, datasets 5.1.0 — GPU: RTX 2080 Ti 11 GB (Turing, sm_75), driver 595.91.07
import sys, time
import torch, bitsandbytes as bnb
from datasets import load_dataset
from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from transformers.utils import logging
logging.disable_progress_bar()
torch.manual_seed(0)

MODE = sys.argv[1] if len(sys.argv) > 1 else "qlora"     # "qlora", or "lora-fp16" for comparison
name = "Qwen/Qwen2.5-1.5B"
tok = AutoTokenizer.from_pretrained(name)

if MODE == "qlora":
    bnb_cfg = BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_quant_type="nf4",             # NormalFloat4
        bnb_4bit_use_double_quant=True,        # also quantize the scale constants
        bnb_4bit_compute_dtype=torch.float16,  # Turing has no hardware bf16 → fp16
    )
    model = AutoModelForCausalLM.from_pretrained(name, quantization_config=bnb_cfg, device_map={"": 0})
else:
    model = AutoModelForCausalLM.from_pretrained(name, dtype=torch.float16, device_map={"": 0})
print(f"[{MODE}] weights on GPU: {torch.cuda.memory_allocated() / 2**30:.2f} GiB")

# Both setups use the same gradient checkpointing so the VRAM comparison is fair
model.gradient_checkpointing_enable(gradient_checkpointing_kwargs={"use_reentrant": False})
cfg = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05, target_modules="all-linear", task_type="CAUSAL_LM")
model = get_peft_model(model, cfg)
n_train = sum(p.numel() for p in model.parameters() if p.requires_grad)
n_all = sum(p.numel() * (2 if p.dtype == torch.uint8 else 1) for p in model.parameters())  # 1 uint8 byte = two 4-bit weights
print(f"trainable {n_train:,} / {n_all:,} ({n_train / n_all:.2%}), dtype adapter: {next(p for p in model.parameters() if p.requires_grad).dtype}")

# Data: wikitext-2, concatenated and cut into 512-token blocks
raw = load_dataset("Salesforce/wikitext", "wikitext-2-raw-v1", split="train")
ids = tok("\n".join(t for t in raw["text"][:4000] if t.strip()), return_tensors="pt").input_ids[0]
SEQ, BS, STEPS = 512, 4, 40
blocks = ids[: (len(ids) // SEQ) * SEQ].view(-1, SEQ)

opt = bnb.optim.PagedAdamW(model.parameters(), lr=2e-4)   # paged optimizer, fp32 state
scaler = torch.amp.GradScaler("cuda")                     # fp16 needs loss scaling
torch.cuda.reset_peak_memory_stats()
model.train(); t0 = time.time()
for step in range(STEPS):
    x = blocks[step * BS:(step + 1) * BS].cuda()
    with torch.autocast("cuda", dtype=torch.float16):
        loss = model(input_ids=x, labels=x).loss
    scaler.scale(loss).backward()
    scaler.step(opt); scaler.update(); opt.zero_grad(set_to_none=True)
    if step % 10 == 0 or step == STEPS - 1:
        print(f"step {step + 1:>2}  loss {loss.item():.4f}")
torch.cuda.synchronize()
print(f"{(time.time() - t0) / STEPS:.2f} s/step, peak VRAM: {torch.cuda.max_memory_allocated() / 2**30:.2f} GiB")

Output of python qlora.py qlora followed by python qlora.py lora-fp16 (library warnings removed):

[qlora] weights on GPU: 1.07 GiB
trainable 18,464,768 / 1,562,179,072 (1.18%), dtype adapter: torch.float32
step  1  loss 3.0523
step 11  loss 2.7242
step 21  loss 2.5036
step 31  loss 2.3735
step 40  loss 2.4673
1.01 s/step, peak VRAM: 5.46 GiB

[lora-fp16] weights on GPU: 2.88 GiB
trainable 18,464,768 / 1,562,179,072 (1.18%), dtype adapter: torch.float32
step  1  loss 2.9875
step 11  loss 2.6646
step 21  loss 2.4292
step 31  loss 2.3208
step 40  loss 2.4175
0.80 s/step, peak VRAM: 6.66 GiB

Reading the output:

• The weights match the calculation in section 1. Qwen2.5-1.5B has 1.31B parameters in the blocks’ linear layers and 233M embedding parameters (shared with lm_head). 1.31B × 4.127/8 + 233M × 2 bytes ≈ 1.14 GB = 1.06 GiB, matching the 1.07 GiB measured. The fp16 version is 2.88 GiB.
• The number of trainable parameters is the same in both setups (18.46 million, 1.18%), and the adapter is fp32 even though the base is 4-bit.
• Peak VRAM differs by only 1.2 GiB, less than the 1.8 GiB difference in the weights alone. For a small model, most memory goes to activations: the logits of one batch alone are 4 × 512 × 151,936 numbers, about 1.16 GiB in fp32. The ratio flips as models grow; at 65B, the weights decide whether it fits on the GPU at all.
• QLoRA is about 25% slower (1.01 versus 0.80 seconds per step), because of the dequantization step in every layer, in both the forward and the backward pass.
• QLoRA’s loss is slightly higher from the very first step (3.052 versus 2.988 on the same batch, with the adapter still at 0). That is the quantization error of the NF4 base, measured directly as loss. Section 8 measures it more carefully.
• The losses can differ in the fourth decimal place between runs, because some GPU kernels are not deterministic.

For this model, plain LoRA also fits in 11 GB. QLoRA becomes necessary as models grow: a 7B base in fp16 already takes 14 GB and does not fit this card; in NF4, the same model is only about 4–5 GB.

The memory table at larger scale

The table assumes every parameter is quantized; real numbers are slightly higher because embeddings usually stay in 16 bits. At 65B, an NF4 base of about 33.5 GB plus the adapter and activations still stays under 48 GB, consistent with the paper.

A caveat when merging

A QLoRA adapter is trained on a quantized base, so it also learns to compensate for that particular base’s quantization error. There are three ways to use it:

1. Keep the NF4 base + adapter: closest to how it was trained, but inference is slow because of dequantization.
2. Merge into the original 16-bit base: fast, but the adapter now meets a different base from training. The shift is usually small, but measure it again.
3. Merge, then quantize again (for example to GGUF): the new quantization error is not the one the adapter learned to compensate for. This is the easiest way to skew results, so always measure before and after.

08What a model loses when it is quantized

Code sample 2: fake-quant on Qwen2.5-0.5B

To see the cost clearly, the code below writes its own fake-quant function: quantize and immediately dequantize the weights of every linear layer in the blocks, using symmetric absmax per group along the input dimension. There are three types, INT8, INT4 and NF4 (the table from section 7), with different group granularities, including one version deliberately without groups. Each configuration is measured on 50 parallel English and Vietnamese passages from FLORES-200 (taken from Belebele):

• bits-per-byte (bpb), the tokenizer-independent measure used in part 10, section 2;
• the average per-token KL between the original and the quantized distribution, KL(p_orig ‖ p_quant), in nats;
• the top-1 flip rate: the percentage of positions where the most likely token changes compared with the original;
• the share of weights rounded to 0.

The measurements run on the GPU, with an fp32 reference because Turing has no hardware bf16.

# Python 3.12, torch 2.14.1+cu130, transformers 5.18.0, datasets 5.1.0
# Runs on the GPU (RTX 2080 Ti); fp32 reference because Turing has no hardware bf16
import torch
import torch.nn.functional as F
from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers.utils import logging
logging.disable_progress_bar()

# --- NF4: 16 levels from quantiles of N(0, 1), built like bitsandbytes (create_normal_map) ---
def nf4_levels(offset=0.9677083):
    icdf = torch.distributions.Normal(0.0, 1.0).icdf
    pos = icdf(torch.linspace(offset, 0.5, 9)[:-1])        # 8 positive levels
    neg = -icdf(torch.linspace(offset, 0.5, 8)[:-1])       # 7 negative levels
    v = torch.cat([pos, torch.zeros(1), neg]).sort().values
    return v / v.abs().max()                                # normalize to [-1, 1]

NF4 = nf4_levels()

def fake_quant(w, kind, group):
    """Quantize then immediately dequantize (fake-quant): symmetric absmax, one scale per group along the input dim."""
    out_f, in_f = w.shape
    g = in_f if group == "channel" else (w.numel() if group == "tensor" else group)
    x = w.reshape(-1, g)                                    # each row is one group sharing one scale
    absmax = x.abs().amax(dim=1, keepdim=True).clamp(min=1e-8)
    if kind == "nf4":
        lv = NF4.to(w.device)
        idx = (x / absmax).unsqueeze(-1).sub(lv).abs().argmin(-1)   # nearest NF4 level
        q = lv[idx] * absmax
    else:
        qmax = 2 ** (int(kind[3:]) - 1) - 1                 # int8 → 127, int4 → 7
        scale = absmax / qmax
        q = (x / scale).round().clamp(-qmax - 1, qmax) * scale
    return q.reshape(out_f, in_f)

name = "Qwen/Qwen2.5-0.5B"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name, dtype=torch.float32).cuda().eval()
linears = {n: m for n, m in model.named_modules()
           if isinstance(m, torch.nn.Linear) and ".layers." in n}  # only the linear layers inside blocks
original = {n: m.weight.data.clone() for n, m in linears.items()}

# 50 parallel en–vi FLORES-200 passages (via Belebele)
texts = {}
for lang, cfg in (("en", "eng_Latn"), ("vi", "vie_Latn")):
    rows = load_dataset("facebook/belebele", cfg, split="test")
    seen, texts[lang] = set(), []
    for r in rows:
        if r["link"] not in seen and len(texts[lang]) < 50:
            seen.add(r["link"]); texts[lang].append(r["flores_passage"])

@torch.no_grad()
def logprobs(text):
    ids = tok(tok.eos_token + text, return_tensors="pt").input_ids.cuda()
    return F.log_softmax(model(ids).logits[0, :-1].float(), -1), ids[0, 1:]

ref = {lang: [(lp.half(), tgt) for lp, tgt in map(logprobs, ts)] for lang, ts in texts.items()}  # keep in fp16 to fit in VRAM

def evaluate():
    res = {}
    for lang, ts in texts.items():
        nll = kl = flips = n_tok = n_byte = 0
        for t, (lp_ref, tgt) in zip(ts, ref[lang]):
            lp, _ = logprobs(t)
            lp_ref = lp_ref.float()
            nll += -lp.gather(1, tgt[:, None]).sum().item()
            kl += (lp_ref.exp() * (lp_ref - lp)).sum().item()          # KL(p_ref ‖ p_q)
            flips += (lp.argmax(-1) != lp_ref.argmax(-1)).sum().item()  # top-1 changed versus the original
            n_tok += tgt.numel(); n_byte += len(t.encode("utf-8"))
        res[lang] = (nll / 0.6931471805599453 / n_byte, kl / n_tok, flips / n_tok)
    return res

configs = [("fp32 (orig)", None, None), ("int8 / channel", "int8", "channel"),
           ("int4 / g128", "int4", 128), ("nf4 / g128", "nf4", 128),
           ("int4 / g64", "int4", 64), ("nf4 / g64", "nf4", 64),
           ("int4 / channel", "int4", "channel"), ("int4 / tensor", "int4", "tensor")]
print(f"{'config':<16}{'to 0':>7}{'bpb en':>8}{'bpb vi':>8}{'KL en':>8}{'KL vi':>8}{'flip en':>9}{'flip vi':>9}")
for label, kind, group in configs:
    for n, m in linears.items():
        m.weight.data = original[n] if kind is None else fake_quant(original[n], kind, group)
    zero = sum((m.weight == 0).sum().item() for m in linears.values()) / sum(w.numel() for w in original.values())
    r = evaluate()
    print(f"{label:<16}{zero:>7.1%}{r['en'][0]:>8.3f}{r['vi'][0]:>8.3f}{r['en'][1]:>8.3f}{r['vi'][1]:>8.3f}"
          f"{r['en'][2]:>9.1%}{r['vi'][2]:>9.1%}")

Output:

config             to 0  bpb en  bpb vi   KL en   KL vi  flip en  flip vi
fp32 (orig)        0.0%   0.864   0.905   0.000   0.000     0.0%     0.0%
int8 / channel     1.5%   0.864   0.904   0.010   0.008     3.1%     4.2%
int4 / g128       18.5%   0.942   1.024   0.337   0.445    27.6%    37.8%
nf4 / g128        11.2%   0.913   0.966   0.212   0.254    21.0%    28.2%
int4 / g64        16.6%   0.925   0.984   0.258   0.336    24.1%    32.7%
nf4 / g64         10.0%   0.907   0.956   0.206   0.227    19.7%    25.5%
int4 / channel    25.5%   1.042   1.215   0.693   1.074    43.0%    58.4%
int4 / tensor     86.1%   5.305   5.181  15.678  14.196   100.0%   100.0%

Reading the output line by line:

• INT8 per channel loses almost nothing in bpb (0.864 unchanged, 0.905 to 0.904), and KL is only 0.01. But 3–4% of positions still change their top-1 token. These are positions where the top two tokens are nearly tied, and a small perturbation is enough to swap them.
• At 4 bits, granularity decides everything. All INT4: with groups of 64, English bpb rises 7%; per channel, 21%; per tensor, the model breaks completely: bpb goes from 0.86 to 5.3 and every top-1 token changes. The “to 0” column explains why: with one scale for the whole matrix, a few unusually large weights stretch the grid so far that 86% of the weights round to 0. This is the weight version of the outlier story in section 5. LLM.int8() is about outliers in activations, while here it is a few unusually large weights, but the mechanism is the same: one shared scale for too many values.
• NF4 beats INT4 at the same group size: with groups of 64, KL is 0.206 versus 0.258 for English and 0.227 versus 0.336 for Vietnamese. The bit cost is the same on both sides, 4 bits plus one scale per group. A grid placed according to the normal distribution puts more levels where most weights are, so fewer weights round to 0 (10% versus 16.6%).
• Perplexity hides many changes. NF4 with groups of 64 raises English bpb by only 5%, a number that sounds acceptable, yet almost 20% of positions change their top-1 token.
• Vietnamese loses more than English in every usable 4-bit configuration (except the per-tensor one, which is broken outright), by all three measures. With per-channel INT4, Vietnamese bpb rises 34% while English rises 21%; flips are 58% versus 43%. This is the same content in two languages, so the difference does not come from the difficulty of the text.

Three limits should be stated clearly. This is a 0.5B model, far more sensitive to quantization than models of 7B and up. The hand-written fake-quant has none of the techniques of GPTQ or AWQ, so it is a lower bound on 4-bit quality. And the top-1 flip here is computed over every position of the text, not over the answers of a task, so it does not directly say how many more questions the model gets wrong.

Evidence from the papers

Papers on larger models and real tasks paint the same picture:

• Accuracy hides flips. Dutta et al. 2024 (“Accuracy is Not All You Need”) measure the flip rate: the percentage of answers that change from correct to incorrect and from incorrect to correct between the original and the compressed model (section 2). Accuracy moves by only 0–2% while flips reach 13.6% (section 1, Figure 1), because the two directions of flips cancel out in the average. On MMLU, 4-bit bitsandbytes quantization gives 8–16% flips; on GSM8K, 12–30%, a figure that pools BnB 8-bit and 4-bit (sections 4.1, 4.4).
• Non-Latin-script languages lose more, and automatic metrics underestimate the damage. Marchisio et al. 2024 find the impact is worst for languages not written in Latin script, and an average 1.7% drop on automatic tasks in Japanese corresponds to a 16.0% drop when human raters evaluate realistic prompts (abstract; section 4.7, Table 8).
• Models trained on more tokens are more sensitive. Huang et al. 2024 find that Llama 3 degrades noticeably at low bit widths (abstract). With the same 4-bit RTN quantization at group size 128, LLaMA3-8B’s WikiText2 perplexity rises from 6.1 to 8.5, while LLaMA2-7B only goes from 5.5 to 5.7 (v3, Table 1 in section 2 and Table 5). Kumar et al. 2024 put this into a scaling law: more pretraining data can make the model after quantization worse (abstract, section 3.1, Figure 2); their experiments only go up to 1.7B parameters and 26B tokens, so this is a trend, not a number for large models. This is the flip side of training far past Chinchilla-optimal, discussed in part 10, section 6.
• In the other direction, 4 bits is usually a good balance. Dettmers & Zettlemoyer 2022 show that for a fixed total number of bits, a larger model at 4 bits almost always beats a smaller model at 16 bits (abstract, Figure 1). At 3 bits, this relationship reverses.

A checklist before using a quantized model

1. Measure on your own task, not just perplexity or a generic benchmark. If the application is in Vietnamese, measure in Vietnamese.
2. Measure flips, not just accuracy: run the same set of questions through the original and the quantized model, and count the answers that change, in both directions.
3. Compare at the same bit cost: compare Q4_K_M with another model’s Q4_K_M, or a large model at 4 bits with a small model at 16 bits using the same memory.
4. Check output formats and long tasks: tool calls, JSON and multi-step reasoning usually break before short question answering does.
5. If you quantize yourself, keep the calibration set and record the method, group size and library versions, so you can reproduce it later.

09Common misconceptions

The two outer rows of Figure 1 are eight misconceptions, each linked to the step it misunderstands:

Summary: two knobs, each cutting a different part of memory

Full fine-tuning an 8B model needs about 128 GB for the model states alone. There are two ways to cut it.

• LoRA cuts the gradients, master weights and Adam state of the base, 12 of the 16 bytes per parameter, by freezing W₀ and learning only ΔW = (α/r)·B·A. B = 0 at initialization, so training starts exactly at the base; once merged, inference is no slower.
• The ΔW of real full fine-tuning is usually not low-rank in the energy sense: for Qwen2.5-0.5B-Instruct, mlp.down_proj needs nearly 700 of 896 singular values to keep 90% of the energy. But the part that matters for behavior sits in the top few directions: keeping rank 8 (7.6% of the energy) already halves the KL to Instruct. So a low-rank update is good enough when the data is moderate, and worse when teaching a whole new domain.
• Attach LoRA to every linear layer, especially the FFN; use a learning rate about 10× that of full fine-tuning; for DPO, turning the adapter off gives you the reference.
• Quantization cuts the weights from 2 bytes to about 0.5 bytes. The hard part is outliers, and every good method uses small per-group scales: “4-bit” is really 4.1–4.9 bits.
• GPTQ and AWQ are algorithms, GGUF is a file format. Weight-only quantization speeds up decode because decode is bound by memory bandwidth, but it does not help prefill.
• QLoRA combines both: an NF4 base with double quantization, a 16-bit adapter, and paged optimizers. On an RTX 2080 Ti, Qwen2.5-1.5B trains with a peak of 5.46 GiB VRAM, about 25% slower than plain LoRA.
• The cost of quantization has to be measured on your own task and language. Perplexity stays almost the same while tens of percent of top-1 tokens have changed, and Vietnamese loses more than English.

The weights are now four times smaller. But when the model serves many users at once, memory gets eaten by the KV cache, which grows with the number of requests and the context length. And inference splits into two very different phases, prefill, bound by compute, and decode, bound by memory bandwidth, as the llama.cpp table in section 6 already hinted. Part 13 will cover those two phases.

References

Papers

• Houlsby et al. 2019 — Parameter-Efficient Transfer Learning for NLP
• Bengio et al. 2013 — Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Jacob et al. 2018 — Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference
• Aghajanyan et al. 2020 — Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning
• Li & Liang 2021 — Prefix-Tuning: Optimizing Continuous Prompts for Generation
• Lester et al. 2021 — The Power of Scale for Parameter-Efficient Prompt Tuning
• Hu et al. 2021 — LoRA: Low-Rank Adaptation of Large Language Models
• Dettmers et al. 2022 — LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
• Frantar et al. 2022 — GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
• Xiao et al. 2022 — SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
• Dettmers & Zettlemoyer 2022 — The case for 4-bit precision: k-bit Inference Scaling Laws
• Dettmers et al. 2023 — QLoRA: Efficient Finetuning of Quantized LLMs
• Lin et al. 2023 — AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
• Sheng et al. 2023 — S-LoRA: Serving Thousands of Concurrent LoRA Adapters
• Kalajdzievski 2023 — A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA (rsLoRA)
• Liu et al. 2024 — DoRA: Weight-Decomposed Low-Rank Adaptation
• Hayou et al. 2024 — LoRA+: Efficient Low Rank Adaptation of Large Models
• Meng et al. 2024 — PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models
• Huang et al. 2024 — An Empirical Study of LLaMA3 Quantization: From LLMs to MLLMs (v1 was titled “How Good Are Low-bit Quantized LLaMA3 Models? An Empirical Study”)
• Biderman et al. 2024 — LoRA Learns Less and Forgets Less
• Marchisio et al. 2024 — How Does Quantization Affect Multilingual LLMs?
• Dutta et al. 2024 — Accuracy is Not All You Need
• Kumar et al. 2024 — Scaling Laws for Precision
• Qwen Team 2024 — Qwen2.5 Technical Report

Blogs and documentation

• John Schulman and Thinking Machines Lab — LoRA Without Regret (2025)
• Hugging Face — PEFT: LoRA and the LoraConfig reference
• Hugging Face — bitsandbytes and Quantization in transformers
• Hugging Face — TRL: DPO Trainer
• llama.cpp — llama-quantize README (commit 5e03bdd, 6 Oct 2026) and PR #1684: k-quants
• Hugging Face — Qwen/Qwen2.5-0.5B, Qwen/Qwen2.5-0.5B-Instruct, Qwen/Qwen2.5-1.5B
• Hugging Face — the facebook/belebele dataset (FLORES-200 passages) and Salesforce/wikitext