Three branches from one base model: LoRA, QLoRA, PTQ and common misconceptions

Three branches from one base model: LoRA, QLoRA, PTQ and common misconceptions A workflow diagram generated by Archify. EX / LoRA misconceptions 02 / LoRA branch 03 / QLoRA (main path) 04 / PTQ for deployment EX / Quant misconceptions Bigger r is better · small r is often enough · LoRA misconceptions Bigger r is better small r is often enough Always worse/equal · vs full FT: it depends · LoRA misconceptions Always worse/equal vs full FT: it depends Model becomes low-rank · only ΔW is low-rank · LoRA misconceptions Model becomes low-rank only ΔW is low-rank QLoRA gives 4-bit LLM · it yields an adapter · LoRA misconceptions QLoRA gives 4-bit LLM it yields an adapter Freeze W₀ · attach B·A, pick r, α · LoRA branch Freeze W₀ attach B·A, pick r, α Train A, B · no gradient for base · LoRA branch Train A, B no gradient for base Merge / keep adapter · W₀ + (α/r)·BA · LoRA branch Merge / keep adapter W₀ + (α/r)·BA Base model bf16 · weights W₀ · QLoRA (main path) Base model bf16 weights W₀ Quantize base · NF4 + double quant · QLoRA (main path) Quantize base NF4 + double quant Train LoRA · dequantize on forward · QLoRA (main path) Train LoRA dequantize on forward Adapter + NF4 base · 16-bit adapter · QLoRA (main path) Adapter + NF4 base 16-bit adapter Calibration (optional) · GPTQ, AWQ, imatrix · PTQ for deployment Calibration (optional) GPTQ, AWQ, imatrix GPTQ / AWQ / k-quant · grouped, with scales · PTQ for deployment GPTQ / AWQ / k-quant grouped, with scales GGUF, safetensors · file formats · PTQ for deployment GGUF, safetensors file formats Inference · llama.cpp, vLLM · PTQ for deployment Inference llama.cpp, vLLM Re-measure · your task, your language · PTQ for deployment Re-measure your task, your language 4-bit means 4 bits · scales cost extra bits · Quant misconceptions 4-bit means 4 bits scales cost extra bits GGUF is a method · it is a file format · Quant misconceptions GGUF is a method it is a file format Always faster · depends on kernel, phase · Quant misconceptions Always faster depends on kernel, phase Same PPL = no loss · answers still change · Quant misconceptions Same PPL = no loss answers still change Legend training or quantization step input and output models calibration data and measurement storage format common misconception