One linear layer in QLoRA and the memory of three fine-tuning setups

One linear layer in QLoRA and the memory of three fine-tuning setups A workflow diagram generated by Archify. 01 / Frozen W₀ branch (main path) 02 / LoRA branch, with gradients 03 / Model-state memory, excluding activations Activation x · fp16 or bf16 · Frozen W₀ branch (main path) Activation x fp16 or bf16 W₀ stored as NF4 · block 64, double quant · Frozen W₀ branch (main path) W₀ stored as NF4 block 64, double quant Dequantize · to compute dtype · Frozen W₀ branch (main path) Dequantize to compute dtype W₀·x · W₀ is not updated · Frozen W₀ branch (main path) W₀·x W₀ is not updated Add · W₀x + (α/r)·BAx · Frozen W₀ branch (main path) Add W₀x + (α/r)·BAx Output h · to the next layer · Frozen W₀ branch (main path) Output h to the next layer A: r × k · random init · LoRA branch, with gradients A: r × k random init B: d × r · initialized to 0 · LoRA branch, with gradients B: d × r initialized to 0 Full FT ≈ 128 GB · 16 bytes per parameter · Model-state memory, excluding activations Full FT ≈ 128 GB 16 bytes per parameter LoRA ≈ 17 GB · bf16 base + adapter · Model-state memory, excluding activations LoRA ≈ 17 GB bf16 base + adapter QLoRA ≈ 6.4 GB · NF4 base + adapter · Model-state memory, excluding activations QLoRA ≈ 6.4 GB NF4 base + adapter drop W₀ state base 2 → 0.5 bytes Legend LoRA adapter, with gradients forward computation activations in and out frozen weights stored as NF4 model-state memory, Llama 3.1 8B