LoRA's own paper claims a 10,000x reduction in trainable parameters and a 3x reduction in GPU memory for GPT-3, and working the real arithmetic for a 70B model's actual dimensions shows why the honest number for a typical production setup is closer to 1,700x, not the maximal headline figure, which is exactly why QLoRA's further trick, quantizing the frozen weights to 4 bits, is what actually lets a 65B model fine-tune on one 48GB GPU

Supervised fine-tuning was introduced earlier in this series as the least controversial step in turning a base model into an assistant: continue training on a smaller, higher-quality dataset with the same cross-entropy loss pretraining used. What that post didn’t need to cover, because RLHF’s memory problem is dominated by holding four models at once, is the question that determines whether any fine-tuning step, SFT, a DPO run, a narrow domain adaptation, is affordable at all: do you update every one of a model’s parameters, or a much smaller number sitting next to them? That question has a real, worked-out answer, and the honest version of that answer is more interesting than the paper’s own headline number, because the headline number is the best case, not the typical one.

Full fine-tuning’s real cost, restated for one number#

Every parameter you intend to update needs the same four tensors this series already priced out: FP32 weights, FP32 gradients, and Adam’s first and second moments, 16 bytes per parameter, full stop. For a 70B model, that’s

$70 \times 10^9 \times 16 \text{ bytes} = 1{,}120\text{ GB}$

before a single activation or KV-cache byte, and before any of the sharding tricks (ZeRO, FSDP) that spread that number across GPUs rather than shrinking it. That figure is the entire reason parameter-efficient fine-tuning (PEFT) exists as a category: if you could full-fine-tune every model you wanted to adapt, on hardware you actually have, nobody would have needed to invent an alternative.

LoRA: freeze W, train a rank-r update next to it#

Hu et al.’s 2021 LoRA paper starts from a real, separately-established observation: Aghajanyan et al. had already shown that the change a pretrained model needs during adaptation to a new task has a surprisingly low intrinsic dimension, even though the model itself has billions of parameters. LoRA takes that observation and turns it into an architecture. For any pretrained weight matrix $W_0 \in \mathbb{R}^{d\times k}$, instead of updating $W_0$ directly, freeze it completely and add a separate, low-rank update:

$h = W_0 x + \Delta W x = W_0 x + \frac{\alpha}{r}BAx$

where $B \in \mathbb{R}^{d\times r}$, $A \in \mathbb{R}^{r\times k}$, and the rank $r \ll \min(d,k)$. $A$ is initialized from a random Gaussian and $B$ from zero, so at the start of training $\Delta W = 0$ and the model is exactly the pretrained one; every gradient step only ever touches $A$ and $B$, never $W_0$. The parameter count for one such matrix drops from $d\times k$ to $r\times(d+k)$, and because $r$ is typically 4 to 64 against hidden dimensions in the thousands, that ratio is enormous.

Checking the paper’s own headline number against real dimensions#

The LoRA paper’s own abstract states two round numbers: a 10,000x reduction in trainable parameters and a 3x reduction in GPU memory, both for GPT-3 175B. Those numbers are real, but they describe the paper’s minimal configuration, LoRA applied to just the query and value projection matrices, at rank $r=4$. It’s worth actually running that arithmetic rather than repeating the round number, because the honest answer for a more typical production configuration is meaningfully different.

Using GPT-3’s real published dimensions, $d_{model}=12{,}288$, 96 layers, LoRA on $W_q$ and $W_v$ only, at $r=4$:

d_model, n_layers, r = 12288, 96, 4
matrices_targeted = 2                     # Wq, Wv only — the paper's minimal config
trainable = n_layers * matrices_targeted * r * (d_model + d_model)
full_finetune = 175e9

print(f"{trainable/1e6:.2f}M trainable")   # 18.87M trainable
print(f"{full_finetune/trainable:.0f}x")   # 9272x

9,272x, in the same regime as the paper’s stated 10,000x, close enough that the difference is almost certainly rounding and the exact parameter count GPT-3 actually has (commonly cited as 174.6B, not an even 175B). That’s a genuine, worth-having confirmation. But now do the same arithmetic for a more realistic production choice: all four attention projections ($W_q, W_k, W_v, W_o$), rank $r=8$, on a 70B model with Llama 2’s real published dimensions ($d_{model}=8{,}192$, 80 layers):

d_model, n_layers, r = 8192, 80, 8
matrices_targeted = 4                     # Wq, Wk, Wv, Wo — a typical real setup
trainable = n_layers * matrices_targeted * r * (d_model + d_model)
full_finetune = 70e9

print(f"{trainable/1e6:.2f}M trainable")   # 41.94M trainable
print(f"{full_finetune/trainable:.0f}x")   # 1669x

1,669x, still enormous, but nowhere near 10,000x. The gap isn’t an error, it’s the real, disclosed shape of the tradeoff: targeting more weight matrices at a higher rank uses more trainable parameters in exchange for closing more of the quality gap to full fine-tuning, and the paper’s own headline number describes the smallest, most aggressive end of that tradeoff, not the setting most teams actually ship with. Quoting “10,000x” without checking which configuration produced it is exactly the kind of number that sounds right and is technically true while describing a corner case.

Which matrices matter more than rank#

The LoRA paper’s own ablation is worth having on hand because it argues against the intuition that “more rank is always better.” Comparing a fixed trainable-parameter budget spent as high rank on few matrices versus low rank spread across more matrices, the paper reports that applying LoRA to all four projection matrices ($W_q,W_k,W_v,W_o$) at a small rank outperforms concentrating the same total budget into just $W_q$ and $W_v$ at a proportionally higher rank. The practical read: which matrices you target is a more important decision than how high you set $r$ once you’re past a small minimum, a genuinely non-obvious result if your mental model of “rank” is “more capacity is strictly better.”

The memory bill, all three ways, for the same 70B model#

Putting LoRA’s memory savings next to full fine-tuning, using this series’ own 16-bytes-per-parameter convention and the same 2-bytes-per-parameter bf16 convention already used for frozen, inference-only models in the RLHF post:

n_params = 70e9

full_finetune_gb = n_params * 16 / 1e9                     # every param gets Adam's full 16 bytes
lora_gb = (n_params * 2 + 41.94e6 * 16) / 1e9               # frozen base at bf16 + tiny trainable state
qlora_gb = (n_params * 0.5 + 41.94e6 * 16) / 1e9            # frozen base at 4-bit NF4 + same tiny trainable state

print(f"Full fine-tune: {full_finetune_gb:.0f} GB")   # 1120 GB
print(f"LoRA:           {lora_gb:.1f} GB")            # 140.7 GB
print(f"QLoRA:          {qlora_gb:.1f} GB")            # 35.7 GB

Worth being honest about rather than glossing over: this series’ own 1,120 GB → 141 GB comparison shows roughly an 8x reduction from full fine-tuning to LoRA, steeper than the paper’s stated 3x. That gap is real and explainable, not a mistake: the paper’s own reported ratio is measured on their actual training setup, where activation memory (which LoRA does not reduce, because gradients still have to flow backward through every frozen layer to reach the trainable adapters) makes up a much larger share of the total than in this back-of-envelope, state-only accounting. The honest takeaway generalizes: LoRA and QLoRA shrink optimizer and weight-storage memory dramatically, but activation memory is governed by batch size and sequence length exactly as it is in full fine-tuning, so very long sequences or very large batches can still push a “small” LoRA run into a memory wall that has nothing to do with how few parameters are trainable.

QLoRA: quantize the frozen part too#

Dettmers et al.’s 2023 QLoRA paper asks the obvious next question: if the frozen base weights in LoRA never receive a gradient, why store them at 16-bit precision at all? QLoRA’s answer combines three real, separately-motivated techniques:

• 4-bit NormalFloat (NF4). A new data type, information-theoretically optimal specifically for weights that are normally distributed, which pretrained neural network weights empirically are. Rather than a generic 4-bit float, NF4’s quantization bins are placed to match a Gaussian distribution’s actual density.
• Double quantization. The quantization constants themselves (the per-block scaling factors NF4 needs to map back to real values) also take memory, roughly 0.5 bytes per parameter at typical block sizes. QLoRA quantizes those constants a second time, a real, disclosed additional saving of about 0.37 bits per parameter on average, small per-parameter but material at 70B+ scale.
• Paged optimizers. Borrowing NVIDIA’s unified CPU-GPU memory paging, so that the rare, transient memory spikes gradient checkpointing causes get paged out to CPU RAM automatically instead of triggering an OOM crash, without the programmer managing that transfer by hand.

Running the same memory arithmetic for the case QLoRA was built to solve, a 65B model on a single 48GB consumer-class GPU:

n_params = 65e9
frozen_nf4_gb = n_params * 0.5 / 1e9              # 4 bits = 0.5 bytes/param
trainable_state_gb = 40e6 * 16 / 1e9              # a LoRA-sized adapter, full 16-byte state

print(f"Frozen base (NF4): {frozen_nf4_gb:.1f} GB")   # 32.5 GB
print(f"Trainable adapter: {trainable_state_gb:.2f} GB")  # 0.64 GB
print(f"Total:             {frozen_nf4_gb + trainable_state_gb:.1f} GB")  # 33.1 GB

33.1 GB, comfortably inside a 48GB card with real headroom left for activations and the paged-optimizer overflow. That arithmetic is the entire reason QLoRA’s own reported result, their Guanaco model family reaching 99.3% of ChatGPT’s quality on the Vicuna benchmark from 24 hours of fine-tuning on one GPU, was possible at all: not a smarter training recipe, a memory bill that finally fit.

DoRA and rsLoRA: two real refinements past the original recipe#

Two follow-ups are worth knowing by name because both are now shipped, real, load-bearing options rather than academic curiosities:

DoRA (Liu et al., ICML 2024 oral) decomposes every pretrained weight matrix into a magnitude component and a direction component, $W = m \cdot \frac{V}{\|V\|}$, and applies LoRA’s low-rank update only to the direction, training the magnitude term separately and directly. The paper’s stated finding is that this decomposition makes LoRA’s own learning pattern resemble full fine-tuning’s more closely than plain LoRA’s does, and the paper reports DoRA consistently outperforming standard LoRA on LLaMA, LLaVA, and VL-BART across commonsense reasoning and visual instruction tuning, with no added inference cost, because the decomposition merges back into a single weight matrix after training exactly the way plain LoRA does.

rsLoRA (Kalajdzievski, 2023) fixes a subtler problem: the standard $\alpha/r$ scaling factor causes the gradient signal through $B\cdot A$ to shrink as $r$ grows, which means naively increasing rank to close the gap to full fine-tuning doesn’t actually help past a point, the larger adapter’s gradients go quietly small. Scaling by $\alpha/\sqrt{r}$ instead of $\alpha/r$ keeps the gradient magnitude roughly constant as rank increases, which is what makes “just use a higher rank” an option that actually works rather than one that silently stalls.

Catastrophic forgetting: the failure mode fine-tuning doesn’t fix by itself#

None of the memory-efficiency tricks above touch a separate, real problem: any further training on a narrow dataset can degrade a model’s general capability, a phenomenon with a real name, catastrophic forgetting, and real measured evidence. An empirical study of catastrophic forgetting during continual fine-tuning tested models from 1B to 7B parameters across domain knowledge, reasoning, and reading comprehension benchmarks, and found forgetting severity scaling with model size within that range, with knowledge-benchmark scores dropping sharply under continued fine-tuning. The same study reports a genuinely useful, non-obvious asymmetry: models that were already instruction-tuned (like Alpaca) forget less under further fine-tuning than base models (like the underlying LLaMA) that hadn’t been instruction-tuned first. Whatever instruction tuning does to a model’s representations appears to make them somewhat more robust to the next round of fine-tuning, not just more aligned.

This connects directly to a mechanism already derived in this series: InstructGPT’s PPO-ptx objective mixes pretraining log-likelihood gradients back into the RL objective at a tuned coefficient $\gamma=27.8$, specifically to buy back general capability the KL penalty alone doesn’t protect. That’s the same underlying problem, forgetting under adaptation, being solved with the same underlying idea, replay some of the original signal during the new training, one level up the post-training stack from where LoRA and QLoRA operate.

The tools real teams reach for instead of writing this from scratch#

Almost nobody hand-writes the LoRA forward pass and a custom training loop in production. Two open-source projects are worth knowing by name because they’re what real teams actually use:

LLaMA Factory, published at ACL 2024 and carrying 25,000+ GitHub stars, is a unified framework supporting 100+ base models through a single interface, with essentially every method mentioned above available as a configuration flag: full fine-tuning, freeze-tuning, LoRA, QLoRA (via AQLM/AWQ/GPTQ/LLM.int8/HQQ/EETQ, not just NF4), DoRA, rsLoRA, LongLoRA, PiSSA, GaLore, and reward modeling/PPO/DPO/KTO/ORPO for the post-training step that follows.

Unsloth takes a narrower, deeper approach: custom Triton kernels rewriting RoPE, the MLP block, and attention specifically for fine-tuning workloads, plus padding-free sequence packing (concatenating multiple training examples into one sequence instead of padding each to the batch’s longest example, a real, disclosed 3x speedup and 30% VRAM reduction from that technique alone). Unsloth’s own published numbers claim 2 to 5x faster training and 30 to 90% less VRAM depending on model and hardware, with concrete real figures like an 8B model’s QLoRA fine-tune fitting in roughly 6GB and a 70B model’s fitting in roughly 41GB.

When full fine-tuning still wins#

None of the above is an argument that PEFT is always correct. Full fine-tuning is still the better call when:

• The domain shift is large enough that the base model’s own representations need to change, not just be steered, a low-rank update by construction can’t express an arbitrary change to $W_0$, only a rank-bounded one.
• You need to merge or stack many task-specific adaptations, and keeping them as separate LoRA adapters causes interference or serving complexity; S-LoRA exists specifically because serving thousands of concurrent LoRA adapters is itself a real, nontrivial systems problem, not a free simplification.
• Compute genuinely isn’t the constraint, and a few points of quality matter more than training-time efficiency, DoRA’s own result, that it narrows but doesn’t fully close the gap to full fine-tuning, is the honest ceiling on how far parameter-efficiency alone can go.

Try it yourself#

Beginner. Using the trainable-parameter formula above, $r\times(d+k)$ per matrix, compute how many trainable parameters LoRA would need for a 7B model ($d_{model}=4096$, 32 layers) applying to all four attention matrices at $r=16$. What fraction of the model’s total parameters is that?

Intermediate. Rerun this post’s own memory-bill comparison for a 7B model instead of 70B, then add QLoRA’s double-quantization saving explicitly (0.37 bits/parameter on the frozen base) and check how much of a difference it actually makes at that smaller scale versus at 70B.

Advanced. rsLoRA’s fix keeps gradient magnitude roughly constant as rank increases by scaling with $\alpha/\sqrt{r}$ instead of $\alpha/r$. Derive why the original $\alpha/r$ scaling causes vanishing gradients as $r$ grows, starting from how $\partial h/\partial A$ and $\partial h/\partial B$ depend on $r$ through the forward equation given earlier in this post.

What this takes to be frontier-job-ready#

Poolside’s real, live posting for fine-tuning and post-training states its state-of-the-art bar directly: “Familiar with, or contributed to the state of the art in multiple of the following topics: Fine-tuning and alignment of LLMs, synthetic data generation, continual learning, RLVR, code generation.” Notice what’s absent from that list: it doesn’t say “knows how to call a LoRA library.” The bar is understanding why each of the choices this post walked through, rank, which matrices, quantization scheme, exists, well enough to make the next one when the library’s defaults don’t fit the problem in front of you.

---

The one-sentence version: LoRA’s real 10,000x parameter-reduction and 3x memory-reduction claims are true for GPT-3’s minimal two-matrix, rank-4 configuration, and working the same arithmetic for a realistic 70B production setup, four matrices, rank 8, honestly lands closer to 1,700x, still enormous, just not the maximal headline; QLoRA’s further trick, quantizing the frozen weights themselves to 4-bit NF4, is the concrete, arithmetic reason a 65B model now fits and fine-tunes on a single 48GB GPU instead of a cluster, and neither trick, nor DoRA’s or rsLoRA’s refinements on top of it, touches the separate, real problem of catastrophic forgetting, which is the same problem PPO-ptx was already built to solve one level up the stack.