Deploying a quantized LLM: GPTQ, AWQ, speculative decoding, and where it breaks

Quantization’s core mechanism doesn’t change once a model is trained: round a high-precision number to a lower-precision grid, accept some error, save memory. What changes at deployment time is the question being asked. During training you’re worried about gradients surviving the round trip; once training is done and you have a finished model, the question becomes purely economic: how much can you shrink these weights, and sometimes the model’s intermediate activations, to run inference cheaper and faster, for a small and measured amount of quality loss.

One piece of vocabulary is worth pinning down before going further, because a failure mode later in this post doesn’t parse without it. A transformer generates text one token at a time, and at every step, attention needs the Key and Value vectors of every previous token in the sequence. Recomputing those from scratch at each new token would be enormously wasteful, so the KV cache simply stores them once and reuses them at every subsequent step.

The economics: why this is a line item, not a nicety#

INT8 weights give roughly 2x memory reduction for a 1-2% quality cost. INT4 gives roughly 4x memory reduction for a 3-5% quality cost. A 70-billion-parameter model in BF16 is 140GB, needing two H100s just to hold the weights; the same model in INT4 is 35GB, fitting on one. That’s not just “a bit cheaper,” it’s four times the request throughput on identical hardware, which is the actual sentence a company’s infrastructure budget is built from.

def serving_capacity(total_gpu_memory_gb, model_params, bytes_per_param):
    memory_per_replica_gb = (model_params * bytes_per_param) / 1e9
    return total_gpu_memory_gb / memory_per_replica_gb

configs = {"BF16 (2 bytes)": 2.0, "INT8 (1 byte)": 1.0, "INT4 (0.5 bytes)": 0.5}
for name, bpp in configs.items():
    print(name, serving_capacity(140, 70e9, bpp), "replicas in a 140GB (2xH100) budget")

# BF16 (2 bytes): 1.0 replicas
# INT8 (1 byte):  2.0 replicas
# INT4 (0.5 bytes): 4.0 replicas

One 140GB budget holds exactly one BF16 replica, two INT8 replicas, or four INT4 replicas. The “4x more requests on the same hardware” claim isn’t a vague scaling argument, it’s this exact division.

GPTQ: quantize one column, measure the damage, compensate the rest#

The naive scheme from the earlier post picked one scale for a tensor and rounded every weight independently, accepting whatever error fell out. GPTQ does something more deliberate: it minimizes the actual output error a layer produces on realistic data, not just how far the rounded weights are from the originals.

$\min_{\hat W} \|WX - \hat W X\|_2^2$

$W$ is the original full-precision weight matrix, $X$ is a calibration batch of real activations passed through the layer, and $\hat W$ is the quantized matrix, constrained to the INT4/INT8 grid. This objective cares about the layer’s output on data that matters, not the weights themselves in isolation; two weight matrices that look quite different entry-by-entry can produce nearly identical outputs on the inputs the model actually sees. GPTQ solves this column by column: quantize one column, measure precisely how much error that introduced, adjust the still-unquantized columns to partially cancel it, then quantize the next column against that adjusted baseline. It’s a sequential, explicit error-compensation loop, not a one-shot rounding pass.

AWQ: protect the outliers instead of splitting them out#

The earlier post’s LLM.int8() fix handled outlier channels by routing them through a separate FP16 pathway while the rest went to INT8. AWQ applies the same underlying insight, that a small fraction of weights matter disproportionately, differently: it identifies the roughly 1% of salient weights ahead of time and protects them before quantization, rather than splitting the computation into two precision pathways at inference time. Same diagnosis as before, a cheaper prescription.

Speculative decoding: spending idle compute instead of idle time#

This one isn’t a quantization technique at all, it just happens to live in the same “make inference cheaper” bucket. A small, cheap draft model proposes several tokens at once, say 4; the large target model verifies the entire batch in a single forward pass, accepting whichever prefix matches its own distribution and discarding the rest, then regenerating from the last accepted token.

The reason this is close to free rather than a genuinely separate cost traces straight back to the compute-bound-vs-memory-bound framework: a normal single-token decode step is memory-bound, meaning the GPU has spare compute capacity sitting idle while it waits on memory bandwidth. Verifying 4 draft tokens in one batched pass costs roughly the same as generating 1 token normally, both are one forward pass through the same 70B model, so speculative decoding is filling compute that was going to be wasted anyway rather than buying new compute.

If $\alpha$ is the per-token acceptance probability and the draft proposes $k$ tokens per round, the expected number of accepted tokens per round is:

$\sum_{i=1}^{k} \alpha^i$

Each additional token is only accepted if every token before it in the round was, which is exactly the geometric-series shape that formula has.

def expected_accepted_tokens(acceptance_rate, k_drafted):
    if acceptance_rate >= 1.0:
        return k_drafted
    return sum(acceptance_rate ** i for i in range(1, k_drafted + 1))

for rate in [0.70, 0.75, 0.80]:
    print(rate, expected_accepted_tokens(rate, k_drafted=4))

# 0.70 -> 1.77 tokens accepted per round of 4 drafted
# 0.75 -> 2.05
# 0.80 -> 2.36

At a typical 70-80% acceptance rate this produces roughly a 2 to 2.5x wall-clock speedup in practice; the exact multiplier also depends on how expensive the draft model itself is to run, which this formula alone doesn’t capture, so treat these numbers as the accepted-token count the speedup is built from rather than the speedup itself.

Real numbers, not hypotheticals#

DeepSeek-V3, served with FP8 quantization, reportedly sustains around 73,700 input tokens per second per node, a concrete published throughput figure directly attributable to these techniques rather than a projection. And if you’re looking for where to start hands-on with any of this, LLM.int8() from the earlier post is the commonly recommended entry point precisely because it requires minimal resources while exposing the full quality-versus-performance tradeoff in one contained example.

Three ways this breaks in production, not in a benchmark#

INT4 plus low-temperature sampling on code generation. Quantization noise shifts the model’s probability rankings slightly, which barely matters when sampling is random but matters a great deal when decoding is near-greedy: an 8% HumanEval drop at temperature 0.1, versus only a 1% drop at temperature 1.0. The same quantized model can look nearly lossless or clearly degraded depending purely on the sampling settings it’s served at, which a training-time-only evaluation would never surface.

Speculative decoding after a model update. The speedup depends entirely on the draft and target models agreeing often. If the target model gets updated and the draft model doesn’t keep pace, acceptance rate can collapse, say from 78% down to 35%, at which point speculative decoding stops being a speedup and becomes a net regression versus not using it at all. It’s a conditional win, not an unconditional one, and a routine update elsewhere in the stack can silently break the condition.

INT8 KV cache on long contexts. Because the cache stores every previous token’s Key/Value vectors and reuses them at every later step, a small rounding error introduced once doesn’t stay isolated, it gets read again at every subsequent step. At 128K tokens, the same small per-token error has had 128,000 opportunities to compound, which is exactly why this failure shows up specifically on long contexts rather than uniformly.

Failure	Trigger	Why it happens
~8% HumanEval drop	INT4 weights + low temperature (0.1)	Near-greedy decoding is far more sensitive to small probability-ranking errors than random sampling
Speculative decoding regression	Target model updated, draft model left stale	Draft/target distribution mismatch collapses the acceptance rate
Long-context quality decay	INT8 KV cache + 128K+ token sequences	Per-token rounding error compounds every time the cached value is reused

Common mistakes#

Evaluating a quantized model only at the sampling temperature used during benchmarking, then serving it at a different one: the HumanEval gap above shows this isn’t a small effect, it’s nearly an order of magnitude difference in apparent quality loss.

Treating speculative decoding as a fixed, permanent speedup rather than a measurement that needs re-checking after either model in the pair changes: the regression case above isn’t rare, it’s the predictable consequence of letting draft and target drift apart.

Applying the same quantization precision uniformly to weights and to the KV cache without separately considering context length: a precision that’s perfectly safe for weights can still be the wrong choice for a cache that gets read tens of thousands of times per sequence.

Assuming post-training quantization is a one-shot, independent-per-weight rounding step: GPTQ’s whole design is a rebuttal to that assumption, sequential, calibrated, and measured against real activations rather than against the weights in isolation.

Try it yourself#

Beginner. Using serving_capacity above, compute how many 70B replicas fit in a 320GB (4xH100) budget at BF16, INT8, and INT4, and confirm the ratios match the 2x and 4x reduction claims.

Intermediate. Using expected_accepted_tokens, compute the accepted-token count at acceptance rate 0.78 versus 0.35, both at $k=4$, and describe in your own words why the second number represents enough of a regression to call it a net loss rather than just “less of a speedup.”

Advanced. GPTQ’s objective is $\min\|WX-\hat WX\|_2^2$, but the column-by-column-with-compensation procedure that solves it isn’t derived above, only described. Work out why compensating the remaining, still-unquantized columns for one column’s rounding error produces a better solution to that objective than quantizing every column independently, in terms of the matmul mechanics from earlier in this series.

---

The one-sentence version: deployment-time quantization is the same rounding-and-error tradeoff as training-time quantization, aimed at a different target, GPTQ and AWQ both spend more care than naive rounding to decide which error to accept, speculative decoding is nearly free because normal decoding wastes compute anyway, and every one of these techniques has a specific, named condition under which it quietly stops being a win, which is exactly why none of them are “set once and forget.”