The roofline model: why LLM decode uses 2% of a GPU's peak, and it's not a bug

Tensor cores were introduced earlier in this series as dedicated matmul-accumulate hardware, distinct from ordinary compute cores. This post gives that idea its actual architectural home, then uses it to answer a question that confuses people every time they first see it: why does an LLM generating text one token at a time use something like 2% of a GPU’s advertised peak throughput, and why is that completely expected rather than a sign something is broken.

What’s actually on the chip#

An SM (Streaming Multiprocessor) is the basic compute engine of an NVIDIA GPU. An H100 is essentially 132 of these replicated across the die, in the SXM5 variant specifically, out of 144 physically present on the full chip. The other 12 are disabled for manufacturing yield: dies with more fabrication defects get sold as the PCIe variant instead, with only 114 SMs enabled. This is worth knowing precisely because it’s a real, recurring source of confusion in practice: “the H100” isn’t one product, it’s a family, and citing a spec without saying which variant you mean is a common way engineers talk past each other in a capacity-planning conversation.

Each SM holds 128 FP32 CUDA cores and 4 fourth-generation tensor cores side by side, sharing the same fast on-chip memory and instruction scheduling. The tensor cores are a small fraction of an SM’s total unit count, but they do disproportionately more work per cycle on matmul-shaped operations specifically, which is the entire reason they exist as separate silicon rather than just more general-purpose cores.

Warps and occupancy are what make memory-bound work survivable at all. A warp is 32 threads executing the same instruction simultaneously. Occupancy is the fraction of an SM’s maximum warp capacity that’s actually active, and it matters specifically because it lets the SM hide memory latency: while one warp waits on a read from HBM, the scheduler switches to a different, ready warp instead of idling. Without enough warps in flight to switch between, every HBM read stalls the entire SM. This is why low occupancy alone can cost roughly 30% of a kernel’s peak throughput, with the compute units sitting idle waiting on memory that a better-occupied kernel would have hidden behind other warps’ work.

Shared memory is the fast tier that makes tiling possible, roughly 228KB per SM, on the order of 100x faster than HBM. This is exactly the memory tier the earlier tiling discussion described loading data into before computing, now placed precisely: it’s SM-local, not chip-wide, and its size relative to HBM’s 80GB is exactly why tiling, computing on small blocks that fit in this fast tier, is the entire discipline of writing a fast GPU kernel.

HBM3, for the SXM5 variant specifically, is 80GB at 3.35 TB/s. The PCIe variant, being a different product, has different memory numbers too, the same variant-confusion problem as the SM count above.

The roofline model, derived to an exact number#

The roofline model is the formal, plottable version of arithmetic intensity, giving that ratio an actual ceiling and a name for the point where a kernel crosses from one bottleneck to the other:

$P_{\text{attainable}} = \min(P_{\text{peak}}, \; I \times B_{\text{peak}})$

$P_{\text{peak}}$ is the hardware’s peak compute throughput for a given precision. $I$ is arithmetic intensity, FLOPs per byte moved. $B_{\text{peak}}$ is peak memory bandwidth. A kernel’s performance is capped by whichever term is smaller. At low intensity, $I \times B_{\text{peak}}$ is the binding constraint, the kernel is memory-bound, and performance scales linearly with intensity because it isn’t yet supplying enough work per byte to saturate the compute units. Past a critical intensity, the ridge point, $P_{\text{peak}}$ becomes binding instead, and no amount of additional intensity helps further.

$I_{\text{ridge}} = \frac{P_{\text{peak}}}{B_{\text{peak}}} = \frac{989 \times 10^{12}}{3.35 \times 10^{12}} \approx 295.2 \text{ FLOPs/byte}$

That’s H100 SXM5 at FP16 dense precision specifically. Change precision or hardware generation and the ridge point moves; FP8 roughly doubles $P_{\text{peak}}$, shifting the ridge point higher. This number is not a universal constant, it’s this exact chip’s exact configuration.

A precision and sparsity trap worth naming, because it’s a real, common mistake. Vendor spec sheets often headline a number like “3,958 TFLOPS” for the H100. That figure isn’t a competing or contradictory number to the 989 used above, it’s four steps up the same precision/sparsity ladder: 989 dense FP16, roughly 1979 sparse FP16 or dense FP8, 3958 sparse FP8. Citing “the H100’s TFLOPS” without stating both precision and sparsity configuration is close to meaningless, and it’s exactly the kind of gap that produces a wildly wrong capacity estimate when someone builds a spreadsheet off the flashiest number on the page instead of the one that matches their actual workload.

Why decode uses almost none of the peak, worked precisely#

def roofline_attainable_performance(peak_flops_per_sec, peak_bandwidth_bytes_per_sec, arithmetic_intensity):
    memory_bound_ceiling = arithmetic_intensity * peak_bandwidth_bytes_per_sec
    attainable = min(peak_flops_per_sec, memory_bound_ceiling)
    verdict = "memory-bound" if attainable < peak_flops_per_sec else "compute-bound"
    pct_of_peak = 100 * attainable / peak_flops_per_sec
    return attainable, verdict, pct_of_peak

H100_PEAK_FLOPS = 989e12   # FP16 dense, SXM5
H100_PEAK_BW = 3.35e12     # bytes/sec, SXM5

for name, intensity in {"Decode, low end": 5, "Decode, high end": 10, "Large training matmul": 1000}.items():
    attainable, verdict, pct = roofline_attainable_performance(H100_PEAK_FLOPS, H100_PEAK_BW, intensity)
    print(f"{name}: I={intensity} -> {attainable/1e12:.2f} TFLOPS, {verdict}, {pct:.2f}% of peak")

# Decode, low end:  I=5    -> 16.75 TFLOPS, memory-bound,  1.69% of peak
# Decode, high end: I=10   -> 33.50 TFLOPS, memory-bound,  3.39% of peak
# Large training matmul: I=1000 -> 989.00 TFLOPS, compute-bound, 100.00% of peak

Generating one new token during decode requires reading essentially every weight matrix in the model once, a memory operation, to perform comparatively few matrix-vector multiplications, a compute operation, landing at roughly 2 to 10 FLOPs/byte. That’s nowhere near the 295.2 ridge point, so decode sits at somewhere between 1.7% and 3.4% of the chip’s peak compute capability. Not because the chip is bad or the kernel is badly written, but because decode’s fundamental data-access pattern is memory-bound by nature. Contrast a large training matmul at an intensity around 1000 FLOPs/byte, achievable with large batch sizes where the same weights get reused across many examples before being re-read: that lands squarely past the ridge point, at 100% of peak.

The single fact that explains three earlier posts at once#

This is worth sitting with, because it retroactively justifies content that looked like three separate techniques. Decode being fundamentally memory-bound is the actual reason FlashAttention reduces HBM reads rather than reducing FLOPs, the actual reason a KV cache exists (avoid re-reading what’s already been read), and the actual reason quantization for inference specifically helps: smaller weights mean fewer bytes to move, which directly raises arithmetic intensity and pushes a kernel closer to the ridge point from below. Three techniques, taught separately, are mechanistically three different answers to the exact same roofline position.

The ceiling isn’t the floor, and that gap is where careers get spent#

The roofline model gives you a theoretical maximum, not a guarantee. Real workloads routinely fall short of it for reasons the two-line equation doesn’t capture: kernel launch overhead, synchronization barriers, partial-tile waste at odd matrix dimensions, register spilling, and imperfect overlap between compute and memory operations. One recent study found achieved BF16 GEMM throughput on a modern GPU varying by 30% or more between adjacent matrix-dimension configurations differing by a single 128-element step, meaning two matrix sizes that look nearly identical on paper can perform wildly differently in practice for reasons that have nothing to do with the roofline ceiling itself. Closing that gap between the roofline’s promise and a kernel’s actual measured throughput is genuinely most of what performance engineering work looks like day to day: not deriving a new ceiling, but finding and removing whatever is keeping a specific kernel far below the one it already has.

In production this gets measured with tools like NVIDIA Nsight Compute, which reads live hardware counters to compute a kernel’s actual achieved arithmetic intensity and throughput directly from silicon, then plots it against the vendor-published or empirically measured roofline, the same equation from above, driven by measurement instead of a hand-typed estimate. A rough version of the same check is fast enough to do in your head before committing to any specific optimization: back-of-envelope arithmetic intensity, compared against the ridge point, tells you which entire category of fix is even relevant, tiling and occupancy work if you’re memory-bound, a lower-precision format with hardware support helps if you’re compute-bound, before you spend an afternoon on the wrong one. Some accounts attribute exactly this discipline, a fast roofline-style mental calculation before any optimization decision, to a performance lead at a frontier lab; I couldn’t independently confirm the specific attribution, so take the named source with a grain of salt, but the practice itself, check which side of the ridge point you’re on before you optimize, is well documented across the performance-engineering literature independent of who said it first.

Failure modes, each traced to its mechanism#

Failure	Mechanism	Cost
Low occupancy	Thread block sized too small, most SMs idle instead of switching between warps	~30% of peak throughput
Warp divergence	An if/else inside a warp forces half the threads to wait while the other half executes, then reverses	~50% efficiency loss
False memory-bound diagnosis	Counting only weight-matrix FLOPs and bytes while ignoring the intermediate $S \times S$ attention-score matrix as a memory cost of its own	Misses the real bottleneck entirely, exactly what FlashAttention’s core insight corrects
Precision/sparsity confusion	Citing peak TFLOPS without stating precision and sparsity configuration	Any roofline built on the wrong peak gives a wrong ridge point and a wrong verdict

This connects to something already established: HBM3 memory failures caused 17.2% of Llama 3 405B’s training interruptions, a real, checkable figure. That’s a hardware failure at exactly the memory tier this whole model is built around, and it’s a reminder that the roofline framework describes healthy hardware behaving correctly. It says nothing about the separate, equally real problem of hardware failing outright, which is a different diagnostic path entirely.

Common mistakes#

Treating a single “TFLOPS” number as if it universally describes a chip, rather than one specific precision, sparsity, and product-variant combination among several: the 989-versus-3958 gap above is a 4x difference hiding behind the same GPU name.

Assuming decode’s low utilization reflects poor engineering rather than an inherent property of the operation: the fix is architectural, FlashAttention, the KV cache, quantization, not “write a faster decode loop,” because no amount of code-level tuning moves an operation’s fundamental arithmetic intensity.

Trusting the roofline ceiling as an achievable target rather than an upper bound: real kernels routinely land well short of it, and the 30%-between-adjacent-matrix-sizes finding above is exactly why closing that gap is its own discipline, not a footnote.

What this takes to be frontier-job-ready#

The technical axis here is concrete and fast: given “this kernel is slow,” know to check occupancy and arithmetic intensity before assuming the algorithm is wrong. The roofline calculation above is quick enough to do by hand in seconds, and it answers which entire category of fix is even relevant before any specific optimization gets attempted, which is the difference between an afternoon and a week.

The operational axis shows up as a direct extension of a requirement already named in this series, debugging across the full stack from hardware errors to training dynamics. An HBM3 hardware failure is exactly the kind of hardware-layer issue that requirement points at, and diagnosing it draws on the same mechanistic understanding built here, not a separate skill set bolted on afterward.

The autonomy axis is the judgment call itself: deciding whether a slow kernel needs a tiling change, an occupancy fix, a precision change, or is already near its roofline ceiling and not worth further chasing, is exactly the kind of call made from a rough, fast, in-your-head calculation, with no fixed script for which answer applies. The skill is running the calculation and correctly reading where it points, under time pressure, without being told which of the four to try first.

Try it yourself#

Beginner. Using roofline_attainable_performance, compute attainable performance and percent-of-peak for a kernel at $I=50$ FLOPs/byte on H100 SXM5, and state whether it’s memory-bound or compute-bound.

Intermediate. FP8 roughly doubles $P_{\text{peak}}$ to about 1979 TFLOPS dense, with $B_{\text{peak}}$ unchanged. Recompute the ridge point at this precision, and explain concretely why a kernel with a fixed arithmetic intensity can flip from compute-bound to memory-bound purely by switching numeric format.

Advanced. Explain, using the attention formula’s three chained operations, why counting only the weight matrices’ FLOPs and bytes understates a kernel’s true memory traffic, and why the intermediate $S \times S$ score matrix needs to be counted separately. Connect this directly to why FlashAttention’s actual fix is avoiding ever materializing that matrix in HBM, not making the matmuls themselves faster.

---

The one-sentence version: an H100 is 132 SMs each pairing ordinary CUDA cores with a few disproportionately powerful tensor cores, fed by HBM that’s enormous compared to a CPU’s RAM but glacial compared to each SM’s own on-chip memory, the roofline model says a kernel’s performance is capped by whichever of compute or memory hits its limit first, and LLM decode sitting at roughly 2% of peak isn’t a bug to fix in the decode loop, it’s the exact same fact that FlashAttention, the KV cache, and quantization were all independently invented to work around. That $P_{\text{peak}}$ number in the ridge-point formula isn’t fixed either, it’s set by whatever numeric format you train in, which is its own engineering discipline with its own silent failure mode.