PagedAttention, chunked prefill, and the launch that cascades

The per-request KV cache math was never the actual production constraint. The real one is aggregate cache across every concurrent user, managed on hardware that fragments if you’re naive about allocation. Take Llama 3 70B’s GQA number from that math, roughly 41GB per request at 128K context. Serve 100 concurrent users and that’s 4.1TB of KV cache, live, simultaneously. That’s not a number you provision for by buying more GPUs and hoping. It’s a number you provision for with an actual memory manager.

PagedAttention: an OS idea, not an ML one#

vLLM’s answer to this is a direct lift from operating systems, not a novel ML idea, and that’s worth knowing because it tells you where to look for good engineering intuition here: OS memory management, not ML papers. KV cache gets divided into fixed-size blocks, 16 tokens each. A block table maps (request_id, block_index) to a physical block address. Blocks get allocated on demand as a sequence grows and freed immediately when a request completes. It’s literal virtual memory paging applied to attention cache, inspired explicitly by the OS’s own demand paging, and it reduces KV cache waste from 60 to 80% down to under 4%.

Read that number again. 60 to 80% waste, down to under 4%. Before PagedAttention, naive allocation, reserving the maximum possible sequence length upfront per request, meant burning the majority of GPU memory on cache that was never used. Not a minor optimization. The difference between serving 20 concurrent users and 80 on identical hardware. If you’re doing capacity planning without accounting for this, your hardware estimate is off by roughly 3 to 4x before you’ve made a single other decision.

Prefix caching, SGLang’s RadixAttention, is the other half, and it’s the one people forget. If many requests share a common prefix, the same system prompt, the same few-shot examples, the same tool definitions injected at the start of every call, you hash that prefix and reuse the already-computed KV blocks across requests instead of recomputing them. At 100M-user scale with a shared system prompt, this isn’t a marginal win. It’s the difference between prefilling that prompt once and prefilling it on every single call.

Two production incidents your dashboard won’t show you#

The failure that pages people at 2am. When KV cache fills under load, vLLM must preempt requests, forcing them to recompute prefill, and p99 latency spikes 10x while p50 looks completely normal. Users see 30-second responses that are invisible in average-based metrics. This is the single most important operational fact in this whole area at scale: your p50 dashboard looks healthy while a meaningful slice of your users have a terrible experience. If your on-call rotation only alerts on average latency, you won’t know you have a problem until support tickets pile up. Alert on p99, not p50, and this isn’t a best-practice platitude, it’s the specific mechanism by which cache pressure hides from average-based monitoring.

The subtler one: prefix cache thrashing. If your system prompt changes slightly per request, a timestamp injected at the top, a per-user variable spliced into what should be a shared prefix, you get zero cache hits where you expected near-100%, and throughput drops roughly 40%. This is a real, recurring production bug pattern: someone adds “personalization” by prepending a user’s name to the system prompt, not realizing they just broke prefix caching for every single request. The fix is almost always to move the variable content to the end of the prompt, not the beginning, because prefix caching only helps for shared leading tokens.

Weight memory is the other half of your hardware bill#

None of the attention-variant compression touches weight memory. GQA and MLA shrink the KV cache, not the 140GB of BF16 weights a 70B model needs just to exist. At scale, weight memory times replica count is the other half of your hardware bill, and it’s controlled by quantization, not attention architecture: BF16 needs two H100s minimum for a 70B model, INT8 halves that to one-and-change, INT4 quarters it to fit on a single H100 at the cost of a few points of quality. That’s not a rounding difference in infrastructure spend, it’s the difference between one GPU and four for identical user-facing capability, and the two specific failure modes that come with it, temperature-dependent quality loss and speculative-decoding acceptance collapse after a model update, are worth knowing cold since they’re exactly the kind of thing an eval suite run at the wrong sampling temperature misses.

Serving systems: the layer that survives real traffic#

Everything above, attention variant, cache manager, quantization scheme, is still just the model. Serving systems is what turns that into something that survives real traffic, and it’s where the actual engineering hours go at scale.

Continuous batching replaces a fixed batch size, where the whole batch waits for its slowest member, with a live request queue that adds new requests in as soon as any prior one finishes. GPU utilization goes up substantially, because you’re never idling a GPU waiting on one straggler in an otherwise-finished batch.

Chunked prefill solves a specific, concrete problem: a 128K-token prefill can take on the order of 10 seconds of compute. Left naive, one long prompt blocks every other pending short request behind it in the queue, someone asking a two-word question waits behind someone who pasted in a whole codebase. The fix is splitting long prefills into roughly 1K-token chunks and interleaving them with ongoing decode steps for other requests, so no single long request can starve everyone else’s latency.

The economics are the actual reason this discipline exists: at billions of tokens per day, a 2x throughput improvement is worth $50M-plus a year. At scale, serving efficiency isn’t an engineering nicety, it’s a line item on the income statement, and the people who own this layer directly move that number.

The launch-day incident, traced through its full causal chain#

The failure mode that defines “handling real scale” as different from “handling a demo”: tested at 3x expected traffic, launch hits 8x, queue grows, p99 hits 30 seconds, emergency scale-out gets triggered, new replicas need to cold-start, loading a 70B model from storage into GPU memory takes on the order of 90 seconds, further delays pile up, cascading failure. Worth internalizing the chain, not just the summary: traffic exceeds tested headroom, queues back up, you try to scale out, new replicas take 90 seconds to load weights, during those 90 seconds you have more load and fewer effective replicas than when the incident started, and the system is now actively working against your own fix. This is why real capacity planning tests meaningfully beyond expected peak, 3x clearly wasn’t enough here, and why cold-start time for your specific model size is a number every serving engineer should know cold. It directly determines how fast you can actually respond to a spike versus how fast you’d like to.

Multi-tenant isolation, briefly: separate queue budgets, rate limiting, and data isolation per customer or workload class, so one customer’s traffic burst doesn’t degrade latency for everyone sharing the fleet. At consumer scale this is less “per enterprise customer” and more “per traffic class,” free tier versus paid, interactive versus batch, but the isolation principle is identical.

FlashAttention: why the mechanism this whole layer runs on isn’t compute-bound#

Everything above assumes attention itself runs efficiently on the hardware, which isn’t automatic. Standard attention doesn’t just do a lot of compute, it does a lot of memory traffic it doesn’t need: read Q, K, V from HBM, write the full $S\times S$ score matrix back, read it back, write softmax weights, read weights and V, write output. Five separate HBM round trips for one attention computation, and the $S\times S$ matrix is what actually kills you. At $S=128K$, $H=64$, BF16, that intermediate tensor alone is roughly 2TB. An H100 has 80GB of HBM. Not slow. Impossible, you cannot fit the intermediate matrix in memory at all at long context, regardless of how much time you’re willing to spend.

FlashAttention tiles Q, K, and V into blocks small enough to fit in SRAM, the on-chip memory roughly 100x faster than HBM, and computes attention block by block using an online softmax, a running max and normalizer updated incrementally as each new K/V block is processed, so the entire row of scores never needs to exist at once for a correct softmax over it. The $S\times S$ matrix is never materialized in HBM at all, and memory cost drops from $O(S^2)$ to $O(S\times d)$.

The insight worth saying out loud: modeling FLOPs alone would suggest the unfused implementation is fine, but once you account for memory bandwidth, the operation can be restructured to avoid materializing intermediate values in slow HBM at all. Standard attention was never compute-bound, the bottleneck was shuffling data that never needed to leave the chip, and FlashAttention’s 5 to 20x speedup comes almost entirely from eliminating that traffic, not from doing less arithmetic. Its backward pass never stores the $S\times S$ matrix either, it recomputes attention weights directly from the stored Q, K, V, a deliberate compute-for-memory trade that only makes sense once the $S\times S$ matrix is understood as the actual constraint.

Version upgrades carry their own silent risk: an FA2-to-FA3 upgrade changing how specific edge cases with custom sparse masks get handled produced a silent quality regression on tasks using those masks, the same category of danger as the GQA broadcast bug, nothing crashes, training proceeds normally, and the regression only shows up in downstream eval on a specific mask pattern nobody thought to re-test. Treat a FlashAttention version bump like any other core-math dependency upgrade: run both implementations on the same inputs and compare outputs token by token before deploying, not a green loss curve as your only signal.

Long-context training: how the capability gets there in the first place#

Extending usable context from, say, 8K to 128K takes four things working together, not one clever trick.

RoPE’s rotation frequency is $\theta_i = 10000^{-2i/d}$, and that base value was chosen for a specific training context length. Positional encodings computed with it don’t automatically behave well at 10 to 15x that length, the rotations become too fine-grained relative to the actual distances involved, distorting the model’s sense of how far apart two tokens are. The fix is increasing the base frequency: Llama 3 used 500,000, up from 10,000, specifically to make positional encodings smoother, meaning more stable and predictable, at long distances.

Context Parallelism exists for one reason: 128K sequences don’t fit in memory even with FlashAttention. FlashAttention solved the $O(S^2)$ intermediate-matrix problem, but the $O(S\times d)$ memory it still needs, multiplied across a full training batch and everything kept for backward, still doesn’t fit on a single GPU at 128K. CP splits the sequence dimension itself across GPUs: each GPU holds $S/G$ tokens, and GPUs ring-communicate their K/V blocks so every token’s query can attend across the full sequence even though no single GPU holds the whole thing, with $O(1)$ additional communication per step. This is a genuinely new axis layered on top of whatever data/tensor/pipeline/expert parallelism you’re already running. Long context isn’t free even with FlashAttention and GQA or MLA already in place.

Staged training, verbatim from Meta’s own report: the context window was extended from 8,192 to 128,192 tokens gradually, first up to 32,768, then up to 128,192. At each stage you verify the model can actually use the longer context, typically with a needle-in-haystack test, before extending further. Jumping straight to the target length risks training instability and gives no diagnostic signal about where a problem originated if quality comes out wrong.

Two failure modes worth knowing by name. A model trained at 4K and deployed at 8K with the wrong base frequency degrades silently on inputs over 4K, nothing errors, the model still generates plausible text, and the degradation only shows up when someone specifically evaluates on long inputs, the same shape of danger as everywhere else in this post. And lost in the middle: models attend well to the beginning and end of long contexts and poorly to the middle, a documented failure mode even after extending context correctly. If you’re asked why a needle-in-haystack test passes at the start and end but fails when the fact is buried in the middle, the correct answer is often “possibly nothing, this is a known limitation, not necessarily a bug,” and knowing that distinction is exactly the kind of judgment these questions are actually probing for.

The loop closes back to serving: a 70B model at 128K context still needs roughly 64GB of KV cache per request, meaning serving it is impractical without quantization even once training got the capability right. Training a correct 128K-context model and shipping a usable feature are two different engineering problems, and solving only the first isn’t shipping.

Kernel engineering: the layer underneath everything else here#

Every technique in this series, GQA’s repeat_interleave, MLA’s low-rank projections, FlashAttention’s tiled online softmax, quantized matmuls, the batching logic above, ultimately executes as an actual GPU kernel. Triton is a Python DSL that compiles to GPU assembly while handling tiling, vectorization, and shared-memory management for you, you write something that reads like NumPy and get a tuned kernel. CUDA is a level below, manual control over the thread hierarchy, full control and full responsibility. CuTe, NVIDIA’s C++ DSL for tensor layout and tiling, is what FlashAttention-3 and -4 are actually written in. Writing a kernel in any of them means deciding explicitly what data goes into SRAM versus stays in HBM, how work gets scheduled across warps, and which operations are structured to actually hit tensor cores rather than falling back to slower general-purpose cores.

This is worth taking seriously rather than treating as generic career advice: performance work that makes abstract, logical changes to a model practical to run is described as the biggest bottleneck and innermost loop of all LLM work, and it’s called out as the most direct path into a frontier lab, backed by a concrete number, every 1% MFU gain over 54 days is worth roughly $200K saved. At this scale kernel-level performance work has a direct, calculable dollar value attached, which is part of why it’s treated as a distinct specialization rather than a niche skill. Worth being precise about sourcing here: a dedicated GPU performance role at one specific lab was named in passing as evidence of this, but I couldn’t independently confirm that exact title against that lab’s own posted roles, so treat the specialization claim as well-supported by the economics above, not by that specific unconfirmed job title.

Three hardware-level failures worth knowing by name, because they explain design choices elsewhere in this series. A shared memory bank conflict, 32 threads accessing addresses with stride 32 all hitting the same bank, serializes what should be parallel and costs a 32x slowdown from what looks like an innocuous indexing choice. Register spilling, a kernel using more registers than an SM has, doesn’t error, it silently spills the overflow into HBM, the exact slow memory this whole area is about avoiding, for a 10x slowdown with no explicit signal beyond a performance profile that looks wrong. Wrong tiling for tensor cores, which require specific tile sizes like 16×16 or 32×32 for FP16, silently drops a kernel out of the fast path entirely into general-purpose cores, a 10x performance drop with no error at all.

Worth knowing where this is heading, not as a footnote: DeepMind’s AlphaEvolve used LLM-in-the-loop evolutionary search to discover a matrix multiplication kernel that outperformed the human-designed implementation it competed against. Kernel engineering is itself becoming a target for automated discovery, and engineers in this space are already using tools like it directly in their own research workflow.

What this takes to be frontier-job-ready#

The technical axis is stated almost verbatim by the labs that hire for it. Poolside’s Data Platform Lead role requires “experience with extreme scale (100s of TB to multi-PB) distributed systems” and “previous hands-on experience shipping and maintaining Kubernetes-based critical infrastructure,” alongside container runtimes and Linux system programming as a separate, explicitly listed requirement, not folded into the Kubernetes line. That’s this entire post’s serving layer, named as a specific, checkable hiring bar rather than a general notion of “systems experience.”

The operational axis is where the launch-day incident above stops being a thought experiment. Anthropic’s own Pretraining Scaling role states plainly that “the work is highly operational, you’ll be deeply involved in keeping our production models training smoothly,” and that during launches the team “often works extended hours and may need to respond to issues on evenings and weekends.” A cold-start cascade during a traffic spike is exactly the kind of event that requirement is written for, not an edge case outside the job description.

The autonomy axis shows up as composure under a very specific, nameable kind of failure. Mistral’s own Applied Scientist language, “you don’t panic when you see OOM errors or when NCCL feels like not wanting to talk,” is describing a baseline temperament, not an aspirational trait, and a p99 latency spike hiding behind a healthy p50 dashboard is precisely the kind of silent, non-obvious failure that temperament requirement exists to prepare you for.

Common mistakes#

Provisioning hardware from per-request cache math instead of aggregate, concurrent-user cache: the per-request number was never the constraint, and skipping PagedAttention-style memory management leaves 60 to 80% of capacity unused.

Building an alerting strategy around average latency: cache-pressure preemption spikes p99 by 10x while leaving p50 untouched, and average-based alerting is specifically blind to this failure by construction.

Adding personalization to a shared system prompt without checking prefix-cache impact: prepending a variable to the front of an otherwise-shared prefix silently drops your cache hit rate to near zero.

Load-testing at a headroom multiplier that turns out to be too low, then scaling out as the fix: cold-start time during a scale-out event adds load before it adds capacity, and if the initial headroom wasn’t enough, the fix arrives too slowly to prevent the cascade.

Trusting a green loss curve after a FlashAttention version upgrade or any core-math dependency change: several of the failure modes in this post are silent by design, and only a direct, token-by-token numerical comparison against the prior version catches them.

Try it yourself#

Beginner. Using PagedAttention’s reported waste reduction (60 to 80% down to under 4%), estimate how many additional concurrent users a fleet could serve on identical hardware after adopting it, starting from a baseline of 20 naive-allocation users.

Intermediate. Work through the launch-day causal chain with different numbers: tested at 5x expected traffic, launch hits 6x, and cold-start time is 30 seconds instead of 90. Does the cascade still occur, and what does that tell you about how much headroom margin actually matters versus cold-start time itself?

Advanced. Using the RoPE base-frequency formula, compute the rotation angle for a token at position 100,000 under both a 10,000 base and a 500,000 base, at a fixed dimension index, and explain concretely why the larger base produces a “smoother” (more slowly varying) rotation at that distance.

---

The one-sentence version: the per-request KV cache number from GQA and MLA was never the real constraint, aggregate cache across concurrent users managed with OS-style paging is, and this entire layer, from PagedAttention through FlashAttention through kernel-level tile sizing, exists because every one of its failure modes is silent by construction: nothing crashes, the loss curve looks fine, the p50 dashboard looks healthy, and the only way to catch any of it is checking the actual number instead of trusting the summary metric. Every one of these serving costs is downstream of a decision made months earlier, at the very start of training: how big to make the model in the first place.