Transformers & Inference
Attention architectures and the systems that serve them.
7 logs in this box-
A hallucination isn't a black box: a linear probe on one mid-layer hidden state can already tell a claim is false before the model finishes the sentence
Azaria & Mitchell's 2023 result that a logistic-regression probe on a single mid-layer hidden state predicts a statement's truth value well above chance, often before generation finishes. Why attention weight going diffuse is a nearly-free uncertainty signal already sitting inside the forward pass. What running the logit lens layer by layer actually shows when a wrong answer locks in partway through the stack. Inference-Time Intervention's real NeurIPS 2023 result, steering along a learned truthfulness direction with no retraining at all. And five concrete code-level places to gate, probe, or penalize a generation before it ships.
-
A retrieval miss in a RAG pipeline throws no exception, it just produces a fluent, confident, wrong answer, and Lewis et al.'s original 2020 paper already named the real reason: a language model's parametric memory and a retriever's non-parametric memory are two structurally different kinds of knowledge that fail in different ways, which is exactly why RAGAS splits evaluation into retrieval metrics and generation metrics instead of grading the final answer alone
Lewis et al.'s original RAG paper (NeurIPS 2020): DPR plus BART, parametric and non-parametric memory combined by design, not an ad hoc workaround. The real bi-encoder-versus-cross-encoder tradeoff worked out mathematically, why every serious pipeline retrieves fast and reranks slow instead of picking one. HNSW and IVF as the two real approximate-nearest-neighbor algorithms behind every production vector index. RAGAS's four real metrics, faithfulness, answer relevancy, context precision, context recall, and why splitting them is what lets you tell a retriever bug from a generator bug instead of just a lower score. Real, concrete failure modes: chunking as the upstream bottleneck nothing downstream can fix, index staleness, and why a retrieval miss fails silently instead of throwing an error.
-
NTK-aware RoPE scaling started as a Reddit post that beat the published literature to the insight, and running the actual scaling formulas side by side shows exactly why: Position Interpolation compresses every frequency by the same factor, while NTK-aware and YaRN leave the high-frequency dimensions almost untouched
The real lineage from Position Interpolation (Chen et al., Meta, 2023) to a May 2023 Reddit post by bloc97 to YaRN's formal NTK-by-parts fix (2.19 perplexity at 131,072 tokens on Mistral 7B) to LongRoPE's evolutionary search past 2 million tokens, ALiBi as the pretraining-time alternative that can't be retrofitted, Liu et al.'s real U-shaped lost-in-the-middle curve, Gemini 1.5's real near-perfect retrieval at 10 million tokens, and the actual 2025 long-context-versus-RAG cost tradeoff, not a resolved debate.
-
I implemented FlashAttention's tiling and online softmax in plain PyTorch, verified the output is bit-for-bit the same math as naive attention, and measured it running up to 1.6x slower on CPU, which is itself real proof the speedup is a GPU memory-hierarchy fact, not an algorithmic one
A runnable reference implementation, correctness-checked against naive attention (max difference 2.98e-07), the exact measured intermediate-tensor size at S=512 (8.39MB down to 1.05MB, an 8x reduction), the precise arithmetic confirming the 2.1TB blowup at Llama 3's real 128,192-token context, and honest CPU timing showing the tiled version losing at every sequence length tested here, with the specific reason why that's expected rather than a bug.
-
PagedAttention, chunked prefill, and the launch that cascades
Why per-request KV cache math was never the real constraint, the monitoring lesson that hides cache pressure from your p50 dashboard, the causal chain behind a launch-day incident, and why kernel engineering has a calculable dollar value attached to it.
-
MHA to MLA: the KV cache math that decides whether you can serve the model at all
Why Llama 2 70B needed GQA and Llama 2 7B didn't, the silent broadcast bug that trains fine and fails eval three weeks later, and why the field didn't stop iterating the moment DeepSeek published MLA.
-
The transformer assembled: why Pre-LN wins, and what attention sinks actually are
Multi-head attention, SwiGLU, and causal masking assembled from pieces already derived in this series, Pre-LN's stability proven rather than asserted, and attention sinks shown to be a forced consequence of softmax's own sum-to-one constraint, not a mystery.