Notes from someone still learning.
Field notes on AI, machine learning, and whatever I'm trying to understand this week, written down so the ideas stay caught.
-
MinHash, bloom filters, and the 5% mix change that moves MATH by 8 points
Why data curation outranks most architecture tweaks in leverage per engineering-hour, the actual MinHash+LSH deduplication algorithm, the bloom-filter decontamination check, and the silent failure mode where a pipeline runs clean and evals regress two weeks later anyway.
-
Chinchilla, compute-optimal training, and why GPT-3 was trained wrong
The power law derived to an exact compute-optimal split, the isoFLOP sweep staff engineers actually run to fit it, why inference cost changes the answer, and the MoE wrinkle that changes what N even means.
-
PagedAttention, chunked prefill, and the launch that cascades
Why per-request KV cache math was never the real constraint, the monitoring lesson that hides cache pressure from your p50 dashboard, the causal chain behind a launch-day incident, and why kernel engineering has a calculable dollar value attached to it.
-
MHA to MLA: the KV cache math that decides whether you can serve the model at all
Why Llama 2 70B needed GQA and Llama 2 7B didn't, the silent broadcast bug that trains fine and fails eval three weeks later, and why the field didn't stop iterating the moment DeepSeek published MLA.
-
The transformer assembled: why Pre-LN wins, and what attention sinks actually are
Multi-head attention, SwiGLU, and causal masking assembled from pieces already derived in this series, Pre-LN's stability proven rather than asserted, and attention sinks shown to be a forced consequence of softmax's own sum-to-one constraint, not a mystery.
-
BPE, glitch tokens, and why the model can't count the r's in strawberry
Byte-pair encoding worked through by hand to a real merge, why vocabulary size is secretly an attention-cost decision, and the convex-hull geometry that explains why specific tokens make a model produce non-sequiturs on command.
-
Why backprop, not forward-mode: the one fact that makes billion-parameter training possible
Forward-mode versus reverse-mode automatic differentiation, derived rather than asserted, LayerNorm's Jacobian shown to have the exact same shape as softmax's, and how round() gets a gradient at all when its true derivative is zero almost everywhere.
-
Autograd is your hand-derived backward pass, automated, and where it silently lies to you
The computation graph as a define-by-run DAG, gradient accumulation at branch points, the missing-zero_grad bug that quietly inflates your learning rate, and the broadcasting mistake that trains a model on the wrong axis without ever raising an error.
-
Loss scaling, FP8, and the failure that never crashes
Why FP16 gradients underflow to exactly zero without a fix, the loss-scaling control loop that recovers them, and how DeepSeek-V3 actually trains in FP8 once naive per-tensor scaling and a 14-bit accumulator both get in the way.
-
The roofline model: why LLM decode uses 2% of a GPU's peak, and it's not a bug
SMs, tensor cores, and HBM given their actual architectural home, the roofline equation derived to an exact ridge point, and why FlashAttention, the KV cache, and quantization are three different answers to the same underlying number.
-
Containers are the process illusion, repackaged, and why Kubernetes doesn't gang-schedule by default
Namespaces and cgroups as os_linux's virtual-memory trick extended to the whole process, why a Kubernetes OOMKilled pod is the exact same OOM killer at a different boundary, and the gang-scheduling gap that silently deadlocks 512-GPU jobs on vanilla Kubernetes.
-
Partial failure is the default: stragglers, deadlocks, and coordinating GPUs that can't see each other
Why one 12%-slow GPU slows every GPU, the fail-stop vs fail-slow distinction that decides whether checkpoint-and-restart helps or hurts, the four ranked mitigation strategies, and the three-axis bar a frontier lab actually hires against.
-
How gradients cross the wire: TCP, RDMA, and why one dead link hangs 16,000 GPUs
Why standard TCP is too slow for gradient sync, how RDMA bypasses the CPU entirely, the ring AllReduce algorithm derived and shown to be bandwidth-optimal, and the real failure modes that freeze an entire cluster at once.
-
Parallel filesystems, checkpoint I/O, and the noise nobody's code caused
How striping lets hundreds of GPUs read the same dataset without one drive becoming the bottleneck, why data loading and checkpointing are opposite I/O problems, and the real incident where someone else's job quietly stole training throughput.
-
Why 'frozen' training jobs usually aren't: processes, memory, and the illusions Linux maintains
Virtual memory as an illusion the OS maintains, why fork() and CUDA don't mix, the GIL's real cost in RL training loops, and how to actually find the OOM killer's fingerprints in the kernel log.
-
Deploying a quantized LLM: GPTQ, AWQ, speculative decoding, and where it breaks
How post-training quantization rounds weights by measuring their actual effect instead of guessing, why speculative decoding is nearly free on memory-bound hardware, and the three specific ways production quantization fails.
-
Gradient descent, Adam, and the memory bill nobody mentions
SGD as the baseline weight update, Adam's two moving averages and bias correction derived from scratch, and why Adam's 16 bytes per parameter is the real reason distributed-training infrastructure exists.
-
What a language model's loss actually measures: cross-entropy, KL divergence, and perplexity
Why softmax output is a genuine probability distribution, why the cross-entropy gradient collapses to one clean subtraction, and what a perplexity number in a model card actually means.
-
Derivatives, gradients, and how backprop actually flows through attention
The chain rule as multiplied rates, gradients as vectors of partial derivatives, and why a linear layer's Jacobian is just its own weight matrix, transposed for the backward pass.
-
Vectors, matrices, and why GPUs are shaped the way they are
Dot products as directional agreement, matrix multiplication as composed transformations, and the memory hierarchy that decides how fast either one actually runs.
-
What a number actually is: two's complement, floating point, and the price of quantization
Why computers count in binary, how negative numbers and fractions get encoded without extra circuitry, and why shaving bits off a model's weights is never quite free.