Math Foundations
Vectors, gradients, and the loss functions under it all.
4 logs in this box-
Gradient descent, Adam, and the memory bill nobody mentions
SGD as the baseline weight update, Adam's two moving averages and bias correction derived from scratch, and why Adam's 16 bytes per parameter is the real reason distributed-training infrastructure exists.
-
What a language model's loss actually measures: cross-entropy, KL divergence, and perplexity
Why softmax output is a genuine probability distribution, why the cross-entropy gradient collapses to one clean subtraction, and what a perplexity number in a model card actually means.
-
Derivatives, gradients, and how backprop actually flows through attention
The chain rule as multiplied rates, gradients as vectors of partial derivatives, and why a linear layer's Jacobian is just its own weight matrix, transposed for the backward pass.
-
Vectors, matrices, and why GPUs are shaped the way they are
Dot products as directional agreement, matrix multiplication as composed transformations, and the memory hierarchy that decides how fast either one actually runs.