Training Systems
Autograd, scaling laws, data pipelines, the loop itself.
8 logs in this box-
CLIP's contrastive pretraining matched a fully-supervised ResNet-50 on zero-shot ImageNet, 76.2% against the prior best zero-shot result of 11.5%, using no labels at all, and every vision-language model since has picked one of exactly two real ways to plug that visual signal into a language model: LLaVA's single projection layer or Flamingo's gated cross-attention, and Qwen2-VL and Llama 3.2 Vision are real, live, frontier-lab proof that both choices still ship today
CLIP's real InfoNCE contrastive loss and its actual 32,768 batch size, worked through by hand. LLaVA's real training recipe: a single linear (later MLP) projection layer, 600K-to-1.2M image-text pairs, and a full training run finishing in about a day on one 8-A100 node. Flamingo's Perceiver Resampler and gated cross-attention, including the real, LoRA-adjacent trick of initializing the gate at zero so new cross-attention layers don't destroy a frozen language model at initialization. Qwen2-VL's naive dynamic resolution and 2D-RoPE versus Llama 3.2 Vision's frozen-text, cross-attention-adapter design, two frontier labs' real, live, and genuinely different 2024 answers to the same fusion question. Object hallucination, catastrophic forgetting of vision ability, and the resolution-versus-context-budget tradeoff as the real failure modes, not toy caveats.
-
Reproducibility on a GPU has two different meanings that get conflated constantly, forcing PyTorch's actually-deterministic mode costs a real, measured 2 to 5x slowdown, and Chinchilla's own paper names the exact way an ablation quietly becomes invalid
Why 'same seed, same result' and 'bitwise-identical every step' are different guarantees with different real costs, the specific atomicAdd-based kernels that make ordinary GPU training non-deterministic by default, Henderson et al.'s real finding that random seed alone produces statistically significant performance differences, and the isoFLOP sweep failure mode Chinchilla's own paper names directly: a proxy model trained on the wrong data mix invalidates the entire extrapolation.
-
DDP's global batch is a promise, and set_epoch() is the one line that keeps it
RANK/WORLD_SIZE/LOCAL_RANK and why rendezvous is all-or-nothing, the exact mechanism by which a forgotten set_epoch() replays the same global shuffle forever without ever raising an error, the global-batch formula tied to Llama 3's actual DP=64 configuration, and why no_sync() exists.
-
MinHash, bloom filters, and the 5% mix change that moves MATH by 8 points
Why data curation outranks most architecture tweaks in leverage per engineering-hour, the actual MinHash+LSH deduplication algorithm, the bloom-filter decontamination check, and the silent failure mode where a pipeline runs clean and evals regress two weeks later anyway.
-
Chinchilla, compute-optimal training, and why GPT-3 was trained wrong
The power law derived to an exact compute-optimal split, the isoFLOP sweep staff engineers actually run to fit it, why inference cost changes the answer, and the MoE wrinkle that changes what N even means.
-
BPE, glitch tokens, and why the model can't count the r's in strawberry
Byte-pair encoding worked through by hand to a real merge, why vocabulary size is secretly an attention-cost decision, and the convex-hull geometry that explains why specific tokens make a model produce non-sequiturs on command.
-
Why backprop, not forward-mode: the one fact that makes billion-parameter training possible
Forward-mode versus reverse-mode automatic differentiation, derived rather than asserted, LayerNorm's Jacobian shown to have the exact same shape as softmax's, and how round() gets a gradient at all when its true derivative is zero almost everywhere.
-
Autograd is your hand-derived backward pass, automated, and where it silently lies to you
The computation graph as a define-by-run DAG, gradient accumulation at branch points, the missing-zero_grad bug that quietly inflates your learning rate, and the broadcasting mistake that trains a model on the wrong axis without ever raising an error.