Post-Training & Alignment
RLHF, DPO, and GRPO: the reward models and rollout loops that turn a base model into a product.
5 logs in this box-
LoRA's own paper claims a 10,000x reduction in trainable parameters and a 3x reduction in GPU memory for GPT-3, and working the real arithmetic for a 70B model's actual dimensions shows why the honest number for a typical production setup is closer to 1,700x, not the maximal headline figure, which is exactly why QLoRA's further trick, quantizing the frozen weights to 4 bits, is what actually lets a 65B model fine-tune on one 48GB GPU
LoRA's real mechanism (freeze W, train a rank-r update BA next to it) worked out against this series' own established 16-bytes-per-parameter accounting, not the paper's rounded claim alone. QLoRA's actual mechanism, NF4 quantization, double quantization, paged optimizers, and the real Guanaco result: 99.3% of ChatGPT's quality on the Vicuna benchmark from 24 hours of fine-tuning on a single GPU. DoRA's magnitude/direction decomposition (ICML 2024 oral) and rsLoRA's rank-stabilization fix as two real, disclosed refinements. A real empirical finding that base models forget more than already-instruction-tuned models under further fine-tuning. And LLaMA Factory (25,000+ stars, 100+ supported models) and Unsloth (claimed 2-5x speedups, 30-90% VRAM reduction) as the tools real teams reach for instead of writing this from scratch.
-
InstructGPT's real reward model accuracy, 69.6% on held-out labelers, sits below the 72.6% humans agree with each other at, which means the ceiling on reward-model calibration was never model capacity, and I trained a real toy reward model that faithfully reproduces exactly the length bias baked into its labels to prove it
The real architecture (Llama 2's two separate reward models, a regression head replacing the LM head), InstructGPT's precise, sourced calibration numbers, RewardBench's real 2,985-prompt evaluation standard, and an actual trained linear reward model whose learned weights track its biased training labels almost exactly, 0.676 and 0.290 against a true 0.7 and 0.3, with 87.4% overall accuracy collapsing to 60.2% specifically on the close-call pairs where the bias actually bites.
-
Claude Opus 4 shipped under ASL-3 in May 2025 because Anthropic's own Responsible Scaling Policy actually triggered, not because a policy document said it should, and the same long-context window this series already covered as a serving win is what makes many-shot jailbreaking possible in the first place
The real ASL-1 through ASL-4 capability thresholds, the actual biosecurity trial result that pushed Opus 4 into precautionary ASL-3 deployment, Anthropic's own February 2025 red-team challenge (339 researchers, 300,000-plus attacks, a jailbreak success rate cut to 4.4%), and the real, quantified tension between jailbreak resistance and over-refusal that XSTest and OR-Bench measure directly rather than assert.
-
An agent's tool interface changes its success rate as much as the underlying model does, SWE-agent's own numbers prove it, and the three capabilities that make an agent useful are exactly what Simon Willison calls the lethal trifecta
Princeton's SWE-agent got more real lift from redesigning the agent's command interface than from swapping models, a result with an exact number attached. Reflexion's 91% HumanEval via verbal self-critique and Poolside's RLCEF, running real production RL at roughly 10,000 code executions a minute, are two genuinely different answers to how an agent gets better after a failure, one costs nothing and forgets, the other costs an actor-learner cluster and never does. And the same three capabilities, private data, untrusted content, external communication, that make an agent worth building are the exact three Simon Willison named as structurally unfixable by prompting, with a real GitHub MCP exploit as proof.
-
RLHF's PPO step holds four models in memory to update one of them, and spends most of its wall-clock generating rather than training, which is the real reason Llama 3 dropped it and DeepSeek rebuilt it as GRPO
InstructGPT's real reward model was 6B parameters, not 175B, kept deliberately small; OpenRLHF's own architecture puts generation at the center of the wall-clock, not the gradient step; DPO's closed-form reward substitution and GRPO's group-relative baseline are two different, real answers to that memory and throughput problem. DeepSeek-R1's actual four-stage pipeline, its documented language-mixing bug, and OpenAI's own April 2025 sycophancy rollback show what happens when the reward signal is wrong instead of merely expensive.