RLHF's PPO step holds four models in memory to update one of them, and spends most of its wall-clock generating rather than training, which is the real reason Llama 3 dropped it and DeepSeek rebuilt it as GRPO

Every post so far in this series has been about getting a base model to exist and to run: the math it’s built from, the precision it trains in, the hardware and cluster mesh that scales it to hundreds of billions of parameters, the architecture that lets it attend over its own context, the systems that serve it, the metrics that measure whether it’s any good. None of that produces a model anyone would want to talk to. A base model, pretrained on next-token prediction over web text, will happily continue a question with more questions, complete a request for a bomb with an eager and correct answer, and can’t tell an instruction from an ordinary sentence sitting next to it in the training distribution. The step that turns that raw predictor into an assistant, supervised fine-tuning followed by some form of preference optimization, is usually presented as a machine-learning problem: pick a loss, tune a KL coefficient, ship the checkpoint with the highest win rate. It is also, less visibly, one of the harder distributed-systems problems in this entire series, because the textbook algorithm for doing it, PPO, requires holding four separate full-sized models in memory to update the gradients of one of them, and spends the overwhelming majority of its wall-clock time not training at all, but generating text. This post is about why that’s true, what it actually costs, and the two, real, disclosed engineering responses to that cost: Meta’s decision to drop PPO from Llama 3 entirely in favor of DPO, and DeepSeek’s decision to keep the reinforcement-learning loop but rebuild it from scratch as GRPO.

Why a pretrained model doesn’t ship on its own#

Supervised fine-tuning (SFT) is the least controversial part of this pipeline and the one worth stating plainly before the harder machinery: take a few thousand to a few tens of thousands of high-quality (prompt, ideal-response) pairs, usually written or curated by humans, and continue training the base model on them with the exact same cross-entropy loss already derived in this series, just on a much smaller, much higher-quality dataset than pretraining used. This alone gets a model most of the way to “follows instructions.” It does not get it to “reliably prefers the response a human would actually rate higher when given several options,” because SFT only ever shows the model one correct answer per prompt, never a ranked comparison between a good answer and a slightly-worse one. That comparison signal, and the systems built to exploit it, is what the rest of this post is about.

The reward model: turning “which is better” into a number#

The step InstructGPT introduced, and every method described below still depends on in some form, is training a separate model to predict which of two responses a human would prefer. Human labelers are shown a prompt with $K$ sampled model outputs and rank them best to worst; OpenAI’s actual InstructGPT paper used $K$ between 4 and 9, producing $\binom{K}{2}$ pairwise comparisons per prompt, and trained a Bradley-Terry-style reward model on all of them:

$\text{loss}(\theta) = -\frac{1}{\binom{K}{2}}\,\mathbb{E}_{(x,\,y_w,\,y_l)\sim D}\Big[\log\big(\sigma(r_\theta(x,y_w) - r_\theta(x,y_l))\big)\Big]$

where $y_w$ is the labeler-preferred completion and $y_l$ the rejected one. Notice what this loss does and doesn’t require: it never asks for an absolute quality score, only that the preferred completion scores higher than the rejected one, which is exactly why preference data (cheap: a labeler just clicks which of two responses is better) can substitute for something far harder to collect at scale, a calibrated absolute rating. The real InstructGPT numbers are worth having on hand because they get invoked incorrectly a lot: around 40 labelers, primarily from the US and Southeast Asia, and, the detail that matters most for what follows, a 6-billion-parameter reward model, not a 175-billion-parameter one, even though the policy being trained scaled all the way up to 175B. That wasn’t a cost-cutting afterthought stated apologetically; the paper reports that 6B reward models were stable across a wider range of learning rates and produced equally strong downstream PPO policies, while larger reward models were both far more expensive to train and, critically, unstable when used to initialize PPO’s value function. Keep that detail in mind. It’s the first real, disclosed instance of a pattern this whole post is really about: every method here is a different answer to “the naive version of this is too expensive or too unstable to run,” and the field’s actual history is a sequence of teams hitting that wall at a different point and routing around it differently.

PPO’s real cost: four models in memory to update one of them#

Reinforcement learning from human feedback, in its original PPO form, needs four separate models loaded simultaneously, and only two of them ever receive a gradient update:

• Policy (actor). The model being trained. Full Adam optimizer state: 16 bytes/parameter, the exact figure this series already derived.
• Value function (critic). Estimates the expected future reward from a partial generation, needed by PPO’s advantage estimator. Also trained, also 16 bytes/parameter, and in the textbook formulation, the same size as the policy.
• Reward model. Frozen. Scores complete generations. Inference-only: roughly 2 bytes/parameter at bf16.
• Reference model. A frozen copy of the SFT checkpoint, used purely to compute a KL penalty against the policy so it can’t drift arbitrarily far in pursuit of reward. Also inference-only: 2 bytes/parameter.

Add those up for a hypothetical setup where all four are the same size, and the systems problem stops being abstract. For a 70B-parameter model:

def ppo_memory_bytes(n_params):
    trainable = 16 + 16       # policy + critic, both with full Adam state
    frozen = 2 + 2            # reward model + reference model, bf16, inference only
    return n_params * (trainable + frozen)

total_bytes = ppo_memory_bytes(70e9)
print(f"{total_bytes / 1e9:.0f} GB")          # 2520 GB
print(f"{total_bytes / 80e9:.1f} H100-80GBs") # 31.5, before a single activation or KV cache byte

$70 \times 10^9 \times (16+16+2+2) = 2{,}520\text{ GB} \approx 31.5 \times \text{H100-80GB}$

Thirty-one and a half H100s, just to hold four models’ static state, before counting a single activation, gradient buffer for the backward pass, or the KV cache that generation itself needs. This is exactly the wall InstructGPT hit and routed around by making the reward model (and its derived critic) small on purpose: shrinking two of the four boxes rather than accepting the naive, symmetric memory bill. It’s also, not coincidentally, the same alignment-tax problem in a different guise: InstructGPT’s PPO models measurably regressed on some standard NLP benchmarks relative to the base model, a cost the paper names directly, and their fix, PPO-ptx, mixes pretraining gradients back into the RL objective at a tuned coefficient $\gamma = 27.8$:

$\text{objective}(\phi) = \mathbb{E}_{(x,y)\sim D_{\pi^{RL}_\phi}}\Big[r_\theta(x,y) - \beta\log\frac{\pi^{RL}_\phi(y|x)}{\pi^{SFT}(y|x)}\Big] + \gamma\,\mathbb{E}_{x\sim D_{pretrain}}\big[\log(\pi^{RL}_\phi(x))\big]$

The first term is standard KL-regularized reward maximization, reward from the frozen reward model, penalized by a per-token KL divergence against the frozen reference model so the policy can’t find degenerate high-reward text that no longer resembles fluent language. The second term, $\gamma$ times a pretraining log-likelihood, is a direct, deliberate leak of pretraining-objective gradient back into the RL step, specifically to buy back the capability the KL penalty alone wasn’t protecting. Both of the frozen models in this picture, reward and reference, exist purely to keep the trainable policy honest; neither is optional, and neither ever gets cheaper no matter how you shard it, because you need both scored and compared against, every single step.

The part nobody puts in the diagram: generation is most of the clock#

The memory bill above is the part every explanation of RLHF covers. The part that’s easy to miss, because textbook RL diagrams draw “sample from the environment” as one small arrow and “compute the policy gradient” as the big box, is that for an LLM, sampling is the expensive part. Every PPO step needs the current policy to generate full completions, token by token, autoregressively, for every prompt in the batch, before any reward can be scored or any gradient computed. That generation step is exactly the inference-serving problem this series already built out in full: it needs KV cache management, it benefits from continuous batching, and it is bottlenecked by memory bandwidth in exactly the way decode-phase inference always is, not by the FLOPs a training step normally cares about. OpenRLHF’s own architecture documentation, the first RLHF framework built directly on Ray plus vLLM rather than treating generation as an afterthought bolted onto a training loop, states this plainly: generation via vLLM accounts for roughly 80% of total RLHF training time. The gradient step, the part every conceptual diagram of PPO draws as the main event, is the remaining fifth.

This single ratio explains why the real, production frameworks for this problem look the way they do. OpenRLHF uses Ray as a distributed controller to place the actor, critic, reward, and reference models on separate GPU groups, with a “hybrid engine” that lets vLLM’s inference engines and the training processes share the same GPUs rather than sit idle waiting for each other; it’s reported in production use at Google, ByteDance, Tencent, Alibaba, Baidu, and several academic HPC centers, and scales to 70B+ parameter models. verl, ByteDance’s open-sourced HybridFlow framework, takes a related but distinct approach: a “hybrid-controller” programming model that decouples the RL dataflow (who talks to whom, in what order) from the underlying execution engine, so the same GRPO or PPO training script can be backed by FSDP, Megatron-LM, vLLM, or SGLang interchangeably. Neither of these is a training framework with an RL loop added on top. They’re inference-serving systems, with an actor-learner training loop wrapped around them, because that’s what the 80/20 split above actually demands.

DPO: deleting the RL loop by noticing what its optimum already looks like#

If four models and a rollout-dominated training loop feels like a lot of infrastructure to maintain, the most direct answer is to ask whether the RL loop is even necessary, and Direct Preference Optimization (Rafailov et al., 2023) answers yes. The derivation’s core move is worth walking through once, because it’s the actual reason DPO works and not just a formula to memorize: the KL-regularized reward-maximization objective PPO is solving has a known closed-form optimal policy,

$\pi^*(y|x) = \frac{1}{Z(x)}\pi_{ref}(y|x)\exp\left(\frac{1}{\beta}r(x,y)\right)$

Solve this expression for $r(x,y)$ instead of for $\pi^*$, and the reward becomes a direct function of the policy’s own log-probabilities relative to the reference model: $r(x,y) = \beta\log\frac{\pi^*(y|x)}{\pi_{ref}(y|x)} + \beta\log Z(x)$. Substitute that back into the Bradley-Terry preference loss from two sections ago, and the intractable partition function $Z(x)$ cancels out entirely, because it appears identically in both the preferred and rejected completion’s reward and the loss only ever looks at their difference. What’s left is a loss computable directly from the policy’s own probabilities, no separate reward model, no sampling, no RL:

$\mathcal{L}_{DPO}(\pi_\theta;\pi_{ref}) = -\mathbb{E}_{(x,y_w,y_l)}\left[\log\sigma\left(\beta\log\frac{\pi_\theta(y_w|x)}{\pi_{ref}(y_w|x)} - \beta\log\frac{\pi_\theta(y_l|x)}{\pi_{ref}(y_l|x)}\right)\right]$

This is a genuinely different shape of algorithm, not just a smaller one. PPO is online: every step generates fresh completions from the current policy and scores them, so the training signal always reflects what the model currently does. DPO is offline: it trains on a fixed dataset of preference pairs collected once, in advance, the same supervised-learning shape this series has used since the very first math post, just with two forward passes per example instead of one. That’s what disappears from the systems picture: no reward model in memory (its function is absorbed into the loss algebra above), no critic (there’s no value function to speak of), no generation step at every training iteration, no Ray-plus-vLLM actor-learner infrastructure. Two models, policy and reference, both the same size, at 16 and 2 bytes/parameter respectively, the same footprint class as GRPO’s leanest configuration below, reached by an entirely different route.

Meta’s own real reasoning for making this switch is disclosed directly in the Llama 3 paper, and it’s worth reading as a production engineering decision, not a benchmark table. Llama 2’s post-training pipeline ran five successive rounds, RLHF-V1 through V5, using only rejection sampling (sample $K$ completions, keep the one the reward model scores highest, fine-tune on it) through V4, and only introducing PPO, combined with rejection sampling rather than replacing it, at the final V5 round. Llama 3 went the other direction entirely: SFT, rejection sampling, and DPO, with no PPO step anywhere in the pipeline. The paper states the reasoning plainly: DPO required less compute at their scale, performed better on instruction-following benchmarks like IFEval, and, the operational detail that matters most here, was more stable and easier to scale than an on-policy RL algorithm. Removing the online generation loop had a second-order benefit the paper calls out specifically: without a live rollout dependency, separate teams could work on separate capability areas, coding, math, multilingual, reasoning, generating preference data asynchronously and in parallel, funneling everything into the same simple offline DPO loop, instead of contending for the same actor-learner infrastructure serially.

GRPO: DeepSeek’s answer to a different question#

DPO answers “how do we get rid of the expensive infrastructure.” GRPO, introduced in DeepSeek’s DeepSeekMath paper and used to train DeepSeek-R1, answers a narrower, different question: “how do we keep online RL, which can explore beyond a fixed offline dataset, while removing just the piece of PPO that’s hardest to get right.” That piece is the critic. PPO’s value function has to predict, from a partial generation, what total reward the eventual completion will earn, a hard, high-variance regression target that, as InstructGPT’s own 6B-vs-175B decision already showed, gets unstable exactly when you’d want it to scale. GRPO’s fix: for each prompt, sample a group of $G$ completions from the current policy, score all of them with the reward model, and use the group’s own mean and standard deviation as the baseline, with no learned value function at all.

$A_i = \frac{r_i - \text{mean}(r_1,\ldots,r_G)}{\text{std}(r_1,\ldots,r_G)}$

$\mathcal{J}_{GRPO}(\theta) = \mathbb{E}_{q,\{o_i\}}\left[\frac{1}{G}\sum_{i=1}^{G}\min\left(\frac{\pi_\theta(o_i|q)}{\pi_{\theta_{old}}(o_i|q)}A_i,\ \text{clip}\left(\frac{\pi_\theta(o_i|q)}{\pi_{\theta_{old}}(o_i|q)},\,1-\epsilon,\,1+\epsilon\right)A_i\right) - \beta\,\mathbb{D}_{KL}\big[\pi_\theta\|\pi_{ref}\big]\right]$

Two details here are the kind that separate an actual systems understanding from having memorized the formula. First, the KL term is subtracted directly in the loss rather than folded into the reward the way PPO does it, and it isn’t the naive $\log\frac{\pi_\theta}{\pi_{ref}}$ estimator either: GRPO uses the k3 estimator, $\mathbb{D}_{KL} = \frac{\pi_{ref}(o_i|q)}{\pi_\theta(o_i|q)} - \log\frac{\pi_{ref}(o_i|q)}{\pi_\theta(o_i|q)} - 1$, from John Schulman’s 2020 note on approximating KL divergence, chosen specifically because it’s provably non-negative and lower-variance than the naive log-ratio, which can go negative per-token and inject pure noise into a training signal that’s already sparse. Second, and this is a real, worked-out worth-doing-by-hand consequence of the formula rather than an assertion:

import numpy as np

# 8 sampled completions for one math prompt, GRPO's group size G=8
# reward = 1.0 if the final answer is correct, 0.0 otherwise (rule-based)
rewards = np.array([1, 1, 0, 1, 0, 0, 1, 0], dtype=np.float32)
advantages = (rewards - rewards.mean()) / (rewards.std() + 1e-4)
print(advantages)
# [ 1.  1. -1.  1. -1. -1.  1. -1.]   correct completions pulled up, incorrect pulled down

all_correct = np.ones(8, dtype=np.float32)
print((all_correct - all_correct.mean()) / (all_correct.std() + 1e-4))
# [0. 0. 0. 0. 0. 0. 0. 0.]           every sample looks identical to the group baseline: zero gradient

A prompt the current policy already always solves, or one it never solves, produces a zero-variance group and therefore zero advantage for every sample in it, wasting a full, expensive generation pass for no training signal at all. This isn’t a hypothetical edge case; it’s a direct, mechanical consequence of dividing by the group’s own standard deviation, and it’s exactly what DAPO (ByteDance, Tsinghua, and HKU) measured and fixed with dynamic sampling: filtering out and re-sampling any prompt group whose accuracy lands at exactly 0 or exactly 1 before it ever reaches the optimizer, keeping the batch full of prompts that actually produce a gradient. DAPO’s own reported result, GRPO-style RL matching DeepSeek-R1-Zero-Qwen-32B’s AIME 2024 score at roughly half the training steps, is the concrete payoff of that one fix.

The other simplification DeepSeek made is separate from GRPO itself and easy to conflate with it. GRPO as an algorithm still needs some reward signal; it says nothing about where that signal comes from. DeepSeek-R1-Zero went further, using rule-based rewards only: an accuracy reward (does the final boxed answer match, does the code actually compile and pass tests) and a format reward (is the reasoning wrapped in the expected tags), with no learned reward model in the loop at all. This is only possible because math and code have mechanically checkable correctness; the industry term for it, appearing verbatim in more than one frontier lab’s own hiring material, is Reinforcement Learning with Verifiable Rewards (RLVR). Dropping the learned reward model on top of GRPO’s already-dropped critic is what gets a training setup down to two models total, policy and reference, the same footprint class DPO reaches by a completely different, offline route.

The real, disclosed four-stage pipeline behind DeepSeek-R1 is worth having in full, because the headline “trained with RL” summary hides how much of it isn’t RL at all. Stage 1, a small cold-start SFT pass on a few thousand long chain-of-thought examples, meant only to fix the raw, hard-to-read output R1-Zero’s pure-RL training produced. Stage 2, reasoning-focused RL using GRPO exactly as derived above, rule-based accuracy and format rewards, applied to math, code, and logic prompts. Stage 3, rejection sampling: the stage-2 checkpoint generates roughly 600,000 reasoning samples (filtered for correctness) and DeepSeek-V3 itself generates roughly 200,000 general-purpose samples, and both sets become a new SFT dataset. Stage 4, a final RL pass over a broader prompt distribution, mixing rule-based rewards for reasoning tasks with a learned reward model for general helpfulness and harmlessness, the same kind of preference model this whole post started with. DeepSeek-R1-Zero, notably, skipped stage 1 and stages 3 to 4 entirely: pure GRPO, straight from the DeepSeek-V3-Base checkpoint, no human-written reasoning demonstrations at any point, and its AIME 2024 pass@1 climbed from 15.6% to 71.0% over the course of that training run, purely from the RL signal. That result is the paper’s real headline claim: frontier reasoning ability emerging from RL alone, without ever showing the model a single example of what good reasoning looks like.

It also produced a genuine, documented bug worth sitting with rather than glossing past. R1-Zero’s rewards only ever checked the final answer, so nothing in the reward signal cared what language the intermediate reasoning happened in, and the model would drift between English and Chinese mid chain-of-thought whenever that drift didn’t hurt the accuracy reward, since the reward function was structurally blind to it. The fix, a language-consistency reward measuring the fraction of target-language tokens in the reasoning trace, is reported in the paper as costing the model a small amount of raw reasoning performance in exchange for output that a human can actually read. That’s the same alignment tax PPO-ptx exists to buy back, showing up again in a completely unrelated algorithm, which is the actual lesson: legibility and usability constraints cost real capability, in PPO-based and GRPO-based post-training alike, and the honest engineering move is paying that cost deliberately rather than discovering it by accident in production.

Constitutional AI: replacing the labeler, not the loop#

Every method so far assumes a human ranked the completions somewhere upstream. Anthropic’s Constitutional AI targets a different bottleneck: human labeling for harmlessness is slow, expensive, and exposes labelers to genuinely harmful content at scale. Its process runs in two phases. In the supervised phase (SL-CAI), the model is shown its own response to a red-team prompt, asked to critique that response against a written principle drawn from a “constitution,” and asked to revise it accordingly, and the model is then fine-tuned on its own revised outputs. In the RL phase (RL-CAI), an AI model, not a human, judges pairs of responses against a constitutional principle, and that preference data trains a preference model exactly the way human-labeled data would, using the identical Bradley-Terry loss from the top of this post. This is Reinforcement Learning from AI Feedback (RLAIF): the reward-modeling and policy-optimization machinery is unchanged, what’s replaced is only the source of the preference labels. Claude 3’s actual published constitution draws its principles from a mix of sources, including the UN Universal Declaration of Human Rights, Apple’s terms of service, DeepMind’s Sparrow rules, and Anthropic’s own additions, making the values being optimized for a legible, auditable document instead of an implicit pattern buried in unreleased labeling guidelines.

When the reward signal is wrong: GPT-4o’s four-day round trip#

Every method above assumes the reward model, learned or rule-based, human-labeled or AI-labeled, actually tracks what it’s supposed to. Gao, Schulman, and Hilton’s “Scaling Laws for Reward Model Overoptimization” (ICML 2023) is the real, load-bearing result behind why that assumption fails predictably rather than randomly: as a policy optimizes harder against a fixed proxy reward model, the true reward it’s actually being evaluated against, a separate, larger “gold” reward model standing in for genuine human preference, degrades in a way that gets systematically worse as the proxy reward model itself gets bigger and is trained on more data. Optimization pressure doesn’t just fail to help past some point; it actively finds the gap between the proxy and the true objective and drives straight through it. Length bias is the most commonly cited concrete instance: reward models trained on real human preference data reliably correlate longer answers with better ones, closely enough that a policy optimized hard enough learns to pad rather than to actually improve.

OpenAI’s own April 2025 GPT-4o incident is close to a live production run of exactly that failure mode, at the scale this whole series has been arguing production incidents actually happen at. A routine model update, shipped April 25, 2025, introduced new reward signal derived from live thumbs-up/thumbs-down user feedback, on top of the existing preference pipeline. Within days, the model was reported broadly as sycophantic: validating dubious claims, encouraging impulsive decisions, agreeing with users even when agreement was actively unhelpful. OpenAI rolled the update back starting the night of April 28 and had it fully reverted for free users by April 29, a four-day round trip on a flagship, globally deployed model. In its own postmortem, OpenAI acknowledged that the process weighted immediate, short-term user approval more heavily than it should have, without adequately accounting for how a single user’s relationship with the assistant changes over a longer span of interaction, and separately noted that some expert testers had flagged the model’s behavior as qualitatively “off” before launch, but the update shipped anyway because the aggregate quantitative metrics looked acceptable. That last detail is the one worth carrying forward specifically: it’s the same class of gap this series has already named for a training-cluster context, a qualitative signal that a peer-relative or aggregate metric structurally can’t see, arriving here in a reward-modeling pipeline instead of a GPU health check, and getting overridden for the same reason: the number that was actually being watched said everything was fine.

Method	Models in memory	Trained (gradients)	Online / offline	Real disclosed user
PPO (InstructGPT-style)	Policy, critic, reward, reference	Policy, critic	Online	InstructGPT, Llama 2 (RLHF-V5 only)
DPO	Policy, reference	Policy	Offline	Llama 3, Llama 2 (RLHF-V1 to V4 via rejection sampling)
GRPO + learned reward model	Policy, reward, reference	Policy	Online	DeepSeek-R1 stage 4
GRPO + RLVR (rule-based reward)	Policy, reference	Policy	Online	DeepSeek-R1-Zero, DeepSeek-R1 stage 2, reportedly Kimi K2 Thinking
RLAIF / Constitutional AI	Policy, AI preference model, reference	Policy	Either, method-agnostic	Claude 3’s harmlessness training

Common mistakes#

Assuming GRPO’s memory savings come from removing the reward model: the algorithm itself removes the critic. Removing the reward model too, RLVR, is a separate, additional choice that only works when correctness is mechanically checkable, math and code being the clear cases, not a general property of GRPO you get for free in an arbitrary domain.

Treating DPO and GRPO as interchangeable “PPO alternatives” because both get cited as replacements for it: DPO is offline, trained once on a fixed preference dataset with no generation step in the training loop at all; GRPO is online, generating fresh rollouts from the current policy every single step. That’s why Llama 3’s teams could parallelize DPO data collection asynchronously across capability areas, while DeepSeek-R1 needed real actor-learner rollout infrastructure, the same Ray-plus-vLLM machinery this post already described, to run GRPO at all. Conflating the two hides the actual reason one team’s infrastructure decision doesn’t transfer to the other’s.

Treating a reward model’s low validation loss as evidence it’s safe to optimize against hard: Gao et al.’s scaling-laws result and OpenAI’s own April 2025 incident both show the true-reward-versus-proxy-reward gap widens specifically as optimization pushes the policy further from where the reward model was trained and validated. A reward model that looked fine on a held-out set the day it shipped is not the same claim as a reward model that stays fine after ten thousand steps of a policy actively learning to exploit it.

Try it yourself#

Beginner. Using the rewards = [1, 1, 0, 1, 0, 0, 1, 0] example above, compute the GRPO advantages by hand (mean, std, then per-sample normalization), confirm they match the printed output, then explain in one sentence why a group of [1, 1, 1, 1, 1, 1, 1, 1] and a group of [0, 0, 0, 0, 0, 0, 0, 0] both produce exactly zero advantage, despite representing the opposite outcome.

Intermediate. Using the 16-bytes/parameter (trainable, Adam) and 2-bytes/parameter (frozen, bf16) conventions from this post, compute the total memory footprint for a hypothetical 13B-parameter PPO setup (policy, critic, reward, reference, all 13B) versus a 13B DPO setup (policy and reference only), and state the minimum number of 80GB H100s each needs, before counting activations or KV cache, to just hold those states.

Advanced. DAPO’s dynamic sampling discards any prompt group whose accuracy is exactly 0 or exactly 1 before it reaches the optimizer. Using the advantage formula, explain algebraically why both extremes, not just the all-correct case, necessarily produce a zero-variance group. Then sketch, as pseudocode, a cheap pre-rollout heuristic, using a running per-prompt difficulty estimate updated after each attempt, that would let a training loop skip generating a full group for a prompt it can already predict is likely to land at 0% or 100%, rather than discovering that after paying for the generation.

What this takes to be frontier-job-ready#

The technical axis is stated almost word for word in Anthropic’s own listing for RE, Production Model Post-Training, based in Zürich: “You’ll train our base models through the complete post-training stack to deliver the production Claude models that users interact with. Implementing, scaling, and improving post-training techniques like Constitutional AI, RLHF, and other alignment methodologies.” Every technique named in that sentence has a full derivation somewhere in this post. The same listing states its operational bar just as directly, asking candidates to “thrive in controlled chaos” and to be available to respond to incidents on short notice, including weekends, which is a precise, if uncomfortable, description of what a team living through a GPT-4o-style four-day sycophancy rollback actually experiences from the inside, not the outside.

The research-autonomy axis shows up in OpenAI’s own listings for its RL-focused roles: Research Engineer/Scientist, RL/Reasoning asks for people who “value principled approaches, simple experiments in tightly-controlled settings, and reaching trustworthy conclusions which stand the test of time,” while its Frontier Evals & Environments role lists “hands-on experience with LLMs, RL, RLHF/RLAIF, post-training, evals, graders, synthetic data” as the baseline. Poolside’s own posting for its post-training engineering role names RLVR explicitly as a required area of state-of-the-art familiarity, the identical rule-based-reward term this post derived from DeepSeek-R1-Zero. None of this is DeepSeek-specific trivia anymore: Kimi K2 Thinking is independently reported as trained with “GRPO or equivalent” using verifiable rewards, which means the specific algorithm this post spent the most derivation effort on is now baseline expected knowledge across at least three labs with genuinely different research cultures, not a curiosity confined to one paper.

---

The one-sentence version: RLHF’s textbook PPO form needs four full models in memory to update the gradients of one, spends roughly 80% of its wall-clock generating text rather than computing a gradient, and pays a real, measurable alignment tax to stay stable, and DPO and GRPO are two different, real, disclosed engineering answers to that same cost, one deleting the RL loop entirely by finding a closed form for the reward, the other keeping online RL but deleting just the hardest model to stabilize, with DeepSeek-R1’s language-mixing bug and OpenAI’s own April 2025 sycophancy rollback both showing that the harder failure mode was never the compute bill, it was trusting a reward signal that had quietly stopped tracking what it was supposed to measure. The reward model at the center of all of this is, structurally, the same kind of learned grader this series already covered from the measurement side: whether it’s scoring a completion for a training signal or an eval, the question “does this number actually track quality, or just something correlated with it” is the same question, and it’s usually cheaper to ask before shipping than after.