NTK-aware RoPE scaling started as a Reddit post that beat the published literature to the insight, and running the actual scaling formulas side by side shows exactly why: Position Interpolation compresses every frequency by the same factor, while NTK-aware and YaRN leave the high-frequency dimensions almost untouched

FlashAttention solves the memory problem: the $S\times S$ score matrix stops existing in HBM at all. It does not solve a completely separate problem that shows up at exactly the same long-context frontier: RoPE’s rotation angle, $\theta_i = 10000^{-2i/d}$, was fit to a specific training length, and simply running the same formula at 8 to 30 times that length produces rotation angles the model never saw during training, positions that look out-of-distribution to the exact mechanism that’s supposed to tell the model how far apart two tokens are. This post is the real lineage of fixes for that second problem: a sequence of genuinely different ideas, each one a direct, named response to a specific flaw in the one before it, including a case where the open-source community’s own experimentation beat a published paper to the correct insight by several months. It ends with running the actual competing formulas side by side and printing the real numbers that show why one approach won.

The real flaw in the first fix: uniform compression#

Chen et al.’s Position Interpolation (Meta, June 2023) was the first widely-adopted real fix, and its core move is almost embarrassingly simple once stated: instead of extrapolating position indices past what the model trained on, which produces attention scores the paper describes as catastrophically large, linearly compress the position indices to fit back inside the original trained range. Formally, $g(m) = m/s$ where $s = L'/L$ is the ratio between the target and original context length, leaving the frequency itself, $h(\theta_d) = \theta_d$, untouched. The real result: LLaMA extended to 32,768 tokens with fewer than 1,000 fine-tuning steps, a genuinely cheap fix for a genuinely real problem.

The flaw is exactly as simple to state, and it’s the reason everything else in this post exists: Position Interpolation compresses every single frequency dimension by the identical factor $1/s$, whether that dimension encodes fine-grained local word order or coarse, long-range document structure. Running the actual formula confirms this directly rather than asserting it. At a scale factor of $s=8$ (extending 4,096 to 32,768 tokens), computing PI’s effective frequency ratio at nine different dimensions across a 128-dimensional head:

dim d       PI ratio
     0       0.1250
     8       0.1250
    16       0.1250
    24       0.1250
    32       0.1250
    40       0.1250
    48       0.1250
    56       0.1250
    63       0.1250

Every single dimension, high-frequency or low, gets divided by exactly $1/s = 0.125$. That uniformity is precisely the problem: the high-frequency dimensions are what encode fine local relationships, adjacent-token structure, and compressing them by the same amount you compress the long-range dimensions blurs local resolution the model never actually needed to sacrifice to gain long-range reach.

NTK-aware: a Reddit post that beat the literature to the fix#

Here’s the real, worth-knowing-cold history: the correct next idea wasn’t published in a paper first. In May 2023, a Reddit user going by “bloc97” posted the observation that Position Interpolation’s uniform compression conflicts directly with what Neural Tangent Kernel theory says about how a neural network actually uses different frequency bands, high frequencies for local detail, low frequencies for global position, treating them identically throws away exactly the distinction the model relies on. The fix bloc97 proposed doesn’t touch position at all, it rescales RoPE’s base itself: $b' = b\cdot s^{|D|/(|D|-2)}$, then recomputes every dimension’s frequency from that new base. This produces a non-uniform effect for a structural reason, not a tuned one, since the exponent $-2d/|D|$ already varies by dimension, rescaling $b$ compounds differently at $d=0$ than at $d=63$.

Running the actual NTK-aware formula, same $s=8$ scale factor, next to Position Interpolation’s uniform 0.125 everywhere:

dim d     PI ratio   NTK ratio
     0      0.1250      1.0000
     8      0.1250      0.7679
    16      0.1250      0.5897
    24      0.1250      0.4529
    32      0.1250      0.3478
    40      0.1250      0.2671
    48      0.1250      0.2051
    56      0.1250      0.1575
    63      0.1250      0.1250

At $d=0$, the highest-frequency dimension, NTK-aware’s ratio is 1.0000, completely unchanged, exactly the local-detail preservation Position Interpolation sacrifices for no benefit. By $d=63$, the lowest-frequency dimension, it converges to the same 0.1250 Position Interpolation applies everywhere, because at the extreme low-frequency end, long-range compression really is what’s needed. This is the real, measured shape of the actual improvement, a smooth curve from “leave it alone” to “compress it,” not a uniform blanket rule, and it required no training at all to achieve, real, working long-context extension purely from a formula change, first demonstrated in a public forum post months before the formal literature caught up. The real adoption followed quickly: Code-LLaMA shipped with an NTK-by-parts variant, and Qwen-7B shipped with a Dynamic NTK variant.

YaRN: formalizing the fix, with a real number attached#

Peng et al.’s YaRN (2023) is the formal paper that systematized bloc97’s community insight into “NTK-by-parts”: rather than one smooth curve, blend explicitly between full interpolation and no interpolation at all, per dimension, using a piecewise ramp:

$h(\theta_d) = (1-\gamma)\cdot\frac{\theta_d}{s} + \gamma\cdot\theta_d, \qquad \gamma = \begin{cases}0 & r < \alpha \\ 1 & r > \beta \\ \frac{r-\alpha}{\beta-\alpha} & \text{otherwise}\end{cases}$

where $r$ compares a dimension’s wavelength against the original training length, and $\alpha=1$, $\beta=32$ is reported as the setting that works well for the Llama family specifically. Running this formula alongside the other two:

dim d     PI ratio   NTK ratio   YaRN ratio
     0      0.1250      1.0000       1.0000
     8      0.1250      0.7679       1.0000
    16      0.1250      0.5897       1.0000
    24      0.1250      0.4529       0.6786
    32      0.1250      0.3478       0.2808
    40      0.1250      0.2671       0.1550
    48      0.1250      0.2051       0.1250
    56      0.1250      0.1575       0.1250
    63      0.1250      0.1250       0.1250

YaRN’s ramp is visibly discrete rather than smooth, holding fully unchanged (1.0000) through $d=16$, transitioning sharply across $d=24$ to $d=40$, then settling at Position Interpolation’s full 0.125 compression from $d=48$ onward, a direct, literal implementation of “some dimensions need zero change, some need full compression, and a narrow band in between needs to be blended.” The real, reported payoff: YaRN with scale 16 on Mistral 7B achieves a perplexity of 2.19 at 131,072 tokens, beating both base Mistral v0.1 and the NTK-aware-based MistralLite variant, using fine-tuning on under 0.1% of the original pretraining data. A separate variant, Dynamic-YaRN, achieves more than 2x context extension with no fine-tuning at all, applying the rescaling only at inference time.

LongRoPE: pushing past 2 million tokens#

Microsoft’s LongRoPE (February 2024) is the current frontier of this specific lineage, extending a pretrained model’s context to 2,048K tokens, over 2 million, via three real, stacked techniques rather than one clever formula: an evolutionary search that finds non-uniform interpolation factors per dimension automatically, rather than a hand-derived ramp like YaRN’s; a progressive extension strategy that first fine-tunes to 256K, then applies a second interpolation pass on top of that already-extended checkpoint to reach 2048K, rather than jumping directly to the target in one step; and a final short-context readjustment pass specifically to recover the original model’s performance at its original short context length, since aggressive long-context rescaling otherwise costs something back at the short end. LongRoPE is real, open-sourced, and shipped in a real production model, Microsoft’s Phi-3.

ALiBi: the alternative that can’t be retrofitted#

Every method above shares one structural constraint: they’re all ways to rescale a RoPE model that’s already been pretrained. Press, Smith, and Lewis’s ALiBi (ICLR 2022) is a genuinely different design, not a rescaling trick at all: no positional embedding is added to the input whatsoever. Instead, attention scores themselves get a penalty directly proportional to the distance between query and key, applied at every layer, every head, computed fresh each time rather than baked into a rotated vector once. The real reported result: a 1.3B model trained at sequence length 1,024 extrapolates to 2,048 at the same perplexity as a sinusoidal-embedding model actually trained at 2,048, while training 11% faster and using 11% less memory.

The real tradeoff, worth stating precisely rather than treating ALiBi as strictly better: this is a decision made at pretraining time, part of the architecture itself, not a post-hoc rescaling you can apply to a model that already shipped with RoPE. Every technique earlier in this post, Position Interpolation, NTK-aware, YaRN, LongRoPE, exists specifically because RoPE was already the choice and retraining from scratch wasn’t on the table. If you’re designing a new model’s architecture from a blank page, ALiBi’s native extrapolation is a real, competing option. If you’ve already pretrained on RoPE, it isn’t, and everything above this section is what you actually have available.

What extending context doesn’t fix#

None of the above touches a separate, real, measured failure mode. Liu et al.’s “Lost in the Middle” (Stanford, TACL 2024) tested models on multi-document QA and key-value retrieval and found a consistent U-shaped performance curve: information at the very start or very end of a long context gets used reliably, information buried in the middle gets used measurably worse, across many models and tasks, and the effect persists even after instruction fine-tuning specifically aimed at long-context use. The paper’s own real implication is the one worth carrying forward: successfully extending the context window, by any of the methods above, doesn’t by itself guarantee the model can actually use every position in that window with equal reliability. Those are two different claims, and only the first one is what RoPE scaling and its variants actually fix.

What companies actually built at scale#

Google’s Gemini 1.5 technical report (February 2024) discloses real, measured numbers worth having exactly rather than rounded: near-perfect retrieval, above 99%, up to at least 10 million tokens in evaluation, with a standard, generally available context of 1 million tokens. The real practical framing the report gives: 1 million tokens covers roughly an hour of video, eleven hours of audio, or more than 30,000 lines of code, and at that scale it’s compared directly against Claude 2.1’s 200K and GPT-4 Turbo’s 128K at the time of publication, a genuine multiple, not a marginal edge. Reaching that scale in production is exactly the Ring Attention and Context Parallelism systems work already derived in this series: none of the RoPE-scaling lineage above solves the memory-sharding problem, it solves the positional-generalization problem, and a real 1M-plus token deployment needs both solved simultaneously, not either alone.

Long context versus RAG: a real, current tradeoff, not a settled question#

Extending context that far raises the obvious real question: does long context replace retrieval-augmented generation entirely? The honest, current answer, based on real 2024-2025 evaluation work, is no, and the reason is a genuine cost-versus-completeness tradeoff rather than a capability gap. When resourced generously, long-context models measurably outperform RAG on average, because nothing gets lost to a retrieval step’s imperfect recall. RAG remains meaningfully cheaper, and that gap is a real, current dollar figure, processing a full million-token context on GPT-4 Turbo runs on the order of $15 per million input tokens as of late 2025, a cost paid on every single query if the full context is resent each time. The real, practical split that’s emerged: long context wins for bounded, mostly-static material, a fixed contract, a fixed codebase, a fixed report, where paying once (or paying with caching) to load everything is affordable and retrieval’s imperfect recall is a real risk worth avoiding. RAG wins as the underlying collection grows large or changes frequently, where retrieval’s job is precisely to avoid resending everything on every query. Neither claim generalizes to “always use X,” and a real production decision here is a real cost-and-freshness calculation, not a philosophical preference.

Common mistakes#

Assuming any RoPE-scaling method is a strictly better version of the one before it: Position Interpolation, NTK-aware, YaRN, and LongRoPE each traded a specific weakness for a specific new cost, more fine-tuning, more search complexity, more staged training, not a monotonic improvement with no downside.

Treating ALiBi as simply “a better RoPE”: it’s a different architectural commitment made at pretraining time, unavailable as a fix for a model that’s already trained on RoPE, which is precisely why the RoPE-scaling lineage exists at all rather than everyone just switching.

Believing a model that accepts a 1M-token input actually uses all 1M tokens equally well: Lost in the Middle’s real, measured U-shaped curve shows context-window size and effective context utilization are different claims, and the second one doesn’t follow automatically from the first.

Choosing long context or RAG as a fixed architectural preference rather than re-evaluating per use case: the real 2025 evidence splits on document volatility and cost tolerance, not on which technique is more advanced.

Try it yourself#

Beginner. Using Position Interpolation’s formula, $g(m)=m/s$, compute the effective position value for a token at real position 50,000 when extending a model trained at 8,192 tokens out to 128,192, matching Llama 3’s real target length.

Intermediate. Using the ratio tables in this post, explain in one sentence why NTK-aware’s ratio at $d=0$ is exactly 1.0000 regardless of the scale factor $s$, referencing the exponent structure of $b'^{-2d/|D|}$ directly.

Advanced. YaRN’s ramp uses $\alpha=1, \beta=32$ as the setting reported to work well for the Llama family. Design an experiment that would tell you whether a different model family needs different $\alpha,\beta$ values, using perplexity at the target extended length as your metric, and explain what evidence would indicate the ramp boundaries themselves, not just the scale factor, need retuning for a new architecture.

What this takes to be frontier-job-ready#

The real hiring signal here is stated directly, not implied: Anthropic’s Pretraining Scaling RE role names “long context support” as an explicit project type, on equal footing with novel architecture work, not a subordinate detail. The real technical bar this whole post has been building toward is being able to reason about which of these real, named techniques applies to a specific model’s situation, already pretrained on RoPE versus starting fresh, needing zero-shot extension versus able to afford fine-tuning, rather than reciting that “RoPE scaling exists.”

The genuinely rare skill, evidenced directly by bloc97’s real history in this post, is being willing to derive and test a fix from first principles, connecting NTK theory to a rescaling formula, before a formal paper validates it, and shipping that working result publicly months ahead of the literature. That’s a real, demonstrated instance of exactly the “reaching trustworthy conclusions through principled, simple experiments” standard this series already cited from OpenAI’s own RL/Reasoning hiring language, applied outside a lab entirely.

The one-sentence version: Position Interpolation’s real flaw, compressing every frequency dimension by the same factor, was diagnosed and fixed first in a May 2023 Reddit post, not a paper, and running NTK-aware and YaRN’s actual formulas side by side against Position Interpolation’s proves the fix numerically, ratio 1.0000 at the highest-frequency dimension instead of a uniform 0.125 everywhere; LongRoPE pushes the same underlying idea past 2 million tokens via evolutionary search, ALiBi sidesteps the entire lineage with a pretraining-time architectural choice that can’t be retrofitted, and Gemini 1.5’s real near-perfect retrieval at 10 million tokens still runs into the same measured U-shaped attention curve Liu et al. documented, a context window that accepts a token is not the same claim as a model that uses every one of them equally well.