MinHash, bloom filters, and the 5% mix change that moves MATH by 8 points

“Clean the data” sounds like janitorial work next to “design a new architecture.” It isn’t, and the gap between those two impressions is worth closing precisely, because it’s backwards. A 5% change in code and math data mix can move the MATH benchmark 3 to 8 points, a bigger capability swing than most architecture changes produce, at a fraction of the engineering cost, and reversible in days instead of requiring a new training run from scratch. This is why frontier labs run dedicated data roles, Anthropic’s RE, Tokens / Pre-training Data position and Poolside’s Data Research role among them, rather than treating data prep as a preprocessing chore owned by whoever’s free.

The deeper reason this matters: a model learns exactly what the data shows it. There’s no separate reasoning module to patch afterward if the data was wrong. Code quality, multilingual performance, math ability, safety behavior, every downstream capability traces back to what was in the corpus and in what proportion. Get the data wrong and no training recipe fixes it, you’re running a correct process on the wrong inputs.

The production pipeline runs collect, deduplicate, quality-filter, decontaminate, mix, tokenize, shard, stream, at petabyte scale on the distributed storage systems and Linux fundamentals covered earlier in this series. This post goes through each stage with the actual mechanics, the real numbers, and the specific way each one breaks silently in production.

Deduplication: MinHash and LSH, as an actual algorithm#

A trillion-token web-scraped corpus has massive near-duplication: the same news article syndicated across a hundred sites, boilerplate legal disclaimers on thousands of pages, near-identical forum threads. Exact-duplicate removal, hashing the whole document and dropping repeats, catches almost none of it, because near-duplicates differ by a sentence, a date, an ad insertion. Pairwise comparison across a trillion tokens is computationally impossible, so you need a cheap way to estimate similarity without ever doing the expensive comparison directly.

MinHash turns “is this document similar to that one” into a cheap comparison. Extract all n-grams from a document, say every 5-word sequence. Hash every n-gram with $K$ different hash functions. For each hash function, keep only the minimum hash value seen across all n-grams in the document. That gives a $K$-dimensional signature, $K$ numbers standing in for the full text. Two documents sharing many n-grams will, with high probability, produce the same minimum hash value for a given hash function, so the fraction of the $K$ hash functions that agree between two signatures is a statistically unbiased estimator of the actual Jaccard similarity between the original documents. An expensive text-overlap computation becomes a cheap comparison of two short vectors.

You still can’t afford to compare every pair of $K$-dimensional signatures across a trillion-token corpus, so LSH (locality-sensitive hashing) banding solves the remaining problem. Divide the $K$-dimensional signature into $B$ bands of $R$ rows each, hash each band separately, and any two documents landing in the same bucket for any band become candidates for being near-duplicates. This turns “compare every pair” into “group documents by hash bucket, only compare within a bucket,” which scales to a corpus of any size because most documents never share a bucket with anything. Candidates then get verified against an actual similarity threshold, typically Jaccard similarity around 0.8, before being confirmed as duplicates and removed. LSH is fast, approximate candidate generation; the threshold check is the expensive but precise confirmation, run only on the small candidate set LSH surfaced.

This runs as Spark jobs processing petabytes of raw text across 10 to 100 machines, hours to days per full deduplication pass over a trillion-token corpus. Not a laptop script, a distributed systems job in its own right, which is exactly why “large-scale GPU clusters + distributed data pipelines” is listed as a hard requirement alongside the ML-specific skills on Poolside’s Data Research posting.

The failure mode that’s genuinely dangerous, because it’s silent. Get the similarity threshold slightly wrong, 0.7 instead of 0.8, and you’re flagging documents as duplicates that are merely similar, not actually redundant. You remove too aggressively, domain coverage in the corpus drops, an entire style of writing or topic area gets thinned out because its documents all resembled each other just enough to trip the lower threshold, and model quality regresses. Nothing errors. No pipeline step fails. You find out two weeks later when evals come back worse than expected, and the root cause is a single threshold parameter set 0.1 too low in a job that ran successfully and reported no errors. This is the single most important thing to internalize about data engineering failures generally: they’re almost never crashes. They’re silent distribution shifts that only show up in downstream eval numbers, often long after the pipeline step that caused them, the same shape of danger as the broadcast bugs and silent regressions covered elsewhere in this series.

Quality filtering: deciding what “good writing” means, at scale#

A model-based classifier, trained on human-rated quality judgments, coherence, informativeness, whether text reads like genuine writing versus spam or auto-generated filler, gets run over the entire corpus to filter or weight documents. This matters more than “quality” sounds like it should, because low-quality text, SEO spam, auto-generated boilerplate, broken OCR output, keyword-stuffed pages, teaches the model to produce that same kind of output. A quality classifier is, in effect, deciding what good writing and good reasoning mean for the entire model at the scale of a trillion tokens, and that decision compounds across every downstream capability. If the classifier was trained on raters who systematically preferred a certain style, register, or language, that preference gets baked into the entire pretraining corpus’s composition, and by extension into the model’s own default voice and judgment.

In current frontier practice, quality filtering isn’t a binary pass-fail gate. It increasingly means scoring documents to construct explicit training curricula, feeding directly into the curriculum stage below rather than being a one-time filter applied once and forgotten.

Decontamination: the mechanism, not just “check for overlap”#

Contamination, eval benchmark answers leaking into the training corpus, inflates benchmark scores by 5 to 15 points, and an organization ends up making a launch decision based on a number that is, functionally, a lie. Not hypothetical: HumanEval coding problems have appeared in public GitHub repositories that get scraped into pretraining corpora, and a team that doesn’t check sees a 15-point “improvement” on a coding benchmark that’s entirely an artifact of the model having memorized the test set, not an actual capability gain. Users notice the real capability hasn’t improved. Trust in the entire eval pipeline erodes from a single missed decontamination pass.

The actual mechanism: for every eval benchmark example, extract all 13-grams, 13-token sequences. Hash each 13-gram and check it against a bloom filter built from the entire training corpus. A bloom filter is the right structure specifically because it lets you check whether an n-gram exists anywhere in a trillion-token corpus in roughly constant time and modest memory, without storing or searching the corpus text directly, a probabilistic set-membership structure built exactly for this kind of large-scale lookup, where checking against the corpus directly would be computationally infeasible. If any 13-gram from an eval example matches something in the training bloom filter, that example gets flagged as contaminated and excluded, either from the training data or from the eval computation, depending on which side of the pipeline you’re protecting.

The n-gram length is a tuning choice with the identical shape as the MinHash threshold above: too short, say 3-grams, and you get constant false positives, because short common phrases like “the results show that” appear everywhere by coincidence, not contamination. Too long and you miss contamination that’s been lightly paraphrased. You’re trading false positives against false negatives on a similarity measure, and the right threshold is an empirical tuning decision, not a fixed universal constant, exactly the same tension as the dedup stage wearing a different hat.

Data mixing: the highest-leverage decision in the whole pipeline#

Once you have a clean, deduplicated, decontaminated pool of data from many sources, web text, code, books, academic papers, multilingual text, curated math, you don’t just concatenate and shuffle. You decide what proportion of the training corpus comes from each source, the mix. This is where the 5%-shift-to-3-to-8-MATH-points number at the top of this post actually comes from: mix ratio is one of the most consequential decisions in the entire pretraining process, and it’s cheap to experiment with relative to almost everything else in the pipeline, a small proxy training run with a different mix costs a fraction of the time and money of the full production run.

This is precisely the work named directly in Anthropic’s own posting for the Tokens / Pre-training Data role: understanding pretraining data trends and scaling laws, optimizing pretraining data mixes, investigating potential new sources of data, and comparing the compute efficiency of different datasets. That’s a genuinely research-flavored engineering job, where the deliverable is an empirically validated mix ratio, arrived at the same way scaling laws get fit: cheap proxy experiments extrapolated to inform an expensive production decision, applied to mix ratio instead of model size.

Curriculum and data repetition: named requirements most explainers skip#

Poolside’s Data Research posting lists these as explicit hard requirements, data curation, deduplication, data mixing, tokenization, curriculum, impact of data repetition, and yet these two concepts get almost no real treatment in most explainers of this topic.

Curriculum is the order data is shown in, not just what’s included. The naive assumption is that pretraining data is one giant shuffled pool and order doesn’t matter, since you’ll see everything eventually anyway. In practice, frontier labs treat ordering as its own lever: composition can shift deliberately across a single run, weighting toward broad, diverse web text earlier, then shifting toward higher-quality, more specialized sources, curated code, math, carefully selected long-form text, later, sometimes concentrated in a final annealing or cooldown phase. The intuition mirrors curriculum learning generally: a model early in training is still forming broad representations and benefits from broad exposure; a model later in training, with representations more settled, gets more out of concentrated high-quality signal than from more of the same web text it’s already mostly learned from. Getting this schedule wrong isn’t a crash, it’s a training run that ends up with a subtly different capability profile than intended, discovered only once evals come back.

Data repetition is how many times the same document or source gets seen across a run, and it’s a real, measured tradeoff, not an obvious “more passes is bad.” With trillion-token corpora, especially for smaller high-quality sources, curated math datasets, textbooks, carefully vetted code, you frequently cannot avoid repetition, there simply isn’t enough unique high-quality data of a given type to fill a multi-trillion-token run without seeing some of it more than once. The engineering question is how many repetitions before returns diminish or actively hurt, the model starting to overfit to a specific source’s quirks rather than generalizing from it. This gets answered empirically, with small-scale ablation experiments measuring the marginal value of an additional pass over a given source, the same proxy-experiment methodology used for scaling laws and data mix, applied to a third axis, repetition count, instead of model size or mix ratio.

Both are worth naming as separate, specific hard requirements rather than folding them into a generic “data mixing” bullet, because they’re genuinely separate empirical questions from what proportion of each source to use, and treating either as an afterthought is exactly the gap between someone who’s built a clean-the-data pipeline once and someone who’s actually run the ablations answering these questions for a real production corpus.

Synthetic data isn’t a separate pipeline#

Poolside’s Synthetic Data posting describes the mission directly: designing and implementing complex pipelines that generate large amounts of data while maintaining high diversity and optimizing available resources, validated through quantitative data ablation experiments. This connects back to everything above rather than standing apart from it. Synthetic data still goes through quality filtering, is the generated content actually good or is it degenerate, repetitive model output. It still needs decontamination, did the generation process leak an eval answer into its own training signal. It still gets folded into the overall mix decision, what proportion of the corpus should be synthetic versus naturally sourced.

There’s one risk unique to synthetic data worth flagging honestly rather than glossing over: a generating model producing increasingly narrow, self-similar output that then gets trained on, further narrowing the next generation’s outputs. That loop, and how labs monitor and prevent it, is a real, distinct topic with its own mechanics, and it’s worth knowing the failure mode exists even without covering the full defense here.

Real production stakes, stated plainly#

A 5% change in code and math data mix can move the MATH benchmark 3 to 8 points, a bigger capability swing for less engineering cost and less risk than most architecture changes, and the direct justification for treating mix ratio as a first-class experimental variable rather than a value someone picks once. Contamination inflates benchmark scores 5 to 15 points, meaning skipping decontamination doesn’t just risk a slightly wrong number, it risks the entire launch decision process operating on a false signal, with the failure only becoming visible once real users notice the model doesn’t actually perform the way its benchmark scores claimed.

Both job postings cited throughout describe this as hands-on, experimental, ablation-driven engineering work, not a preprocessing chore handed to whoever’s available, “your number one mission would be to improve the quality of the pretraining datasets by leveraging your previous experience, intuition and training experiments.” Worth taking literally if you have a data-engineering rather than pure-ML background: Spark and Airflow experience is described as mapping directly onto what these roles require, meaning the distributed-systems skills, large-scale batch processing, pipeline orchestration, handling petabyte-scale jobs reliably, are treated as directly transferable into a frontier-lab pretraining data role, not a separate bar you clear first before the “real” ML work starts.

Where this actually gets tested#

“Walk me through how you’d deduplicate a trillion-token corpus” tests whether MinHash plus LSH is known as an actual algorithm, signature generation, banding, candidate verification, or just the phrase “we’d deduplicate it.” “Your dedup pass ran successfully with no errors, but evals two weeks later are worse than the previous corpus version, where do you look” has one correct answer, the threshold-too-aggressive failure mode, check whether domain coverage dropped, not whether the job crashed, because it didn’t. “How would you detect whether your training corpus contains eval benchmark answers” wants n-gram extraction plus a bloom filter check against the full corpus specifically, not a vague “we’d search for overlap.” “A coding benchmark jumped 15 points after a data refresh, what’s your first hypothesis” wants contamination, checked before the result is trusted as a genuine gain, not celebrated as a win. “How do you decide how many times a high-quality but limited-size source should be repeated across a run” wants small-scale ablation experiments measuring marginal value per additional pass, the same proxy-experiment methodology used everywhere else in this pipeline.

What this takes to be frontier-job-ready#

The technical axis is the algorithms themselves, treated as things you could actually implement, not just name: MinHash’s signature generation and LSH’s banding trick, the bloom filter’s role in decontamination, and knowing precisely why each threshold, similarity cutoff, n-gram length, is a tuning decision rather than a fixed constant.

The operational axis is the silent-failure discipline running through this entire post. A dedup or decontamination job that reports success tells you nothing about whether it made the right call, and the actual skill is knowing to check domain coverage and eval numbers after a pipeline change, not just checking that the pipeline didn’t crash.

The autonomy axis is the ablation-driven work Poolside’s own postings describe directly, running the cheap proxy experiment that answers a mix, repetition, or curriculum question for your specific corpus, rather than reaching for a published ratio that was never validated on your actual data. There’s no fixed script for any of these three questions, only the discipline of measuring rather than assuming.

Common mistakes#

Treating a pipeline stage’s successful completion as evidence it did the right thing: dedup, decontamination, and quality filtering all fail silently by construction, a job with zero errors can still have quietly wrecked domain coverage or missed a contaminated eval set.

Setting a similarity or overlap threshold once and never revisiting it: both the MinHash Jaccard threshold and the decontamination n-gram length are empirical tuning decisions specific to a corpus and use case, not universal constants to copy from a paper.

Celebrating a benchmark jump without checking contamination first: a 15-point improvement is exactly as consistent with a memorized test set as with genuine capability gain, and the two are indistinguishable without an explicit decontamination check.

Treating data mixing, curriculum, and repetition as one undifferentiated “data mixing” decision: they’re three separate empirical questions, proportion, order, and repeat count, each requiring its own ablation, and collapsing them into one loses exactly the leverage that makes each one worth tuning independently.

Try it yourself#

Beginner. For a toy corpus of 5 short documents, compute MinHash signatures by hand using $K=3$ hash functions and 2-word n-grams, and verify that two near-duplicate documents produce more matching signature values than two unrelated ones.

Intermediate. Given a training corpus’s bloom filter and a benchmark’s 13-gram set, describe what a false positive in the decontamination check would look like, a clean eval example incorrectly flagged as contaminated, and explain how that risk changes as you shrink the n-gram length from 13 down to 5.

Advanced. Design a proxy-experiment methodology, following the same shape as an isoFLOP sweep, to determine the optimal number of repetitions for a fixed, scarce high-quality data source, given a fixed total token budget that also includes an abundant lower-quality source competing for the same training tokens.

---

The one-sentence version: deduplication compresses an impossible pairwise-comparison problem into MinHash signatures and LSH buckets, decontamination checks 13-grams against a bloom filter because searching a trillion-token corpus directly isn’t an option, data mixing is the single highest-leverage decision in the whole pipeline because a 5% shift can move a benchmark further than most architecture changes, and every one of these stages shares the exact same danger: none of them crash when they get it wrong, they just quietly change what the model learns, and the only way to catch it is checking the actual eval numbers, not the pipeline’s exit code. Once that data is clean and mixed, getting it, and every gradient computed from it, moving correctly across thousands of GPUs is an entirely separate class of silent failure.