Parallel filesystems, checkpoint I/O, and the noise nobody's code caused
How striping lets hundreds of GPUs read the same dataset without one drive becoming the bottleneck, why data loading and checkpointing are opposite I/O problems, and the real incident where someone else's job quietly stole training throughput.
Everything in the previous post assumed training data was already in memory. It has to get there from somewhere, usually a disk, often thousands of disks working together. A filesystem is the abstraction that lets a program say “open this named file” instead of “read these specific physical sectors on this specific drive.” A parallel filesystem extends that so hundreds of GPUs can read training data or write checkpoints simultaneously, without one drive becoming a bottleneck for everyone.
A single GPU reading training data, or a single checkpoint write, is a solved problem on a laptop. It stops being solved the moment hundreds of GPUs need to do it at once: a 512-GPU training cluster requires sustained storage throughput of 400 to 600 GB/s during data-loading phases. This isn’t a rounding-up-for-effect number either. Meta’s Llama 3 training infrastructure achieved 380 GB/s sustained throughput using a parallel filesystem backed by more than 1 petabyte of NVMe capacity, a real, checkable figure from a real, named training run.
Striping: many drives, one logical file
Parallel filesystems (Lustre, IBM Spectrum Scale/GPFS, WekaFS) split a single logical file across many physical storage targets, OSTs in Lustre’s terminology, so many drives can be read from or written to in parallel for one file, rather than one drive serving the whole file sequentially.
A published benchmarking study on the Argonne Leadership Computing Facility’s Polaris system gives a concrete sense of scale: its Lustre-based parallel filesystem holds 100 petabytes across 40 metadata servers and 160 object storage targets, with 650 GB/s of aggregate bandwidth. The default striping settings weren’t left alone either, the study explicitly widened them: all files were reconfigured to stripe across every available OST, with a 64MB stripe size. That’s a standard, real tuning move, not a hypothetical one: the default striping configuration is rarely optimal for the concurrent, bursty write pattern a checkpoint save actually produces, so engineers explicitly widen the stripe rather than trust the out-of-the-box setting.
Two I/O patterns, two different bottlenecks
Different phases of training impose genuinely different access patterns on storage. Dataloaders, tokenizer outputs, and runtime caches rely on many small, latency- and IOPS-sensitive reads. Checkpointing is the opposite: a few enormous, bandwidth-dominated writes. This is the filesystem-layer version of the compute-bound-versus-memory-bound distinction from earlier in this series: a filesystem tuned to serve one pattern well is not automatically good at the other, and treating both as one undifferentiated pool of shared storage amplifies contention between them rather than avoiding it. Production teams increasingly separate the two physically, one storage tier for data loading, a different one for checkpoint writes, rather than pretending “storage” is a single interchangeable resource.
A real, documented failure: the invisible-noise problem
Here’s a genuinely useful incident to sit with, from a published large-scale training report. Early production runs showed substantial run-to-run variation in tokens per second, despite identical model and parallelization settings, nothing about the job itself had changed. Digging into application and system metrics traced the dominant residual source of variability to I/O interference on the shared global filesystem, caused by unrelated workloads producing transient bandwidth and metadata slowdowns.
This is the direct filesystem analog of the OOM killer from the previous post: an invisible, shared-resource failure that looks like random noise in a training-throughput graph until you specifically know to check whether someone else’s job on the same shared filesystem, not your model, not your code, not your GPUs, is the actual cause.
Checkpoint I/O is its own engineering problem
Large models get sharded and parallelized across GPUs using data-, tensor-, and pipeline-parallelism plus ZeRO-style redundancy elimination, collectively “4D parallelism.” That leaves a fragmented application state: each GPU holds either a replica or a unique shard of the model or optimizer state. Frameworks like DeepSpeed checkpoint this as one file per shard, producing an N-by-M file layout, hundreds of files per rank, that all need to write concurrently without overwhelming the filesystem’s metadata servers. That’s a specifically filesystem-level engineering problem, not just “save the model to disk.”
The scale here is worth stating plainly: checkpointing a 25-billion-parameter mixture-of-experts model can produce a 400GB checkpoint. Some production platforms mount checkpoints via HDFS-FUSE and use a striped parallel read specifically to speed up checkpoint resumption when many worker nodes need to download the same large file concurrently at job startup, the exact same striping idea from earlier in this post, applied to reads instead of writes.
Object storage: the durable, cheaper second tier
Parallel filesystems are expensive and tuned for active, high-throughput training, not for holding onto data forever. The standard architecture writes actively to parallel storage during training, then asynchronously replicates critical checkpoints to object storage for durability, a tiering approach that trades some of the parallel filesystem’s performance for object storage’s economics, roughly a tenth of the cost per terabyte.
Standard filesystem-layer checklist
| Symptom | Likely cause | Standard fix |
|---|---|---|
| Training throughput varies run-to-run with identical settings | Shared filesystem contention from unrelated jobs | Isolate data-loading and checkpoint I/O onto separate storage tiers; monitor per-job I/O attribution |
| Checkpoint saves take unexpectedly long | Default striping too narrow for concurrent multi-rank writes | Widen stripe count and size explicitly, e.g. stripe across all available OSTs |
| Checkpoint resumption is slow at job startup | Sequential download of one large file to many nodes | Striped parallel read (e.g. striped HDFS-FUSE) |
| “Too many open files” during data loading | Thousands of individual file handles, one per shard | mmap-based readers; raise ulimit -n as a stopgap, not a fix |
Common mistakes
Treating “storage” as one undifferentiated resource: data loading and checkpointing have opposite I/O profiles, and tuning a filesystem for one at the expense of the other is a common, avoidable source of contention.
Assuming a throughput regression means a code or model change: the invisible-noise incident above is a real case where nothing about the job changed at all, and the actual cause was a neighbor on the same shared filesystem.
Leaving default striping settings in place for checkpoint-heavy workloads: the default is tuned for a generic access pattern, not for the specific concurrent-write burst a multi-rank checkpoint save produces.
Treating checkpoint files as a single artifact rather than what they actually are under 4D parallelism, hundreds of files per rank, each needing coordinated, striped I/O to write and read back efficiently at scale.
Try it yourself
Beginner. Using the Polaris figures above (100PB, 160 OSTs, 650 GB/s aggregate), estimate the average bandwidth available per OST if load were perfectly balanced, and compare it to the study’s actual per-file stripe width of “all available OSTs.”
Intermediate. Sketch, as a design decision rather than code, how you’d separate data-loading storage from checkpoint storage for a 512-GPU job, given the 400-600 GB/s data-loading requirement cited above and a 400GB checkpoint that needs to write in well under a minute.
Advanced. The invisible-noise incident was diagnosed by correlating application-level throughput metrics with system-level I/O metrics. Design a monitoring approach that would catch this class of problem automatically, flagging “throughput dropped and it’s not your job” as a distinct alert from “throughput dropped and it is.”
The one-sentence version: a parallel filesystem’s entire trick is striping one logical file across many physical drives so many GPUs can read or write it at once, data loading and checkpointing are opposite I/O problems that punish being treated identically, and the most expensive failures at this layer are the quiet ones, contention from a neighbor’s job or a checkpoint layout nobody engineered on purpose, that look exactly like your own code regressing until you know to check somewhere else first. Once the data is off disk, it still has to cross between machines, which is its own layer with its own way of hanging the whole cluster at once.