Why 'frozen' training jobs usually aren't: processes, memory, and the illusions Linux maintains

An operating system is a referee, not a worker. A CPU core can only truly execute one instruction stream at a time, and RAM is one shared, finite pool. Without a referee, every running program would have raw access to that pool and could corrupt any other program’s data. The OS’s actual job is to give every program the illusion that it owns the whole machine privately, while secretly managing the real, shared, limited hardware underneath.

A process is one running program with its own private illusion of memory: it gets what feels like an enormous, private address space starting at 0, even though the real data is scattered wherever physical RAM had room. That illusion is virtual memory. A thread is a worker inside a process, sharing that process’s memory directly, so two threads can see and modify the same variable; two separate processes cannot, unless they deliberately set up a way to share.

Nearly every failure mode below has the same shape: a piece of hardware or OS reality (finite RAM, one physical GPU context, a fixed descriptor count, a single interpreter lock) leaking through that illusion. Recognizing the shape is what turns “the job randomly froze, no idea why” into a specific, locatable cause.

Page tables: the mechanism behind the illusion#

The OS keeps a per-process lookup table, the page table, mapping virtual addresses (what the program thinks it’s asking for) to physical addresses (where the data actually lives). Every memory access gets silently translated through it by CPU hardware:

$\text{PhysicalAddr} = \text{PageTable}\left[\left\lfloor \frac{\text{VirtualAddr}}{\text{PageSize}} \right\rfloor\right] + (\text{VirtualAddr} \bmod \text{PageSize})$

Memory is divided into fixed-size pages, typically 4KB on Linux. The floor division picks out which page a given address falls into; the remainder is the offset within that page. It’s an apartment lookup: the page table says which physical building a “building number” (virtual page) currently maps to, and the remainder says exactly where in that building to look.

Worked example. An 8GB process with 4KB pages has $8\times10^9/4096 \approx 1.95$ million pages. Its page table, at roughly 8 bytes per entry, is about 15.6MB, not 8GB. That gap, table size versus actual memory size, is exactly what makes the next section’s trick cheap.

fork() and copy-on-write#

fork() creates a child process that starts as a duplicate of the parent, including a copy of its entire virtual memory. Naively that sounds expensive. Copy-on-write makes it cheap: instead of duplicating memory immediately, the OS points the child’s page table at the same physical pages the parent uses, marked read-only. Only when either process writes to a shared page does the OS copy that one page, lazily, one at a time. The cost of fork() itself is proportional to page table size (cheap), not memory size (expensive); the deferred cost, actually copying a page, only happens for however many pages get written to, which for a child that immediately calls exec() to become a different program can be close to zero.

import os, sys

large_data = list(range(10_000_000))
pid = os.fork()

if pid == 0:  # child
    large_data[0] = -999   # this write triggers copy-on-write for this one page
    print(f"Child PID {os.getpid()}: large_data[0] is now {large_data[0]}")
else:  # parent
    os.waitpid(pid, 0)
    print(f"Parent PID {os.getpid()}: my large_data[0] is still {large_data[0]}")

Parent PID 518: allocated large_data before forking
Child PID 519: I have my OWN virtual view of large_data
Child PID 519: large_data[0] is now -999
Parent PID 518: my large_data[0] is still 0

The parent’s value stayed 0, proof the write didn’t propagate back. Separate physical memory came into existence exactly when the child wrote, not before, not for the other 9,999,999 list entries.

Why fork() corrupts CUDA, and why spawn fixes it#

A CUDA context holds live handles and driver-level state pointing at singular GPU resources. Copy-on-write can safely duplicate ordinary CPU memory because a page is just bytes, duplicating it changes nothing about what it means. A CUDA context isn’t like that: there’s exactly one real GPU context, and forking copies the process’s references into it without ever telling the GPU driver a second process now holds them. The child ends up with a corrupted, half-valid reference to something the parent still exclusively controls. This isn’t a bug that could be patched, it’s a structural mismatch between what copy-on-write can handle and what a CUDA context actually is.

spawn sidesteps the problem entirely rather than working around it: it starts a genuinely fresh process from scratch, running the target program’s startup code from the beginning, so it never inherits a copy of an already-initialized CUDA context, because it never had one to begin with.

import torch.multiprocessing as mp
mp.set_start_method('spawn', force=True)   # must be called BEFORE any CUDA call

from torch.utils.data import DataLoader

train_loader = DataLoader(
    dataset, batch_size=32, num_workers=4,
    multiprocessing_context='spawn',   # Linux defaults to 'fork' -- override it explicitly
    pin_memory=True,
)

Linux’s default multiprocessing start method is fork, and PyTorch’s DataLoader uses it unless told otherwise. A training script can work fine for weeks until someone adds a CUDA call before the DataLoader gets built, and then it breaks unpredictably. This is a real, common, hard-to-diagnose class of bug precisely because the ordering constraint, spawn or fork must be set before any CUDA call, isn’t visible anywhere in the error message you eventually get.

The GIL: threads that don’t actually parallelize#

CPython’s Global Interpreter Lock allows only one thread to execute Python bytecode at a time, which exists to prevent two threads from corrupting shared memory by writing to it simultaneously. The side effect: Python threads cannot run Python code in true parallel, regardless of core count.

def cpu_bound_work(n):
    total = 0
    for i in range(n):
        total += i * i
    return total

Timed with 4 threads each doing an equal share of the same total work versus a single-threaded baseline:

Single-threaded baseline (1 unit of work): 1.11s
4 threads doing 4x the work: 4.32s   ->  ~3.9x the single-thread time

No parallelism at all, confirming the GIL serializes bytecode execution exactly as advertised. Running the equivalent comparison with multiprocessing instead of threading is supposed to show a clean win, since separate processes have separate interpreters and therefore separate GILs, but on a machine with few available cores, or at a problem size small enough that per-process spawn overhead dominates, that win doesn’t always show up cleanly in the raw numbers. That’s worth knowing before trusting a microbenchmark: check nproc and problem size before concluding multiprocessing “didn’t help.” The mechanism, separate interpreters mean separate locks, is correct regardless of whether one particular timing run shows it clean.

The production diagnostic tool is py-spy, and it matters specifically because most Python profilers run inside the target process, slowing it down and making them unsafe to attach to a live job. py-spy reads the target process’s memory directly from outside, without running inside it:

py-spy top --pid <PID>        # live view of where time is going
py-spy top --pid <PID> --gil  # restrict to threads currently holding the GIL

Total Samples 991
GIL: 2.20%, Active: 19.78%, Threads: 16
%Own   %Total  OwnTime  TotalTime  Function (filename:line)
8.79%  8.79%   0.330s   0.330s     readinto (socket.py:586)

This isn’t an abstract concern. Anthropic’s ML Systems team has built instrumentation specifically to detect and eliminate Python GIL contention in training code, because RL rollout loops with Python reward models can block on the GIL long enough to leave the GPU idle, with one reported figure putting the throughput loss at around 40%. The GPU was ready to keep working; a lock held by unrelated Python code kept it waiting.

The OOM killer#

Virtual memory is an illusion, physical RAM is finite. When the sum of what every process is actually using exceeds real RAM, Linux’s last resort is the OOM killer: pick one process, typically the largest memory user, and forcibly terminate it. In a 512-GPU distributed job, if the killed process is one specific rank, the other 511 are left waiting forever at a synchronization point the dead rank can never reach. Nothing crashes loudly; the job just stops making progress.

[Mon Jul 13 14:22:07 2026] Out of memory: Killed process 48213 (python3) total-vm:118274816kB, anon-rss:84213504kB, ...

dmesg -T | grep -i "killed process"     # human-readable timestamps
journalctl -k | grep -i "out of memory" # systemd systems

The diagnostic move is extracting the PID from that log line and cross-referencing it against your job launcher’s rank-to-PID map, since the kernel log has no concept of “training rank,” only OS process IDs. That’s the difference between “job appears frozen” as a symptom and “rank 7’s process 48213 was killed at 14:22:07, here’s exactly how much memory it was holding” as a diagnosis.

The NCCL hang#

A distributed job “appears frozen” specifically when timeout and error handling aren’t configured. Configured correctly, the real symptom is a loud, specific error, not silence:

[Rank 1] Watchdog caught collective operation timeout:
WorkNCCL(SeqNum=15173, OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1802710 milliseconds before timing out.

export TORCH_NCCL_ASYNC_ERROR_HANDLING=1
export TORCH_NCCL_DUMP_ON_TIMEOUT=1
export NCCL_DEBUG=WARN      # WARN in production; INFO floods logs at scale
export NCCL_TIMEOUT=1800    # seconds -- set explicitly rather than relying on the default

Worth flagging as the kind of detail that goes stale fast: the async-error-handling environment variable was renamed from NCCL_ASYNC_ERROR_HANDLING to TORCH_NCCL_ASYNC_ERROR_HANDLING in PyTorch 2.2+. Copying a config block from an older tutorial without checking the version it targeted is a quiet, specific way to end up back at silent hangs instead of loud timeouts.

File descriptor exhaustion#

Every process has an OS-enforced cap on simultaneously open files (ulimit -n). A DataLoader that opens thousands of file handles without closing them will hit it.

Soft limit: 1024, Hard limit: 4096
Lowered soft limit to 50 for this demo
Hit the exact production failure after opening 47 files:
OSError: [Errno 24] Too many open files: '/tmp/dummy_fd_47.txt'

Two quick fixes and one real one: ulimit -n 65536 before launching the job is quick but session-scoped; an entry in /etc/security/limits.conf is persistent; the architectural fix is not opening thousands of individual file handles in the first place, using mmap or a memory-mapped Arrow/Parquet reader that keeps far fewer descriptors open regardless of dataset size, rather than treating the ulimit bump as the solution instead of a stopgap.

Pinned memory and mmap#

Ordinary (“pageable”) memory can be moved around in physical RAM, or swapped to disk, at the OS’s discretion. Pinning memory tells the OS never to move it while it’s needed, which matters because a GPU’s DMA (Direct Memory Access) hardware needs a fixed, stable physical address to read from directly, without CPU supervision. Pinned transfers run roughly 3x faster than pageable ones for host-to-device copies.

batch = batch.pin_memory()
batch = batch.to(device, non_blocking=True)   # async transfer

The gotcha worth knowing before it costs you a debugging afternoon: non_blocking=True only actually produces async behavior when the source tensor is pinned. Calling it on an unpinned tensor silently falls back to a blocking copy, no error, no warning. It shows up as “why isn’t this faster,” never as a crash.

mmap extends the same virtual-memory idea to files: it maps a file directly into a process’s address space, and the existing paging machinery loads pieces of it into physical RAM only as they’re actually accessed, “zero-copy” because there’s no separate buffer being filled. The file becomes part of the process’s addressable memory directly, rather than something read into a copy of itself.

Where this actually breaks#

Symptom	Real cause	What to check first
DataLoader corrupts CUDA state, but only on GPU machines	fork copying a CUDA context that can’t be safely duplicated	Multiprocessing start method, before assuming a data bug
Job “freezes” but nothing crashed	One rank OOM-killed, the other 511 wait forever	dmesg -T / journalctl -k for a kill, before assuming a code bug
GPU utilization is low but nothing crashed	Python-side GIL contention blocking the pipeline that feeds the GPU	py-spy top --pid <PID> --gil, before assuming a data problem
”Too many open files” during data loading	Thousands of individual file handles never closed	mmap-based readers, ulimit -n only as a stopgap
Distributed job hangs with no error at all	Timeout/error handling env vars unset or using a renamed variable	The exact env var names for your installed PyTorch version

Common mistakes#

Assuming a “frozen” job means a code bug: the far more common cause at scale is one silently killed process leaving the rest of a coordinated job waiting forever, and the evidence for that lives in the kernel log, not your application log.

Calling fork()-based multiprocessing (or leaving PyTorch’s default) after a CUDA context already exists: the failure is structural, not a rare edge case, and it’s entirely avoidable by setting the start method before any CUDA call, every time.

Trusting non_blocking=True without checking that the source tensor is actually pinned: the silent fallback to a blocking copy means the code looks async and isn’t, and nothing will tell you.

Copying environment-variable-based configuration (NCCL settings especially) from documentation without checking which framework version it targeted: variable names change between versions, and an unset or misnamed variable turns a loud, diagnosable error into a silent hang.

Try it yourself#

Beginner. Using the section 5’s address-translation formula, compute which page number and offset the virtual address 0x2050 falls into, assuming a page size of 0x1000 (4096 in decimal), by hand.

Intermediate. Modify the fork example so the parent writes to large_data[0] instead of the child, predict what the child’s value will show before running it, then confirm: copy-on-write triggers on either side’s write, not only the child’s.

Advanced. Research what system calls multiprocessing_context='spawn' actually issues under the hood, and explain, using the CUDA-context-corruption mechanism above, why a pipeline that works perfectly on CPU-only code can start failing mysteriously the moment GPU operations are introduced, if it was still using fork.

---

The one-sentence version: virtual memory, copy-on-write, the GIL, and file descriptor limits are all the same kind of thing, a finite hardware or OS resource hiding behind a seamless-feeling illusion, and the entire diagnostic skill at this layer is not trusting where a symptom appears, a “frozen” job is usually a dead rank, an idle GPU is usually a Python lock, and the fix is always checking the specific log or tool that reveals the real location rather than guessing from the symptom’s surface. The other place that exact same illusion-versus-reality shape shows up is the filesystem underneath all of this, which has its own version of “someone else’s job just stole your throughput.”