Containers are the process illusion, repackaged, and why Kubernetes doesn't gang-schedule by default

A container isn’t a new idea sitting next to the process and memory model from earlier in this series. It’s that exact model, extended. Recall that a process already has its own private, illusory view of memory: virtual memory, backed by page tables, translated transparently by the CPU. A container gives a process the same trick for everything else. Instead of just memory, a contained process gets its own illusory filesystem (it sees a root directory that isn’t the real machine’s root), its own illusory process table (inside the container it might be PID 1, while the real machine has it at PID 48213), and its own illusory network stack (its own IP, its own interfaces). None of this virtualizes hardware the way a VM does. It’s the same illusion fork()’s copy-on-write already relies on, stretched from “you have your own memory” to “you have your own everything.”

Two Linux mechanisms build the whole thing. Namespaces provide the isolation: a PID namespace gives a process its own numbering starting from 1, a network namespace gives it its own interfaces and routing table, a mount namespace gives it its own filesystem view. cgroups (control groups) provide the resource limiting: how much CPU, memory, and I/O bandwidth this specific group of processes is allowed to consume, enforced by the kernel, not by convention or a polite request.

The payoff is that “works on my machine” becomes structurally impossible rather than just less common. The exact CUDA version, cuDNN version, and every Python dependency ship inside the image, so a container that runs on a laptop runs identically on a production training node, because it never relies on the host’s installed software, only its kernel.

The cgroup memory limit is the OOM killer, at a smaller boundary#

This is worth making explicit because it’s easy to treat as a brand-new failure mode when it isn’t one. The OOM killer picks the largest memory consumer when physical RAM runs out and kills it. A container’s memory limit is a cgroup memory limit, and when a containerized process’s cgroup exceeds it, the exact same kernel mechanism fires, just scoped to the cgroup boundary instead of the whole machine. In Kubernetes specifically this produces a very particular, well-known signature: the pod’s status reason becomes OOMKilled, and the exit code is 137, which is $128 + 9$, SIGKILL, the same signal the OS-level OOM killer always sends. Seeing OOMKilled in kubectl describe pod isn’t a new class of bug to learn. It’s the identical mechanism from earlier in this series, visible at a different layer.

Kubernetes and Slurm solve different problems#

Kubernetes orchestrates containers across machines: scheduling (which node runs which container), health checks (restart one that stops responding), autoscaling, service discovery. Its whole design assumes a failed container is an independent, stateless event: kill it, start a fresh one, nothing else needs to know or care. That assumption is exactly right for serving traffic and exactly wrong for one specific workload.

Slurm is a different animal, an HPC batch scheduler built for job queues, gang scheduling, and topology-aware allocation, predating containers entirely and built for “run this one enormous batch job across exactly the right set of machines, all at once,” not “keep this web service running.”

Gang scheduling: why “all 512 or none” isn’t optional#

A job requesting 512 GPUs needs all 512 simultaneously free before it starts, or it needs to wait until they are. Here’s why partial allocation is actively worse than not starting at all: if 400 GPUs get allocated and those processes begin initializing NCCL while waiting for the remaining 112 to join, they wait forever, because NCCL’s initialization is itself a barrier: every rank must be present before the communicator can form. That’s the same “everyone or nobody” property that makes AllReduce fragile to one missing participant, now showing up before training has even started. Slurm’s gang scheduling exists specifically to prevent this: it holds the whole request until all 512 are free, then allocates them atomically.

Topology matters just as much as headcount. A --constraint=nvlink flag ensures GPUs in the same tensor-parallel group land on machines connected via the fast intra-node NVLink fabric, not scattered across the slower inter-node network. Get this wrong and the MFU-halving straggler penalty from earlier in this series applies to the entire job, permanently, not intermittently. It’s not a fault that resolves or gets mitigated. It’s a placement mistake baked in for the run’s whole duration.

Vanilla Kubernetes has no concept of this, which is why Volcano and Kueue exist#

Kubernetes’ default scheduler admits pods one at a time, greedily, with no notion of “these 512 pods must be admitted together or not at all.” Even Google, the company that originated Kubernetes, needed custom engineering on top of it to manage workloads across 50,000 TPU chips. Third-party schedulers close the gap: Volcano and Kueue are the two most common answers, and current practice increasingly runs both together, Kueue handling multi-tenant quota and admission control, Volcano handling the actual gang-scheduling guarantee underneath it. NVIDIA’s own KAI Scheduler is a newer, GPU-topology-aware alternative, built specifically for homogeneous NVIDIA clusters, though less broadly deployed than Volcano as of this writing. Volcano’s gang scheduling has been measured to cut training job completion time by 40% on shared clusters versus the naive, non-gang-aware default, which is the concrete number behind an otherwise abstract-sounding problem.

What a gang-scheduling failure actually costs#

$\text{Wasted GPU-hours} = N_{\text{allocated}} \times T_{\text{wait}}$

Every GPU sitting in a partial allocation is fully reserved, fully billed on cloud infrastructure or fully unavailable to any other job on owned infrastructure, doing zero useful work for the entire wait. Worked example. 400 GPUs allocated, waiting 25 minutes (about 0.417 hours) for the remaining 112:

$400 \times 0.417 \approx 167 \text{ GPU-hours}$

At a representative $2 to$4 per GPU-hour for high-end training hardware, that single gang-scheduling failure costs somewhere between $330 and$670 in pure idle time, before the job has trained on a single batch. “Wasted GPU-hours” isn’t a vague inefficiency, it’s a specific, computable dollar figure, and it recurs every time a job like this gets scheduled without gang-scheduling guarantees.

The actual configuration, not just the theory#

FROM nvidia/cuda:12.4.1-cudnn-runtime-ubuntu22.04

# System deps first: one layer, cached unless this line itself changes
RUN apt-get update && apt-get install -y python3-pip git

# Python deps in their own layer, so editing train.py below doesn't
# force a slow reinstall of every package
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Application code last: the layer most likely to change on every iteration
COPY train.py .
CMD ["python3", "train.py"]

Docker caches each instruction’s result, and a cache miss on any layer invalidates every layer after it. Ordering rarely-changing system dependencies first and frequently-edited code last means a typical iteration only re-runs the cheap final COPY, not the expensive apt-get/pip install above it.

#!/bin/bash
#SBATCH --job-name=distributed_training
#SBATCH --nodes=64
#SBATCH --gpus-per-node=8              # 64 * 8 = 512 GPUs total
#SBATCH --constraint=nvlink            # tensor-parallel groups share NVLink fabric
#SBATCH --exclusive                    # no other job shares these nodes
#SBATCH --time=48:00:00

# Slurm will not start this job until all 512 GPUs across all 64 nodes
# are simultaneously available -- the gang-scheduling guarantee itself,
# enforced by the scheduler, not by application code.
srun python3 train.py --world-size=512

Both the gang-scheduling guarantee and the topology correctness are declarative here: no application code enforces either, the scheduler simply refuses to start the job otherwise.

# Volcano's PodGroup CRD approximating Slurm-style gang scheduling on Kubernetes
apiVersion: scheduling.volcano.sh/v1beta1
kind: PodGroup
metadata:
  name: distributed-training-job
spec:
  minMember: 512          # Volcano admits NO pod until all 512 can be scheduled
  queue: training-queue
---
apiVersion: batch/v1
kind: Job
metadata:
  name: training-worker
  annotations:
    scheduling.k8s.io/group-name: distributed-training-job
spec:
  parallelism: 512
  template:
    spec:
      schedulerName: volcano
      containers:
      - name: worker
        image: my-training-image:latest
        resources:
          limits:
            nvidia.com/gpu: 1
            memory: "80Gi"   # a live cgroup limit; exceeding it triggers OOMKilled

minMember: 512 is the direct fix for vanilla Kubernetes’ gap: without it, Kubernetes happily schedules pods one at a time and lets 400 sit waiting on 112 forever. With it, Volcano holds back every pod until all 512 can be admitted simultaneously, converting a silent hang into either “the job starts” or “the job stays pending,” never a partial, deadlocked middle state.

Where this fails, and why the fix isn’t “just restart the one pod”#

Kubernetes preempts one pod in a 512-GPU job to schedule a higher-priority workload. That rank disappears from the NCCL communicator mid-collective. This is now a barrier with a permanently-missing participant, indistinguishable from a deadlock from outside, and the entire job has to restart from its last checkpoint, not just the preempted pod, because the NCCL communicator itself is now in a broken state, not just one worker. Losing one pod out of 512 sounds like a 1/512 problem. It isn’t. It’s a whole-job problem, for the same barrier reasons as everywhere else in this series.

Choice	Strength	Weakness	Best for
Slurm	Native gang scheduling, topology-aware placement, mature HPC/MPI integration	Weaker general-purpose ecosystem, no native service discovery or autoscaling	Tightly-coupled, synchronous GPU training
Vanilla Kubernetes	Rich ecosystem (Helm, monitoring, CI/CD, cloud-native tooling)	No native gang scheduling; partial-allocation deadlocks are a real, documented failure	Serving, CI, workloads with independent failure domains
Kubernetes + Volcano/Kueue	Gang scheduling and the full Kubernetes ecosystem together	Two more operators to install and maintain; real integration effort	Organizations standardizing on Kubernetes for both training and serving

Common mistakes#

Assuming a container is a lightweight VM: it isn’t, it shares the host kernel, so a kernel-level vulnerability or crash affects every container on that host, unlike genuine hardware virtualization.

Running Kubernetes for tightly-coupled training without a gang scheduler, then being surprised by partial-allocation hangs that look identical to the deadlocks covered earlier in this series: the surprise is avoidable, the failure mode is well known and has a name.

Treating OOMKilled as a mysterious Kubernetes-specific bug instead of the already-understood OOM-killer mechanism at a cgroup boundary: the fix (raise the limit, reduce memory use, or both) is the same fix as everywhere else this mechanism shows up.

What this takes to be frontier-job-ready, on the axes that actually get checked#

Given “a training job is stuck pending,” know to check whether the scheduler is gang-aware before assuming a resource shortage: a job requesting 512 GPUs on a cluster with 600 free can still hang forever if no single scheduling decision reserves all 512 atomically. Given a pod showing OOMKilled, know it’s the same OOM mechanism from earlier in this series at a cgroup boundary, not a new failure category needing new instincts. That’s the technical axis, and it’s necessary but, as covered in the previous post, not sufficient on its own.

The operational axis shows up here in a specific, verifiable way: at least one frontier lab’s infrastructure role lists “container runtimes + Linux system programming” as a hard requirement separate from “Kubernetes-based critical infrastructure” experience, treating operating these systems in production and understanding Kubernetes as an API surface as two distinct, both-required competencies, not one skill that subsumes the other.

The autonomy axis shows up as a real trade-off with no textbook answer: choosing between Slurm, vanilla Kubernetes, and Kubernetes-plus-Volcano/Kueue for a given workload, the table above, is exactly the kind of judgment call made under the specific constraints of a specific cluster and workload, without a manager specifying which one to pick. There’s no universally correct choice here, only a defensible one given the actual failure mode you’re optimizing against.

Try it yourself#

Beginner. A cluster has 600 free GPUs. A job requests 512 with no gang-scheduling guarantee in place. Describe a concrete allocation sequence under which this job never starts despite the cluster having enough total capacity.

Intermediate. Using the wasted-GPU-hours formula, compute the cost of a gang-scheduling failure where 480 GPUs wait 40 minutes for the remaining 32, at $3/GPU-hour, and compare it to the cost of running Volcano’s PodGroup admission check instead.

Advanced. Explain, in terms of the NCCL communicator’s internal state, why losing one pod out of 512 to preemption requires restarting the entire job from checkpoint rather than just replacing that one pod and having it rejoin. Connect this explicitly to why a barrier has no notion of “close enough.”

---

The one-sentence version: a container is the process illusion from earlier in this series (private memory, now private filesystem, process table, and network too) built from namespaces for isolation and cgroups for enforcement, a Kubernetes OOMKilled pod is the exact same OOM killer at a smaller boundary, and the entire Slurm-versus-Kubernetes question comes down to one property: does this workload treat a missing participant as routine (serving) or as a whole-job failure (synchronous training), because vanilla Kubernetes was only ever built for the first case. Getting the scheduling right only guarantees the GPU is yours; what it actually does with the cycles it’s given is a separate question entirely.