MLOps & Training Frameworks

The rest of Part 10 gets electrons to the GPU, keeps the node healthy and recovers it when it dies. This chapter covers what runs on that fleet to turn its arithmetic into a trained model. That is where a power-bound, depreciation-clocked data center either earns its megawatts or wastes them. A cluster can be perfectly provisioned (Chapter 10.5), perfectly scheduled (Chapter 10.1) and perfectly observed (Chapter 10.6) and still run at 30% model-FLOPS-utilization because someone chose the wrong parallelism layout or let checkpoint overhead eat a fifth of every run. The framework and the control plane set the conversion efficiency between the capex you bought and the weights you ship.

Three choices dominate. The training framework (FSDP2, DeepSpeed/ZeRO or Megatron-Core) sets which parallelism strategies you can express cheaply and how much team time goes to plumbing instead of research. The orchestration plane (Kubernetes-native, Ray or SkyPilot-over-Slurm) owns the job lifecycle, so it decides how gang scheduling, topology placement and elastic recovery behave when a node fails, not if. Reproducibility and checkpointing decide whether a run is a scientific artifact you can re-derive or a one-off you can never explain. Inference serving (vLLM, TensorRT-LLM, SGLang, disaggregation, goodput-optimal scheduling) is a different discipline with different SLOs. Chapter 10.11 owns it end to end; this chapter stops at the trained checkpoint and its handoff to a release.

A single accelerator cannot hold a frontier model. A dense 70B model in BF16 needs about 140 GB for weights alone. Training state (optimizer moments, gradients, master weights) pushes the real footprint to roughly 16 bytes per parameter with mixed-precision Adam: 2 bytes BF16 weight, 2 bytes BF16 gradient, and 12 bytes of FP32 master weights and Adam moments. That is two bytes per parameter more than the ~14 bytes/parameter checkpoint recipe in Chapter 9.4, which leaves out the gradient because the checkpoint is taken after the optimizer step has consumed it. The 14-byte figure is one recipe; FSDP2 with FP32 parameters writes about 12. Even a GB200 GPU with 186 GB usable HBM, or the larger HBM3E stacks on GB300-class parts, cannot hold a 70B+ training job, let alone a 400B+ one. A framework's first job is to shard that state across the fleet; its second is to keep the GPUs busy while doing so. Three families dominate production in 2026, on a spectrum from 'least code, least control' to 'most code, most control.'

PyTorch FSDP2 (Fully Sharded Data Parallel, the rewritten v2 that replaced the FlatParameter model with per-parameter DTensor sharding) is the path of least friction for PyTorch-native shops. It shards parameters, gradients and optimizer state across the data-parallel group. It rebuilds each layer's full weights on demand with an all-gather that prefetches the next shard while the current one computes, the same idea as DeepSpeed ZeRO-3, expressed natively in the framework. FSDP2 is the right default for mid-scale training (hundreds to low thousands of GPUs) and for teams that want minimal abstraction over their model code. Its weakness is that parameter sharding is a data-parallel technique. Once you need tensor or pipeline parallelism to split a single layer across GPUs, FSDP2 alone is not enough: you compose it with PyTorch's device-mesh primitives or reach for Megatron.

DeepSpeed/ZeRO (created at Microsoft, now a PyTorch Foundation-hosted project) pioneered the partition-the-redundancy idea: ZeRO-1 shards optimizer state, ZeRO-2 adds gradient sharding, and ZeRO-3 adds parameter sharding (the FSDP equivalent). Its durable niche in 2026 is the offload-heavy lane. ZeRO-Infinity streams parameters and optimizer state to CPU DRAM and NVMe, so a memory-starved cluster can train a model that would not otherwise fit. That buys fit at the cost of throughput: the offload traffic contends with compute and drags MFU down, but the job runs on hardware that could not otherwise hold it. DeepSpeed also keeps strong MoE support and a large installed base, so 'we already have DeepSpeed expertise' is a legitimate reason to stay.

Megatron-Core (NVIDIA) is the frontier-scale framework, the place to go when every absolute MFU point you gain cuts the GPU-hours a fixed training workload needs across the whole fleet. Megatron implements true model parallelism: tensor parallelism (splitting individual matmuls across GPUs), pipeline parallelism (splitting layers into stages), and now expert and context parallelism, with hand-tuned communication-overlap kernels. It carries the most code and the heaviest operational burden, and it repays that burden with the highest realized utilization at the largest scales. In 2026 these families increasingly compose rather than compete. Megatron-FSDP layers FSDP-style sharding under Megatron's tensor, pipeline and expert dimensions, while TorchTitan composes FSDP2 with PyTorch's own tensor, pipeline and context parallelism. Most frontier labs run a hybrid rather than a single pure framework.

Precision is the other framework-level lever. NVIDIA's Transformer Engine runs FP8 training on Hopper and later GPUs, and MXFP8 and NVFP4 training on Blackwell; Megatron-Core builds on it, and TorchTitan ships MXFP8 and NVFP4 recipes through torchao (both as of October 2026). DeepSeek-V3 was trained in FP8 mixed precision (technical report, December 2024). Low precision speeds up the matrix multiplies, but the recipes keep master weights and optimizer state in higher precision: DeepSeek-V3 kept FP32 master weights and BF16 Adam moments. The 16 bytes per parameter therefore shrinks far less than the compute time does. The price is validation: run a loss-curve parity check against a BF16 baseline before you trust a lower-precision recipe with a full run.

An MFU figure means little until you name the peak it is measured against. On Hopper the dense FP8 peak is twice the dense BF16 peak, so the same run shows half the MFU against an FP8 denominator; the MFU figures in this chapter are against dense BF16. Chapter 7.10 owns the precision ladder, the dense-versus-sparse peaks, and how FP8, MXFP8 and NVFP4 training scale their tensors (per-tensor, block and microscaling).

Once you have a framework, the decisive choice is the parallelism layout: how you map the model and the batch onto the physical fleet across what used to be three dimensions and is now four or five. Each dimension trades communication for memory along a different axis, and the best mapping depends on your scale-up domain size, back-end fabric and model shape. There is no portable default; a layout tuned for 8-GPU HGX nodes is wrong on a 72-GPU NVL72 domain.

• Data parallelism (DP): replicate the model, split the batch, and all-reduce gradients each step. It is the easiest to reason about, but every replica needs the whole model, which is why it is paired with sharding (FSDP/ZeRO) to remove the redundancy.
• Tensor parallelism (TP): split an individual matmul (and its layer) across GPUs. Communication is heavy and latency-sensitive (an all-reduce inside every layer), so TP must stay inside the scale-up domain: NVLink, not the scale-out fabric. This dimension couples the layout to hardware most directly. The domain is a ceiling, not a target: attention heads must split evenly across the TP group, and each GPU's share of a matmul shrinks as TP grows, so TP is usually kept at 8 or below, even across a 72-GPU NVL72. NVIDIA's own GB200 pretraining recipes (NeMo 25.04, April 2025, FP8) run TP between 1 and 8. The bigger domain pays off through wide expert and context parallelism and NVLink-local sharding groups (Chapter 8.5).
• Pipeline parallelism (PP): split the layers into stages on different GPU groups and stream micro-batches through. Communication is cheap, since only activations cross stage boundaries, but it creates the pipeline 'bubble' of idle time at fill and drain. Plain 1F1B schedules cap activation memory but leave the bubble unchanged. Interleaved 1F1B (several virtual stages per GPU) divides the bubble by the interleave factor at the cost of proportionally more stage-to-stage traffic. Zero-bubble schedules (ZB-H1/H2, DeepSeek's DualPipe) fill most of the rest by splitting the backward pass, the fuller versions at the cost of extra memory.
• Expert parallelism (EP): for Mixture-of-Experts models, place different experts on different GPUs and route tokens to them with an all-to-all. The all-to-all is the bottleneck. A wide NVLink domain (EP32 on NVL72 versus EP8 on a node) is what makes wide expert parallelism throughput-positive instead of a network disaster (Chapter 8.5).
• Context parallelism (CP): split along the sequence-length dimension so a single very-long-context example fits. It is the fifth dimension, and increasingly necessary as context windows grow.

Getting the layout wrong has concrete, compounding costs. TP that spills out of the NVLink domain onto the scale-out fabric collapses MFU, because the latency-sensitive in-layer all-reduce now crosses a slower hop. PP with too few micro-batches spends a large fraction of each step in the bubble. EP that is too wide for the fabric saturates the all-to-all. The layout is where the abstract 'parallelism strategy' meets the topology of the building, and it is the tuning decision that moves MFU most.

A framework runs one job. The orchestration plane runs the fleet of jobs. It places them topology-aware, gang-schedules every rank so a job gets all its GPUs or none (a half-placed synchronous job deadlocks instead of starting slowly), preempts and re-queues, and re-forms the job when ranks die. This is the boundary with the scheduling plane of Chapter 10.1 and Chapter 10.2: the scheduler decides where a job lands, while the training-orchestration layer above it decides how the job's lifecycle is driven and how pre-processing, training, eval and checkpointing are wired together. Three patterns dominate.

Kubernetes-native (Kubeflow Trainer with Kueue for quota and all-or-nothing admission, on a gang-capable scheduler such as Volcano or NVIDIA's KAI, with topology-aware placement and increasingly Dynamic Resource Allocation for GPUs) suits operators who already run Kubernetes and want training to be one more workload on the same control plane. Kubeflow Trainer's v2.1 release (November 2025) brought a distributed data cache, Kueue-based topology-aware scheduling and Volcano gang scheduling. Its v2.3 release (August 2026) introduced an OptimizationJob resource to drive hyperparameter search. The strength is operational unification: one cluster, one RBAC model, one observability stack. The weakness is that vanilla Kubernetes was never built for gang-scheduled, topology-sensitive, all-or-nothing jobs, which is why Volcano, Kueue and KAI exist to retrofit that behavior.

Ray is the application-framework answer: an actor/task runtime that expresses distributed training, RL rollouts and data pipelines as Python and handles fine-grained orchestration inside a job. The 2026 norm is hybrid: Kubernetes (or KubeRay) for coarse-grained resource management, and Ray for fine-grained application orchestration inside the allocation. Ray is the natural home for RL and post-training, where a run is a choreography of inference-heavy rollout generation feeding a smaller trainer (Chapter 1.4) rather than one monolithic SGD loop.

SkyPilot over Slurm or cloud targets operators whose problem is cost arbitrage and burst, not cluster operations. It abstracts a job over heterogeneous capacity (multiple clouds and neoclouds, spot and on-demand), chases the cheapest available GPUs, and handles provisioning and recovery. For a tenant renting from neoclouds (Chapter 10.9) instead of running a Kubernetes estate, SkyPilot is often the right top layer. Slurm, developed by SchedMD (which NVIDIA acquired in December 2025, with a commitment to keep it open source and vendor-neutral; Chapter 10.1), remains the HPC-heritage batch scheduler underneath many of these clusters and is entirely viable on its own for a homogeneous, single-tenant training estate.

Above the orchestrator sits the layer that makes training reproducible: the training control plane, which has three capabilities. Experiment tracking (MLflow, Weights & Biases and their kin, open source and commercial) records every run's hyperparameters, metrics, hardware, code version and artifacts, so 'which config produced this checkpoint?' is answerable months later. Lineage and artifact management tie a shipped model back to the exact dataset snapshot, tokenizer and base checkpoint it descended from, the same provenance discipline the data-governance regime of Chapter 10.10 requires for legal defensibility. Sweep orchestration drives hyperparameter search with grid, random and Bayesian/bandit methods (Hyperband, ASHA, population-based training) that adaptively kill underperforming trials, so the GPU budget goes where it earns the most signal.

The hard question is where the budget goes. At frontier scale you cannot sweep the full hyperparameter space at the target model size, because a single run is the budget. The workable pattern is scaling-law-guided search: sweep cheaply at small scale, fit the scaling laws (plus the maximal-update parameterization, muP, that lets a learning rate found at small width transfer to large), and test the transfer assumptions before extrapolating the few choices that matter to the full run. An operator that sweeps blindly at scale burns megawatts on trials a well-designed small-scale sweep could have eliminated. One that skips small-scale search entirely gambles the full-run budget on untuned hyperparameters. The control plane is what makes the cheap-search-then-extrapolate workflow auditable.

Checkpointing is the mechanism that most directly converts the fleet's raw availability into goodput: useful work delivered within the job's quality limits or, as a time fraction, the share of wall-clock time that advances the model instead of being lost to faults and recovery. Synchronous coupling raises the stakes: SemiAnalysis reported roughly 7 days as the typical MTBF for a 512-H100 cluster at a top-tier operator. That is one operator's figure, not a per-GPU scaling law, so derive your job's interruption distribution from your own fleet telemetry before choosing a checkpoint cadence. Two costs bracket the choice. Checkpoint too rarely and a failure throws away hours of progress (lost-work cost). Checkpoint too often and the write stall, or the async drain bandwidth, eats into step time (overhead cost). Chapter 9.4 derives checkpoint cadence from measured whole-job interruptions and checkpoint/restart cost; as whole-job MTBF shrinks, the cadence tightens.

The modern answer is asynchronous, multi-tier checkpointing, one of the larger goodput levers. Tier the writes. Capture state into local memory or NVMe, commit complete shards and metadata, replicate to peers outside the failed node, and drain asynchronously to durable parallel-filesystem or object storage outside the failure domain. Overlap the drain with computation, but measure capture pauses and bandwidth contention, because GPUs can still wait. A local or peer copy counts only if its complete, committed state survives the failure you are recovering from. Google reports multi-tier checkpointing restores in under a minute across clusters of thousands of nodes, against up to 30 minutes from object storage alone, and a 6.59% ML-goodput gain on one 35K-chip TPU v5p workload. Both are vendor-reported. Detection, fencing, replacement allocation and the first valid step add separate clocks, as the recovery trace in Chapter 10.7 shows.

The full engineering of checkpoint storage, bandwidth sizing (the assumed 14 bytes/parameter serialized-state recipe) and elastic restart lives in Chapter 9.4; the KV-cache side of the memory hierarchy is Chapter 9.7. This chapter's requirement is narrower: your framework and orchestrator must support async, distributed and elastic checkpointing, or the most expensive part of your fleet idles during every recovery.

How far one failure spreads is a framework property. torchft, paired with TorchTitan's fault-tolerant HSDP training loop, lets a job continue after a replica group fails and restores that group from a healthy peer. It covers only the replicated dimension. NVIDIA's Resiliency Extension can restart a job in-process or in-job, and it does so without asking Slurm for new nodes. Test which failures each one absorbs on your stack (Chapter 10.7).

Reproducibility in large-scale training is a spectrum, and where you sit on it is a deliberate cost decision. Bit-exact reproducibility (re-run the job, get byte-identical weights) requires pinning the entire stack (framework, CUDA, cuDNN, NCCL, driver), forcing deterministic kernels, fixing every RNG seed and capturing the exact data order. Deterministic kernels can cost throughput, and alternative reductions can change rounding order. Pay that MFU cost only where exact replay is required, and test it on the fixed stack. PyTorch documents no complete reproducibility guarantee across releases or platforms. Statistical reproducibility (re-run and land within noise on every eval) is a different target with explicit numerical and quality tolerances. It tolerates non-deterministic kernels but still requires pinned versions, captured seeds and data order, and recorded hyperparameters.

Skimping has concrete, recurring costs. A run that drifts because someone bumped the NCCL version mid-sweep produces results you cannot compare. A collective-library upgrade can also change correctness, not only speed. ROCm 10.1.0's release notes warn that RCCL's Low Latency (LL) all-reduce can return wrong results on MI350-series GPUs; the workaround restricts protocols to LL128 and Simple (Chapter 10.4). NVIDIA's Resiliency Extension tells users to avoid NCCL 2.28.3–2.28.8 for in-process restart. Treat a driver, CUDA/ROCm or NCCL/RCCL change as a release: replay a short numeric check against a pinned baseline and re-test recovery before it reaches a production run. A model you cannot trace back to its dataset snapshot is one you cannot defend in a copyright dispute (Chapter 10.10) or reproduce after the person who trained it leaves.

The minimum reproducibility standard for a serious operator is pinned and recorded software versions, captured seeds, captured data-loader order, a recorded full config, immutable artifact lineage from dataset to checkpoint to eval, and optimizer and data-loader state for resumption. It is cheap to maintain if the control plane enforces it from day one and ruinously expensive to reconstruct afterward.

The five layers of this chapter form a chain. The useful training a fleet delivers is roughly MFU times goodput, so it is the product of the links, not the strength of the best one. The framework sets the ceiling on MFU. The parallelism layout determines how close you get to that ceiling on your topology. The orchestrator determines how much wall-clock goes to training versus queuing, mis-placement or re-forming. Checkpointing determines how much of a failure you lose, and reproducibility determines whether any of it is a repeatable result. A best-in-class framework at 54% MFU, on a topology-blind orchestrator that places tensor-parallel traffic across the spine and with synchronous checkpoints stalling every step, can deliver less useful training than a simpler stack where every link is merely competent. Find the weakest link and fix it, because that one bills megawatts for nothing.

The economic tie-back is direct. Price reliability from your fleet's measured badput categories. In the guide's 90%-versus-96% goodput scenario (Chapter 14.1), the gap on a gigawatt training estate can be worth hundreds of millions of dollars of effective compute per year; your own gap comes from your own measurements. Whatever it is, it is won or lost in this chapter: in the parallelism layout, the gang scheduler's topology awareness, the checkpoint tier and the discipline of pinned, traceable runs. The goodput-versus-availability reframing that governs facility redundancy (Chapter 12.2) governs the software stack too: spend on what protects useful training time, not on nines the synchronous workload does not value.