Provisioning, Bring-Up & Infrastructure as Code

By the time this chapter matters, the capital decisions are made. The silicon is bought (Part 7), the racks are integrated and leak-tested at the factory (Chapter 7.14), the fabric is cabled (Part 8), and the node software stack of driver, CUDA/ROCm, NCCL and firmware is specified (Chapter 10.4). What remains is the unglamorous seam that turns all of that into a node a scheduler can place a job on: provisioning and bring-up.

Nobody buys a data center to admire its PXE pipeline, but every day a rack sits powered and unprovisioned is a day of depreciation with no revenue. A fleet earning roughly $12–13B per GW per year defers about $23–25M of contracted revenue a week for every 100 MW still waiting on its first paid job, and about $30k a week for each NVL72-class rack (Chapter 2.2 works through the revenue timing).

One decision repeats at every layer of that seam: whether physical hardware is handled as pets or as cattle. In the pet model, a human follows a runbook: SSH into a BMC, mount an ISO, install an OS, run a few checks, bless the node. That works at ten nodes and fails at ten thousand. In the cattle model, every node is described declaratively, provisioned by a pipeline, validated by an automated gate and reimaged on demand. Only that model survives fleet scale and the generational refresh that defines the 2026 era.

We trace the bare-metal lifecycle (BMC/Redfish to cluster-join), the golden-image versus config-management fork, the burn-in and acceptance gates a node must clear, the build-versus-buy choice on the management platform, and the day-2 loop that makes provisioning continuous rather than one-shot.

A bare-metal node has a deterministic lifecycle, and every transition should be machine-driven rather than human-driven. The canonical sequence, as implemented by MAAS, OpenStack Ironic and every neocloud's home-grown equivalent, is enroll → power-control → network-boot → image → configure → validate → join → (eventually) decommission. Each step is an out-of-band (OOB) or in-band operation with its own protocol, and the protocol choice carries consequences.

Enroll and power-control happen out-of-band, through the BMC. The baseboard management controller is a small always-on computer beside the host, reachable on a separate management network even when the host is powered off. Historically you spoke to it with IPMI, a 1998-era protocol with a long CVE history and weak authentication. The modern path is Redfish, a RESTful, JSON, schema-defined DMTF standard that does the same job (power on, off and cycle, set a one-time PXE boot, read sensors, mount virtual media) but can be scripted and secured, and is increasingly the only thing a 2026 platform targets.

The choice has teeth. A fleet still leaning on IPMI inherits a fragile, hard-to-secure management plane, and that becomes a security liability because the BMC is a root-of-trust target (see Chapter 11.4). A fleet standardized on Redfish gets one uniform, auditable OOB API across Dell iDRAC, HPE iLO, Lenovo XCC and Supermicro. Redfish is also how power-smoothing and capping controls reach GB300-class racks at run time, so the management plane you build for provisioning is the one you later use for grid-friendliness (Chapter 4.5).

Network-boot and image happen in-band, over the provisioning fabric. Once the BMC sets a one-time PXE boot and powers the node on, the node network-boots into an ephemeral environment (a small in-memory Linux). That environment writes the real OS image to local disk (MAAS uses curtin, Ironic uses its deploy ramdisk) and hands off to cloud-init for first-boot configuration. Configure applies the node's role: storage layout, network (above all the fabric NIC and RDMA setup) and the pinned node-software stack. Validate runs the acceptance gates described below. Only after the gates pass does the node join the scheduler's pool and become eligible to carry work.

The entire choreography must be expressible as code, parameterized by an inventory, and idempotent, so that re-running it on a flaky node is safe and produces an identical result. MAAS describes it as Allocate → PXE boot → ephemeral environment → curtin → cloud-init → reboot → deployed.

Inside the declarative camp sits a second fork that practitioners argue about endlessly. Both answers are defensible, depending on what you optimize for: immutable golden images or live configuration management.

The golden-image model bakes a complete, tested OS into one immutable, versioned, signed artifact: kernel, pinned GPU driver, CUDA/ROCm, NCCL, container runtime, firmware-update tooling and health agents. Provisioning a node means writing that exact image and rebooting, so no per-node configuration step can drift. The payoff is the strongest reproducibility and the smallest drift surface: a node is running golden-image v2026.06.2 or it is not, and 'not' is a clean, detectable state. The cost is that every change, whether a driver bump, a security patch or an agent update, means building, testing and rolling a new image across the fleet. That makes small changes heavyweight and pushes you toward batched releases.

The configuration-management model (Ansible, Puppet, Salt) starts from a thinner base image and converges the node to a desired state by running roles or manifests in place. It is lighter for incremental change (patch one package, re-run the play) and more flexible for heterogeneous fleets. The cost is drift. A long-lived mutable node accumulates state no manifest captures: a hand-fixed config from an incident, a half-applied update, a package installed to debug something at 3 a.m. Over months no two nodes are truly identical. For a fleet whose headline failure mode is 'a marginal node silently degrades a synchronous job', drift is a goodput tax, not a hygiene issue.

The 2026 consensus for AI fleets leans toward immutable golden images for the OS and node-software stack, with thin config management only for per-node state such as hostname, fabric addressing and role labels. The reasoning matches the node-software stack: a synchronous training cluster wants every node bit-for-bit identical, and the cheapest guarantee is to make each node a disposable instance of a versioned artifact rather than a long-lived entity you keep patching. The table below lays out the tradeoff.

Provisioning a node is not the same as trusting it. The costliest provisioning mistake on an AI fleet is admitting a marginal node into a synchronous job: one that boots fine and passes a smoke test but fails under sustained thermal and bandwidth load. The whole job moves at the speed of its slowest straggler and restarts from a checkpoint when any node fails. So one bad node does not degrade gracefully; it strands hundreds or thousands of healthy GPUs (Chapter 10.7).

The defense is a burn-in and acceptance gate: a battery of stress and benchmark tests, with explicit pass/fail thresholds, that a node must clear before it enters the schedulable pool. It is the provisioning-side counterpart to the factory burn-in done at integration (Chapter 7.14), and it must be repeated, because nodes that passed at the factory routinely fail again after transport, install and the first thermal cycle in the actual hall.

The mature reference is a phased, subsystem-by-subsystem validation that builds from the single GPU to the full cluster, codified by practitioners like Together AI. The phases run in order: preparation/config, GPU validation, NVLink/NVSwitch, network fabric, storage, end-to-end model build, then continuous observability. A failure is thereby localized to the cheapest layer that can explain it, instead of surfacing as a mysterious slowdown in a full training run. Each phase has specific tools and thresholds.

The fabric gate anchors to a measured baseline, not a portable ratio. Together AI’s August 2024 cluster-testing guide gave roughly 370 GB/s against 400 GB/s as a rule of thumb, without a transferable rank, rail or normalization basis. For each purchased node or tray, freeze a golden-node baseline and tolerance across the topology, NCCL build, collective, message-size range and scale you will run. A result below that tolerance triggers diagnosis of optics, cabling, routing, congestion, NCCL configuration, firmware and PCIe/NUMA placement before admission, because one weak node can slow a whole synchronous job.

On an NVL72-class rack the unit of acceptance is the rack, not the tray. Its 72 GPUs share one NVLink domain through nine switch trays and a copper backplane, so a tray-by-tray gate can pass every tray and still admit a rack with a weak switch tray or a damaged backplane cable. SemiAnalysis’s August 2025 benchmarks blamed early GB200 rack downtime on that backplane and on diagnostics too immature to find faults quickly (Chapter 10.4). Confirm that every GPU has registered with the fabric and has all its NVLinks up, that the IMEX domain is up on every tray and that the switch trays report healthy. Then run multinode nvbandwidth and an NCCL all-reduce across all 72 GPUs against a golden-rack baseline. Chapter 13.7 engineers the fabric side of that gate.

Two more checks belong in the soak. A liquid-cooled rack needs its loop watched at full load: log CDU flow, coolant ΔT and the leak sensors, since a badly seated cold plate or a manifold branch short of flow appears only at sustained power. Every node, air- or liquid-cooled, also gets a silent-data-corruption screen, because a GPU that returns wrong answers raises no error and nothing else in the gate would flag it (Chapter 13.8 covers both checks).

Burn-in has no universal hour count or pass bar. Published reference points run from SemiAnalysis’s 2024 advice of at least 3–4 weeks of factory burn-in down to the 8-hour GPU and network burn-in that opens its 2026 rental evaluations (Chapter 13.8). Set the duration from the OEM’s procedures, the workload’s thermal and power cycles, the contract’s acceptance terms, fleet size and the defect-discovery curve you observe, and write down when you will stop. Freeze pass, re-soak, quarantine and vendor-return criteria in the contract before testing; a timer expiring is not acceptance. ATP-10A above keeps the failed results alongside the repair and the successful rerun, and a missing record keeps the node fenced.

Everything above can be assembled from open components or bought as an integrated stack. This is the provisioning-layer instance of the build-versus-buy fork that recurs across the platform (Chapter 10.1). The open, assembled path wires together a bare-metal provisioner (MAAS or OpenStack Ironic), an IaC layer (Terraform driving Redfish/Ironic, plus Ansible/cloud-init), a scheduler (Slurm or Kubernetes) and your own health-check and observability glue. The integrated vendor path buys a turnkey product from NVIDIA. Base Command Manager provisions and manages the cluster. Mission Control layers autonomous validation and hardware recovery on top. It supports DGX B200/B300 and DGX and OEM GB200/GB300 NVL72 systems; Vera Rubin is not yet listed (Mission Control 2.3.1, October 2026). Both ship the bring-up, health-check and recovery workflows as a supported product.

The tradeoff is the familiar one. The integrated path compresses time-to-first-job, because the workflows are pre-built, validated against the reference architecture and supported. The price is lock-in: one vendor's hardware and operating model. Base Command Manager costs nothing for an operator's own clusters; the free licence covers up to eight accelerators per system, comes without NVIDIA support and renews yearly. The free licence excludes any organization that resells services built on it, and that clause can catch a neocloud. The premium sits in support, and in the Run:ai, UFM, NetQ and autonomous-recovery features that Mission Control adds.

The open path preserves fleet flexibility: heterogeneous vendors, a second-source GPU such as AMD MI355X on ROCm (version 10.1 as of this writing; the qualified tuple is in Chapter 10.4) and bespoke validation gates. It avoids vendor support and platform licensing, at the cost of carrying the integration and reliability engineering yourself, which is real headcount, not a weekend project.

A workable decision rule: a single-vendor, homogeneous, NVIDIA-reference fleet that values speed-to-revenue over flexibility leans buy. A multi-vendor neocloud that competes on bring-up speed and tenant flexibility, or anyone hedging GPU lock-in, leans build. Even then, the open pipeline's scheduler has a GPU vendor behind it: NVIDIA has owned Slurm's developer, SchedMD, since December 2025 (Chapter 10.1). Many large operators do both, buying the integrated stack for the homogeneous reference pods and building the open pipeline for heterogeneous and experimental capacity.

In September and October 2026 a middle path appeared: NVIDIA open-sourced DSX AI Factory components that grew out of its DGX Cloud operations. The Cluster Readiness Engine (open-sourced in September 2026 under Apache-2.0, v0.7.0 announced at OCP in October 2026) runs workload-simulation stress tests before training and reports failing nodes. It never cordons or modifies a node, so quarantine stays with the operator. Separately, the Rack Management Service handles rack-level firmware. A Cluster API provider for NVIDIA's Infrastructure Controller takes bare metal to a cluster, which NVIDIA says takes hours instead of days. Config Manager configures switches but is in Developer Preview and not recommended for production. All of it is open code that targets NVIDIA hardware. Check each piece's maturity and the hardware it supports before you build an acceptance gate on it.

A successful initial deployment tempts teams to file provisioning under done. Two facts say otherwise. Meta’s Llama 3 405B run on up to 16,384 H100s was interrupted about once every three hours (Chapter 10.7), so repair never stops. And a generational hardware refresh arrives on a 12–18 month cadence. Provisioning is therefore a continuous loop, and the same declarative pipeline that did day-0 bring-up runs it. The day-2 cycle is: detect a failing or suspect node → cordon and drain its jobs → wipe and re-image from the current golden image → re-run the acceptance gate → re-admit to the pool (or route to break-fix).

Done well, this loop also catches wear before it reaches a job. Reimaging resets software and configuration drift to the golden baseline; it does not repair hardware. Periodic re-validation, including a pre-allocation burn-in before a node goes to a new tenant, finds hardware that has degraded (rising correctable-ECC rates, weakening optics, throttling) and routes it to break-fix before it strands a synchronous job. Validation-first fleet management has replaced reactive break-fix as standard practice for that reason.

The day-2 loop is also where the density-ramp thread reappears. A refresh is not a like-for-like swap. A hall provisioning GB300 NVL72 today must re-image and re-validate against a new node-software stack, new firmware and new fabric parameters, and again against a heavier power and thermal load as Vera Rubin NVL72 ramps (OEM operating profiles of 188 kW Max Q / 228 kW Max P; NVIDIA's DSX reference design plans 330 kW TDP per cabinet as the facility basis for power and cooling). Rubin Ultra Kyber (~600 kW; NVIDIA targets H2 2027; SemiAnalysis reported 2028 in July 2026) follows. A declarative pipeline absorbs this as a new image version and a new acceptance threshold, while an imperative, pet-based process treats every refresh as a fresh manual project, and the difference compounds across the asset's life. The provisioning architecture you choose at day-0 is, in effect, a bet on how cheaply you can absorb every refresh after it. → refresh execution economics in Chapter 14.9.

The same provisioning failures recur, and each traces back to treating hardware as pets or skipping a gate:

• Bring-up that never proves a compatible node state. In a runbook-driven fleet, nodes provisioned weeks apart differ in driver, NCCL or firmware version with no mixed-tuple qualification. The cost stays invisible until a synchronous job hangs on a collective or silently runs at degraded bandwidth. Debugging is slow because the difference is buried in version skew no one tracked (Chapter 10.4).
• Skipping or shortcutting burn-in to hit a deadline. Nodes that passed a smoke test but never soaked under sustained load get admitted. The marginal ones leak into production and strand synchronous jobs, at a cost far above the GPU-hours the shortcut saved.
• Mutable long-lived nodes that drift. In a config-management fleet, nodes are patched in place for years while incident-time hand-fixes escape the desired-state record. Repair requires reconciling the desired and observed state, reimaging where needed, and retaining hardware and acceptance history; a golden image alone does not fix a physical fault.
• Provisioning treated as one-time. A pipeline built for day-0 and abandoned leaves the day-2 loop (drain, reimage, re-validate, re-admit) manual. The fleet then falls behind whenever repairable failures arrive faster than the recovery pipeline can process them, and goodput bleeds out one un-recycled node at a time.

Make admission the final step of a recorded provisioning operation, and repeat the applicable gates after repair. A fenced node costs capacity you can count. A node admitted on a boot-success flag moves that uncertainty into a customer job, where its failure spends everyone else’s GPUs too.