Object Storage, Data Lakes & the Capacity Tier

Every tier above this one (node-local NVMe scratch, the parallel file system, the checkpoint store) is sized to a working set. Object storage is sized to everything. It is the only layer in the building that holds the full training corpus, the complete checkpoint lineage across every run, the model registry, the eval artifacts, the inference logs, and the cold copies of weights waiting to be paged onto a GPU. That makes it the largest tier by capacity and usually the cheapest per terabyte, which breeds a dangerous instinct: treating it as a passive archive behind the "real" storage that never touches the GPUs. In the 2026 AI data center that instinct is wrong, and acting on it is expensive.

Object storage has stopped being the cold tier and become a serving tier. Modern data loaders stream Parquet and WebDataset shards directly from S3-compatible buckets into the training loop. Model servers page multi-hundred-gigabyte weight files out of object storage on cold start. Lakehouse engines query Iceberg tables in place, without a copy. Once a GPU waits on an object GET, the bucket's read latency and aggregate throughput limit goodput, the share of wall-clock time that advances the job, and an under-provisioned capacity tier idles the most expensive hardware in the building. The forks ahead are on-prem vs cloud, HDD vs flash, replication vs erasure coding, and archive vs serving, each with a downstream cost. The data-loader path that consumes this tier is Chapter 9.5, the file-system tier above it is Chapter 9.2, and the sizing math that ties them together is Chapter 9.8.

Three properties make object storage attractive as a capacity tier. The first is a flat key-value namespace: no directory tree, no POSIX semantics, no in-place updates. You replace an object whole, by key, though a reader can GET any byte range within it. The second is massive horizontal scale: object count and namespace throughput grow within the chosen service's documented limits. The third is an HTTP/REST access path, and the S3 API is the de-facto wire protocol the whole ecosystem speaks. Whether a given implementation can also serve the performance tier depends on its delivered reads and request tails.

The cost of these properties is latency and small-object overhead. A GET carries request setup, so a workload of millions of tiny objects pays a per-request cost that crushes throughput. Range GETs let a reader fetch only the bytes it needs (a Parquet reader pulls the footer, then just the column chunks in play), but each range is still a priced, latency-bearing request. That is why the data-loader layer packs many small samples into large sequential shards (WebDataset tar, Parquet, MDS) before they reach the bucket. The format choice in Chapter 9.5 exists largely to make object storage fast.

In the AI lifecycle, object storage carries five payloads that share a bucket but have different access patterns. The training corpus is read-heavy, large-sequential, and fans out across thousands of GPUs. Checkpoints are write bursts (incast) and then mostly cold, until a restart makes one of them suddenly latency-critical (Chapter 9.4). The model registry is small in object count, but each object is huge and is read on cold start. Eval and inference logs are append-heavy and queried analytically. The data lake is the governed, queryable view over the raw corpus. Sizing the bucket to the average of these patterns guarantees you under-serve the peak of each one.

The first and least reversible decision is where the petabytes physically live. Data has gravity: once a multi-petabyte corpus lands somewhere, moving it costs money, time and egress fees, and every workload that touches it is pulled toward it. The real choice is not "AWS vs MinIO". It is who owns the durability design and the separate legal availability commitment, who pays for reads, and how locked in the data gravity leaves you over the asset's life. Chapter 9.8 develops the data-gravity economics quantitatively.

Cloud object (S3, GCS, Azure Blob, and the regional sovereign clouds) gives you a published durability design target, distinct from the service-availability SLA, and both are specific to the storage class you choose. It also gives you elastic capacity within service limits and no storage hardware to buy. The price is a per-GB-month storage rate, per-request charges and, the line that dominates AI economics, egress fees when bytes leave the cloud on a billed route, plus the storage class's request and retrieval charges. If your GPUs are in the same cloud, a same-region service path can avoid transfer charges, but cross-region and other billed paths still incur transfer or retrieval costs, so price the exact service boundary. If your GPUs are in a colo or self-build and your data is in the cloud, you pay to feed your own training loop, and data gravity makes it progressively harder to leave.

On-prem object inverts the trade. Ceph RADOS Gateway is the established open-source distributed option. MinIO's open-source repository was archived as unmaintained in April 2026, so MinIO now means the commercial AIStor product. VAST, Everpure (formerly Pure Storage) FlashBlade, Cloudian, Dell ObjectScale (formerly ECS) and the parallel-FS vendors' native S3 endpoints complete the list. You own the capex and the operational burden, you set your own durability scheme, and there is no egress fee to feed a co-located GPU fleet. For a durable, large, self-hosted training operation the on-prem bucket is usually the lower-TCO floor. For spiky, multi-region or cloud-native inference, the managed bucket wins on elasticity and reach.

The second fork is the one that has changed most since 2024: what media sits behind the bucket. Classic object storage is HDD-backed. HDDs are cheap per terabyte and dense, but slow. A nearline drive waits about 4 ms on average for the platter to come around, and the seek adds more. Together that makes a random read cost roughly 11–12 ms. Sequential throughput is a few hundred MB/s per spindle. That is fine for a true cold archive and fatal for a serving tier whose loader deadline it misses, so test aggregate sequential demand and request tails before choosing HDD, flash or a cache. The new pattern is S3-over-flash: an S3-compatible namespace whose hot data lives on NVMe SSD and delivers single-digit-millisecond GETs and an order of magnitude more throughput. The data loader can then stream training shards directly from object storage at line rate instead of staging them through a separate file system first.

The cloud made this concrete with S3 Express One Zone: up to 10x faster data access than S3 Standard at single-digit-ms latency, with request prices about 92% (GET) and 77% (PUT) lower, but with per-GB upload and retrieval charges on every byte. The speed comes from collapsing to a single availability zone, which gives up the multi-AZ durability that defines S3 Standard. That is the flash fork in miniature: you pay for speed in dollars (storage at roughly $0.11/GB-month, several times the Standard rate) and in durability blast radius (one AZ, not three).

On-prem, the same shift is driven by high-capacity QLC SSDs. The 245.76 TB class is shipping as of mid-2026: Micron's 6600 ION since May, and Kioxia's LC9 qualified in Dell ObjectScale at 9.83 PB raw per 2U. Solidigm's 245 TB class is a vendor roadmap target for end-2026, and a 512 TB DapuStor prototype exists but is not a product. Samsung introduced the BM1773 in September 2026 and calls it the first 245.76 TB drive in the smaller E3.S form factor; it has given no shipping date. Size rebuild windows on a drive you can buy. Micron's shipping 6600 ION writes at about 3.0 GB/s, so refilling one 245.76 TB drive after a failure takes about 23 hours at full write speed. Samsung rates the BM1773 at 4.5 GB/s sequential write and 0.6 DWPD, which would cut that to about 15 hours. Plan erasure-coding repair traffic around the write speed of whichever drive you deploy. The newest shipping Gen5 parts read at about 13.7 GB/s sequential (the fleet-typical Gen4 figure in this chapter's table is about 7 GB/s), against about 300 MB/s for an HDD. For a Gen4 QLC drive such as Solidigm's D5-P5336, random-read latency is about 110 µs versus about 11–12 ms for an HDD. That density and those watts per terabyte let an all-flash object tier out-pack an HDD rack several-fold.

The economics reversed during the NAND shortage. In VDURA's Q3 2026 index (August 2026), a 30 TB QLC SSD cost about 15x a 30 TB nearline HDD ($18,080 vs $1,216), even though HDD prices had more than doubled in a year. VDURA sells hybrid flash and disk systems, so it has an interest in the comparison. That puts the 2025 QLC-vs-HDD crossover story on hold. The default is cold data on HDD again, unless the latency or density case pays for the flash premium.

A second on-prem change is S3 over RDMA. The request stays an S3 request, but the GET or PUT payload moves directly between the storage server and GPU or host memory. NVIDIA's cuObject libraries provide the plumbing: v1.0 shipped in January 2026, and by September 2026 the client was at v1.3 and the server at v2.0. Dell ObjectScale 4.3, IBM Storage Scale 6.0.1, the open-source Versity Gateway and MinIO AIStor (stable channel since August 2026) all support it. Dell's own tests claim up to 230% more throughput and up to 98% less client CPU than S3 over TCP, and those are vendor figures, not independent ones. This is an extension, not the AWS S3 API. Within MinIO, RDMA covers just three operations: GetObject, PutObject and UploadPart. Each get or put call to the cuObject server is capped at 1 GiB. Clients need the vendor's SDK, and the network has to be lossless RDMA: RoCE v2 with PFC, or InfiniBand. Test it when client CPU, rather than the bucket, limits your direct reads. Public cloud buckets do not offer it. The GPUDirect path it rides on is in Chapter 9.3.

Not every object earns flash. The corpus shard read every epoch does; the checkpoint from a run that finished four months ago does not; the inference log from last quarter belongs in deep archive. Lifecycle and tiering policies keep the average cost low while the hot path stays fast. They automatically demote objects down a temperature ladder (hot flash → warm standard → cool infrequent-access → cold/archive) as access frequency drops, and the cloud's intelligent-tiering variants do this without an explicit policy by watching access patterns.

The subtlety is in retrieval. Deep-archive classes are cheap to store but slow, and sometimes expensive, to read back. That is fine for a compliance copy and disastrous for a checkpoint you might restart from. The anti-pattern is tiering by age alone: a six-month-old checkpoint looks archival until a model regression makes it the one you must restore now, and a multi-hour glacier retrieval becomes the long pole in your recovery. Tier by access likelihood and recovery-time requirement, not by calendar age, and keep anything on a recovery path in an instant-retrieval class even if it is cold.

A bucket full of Parquet files is a data lake. Add a transactional metadata layer that gives you schema evolution, time travel, ACID commits and partition evolution, and it becomes a lakehouse. That metadata layer is the open table format. For AI fleets it matters because the same governed corpus must be queryable by the data-prep pipeline, the training loader, the eval harness and the analytics engine, without copying it four times. The format is what lets many engines read one copy of the data in place.

This guide chooses Apache Iceberg as the interoperability standard for a new AI fleet. AWS gives Iceberg first-class support, and S3 Tables use Iceberg, although Athena's general CTAS default remains Hive. Snowflake added native Iceberg tables. Databricks acquired Tabular (Iceberg's creators) and uses Delta Lake UniForm to expose Iceberg-compatible metadata, so a single Delta table reads as either format. Iceberg v3 adopts deletion vectors compatible with Delta's and adds row lineage, narrowing the gap so one copy of data can serve both.

So pick Iceberg unless you are all-in on the Databricks/Delta stack, where Delta with UniForm gives you interop anyway. Hudi earns the pick where streaming upserts and CDC merges dominate the write path and you are prepared to own its compaction and table-services operations. One condition applies to every choice: name the writer, reader and catalog releases before committing the corpus. UniForm exposes read metadata without authorizing arbitrary Iceberg writers to mutate a Delta table, and schema evolution, deletes and partition changes must work through every required engine, or the shared copy becomes another conversion pipeline. Betting on a proprietary or orphaned format is the expensive mistake. It strands your catalog and forecloses multi-engine access, which is the reason to run a lakehouse at all. The governance and lineage regime above the lake is Chapter 10.10.

Before enabling cleanup, restore a catalog snapshot, together with the data and delete files it references, into an isolated namespace. Verify that every authorized reader returns the same record IDs. Compaction needs space for the old and replacement files to coexist, and snapshot retention can keep both live. Expiring a snapshot and removing orphan files are different operations, and an age threshold shorter than an active writer or reader can destroy required data. For a required deletion, trace the record through table snapshots, source objects, training shards, retrieval indexes, replicas and caches, and record the completion watermark the data owner requires. Hold the release while any required consumer still returns the record. Sources: Iceberg maintenance, Delta UniForm and Hudi table types.

"Eleven nines" is a marketing number until you ask how it is achieved, because the same durability target costs very differently under different protection schemes. The two primitives are replication (store N full copies; simple and fast to repair, but N-x the raw capacity) and erasure coding (split each object into k data plus m parity shards spread across failure domains, so it survives m losses at a fraction of replication's overhead). At large corpus scale, erasure coding can save substantial media: triple replication of an illustrative 10 PB corpus needs 30 PB before reserve, and 20 PB of that buys protection rather than new logical capacity. Under the (8,3) code from the worked example below, the same 10 PB occupies about 14 PB, again before reserve. The price is CPU on every write and every reconstruction read, plus repair traffic that touches many nodes, so test foreground reads with a failure injected before you bank the saving.

The failure domain the shards span is the real durability lever: count the shards lost to each drive, node, rack or zone event. Reconstruction needs at least k surviving shards of the object, so merely spanning zones does not establish zone survival. S3 Standard's eleven nines come from multi-AZ erasure coding plus continuous background integrity scanning (checksums, auditors, automated re-replication). S3 Express One Zone deliberately gives up the multi-AZ span for latency, so its durability blast radius is a single AZ.

Durability marketing hides two failure modes: silent data corruption and operator error. End-to-end checksums and background scrubbing catch bit rot, so verify your object store does both, because a corrupted training shard poisons a run silently. Eleven nines of durability is also no protection against an rm, a buggy lifecycle rule or ransomware. That is what object lock / immutability (WORM) and versioning are for. For checkpoints and the model registry, an immutability window plus versioning is the difference between a recoverable incident and a destroyed lineage. The SDC-detection discipline connects to fleet health in Chapter 10.6.

The last role completes the case for treating object storage as a serving tier rather than an archive: object storage is where models live between requests and how they reach every GPU in the fleet. A model server on cold start pages a multi-hundred-gigabyte weight file out of the bucket onto the GPU. An autoscaler spinning up a new replica reads the same weights. A fleet-wide model rollout fans the same objects out to thousands of nodes at once. In each case the bucket's read latency becomes cold-start latency, and its aggregate fan-out throughput caps how fast you can scale a model up to meet a traffic spike. Both are user-facing inference SLO concerns, not archive concerns.

So the inference-era capacity tier is increasingly flash-fronted, and the new inference memory hierarchy (Chapter 9.7) puts an object/Ethernet-flash layer explicitly at its base. The KV-cache and weight tiers above it page to and from object storage, and a slow bucket throttles cold-start and cache rehydration. If object storage is your model-distribution backbone, size it for the fan-out burst of a fleet-wide rollout, with thousands of simultaneous reads of the same large objects, not for steady-state archive throughput. Under-provision it and both your cold-start time-to-first-token and your ability to absorb a traffic surge degrade. The model-serving and cold-start mechanics live in Chapter 9.7. The read amplification of a synchronized fan-out is a fabric co-design problem shared with Chapter 8.5.

Choose the bucket, media and table format that meet both the active read path and the recovery requirements of the data you retain. Keep data on HDD while it still reaches the loader on time. Buy flash when the premium earns its keep: on measured request tails, on a fleet-wide model rollout that reads the same objects from thousands of nodes at once, or on a site short of floor space and power. Bank an erasure code's capacity saving over replication only once the layout has passed the lost-zone replay above with repair running. A shared Iceberg, Delta or Hudi table removes duplicate copies only if all of your engines read the same commits and deletions.