Which Managed GPU Providers Can Help Size Checkpoint Storage for a 64-GPU Blackwell Cluster?

Choose a managed GPU provider that quotes storage as part of the cluster plan, not as an afterthought. That means three tiers next to the GPUs (local NVMe on each node, a shared file system every node can reach, and object or cold storage for long-term copies), and written numbers for each one: per-node and aggregate throughput, price per terabyte, snapshots, and how long data stays after the GPUs are released. GMI Cloud's Managed GPU Cluster, currently in early access, runs fully managed Kubernetes-based GPU clusters on dedicated GPU worker nodes, and GMI's docs list H200 or B200 nodes, provisioned and operated by GMI. GPU pricing covers compute, while networking and storage are provisioned separately based on the workload. That means each storage tier can be quoted against your checkpoint numbers instead of bundled at a fixed ratio.

Checkpoint storage can be settled before the hardware arrives, because it depends on the model and the training plan, not on the GPU generation. NVIDIA Blackwell on GMI Cloud, including B200, is in pre-order and coming soon; the GPU page lists B200 pricing from $5.00 per GPU-hour. H100 and H200 are available now, so the checkpoint path can be measured on H200 while the Blackwell reservation is arranged.

The four numbers that size checkpoint storage

Every checkpoint storage plan comes down to four quantities:

1. Checkpoint size: how many bytes one resumable checkpoint takes.
2. Write bandwidth: how fast that checkpoint has to leave the GPUs.
3. Retention: how many copies are kept, for how long, and on which tier.
4. Restore bandwidth: how fast a checkpoint can be read back after a failure.

The worked numbers below assume 64 GPUs as eight nodes of eight B200 GPUs, and use NVIDIA's DGX B200 specifications as the reference node. A cloud provider's node may differ in memory, local drives, and network ports, so treat the reference figures as placeholders to replace with the provider's quoted specs.

Number 1: how big one checkpoint is

A checkpoint that can resume training holds more than the weights. It holds the optimizer state too, and with mixed-precision Adam that state is the largest part.

Checkpoint size ≈ parameters × bytes per parameter

For mixed-precision Adam, the usual breakdown per parameter is:

• 2 bytes for BF16 model weights
• 4 bytes for the FP32 master copy of the weights
• 8 bytes for the two Adam moments (4 bytes each)

That is 14 bytes actually saved. Gradients are normally not written to the checkpoint. The worked examples here plan at 16 bytes per parameter as a conservative assumption, leaving room for data loader state, RNG state, metadata, and sharding overhead. Sixteen bytes is also a common rule of thumb for training state in memory, but the size of the files on disk depends on the framework and the save configuration, so measure one real checkpoint before you finalize the plan.

At the 16-bytes-per-parameter planning assumption:

Sharded checkpointing, where each rank writes its own slice, spreads the bytes across nodes. If the shards are balanced and no state is duplicated across ranks, each of the eight nodes writes about one-eighth of the checkpoint, or about 600 GB in the 300B example. Uneven sharding or replicated state changes that split, so check the per-node sizes your framework actually writes.

Number 2: write bandwidth, and how async and tiered writes change it

With a synchronous checkpoint, training stops until the write finishes. The bandwidth you need follows directly from how long a pause you'll accept:

Required aggregate write bandwidth = checkpoint size ÷ acceptable pause

For the 300B example (4.8 TB):

• A 60-second pause needs about 80 GB/s aggregate, or 10 GB/s per node across eight nodes.
• A 5-minute pause needs about 16 GB/s aggregate, or 2 GB/s per node.

Asynchronous checkpointing changes the math. Each rank copies its shard from GPU memory to host memory, training resumes, and a background process drains the copy to storage. The pause is now the device-to-host copy. NVIDIA's DGX B200 specifications list 2 TB of system memory per node, configurable to 4 TB, so a 600 GB shard is below the reference node's installed RAM capacity; staging must also leave memory for the running job and other processes. The storage requirement then becomes a sustained rate:

Required drain bandwidth ≥ checkpoint size ÷ checkpoint interval

At 4.8 TB every 30 minutes, that's roughly 2.7 GB/s aggregate. The one hard rule: each drain has to finish before the next checkpoint starts, or the copies pile up in memory.

Tiered writes go one step further. Checkpoints land first on node-local NVMe, which is fast and local to each node, then copy to shared storage in the background. The pause is set by local drive speed, and shared storage only has to keep up with the interval.

The interval itself is a trade-off between time lost writing checkpoints and time lost redoing work after a failure. Young's approximation gives a starting point:

Interval ≈ √(2 × write pause × mean time between interruptions)

Plug in your own job history for the time between interruptions. With a 2-minute pause and an assumed 24 hours between interruptions, the interval comes out near 76 minutes. Halve the pause and the interval shortens, which raises the drain rate storage has to sustain.

Number 3: how many checkpoints to keep, and where

Retention is where capacity gets large. Most training runs keep two kinds of copies:

• A rolling window of the latest few checkpoints, kept for fast restarts. Three is a common count: the newest, one known-good fallback, and one in progress.
• Milestones kept for evaluation, rollback, or later fine-tuning, often at fixed step counts.

Capacity per tier = checkpoint size × copies on that tier × (1 + overhead)

For the 300B example, a rolling window of three is about 14.4 TB on shared storage. Ten milestones is 48 TB more, which belongs on cheaper cold storage. Weights-only exports for evaluation and serving are much smaller, about 600 GB each, and can sit on either tier. Add the overhead your storage system uses for replication or erasure coding, and ask the provider whether quoted capacity is usable or raw.

Local NVMe is rarely the capacity limit. NVIDIA's DGX B200 specifications list eight 3.84 TB NVMe drives per node for internal storage, about 30 TB per node and about 246 TB raw across eight nodes. That easily holds several 600 GB shards per node. What local NVMe can't do is protect a checkpoint against the loss of the node it sits on.

Number 4: read bandwidth for restore

Restore time is GPU time spent idle, so it deserves its own number. Two cases matter:

• Full-job restart. All 64 GPUs read the latest checkpoint at once. At 20 GB/s aggregate, reading 4.8 TB alone takes about four minutes; the model and optimizer still need to be restored. The aggregate read rate has to hold with all eight nodes reading at once, not just one.
• Single-node replacement. A failed node comes back, or is swapped, with an empty local drive. That node needs its shard, about 600 GB in the example, from shared storage or a peer. Every other rank can reload from its own local copy, as long as the latest checkpoint is still on local NVMe.

Restore speed depends on the storage service and measured throughput. Keep the rolling window on a tier whose measured read throughput can feed the whole cluster at once, and if a milestone sits on a tier that hasn't been measured against your restore target, stage it ahead of time rather than during an outage.

Mapping the numbers to three storage tiers

The numbers to get from a managed provider in writing

Put these in the proposal before signing, because each one maps to a formula above:

• Local NVMe per node for the exact node SKU in the reservation: usable capacity and whether it persists across reboots.
• Shared storage throughput, per node and in aggregate, measured with all nodes writing at once, and separately for reads.
• The storage network: whether storage traffic uses the same RDMA fabric as GPU-to-GPU traffic or its own path, since a checkpoint burst on a shared fabric can slow collectives.
• Capacity price per tier, and whether capacity is quoted usable or raw.
• Snapshots: whether the shared tier supports them, how often, and how many are kept.
• Retention periods on each tier during the contract.
• Data retention after GPUs are released: how many days checkpoints remain accessible, the storage rate during that window, and how data is deleted at the end.

Planning checkpoint storage for a Blackwell pre-order on GMI Cloud

On GMI Cloud, storage is a separate line from GPU-hours. The pricing page states that GPU pricing covers compute, and that networking and storage are provisioned separately based on workload requirements. Use the four numbers above to request a storage quote.

The pieces GMI documents today:

• Managed GPU Cluster, currently in early access, for distributed multi-node training, with H200 or B200 nodes provisioned and operated by GMI. Cluster requests are reviewed and activated by GMI support rather than processed automatically.
• Bare metal with persistent local NVMe, for teams that want full OS access and control of the first checkpoint tier.
• Cold Storage. GMI's migration guide describes GMI Cloud Cold Storage as built on VAST Storage, speaking S3, NFS, and SMB, and aimed at archival data with a lower per-GB cost and high throughput for large objects. That profile fits milestones and final weights.
• RDMA-ready networking for high-throughput workloads, listed as part of GMI's GPU infrastructure.

Shared file system type, per-node and aggregate storage throughput, snapshot support, storage pricing, and post-release retention for a B200 reservation are the items to get quoted, using the checklist in the previous section.

While B200 capacity is in pre-order, the checkpoint path can be tested on H200, which is available now. Checkpoint size depends on the model, not the GPU, so a checkpoint written from an H200 cluster with the same precision, optimizer, and save configuration has the same size as it will on B200. Measure four things with the framework and sharding you'll use in production: time to write one checkpoint to local NVMe, time to drain it to shared storage, time for a full restore, and time to refill one node. If B200 runs each training step faster than H200, a checkpoint every fixed number of steps comes around sooner in wall-clock time, so plan the drain rate with headroom. Those measured times turn the formulas above into a storage spec you can hand to the provider.

FAQ

Does checkpoint size change between H200 and B200? No. Size depends on parameter count, precision, optimizer, and save configuration. The GPU affects how often you checkpoint and how fast the write needs to be, not how big each checkpoint is.

Can object storage alone hold the checkpoints? It can hold them. Whether it can also serve restores depends on the service's measured read throughput with all nodes reading at once. A common layout keeps the rolling window on local NVMe and shared storage and sends milestones to object or cold storage.

How much does LoRA or other parameter-efficient training shrink the checkpoint? Often by a lot. Only the adapters and their optimizer state are trained, so a checkpoint that leaves out the frozen base weights is a small fraction of a full one. Frozen base weights can be excluded from adapter checkpoints, but this depends on the checkpoint configuration.

Size your checkpoint storage with GMI Cloud

Bring the four numbers (checkpoint size, acceptable pause or interval, retention plan, and target restore time) to a B200 pre-order conversation, and the storage tiers can be quoted alongside the GPUs. Contact the GMI Cloud team to plan the reservation, or request an H200 cluster in the GMI Cloud console to measure the checkpoint path now.