H100 vs H200 vs Blackwell (GB200): Choosing GPUs for Training and Inference

Choosing between the H100, H200, and Blackwell GB200 starts with your model’s memory footprint and ends with cost per token or time to train, not with peak FLOPS on a spec sheet. Each generation solves a different bottleneck, so the right GPU depends on what limits your workload today.

• H100 vs H200 is a memory decision: Both use the same Hopper compute, but the H200 adds 76% more memory (141 GB vs 80 GB) and 43% more bandwidth (4.8 TB/s vs 3.35 TB/s) within the same power profile.

• Blackwell changes the architecture: B200 and GB200 GPUs bring up to 8 TB/s of bandwidth per GPU, native FP4 support, and fifth-generation NVLink at 1.8 TB/s, twice Hopper’s 900 GB/s.

• GB200 NVL72 is a rack, not a card: It links 72 Blackwell GPUs and 36 Grace CPUs into one NVLink domain with 13.4 TB of HBM3e, which NVIDIA rates at 30× faster real-time trillion-parameter inference and 4× faster large-model training than H100.

• Cost per token beats hourly price: On GMI Cloud, an H200 costs about 23% more per hour than an H100 (on-demand rates as of October 2026), so it lowers cost per token whenever it delivers at least 1.23× the throughput on your workload.

• Configuration matters as much as silicon: NVIDIA reports that clusters built from identical hardware routinely land 8 to 12% below reference throughput because of stack settings, so the provider’s tuning shapes the result you actually get. As open models push past a trillion parameters, matching the GPU generation to the workload has become one of the biggest levers on AI infrastructure cost.

Model sizes have outgrown the hardware many teams standardized on. In almost every month of 2026, the largest open-weight model came from a Chinese lab, with monthly ceilings between 754 billion and 2.78 trillion parameters, according to the data GMI Cloud analyzed in its look at where the value goes in open-weight inference. A model of that size does not fit on a single 80 GB GPU, or even on a single eight-GPU Hopper server.

At the same time, the H100 remains one of the most widely deployed and cost-effective AI accelerators, the H200 has become a standard choice for memory-hungry inference, and Blackwell systems such as the GB200 NVL72 now lead the industry benchmarks. Teams no longer pick the newest GPU by default. They pick the one that removes their specific bottleneck at the lowest cost per unit of work.

This GPU comparison for AI explains what actually differs between H100 vs H200 and the Blackwell generation, how memory, bandwidth, and interconnect shape training and inference performance, and how to choose the right hardware for your workload on GMI Cloud.

The Three Generations at a Glance

H100: The Established Workhorse

The NVIDIA H100 is built on the Hopper architecture and introduced the FP8 Transformer Engine that most modern LLM training and serving stacks now rely on. The SXM version carries 80 GB of HBM3 at 3.35 TB/s, delivers 3,958 TFLOPS of FP8 Tensor Core performance with sparsity, and connects to other GPUs at 900 GB/s over fourth-generation NVLink. Its software ecosystem is the most mature of the three, and its supply is the most abundant, which keeps hourly prices low.

H200: Same Compute, Much More Memory

The H200 uses the same Hopper GPU and the same Tensor Cores as the H100, so raw compute is identical. What changes is memory. The H200 carries 141 GB of HBM3e at 4.8 TB/s, nearly double the capacity and 1.4× the bandwidth of the H100, within the same power profile. NVIDIA reports up to 2× faster inference than the H100 on LLMs such as Llama 2, because larger and faster memory lets the GPU run bigger batches and keep more of the model and its KV cache on a single device.

Blackwell B200 and GB200 NVL72: A New Architecture and a Rack-Scale System

Blackwell is a new architecture rather than a memory refresh. A B200 GPU in an HGX B200 server carries 180 GB of HBM3e at 8 TB/s, adds a second-generation Transformer Engine with native FP4 support, and more than doubles Hopper’s FP8 throughput. Fifth-generation NVLink raises GPU-to-GPU bandwidth to 1.8 TB/s.

The GB200 is the Grace Blackwell Superchip, which pairs two Blackwell GPUs with one Arm-based Grace CPU. The GB200 NVL72 puts 36 of these superchips into a single liquid-cooled rack: 72 Blackwell GPUs and 36 Grace CPUs joined in one NVLink domain with 130 TB/s of aggregate GPU bandwidth, 13.4 TB of HBM3e, and up to 1,440 PFLOPS of FP4 compute with sparsity. To the software, the rack behaves much like one very large GPU.

Figures come from NVIDIA specifications. Sparse Tensor Core figures are roughly double the dense figures, and per-GPU values for the GB200 NVL72 are the rack totals divided across 72 GPUs.

Memory Is the First Filter

How Much Memory Your Model Needs

Before comparing speed, check whether the model fits. Model weights need about 2 bytes per parameter at FP16 or BF16, 1 byte at FP8, and half a byte at FP4. A 70-billion-parameter model therefore needs about 140 GB at BF16 and about 70 GB at FP8, before any room for the KV cache, activations, or batching. The table below shows the smallest practical configurations for common model sizes at FP8.

A model that fits inside one server, or one NVLink domain, avoids the slower network hop between nodes. That is the single biggest reason the H200 often beats the H100 on large-model inference, and why rack-scale systems matter for trillion-parameter models.

KV Cache and Long Context

Weights are only part of the memory bill. Every active request also keeps a KV cache that grows with context length and batch size. Long-context chat, retrieval-augmented generation, and agent workflows that carry long histories can need as much memory for KV cache as for the weights themselves. On an H100, a 70B model at FP8 leaves only about 10 GB for everything else, which limits batch size and concurrency. The H200’s extra 61 GB, and Blackwell’s 180 GB or more per GPU, turn directly into more concurrent users per GPU.

Inference: Throughput, Latency, and Cost per Token

Why Decoding Is Bandwidth-Bound

LLM inference has two phases. Prefill processes the prompt in parallel and is mostly compute-bound. Decode generates one token at a time and has to read the model weights from memory for each step, which makes it mostly bandwidth-bound. Because most serving time is spent in decode, memory bandwidth is often a better predictor of inference throughput than peak FLOPS. This is why the H200’s 43% bandwidth gain matters more for serving than for training, and why Blackwell’s 8 TB/s per GPU, together with FP4, raises inference throughput so sharply.

The Cost per Token Math

Hourly price only becomes meaningful once you divide it by throughput. GMI Cloud’s price-to-performance formula converts any GPU rate into a cost per million tokens.

cost per 1M tokens = (GPU price per hour ÷ tokens per second) × (1,000,000 ÷ 3,600)

On GMI Cloud, the published on-demand rates start at $2.60 per GPU-hour for the H100 and $3.20 for the H200 (as of October 2026), a 23% premium. That means the H200 becomes cheaper per token as soon as it serves at least 1.23× the tokens per second of an H100 on the same model. If it reaches 1.5×, which is realistic for memory-bound workloads that benefit from larger batches, its cost per token is about 18% lower. The same test applies to Blackwell: Divide its hourly price by the H100 price, and that ratio is the throughput gain it needs to win on cost. On the same date, the B200 is listed from $5.00 per GPU-hour and the GB200 from $8.00, so they need roughly 1.9× and 3.1× an H100’s throughput on your workload to lower cost per token. Check the GMI Cloud pricing page for current rates before running the numbers. Prices can change with promotions and commitment terms, so confirm the rate that applies to you in the GMI Cloud console.

Where GB200 NVL72 Pulls Ahead

The GB200 NVL72 earns its place on the largest and most interactive workloads. NVIDIA rates it at 30× faster real-time inference than H100 for trillion-parameter LLMs, with 10× greater performance for mixture-of-experts architectures. The gain comes from the combination of FP4 compute, 8 TB/s of bandwidth per GPU, and a 72-GPU NVLink domain that lets a huge MoE model spread its experts across GPUs without crossing a slower network.

Independent benchmarks point the same way. In MLPerf Inference v6.0, published in April 2026, NVIDIA’s Blackwell Ultra GB300 NVL72 delivered the highest throughput across all new workloads, and a configuration of four GB300 NVL72 systems reached roughly 2.49 million tokens per second on DeepSeek-R1 in the offline scenario. Coverage of the round also highlighted that software optimizations alone doubled NVIDIA’s throughput on the same hardware, a reminder that the serving stack matters as much as the chip. GMI Cloud’s guide to the GB300 NVL72 for rack-scale inference covers how the next Blackwell step extends memory to 288 GB per GPU.

For smaller models, the case is weaker. A 7B or 13B model that fits comfortably on one H100 rarely needs a rack-scale system, and the added cost per hour can outweigh the throughput gain.

Training: Time to Train and Scale

Fine-Tuning and Mid-Size Training

For fine-tuning, LoRA adaptation, and training models up to the tens of billions of parameters, the H100 remains a strong and cost-effective choice. Its FP8 Transformer Engine, mature support in frameworks such as PyTorch, NeMo, and DeepSpeed, and wide availability make it easy to scale from one node to many. The H200 helps when larger batch sizes or longer sequences would otherwise force you onto more GPUs, since the same compute with more memory can reduce the degree of model parallelism you need.

Large-Scale Pretraining

For pretraining large models, Blackwell changes the economics. NVIDIA rates the GB200 NVL72 at 4× faster training than H100 for large language models at scale, measured on a 1.8-trillion-parameter MoE model, thanks to its second-generation Transformer Engine, higher FP8 throughput, and 1.8 TB/s NVLink between all 72 GPUs in the rack. Faster training also means fewer GPU-hours per run, which can offset a higher hourly rate.

Real deployments show how much infrastructure choices matter for training budgets. Mirelo AI, which trains foundational models on GMI Cloud’s dedicated GPU infrastructure, reports 40% lower training costs and 20% faster training time.

Same GPU, Different Results

Two clusters built from the same GPUs can perform very differently. In its lessons from the Exemplar Cloud program, NVIDIA notes that validation requires reaching 95% of reference architecture throughput, yet partner clusters on identical H100, GB200 NVL72, or GB300 NVL72 hardware routinely fall 8 to 12% short because of compounding kernel, hypervisor, BIOS, and NCCL settings. In one case, GB200 NVL72 FP8 pretraining ran 12% slower in a virtual machine than on bare metal.

That gap is why provider validation matters. GMI Cloud has been recognized as an NVIDIA Exemplar Cloud on GB300 NVL72 systems for training, meeting NVIDIA’s performance thresholds against its reference architecture. When you compare GPU offers, ask for validated throughput on a workload like yours, not just the GPU model and hourly rate.

A Decision Framework for Choosing Your GPU

Choose the H100 When

The H100 is the right default for models that fit comfortably in 80 GB, for fine-tuning and mid-size training, and for teams that want the most mature software stack at the lowest hourly price. It also suits development, evaluation, and production serving of models up to roughly 34B parameters at FP8, where memory is not the bottleneck.

Choose the H200 When

Choose the H200 when memory is what limits you. It fits 70B-class models on a single GPU with room for KV cache, holds 671B-class MoE models within one eight-GPU server, and supports longer contexts and higher concurrency without code changes, since it runs the same CUDA stack as the H100. For memory-bound inference, it often delivers a lower cost per token than the H100 despite the higher hourly rate.

Choose Blackwell When

Choose a B200 server when you want Blackwell’s bandwidth and FP4 support for large models that fit within eight GPUs. Choose the GB200 NVL72 for frontier-scale pretraining, real-time inference on trillion-parameter or very large MoE models, and workloads where the 72-GPU NVLink domain removes network bottlenecks. Plan for FP4 to unlock the largest gains, and validate output quality at lower precision before moving production traffic.

Running the Choice on GMI Cloud

GMI Cloud offers the full range on dedicated NVIDIA GPU infrastructure, from H100 and H200 to B200, GB200 NVL72, and GB300 NVL72, inside GMI-operated data centers built on NVIDIA reference architectures. H100 and H200 capacity is available on demand, while the GB200 is available now from $8.00 per GPU-hour, the B200 is listed from $5.00 per GPU-hour with limited availability, and the GB300 is open for pre-order (GMI Cloud pricing page, as of October 2026). The Cluster Engine handles multi-node orchestration at the infrastructure layer, with bare metal and root access when you need custom stacks.

If you would rather not choose hardware at all, the same platform serves models through serverless APIs in the model catalog, so you can benchmark a model there first and move to dedicated GPUs once your traffic and throughput numbers justify it.

Conclusion

The H100 vs H200 decision is about memory: Same compute, but the H200’s 141 GB and 4.8 TB/s let larger models and longer contexts run on fewer GPUs, often at a lower cost per token. The Blackwell generation is a bigger step, with FP4, 8 TB/s per GPU, and in the GB200 NVL72, a 72-GPU NVLink domain built for trillion-parameter training and real-time inference. Start from your model’s memory footprint, measure throughput on your own workload, divide by the hourly rate, and choose the GPU that delivers the lowest cost per token or per training run. GMI Cloud gives teams access to every step of that path, from on-demand H100 and H200 GPUs to validated Blackwell rack-scale systems.