How does GB200 NVL72 perform on large-model inference compared to a cluster of H100s, and what throughput gain should I expect?

Teams comparing GB200 NVL72 to a cluster of H100s often reduce it to "newer, faster chips" and expect a flat speedup. That framing misses where the real gain comes from, because a NVL72 rack and a networked H100 cluster differ most in how their GPUs talk to each other, not just in per-chip speed. GB200 NVL72 outperforms an H100 cluster on large-model inference mainly because its 72 GPUs share a single high-bandwidth NVLink domain, which removes the cross-node network bottleneck that limits multi-node H100 clusters, so the throughput gain grows with model size and interconnect demand. This guide explains the architectural difference, why the gain is workload-dependent, and how to size the expected uplift for your own model.

The gain is interconnect, not just faster chips

An H100 cluster serves a large model by splitting it across multiple servers connected over a network. Every time the model's layers need to exchange activations across those nodes, they cross a slower inter-node link, and that communication overhead caps throughput on models too big for one server.

GB200 NVL72 changes the topology, not just the silicon. A GB200 NVL72 rack connects all 72 GPUs in one NVLink domain at bandwidth far higher than the inter-node networking in a typical H100 cluster, so models that must be sharded across many GPUs avoid the network bottleneck that throttles a multi-node H100 deployment. For a model that fits comfortably on a single H100 server, this advantage barely shows up. For a model large enough to span many GPUs, the in-rack interconnect is often the difference between good and poor scaling, which is why the expected gain is not a fixed multiple but a function of how much your model relies on GPU-to-GPU communication.

Why the throughput gain is workload-dependent

Two teams can compare the same rack against the same cluster and see very different uplift. Three factors decide it:

• Model size relative to a single node: A model that fits on one H100 server sees little NVL72 benefit. A model that must shard across nodes sees the largest gain, because that is exactly where the cluster's network becomes the bottleneck.
• Parallelism strategy: Tensor and expert parallelism generate heavy cross-GPU traffic. The more your serving relies on them, the more the NVLink domain helps.
• MoE and long context: Mixture-of-experts routing and long sequences increase interconnect and memory-bandwidth demand, both of which favor the single-domain rack.

The honest expectation is a shape, not a headline number. Vendor throughput multipliers for rack-scale Blackwell are stated for specific model and configuration setups, so treat any single "X times faster" figure as directional until you measure it on your model. The gain rises with how communication-bound your workload is, and falls toward parity when the model fits a single node.

What to expect by workload type

Use the frame below to set a realistic expectation before you benchmark. The uplift column is directional, not a quoted number.

Workload profile	Where the bottleneck sits	Expected GB200 NVL72 uplift vs H100 cluster
Model fits one H100 server	Per-chip compute	Modest; gain is mostly generational
Model shards across few nodes	Some inter-node traffic	Meaningful; NVLink cuts communication cost
Model shards across many nodes	Network is the bottleneck	Largest; single NVLink domain removes the cap
MoE or long-context at scale	Interconnect and bandwidth	Largest; rack topology is the main lever

The pattern is consistent: the more your large-model inference is limited by GPU-to-GPU communication, the more a GB200 NVL72 rack outperforms a networked H100 cluster. When it is limited by per-chip compute on a model that fits one server, the gap narrows to a generational difference.

Sizing the GB200 versus H100 decision on GMI

Since the expected gain depends on your model's communication profile, the practical step is measuring it on real hardware rather than assuming a multiple. We are an AI-native GPU and inference cloud that publishes dedicated NVIDIA GPU list pricing and carries both H100 and GB200 NVL72, so you can compare a cluster approach against a rack approach directly.

We currently list H100 at from $2.00 per GPU-hour and GB200 NVL72 at from $8.00 per GPU-hour Available Now, so you can benchmark your large model on both an H100 cluster and an NVL72 rack and measure the actual throughput gain for your parallelism strategy. Verify current rates and rack availability on our GPU infrastructure (https://www.gmicloud.ai/en/gpus) and pricing (https://www.gmicloud.ai/en/pricing), since Blackwell stock and pricing move quickly, and confirm rack quantity and region rather than assuming elastic supply. Benchmark with the sharding and parallelism you will run in production, because that is what determines how much the NVLink domain helps. When the workload is sustained large-model serving, our Prime Inference (https://www.gmicloud.ai/en/models/prime-inference) provides reserved GPU capacity with single-tenant isolation and warm serving, so you hold NVL72-class capacity for steady traffic instead of competing for rack stock at peak. Start a comparison in our console (https://console.gmicloud.ai) or contact our sales team.

Expect the gain where the network is the bottleneck

If you expect a flat speedup from GB200 NVL72 over an H100 cluster, you will over-estimate the gain on models that fit a single server and under-estimate it on models that shard across many nodes. Size the decision by the bottleneck: measure how communication-bound your large model actually is, then benchmark it on both a rack and a cluster with your real parallelism strategy. GB200 NVL72's advantage is the single NVLink domain, so expect the largest throughput gain where cross-node networking limits your H100 cluster today, and a smaller, generational gain where it does not.