NVIDIA GB300 NVL72 Explained: 288GB HBM3e Rack-Scale Inference

The GB300 NVL72 is not a faster single GPU. It is a rack wired so that 72 Blackwell Ultra GPUs behave like one accelerator with one shared memory space. When your model no longer fits on a node and the cross-node network becomes the bottleneck, the question stops being "which card" and becomes "how large is the coherent domain," and that is exactly what this rack is built to answer. This article covers what the GB300 NVL72 changes for inference, when the rack is worth it, and a pooled memory worked example you can map onto your own model.

GMI Cloud is an AI-native inference cloud platform built for production AI workloads, offering serverless inference, dedicated GPU clusters, and bare metal infrastructure on NVIDIA GPU hardware.

What the GB300 NVL72 actually changes for inference

A conventional 8-GPU server gives you a fast NVLink island of eight accelerators, after which traffic crosses InfiniBand or Ethernet to reach the next node. That hop is an order of magnitude slower than NVLink and it is where large-model inference stalls. The GB300 NVL72 removes the hop by extending the NVLink fabric across the whole rack, so all 72 GPUs sit inside a single coherent NVLink domain.

The numbers that matter for serving:

• Per-GPU memory: 288GB of HBM3e on each Blackwell Ultra GPU, up from 192GB on the GB200 generation. That extra capacity is what lets a single GPU hold more of a model and a far larger KV cache.
• Pooled capacity: roughly 20TB of HBM across the rack once you count all 72 GPUs, addressable as one memory space rather than 72 separate pools you have to shard around.
• NVLink fabric: about 130 TB/s of all-to-all bandwidth inside the domain, so tensor and expert parallel traffic stays on NVLink instead of dropping to the slower cross-node network.

The practical read is simple. On a node-bound setup, a trillion-parameter model is split across many servers and every token of inference pays the network tax between them. Inside the rack-scale domain, that same model is split across GPUs that talk to each other at NVLink speed, so the parallelism overhead that usually caps inference throughput at scale largely disappears.

Why GB300 NVL72 memory bandwidth is the real lever

For decode-heavy serving, throughput is gated by how fast the hardware can stream weights and KV cache, not by raw FLOPs. The GB300 NVL72 memory bandwidth, both the per-GPU HBM3e bandwidth and the aggregate NVLink fabric, is what keeps a sharded model fed. When the GB300 NVL72 memory bandwidth inside the domain exceeds what any cross-node network can deliver, the rack sustains token rates that a multi-node cluster cannot match at the same model size. This is the core reason NVLink domain inference behaves differently from cluster inference: the interconnect stops being the limit.

A pooled memory worked example

Numbers make the case better than adjectives. Take a 2T-parameter mixture-of-experts model served at FP4 for inference.

1. Weights. At roughly 4 bits per parameter, 2T parameters need about 1TB just for weights. That does not fit on any single GPU, and on an 8-GPU node it consumes most of the available HBM before any cache is allocated.
2. KV cache headroom. Long-context serving needs large KV cache. Reserve, say, 4TB across the domain for concurrent sessions. Weights plus cache is now around 5TB.
3. Fit check. The rack's ~20TB pooled HBM holds the 1TB of weights and the 4TB of cache with room left for activations and additional concurrency, all inside one NVLink domain.

The point is not just that it fits. On a node-bound deployment the same model is fragmented across many servers, and every decode step of inference crosses the network for tensor and expert exchanges. Inside the rack, those exchanges stay on the fabric, so the effective decode throughput per token climbs and the cost per million tokens of inference falls even though the rack rents at a frontier-tier rate. Pooled memory turns "this model is too big to serve efficiently" into "this model fits in one domain."

When rack-scale inference is worth it, and when it is not

The GB300 NVL72 is not an automatic upgrade. It earns its place only for a specific class of workload.

Configuration	Pooled HBM	Interconnect	Best fit
H200 (single GPU)	141GB HBM3e	4.80 TB/s	7B to 70B inference that fits one card
B200 (single GPU)	192GB HBM3e	8.0 TB/s	Large models, high single-node throughput
GB200 NVL72 (rack)	~13.5TB pooled	~130 TB/s NVLink	Frontier pooled training and inference
GB300 NVL72 (rack)	~20TB pooled	~130 TB/s NVLink	Trillion-parameter, long-context rack-scale inference

Rack-scale inference pays off when:

• The model plus its KV cache does not fit in a single 8-GPU node, forcing slow cross-node sharding.
• You serve very long context or very large batches where KV cache dominates memory.
• Your serving is bound by interconnect, not compute, so collapsing the network hop into NVLink domain inference raises tokens per second directly.

It is the wrong call when a model fits comfortably on a single B200 or H200, or when traffic is bursty and a full reserved rack would sit underused. A 70B model that fits on one H200 gains nothing from a domain it never touches, and you would be paying frontier rates for idle capacity. Serverless inference and dedicated capacity solve different problems here: serverless absorbs spiky, API-style demand without you reasoning about hardware, while a reserved rack is for sustained, predictable, very-large-model serving you have already sized.

Running the GB300 NVL72 without losing the fabric to overhead

A rack-scale design only delivers if your workload sees the full fabric. Virtualization layers commonly skim bandwidth and add latency before traffic reaches the GPU, which is the opposite of what you want when the whole value proposition is interconnect speed. GMI Cloud bare metal instances run with no hypervisor, delivering 100% of the advertised NVLink and HBM3e bandwidth to your serving stack instead of a virtualized fraction of it.

GMI Cloud is best suited for teams serving trillion-parameter or long-context models that have outgrown a single node and need a coherent NVLink domain rather than a network-stitched cluster. As an NVIDIA Preferred Partner, GMI Cloud pairs frontier-class hardware with a 99.99% availability SLA and a preconfigured stack including CUDA, TensorRT-LLM, and vLLM, so the time from provisioning to first token stays short. For reference, GMI Cloud lists GB200 NVL72 rack capacity at $8.00/GPU-hour; the GB300 NVL72 sits in the same frontier tier, and you should confirm current rates directly rather than assume a number.

Best for: trillion-parameter and mixture-of-experts models where pooled HBM and NVLink domain inference remove the cross-node bottleneck. Best for: long-context, large-batch serving where KV cache outgrows a single node. Not ideal for: models that fit on one B200 or H200, or bursty traffic better matched to serverless.

To verify current configurations and rates before you commit a rack, the live details are documented at docs.gmicloud.ai and you can provision through console.gmicloud.ai.

Size the domain to the model, not the rack to the spec sheet

The GB300 NVL72 is the right rental when your model has outgrown a node and the network has become the limit, and the wrong one when a single card still has headroom. Before you reserve a rack, profile a real serving run and find where it stalls. If the answer is cross-node interconnect on a model too large to fit one node, the pooled HBM and NVLink fabric pay for themselves in throughput. If the model already fits and the fabric would sit idle, rent the single card that removes your actual constraint and revisit the rack when the model, not the spec sheet, demands it.