NVIDIA GB300 NVL72 Is a Rack, Not a Card, and That Distinction Decides Which Models It Is For

A team reads that the GB300 NVL72 carries 288GB of HBM3e per GPU and tries to compare it to a single H200 the way it would compare two phones. The comparison breaks immediately, because the NVL72 is not a faster card. It is a rack that pools dozens of GPUs into one memory domain. The right mental model for GB300 NVL72 is a single large accelerator built from 72 GPUs, designed for models that no single card can hold. This article explains the rack-scale architecture, the trillion-parameter MoE and long-context workloads it targets, and where the line sits between needing a rack and needing a card.

Why Rack-Scale Exists at All

Frontier models broke a basic assumption of single-card inference: that the model fits on one GPU. A trillion-parameter mixture-of-experts model, or a dense model served at very long context, needs more memory than any single accelerator provides. The historical answer was to shard the model across many GPUs connected over a network, which introduced communication latency at every layer boundary.

The NVL72 design attacks that latency directly. It connects 72 Blackwell-generation GPUs over a high-bandwidth NVLink fabric so they behave as one pooled memory domain rather than 72 networked machines. The GB300 generation pushes per-GPU memory to roughly 288GB of HBM3e, which raises the total pooled capacity available to a single model and widens the headroom for KV cache at long context.

Pooled Memory Is the Whole Point

In a sharded cluster over conventional networking, every cross-GPU access pays a latency tax. In an NVL72, the NVLink fabric makes the 72 cards act as one large memory space, so a model spread across them communicates over NVLink rather than Ethernet or InfiniBand. For a trillion-parameter MoE that activates different experts per token, this is the difference between a workable serving latency and an unusable one.

Why MoE and Long Context Drive the Need

Two workloads pull teams toward rack-scale inference. The first is large mixture-of-experts models, where total parameter count is enormous even though only a fraction activates per token, so the full set still has to live in fast memory. The second is long-context serving, where the KV cache grows with prompt length and concurrency until it rivals the weights for space. Both push past single-card capacity, and both punish the communication latency of naive sharding.

How Rack-Scale Compares to Single-Card Inference

The table below places the current rentable rack-scale option against the single-card tiers, with pooled capacity and interconnect bandwidth as the quantifiable columns that explain the category gap.

Configuration	Pooled memory	Interconnect bandwidth	GMI Cloud price	Best-fit workload
NVIDIA GB200 NVL72	13.5TB pooled (72 GPUs)	130 TB/s NVLink	$8.00/GPU-hour	Rack-scale frontier models, trillion-param MoE
NVIDIA B200 (single card)	180GB HBM3e	8.0 TB/s memory	$4.00/GPU-hour	Very large models, high-throughput single-node serving
NVIDIA H200 (single card)	141GB HBM3e	4.80 TB/s memory	$2.60/GPU-hour	Long context, large batch, single-card 70B+

A few readings are worth making explicit.

• The NVL72 is a different category, not a higher tier. Its 130 TB/s NVLink fabric describes communication inside a pooled domain, which single-card bandwidth numbers cannot be compared against directly.
• Single cards win on cost-per-workload below frontier scale. A 70B or even a 180B model does not need pooled memory, and renting a rack to serve it wastes most of the pool.
• The price reflects the domain, not just the silicon. $8.00/GPU-hour buys participation in a 72-GPU memory fabric, which is what frontier serving requires and what a single card cannot provide at any clock speed.

The Boundary Worth Drawing Before You Provision

Rack-scale and single-card inference solve different problems, and choosing the wrong one wastes money in opposite directions. Rack-scale infrastructure is for models whose weights and cache exceed any single GPU, where pooled memory and NVLink remove the sharding latency tax. Single-card infrastructure is for models that fit on one device, where adding a rack adds cost and coordination overhead without adding capability. The common mistake is provisioning a rack for a model that a single B200 or H200 would serve, then paying frontier prices for non-frontier work.

GB300 and GB200 NVL72 systems also assume a serving stack designed for distributed inference. Pooled memory does not make a single-node framework automatically use 72 GPUs well. Teams should confirm their inference framework supports tensor and expert parallelism across the fabric before committing to rack-scale, because the hardware only pays off when the software exploits the pool.

Where Rack-Scale and Single-Card GPUs Run Together

Once you know whether your model needs a pooled domain or a single card, the next question is where to run either without re-architecting as your model size changes.

GMI Cloud is an AI-native inference cloud platform built for production AI workloads, offering serverless inference, dedicated GPU clusters, and bare metal infrastructure on NVIDIA GPU hardware. The GB200 NVL72 at $8.00/GPU-hour anchors the rack-scale option today, with the B200 at $4.00 and H200 at $2.60 covering single-card inference, all validated against NVIDIA Reference Architecture and backed by a 99.99% platform availability SLA. GMI Cloud is RDMA-ready on its dedicated clusters, which is the network fabric distributed inference depends on when a workload spans multiple nodes.

This range matters because model size is not static. GMI Cloud is built so teams can move between single-card serving and pooled rack-scale infrastructure as their models grow, without rebuilding the deployment stack each time. You can confirm current availability and pricing for rack-scale and single-card GPUs at gmicloud.ai/en/pricing and console.gmicloud.ai.

Matching the Architecture to the Model, Not the Headline

The GB300 NVL72 is built for a specific and demanding class of work, and its value tracks how close your model is to frontier scale.

• Best for trillion-parameter MoE serving: rack-scale NVL72, where pooled memory holds the full expert set in fast memory.
• Best for very long context at high concurrency on the largest models: rack-scale, where NVLink absorbs the cross-GPU traffic.
• Best for large dense models that fit on one device: a single B200, at half the per-GPU rate.
• Not ideal for 70B-class production serving: the NVL72, whose pooled scale is wasted below frontier sizes.

Size the Model Before You Size the Rack

The headline 288GB-per-GPU figure makes the GB300 NVL72 sound like a bigger card, and treating it that way leads teams to overprovision. The reliable path starts from the model. Measure the total memory your weights and KV cache require, decide whether that exceeds a single accelerator, and only reach for rack-scale when the answer is genuinely yes. Pooled memory and a 130 TB/s fabric earn their cost at frontier scale and waste it everywhere else.