What GPU, CPU, Memory, and Interconnect Resources Should You Confirm in a Smaller GB300 NVL72 Allocation Before a Pilot?

A GB300 NVL72 rack holds 72 Blackwell Ultra GPUs and 36 Grace CPUs across 18 compute trays, and each tray carries four GPUs, two Grace CPUs, its own memory, network adapters, and local drives. Because those resources come packaged together, a smaller allocation reads most clearly in tray units, with each resource stated as a number in the pilot quote: the GPUs and their NVLink partition, the Grace cores and host memory, the scale-out and storage network paths, and local and persistent storage. GMI Cloud takes GB300 NVL72 pre-orders through our sales team for long-context, high-capacity training and large-scale reasoning, and the lines below are what that conversation should settle. Every GPU allocation on GMI Cloud is single-tenant and dedicated to one customer.

A GPU count alone doesn't show whether the pilot can run the tests you care about. Two trays in one NVLink partition and two trays in separate partitions have the same eight GPUs, yet only the first lets a parallel group span all eight over NVLink. Scale-out ports work the same way: a multi-node test measures RDMA performance only when those ports reach an RDMA fabric. Otherwise NCCL can fall back to slower transports such as IP sockets, and the numbers stop describing the production setup.

The compute tray sets the size of every other resource

NVIDIA's enterprise reference architecture for NVL72 AI factories describes the GB300 NVL72 compute tray, and the GB300 NVL72 product page gives the rack totals. The two-tray column doubles the tray figures, and the full-rack column lists NVIDIA's rack totals.

NVIDIA lists 800 Gb/s of network connectivity per GPU through the ConnectX-8 SuperNIC, and the reference architecture describes the E1.S drives as a fast local cache. A quote that names only "8 GPUs" leaves the CPU, memory, network, and storage lines to assumption.

GPU lines: generation, tray mapping, and partition

The GPU section of the quote should answer four things.

• The system. GB300 NVL72 with Blackwell Ultra GPUs, named as such. GB200 NVL72 racks and HGX B300 servers are different systems with different memory and topology, and a pilot on either won't stand in for a GB300 NVL72 result.
• Tray mapping. The GPU count and the trays those GPUs sit on. Four GPUs share one tray's CPUs, memory, and network adapters, so a count that isn't a multiple of four covers part of a tray, and the quote should say which host resources come with it.
• NVLink partition. If a tensor-parallel or expert-parallel group spans more than four GPUs and should communicate over NVLink, every GPU in that group needs to sit in the same partition. GPUs in different partitions can't reach each other over NVLink.
• Access level. Bare metal or containers. On GMI Cloud, Bare Metal GPU comes with full root access, hardware-level control, and SLA-backed delivery, and Container Service runs Kubernetes-based GPU environments. Root access lets your engineers read the GPU topology and NVLink status directly, which matters when a pilot result looks wrong.

Model state usually sets the tray count. Assume a model whose FP8 weights take 1.4 TB, kept fully resident in HBM and sharded evenly across the GPUs. One tray's 1,152 GB can't hold them. Two trays in one partition give 2,304 GB, which leaves about 900 GB, or roughly 113 GB per GPU, for KV cache, activations, and runtime overhead before concurrency is added. Whether that is enough depends on the runtime, context length, and batch size the pilot targets.

Grace CPU and host memory lines

Quotes for GPU capacity often say little about the CPU side, yet a GB300 NVL72 tray brings 144 Arm cores and about 1 TB of CPU memory with its four GPUs. NVIDIA counts the rack's HBM and CPU memory together as 37 TB of fast memory, which places host memory inside the system design.

Confirm three things here:

• Core allotment. On a bare-metal tray, the full 144 cores come with the hardware. On a container allocation, the quote should state the CPU limits set for your containers. Tokenization, video decoding, data loading, and request handling all run on these cores, and an input pipeline starved of CPU makes the GPUs look slower than they are.
• Host memory allotment. The full tray's CPU memory, or the stated container limit. Pinned buffers for data loading, CPU-side checkpoint staging, and any KV-cache offload all draw on it.
• Software baseline. Grace uses the Arm architecture, so the quote should name the OS image, driver, and CUDA versions and say whether you supply your own image. Your own containers need aarch64 builds ready before the pilot window opens, so bring-up time goes to testing rather than rebuilding.

Interconnect and storage: three networks, three line items

NVLink, inside the partition. NVIDIA rates the full rack at 130 TB/s of NVLink bandwidth across all 72 GPUs, through nine NVLink Switch trays. A smaller allocation reaches only the GPUs in its own partition, so the line item is partition membership. An NVLink all-reduce across the partition on the first pilot day records a baseline for that GPU set and message size.

East-west scale-out, through ConnectX-8. Each GPU has its own ConnectX-8 adapter. These ports carry traffic that leaves the NVLink partition, and an RDMA test runs over them. The quote should say whether the ports in your allocation are connected to an RDMA fabric and which other trays or nodes they can reach. GMI's GPU infrastructure runs RDMA-ready networking for high-throughput workloads.

North-south, through BlueField-3. BlueField-3 DPUs carry storage, data ingest, and external traffic for the trays. The quote should name the storage tier the pilot reads from and writes to and which networks can reach it. GMI runs isolated VPC networking, so the subnets and storage endpoints inside the pilot's VPC can be listed by name.

Storage gets its own lines too. Local E1.S drives handle scratch and cache, and datasets and checkpoints sit on a persistent tier whose capacity and retention after the pilot ends belong next to the tray lines.

Match each pilot test to the resource it needs

Read the quote backward from the tests you plan. A missing line shows up as a test that can't run as intended.

Pilot quote line items, filled in

The values below are a worked example for a two-tray inference pilot, not a standard offer.

• System and status: GB300 NVL72, pre-order, two compute trays, eight Blackwell Ultra GPUs
• Partition: both trays in one NVLink partition
• Access: bare metal with root access
• Host: four Grace CPUs, 288 cores, about 2 TB CPU memory, all assigned to the customer
• Scale-out: eight ConnectX-8 ports connected to the RDMA fabric
• Storage: persistent storage tier and capacity for datasets and checkpoints, plus eight local E1.S drives as scratch
• Software: OS image, driver, and CUDA versions named
• Region and window: site region (GMI runs GPU infrastructure across the US, APAC, and EU) and pilot start and end dates

FAQ

How should the pilot use Grace CPU memory alongside HBM?

Plan for weights and the active KV cache to sit in HBM, and treat host memory as the tier for pinned data-loading buffers, checkpoint staging, and KV-cache offload where your serving runtime supports it. Offload works differently from one runtime to the next: some use host memory to keep reusable prefixes close at hand, which raises cache hit rates rather than the context a single request can hold. Run the pilot with offload on and off at your target context length and compare latency and throughput, so the full-rack plan counts host memory only for the gain you measured.

Is local NVMe on the trays enough storage for a pilot?

That depends on the size of the model files against the usable capacity of the four E1.S drives per tray. When the files fit with room left for scratch data, a short inference pilot that loads one model can run from local disk. A checkpoint-time test tells you the most when it writes to the same kind of persistent storage the full rack will use.

Turn your pilot plan into quote lines

Send us the model size and precision, the largest parallel group your jobs use, the pilot tests you plan to run, and the storage each test needs. Those four inputs map directly onto the tray, partition, network, and storage lines above. NVIDIA H100 and H200 are available now for preparation work, such as data pipelines and storage layout, that can start before Blackwell delivery. Talk to our sales team about the GB300 NVL72 pre-order.