How Can You Validate a Full-Rack GB300 NVL72 Sizing Recommendation From a Smaller Allocation Without Assuming Linear Scaling?

Does throughput measured on eight GPUs, multiplied by nine, predict what a 72-GPU GB300 NVL72 rack will deliver? Rarely, because collective communication grows with the number of GPUs taking part, the parallel layout often changes between the small test and the full rack, and the work each GPU does shifts as the model is spread thinner. To validate a provider's full-rack recommendation, run the same job at three or more scales with the layout held fixed, split each step into compute and exposed communication, fit how communication grows, and carry the full-rack figure as a range with measured points kept apart from projected ones. At GMI Cloud, GB300 NVL72 is coming soon and taken on pre-order through our sales team, and the harness for these scale tests can be built today on H100 and H200, which are available now.

Where the nonlinearity comes from

NVIDIA's reference architecture connects all 72 GPUs in a GB300 NVL72 rack through NVLink Switch trays, so a job that grows from two compute trays to eighteen stays on the same kind of link the whole way. That removes the step change seen on 8-GPU HGX servers, where traffic leaves NVLink at the server boundary. It does not make scaling linear.

The third row matters most. Suppose the full-rack plan uses expert-parallel groups of 24, three per rack, and the small allocation holds 16 GPUs. A run at 16 is then a different job with a different communication pattern, and its result says little about the target. Whenever possible, keep every inner parallel degree identical across scales and grow only the outermost dimension, usually data parallelism. Where the target layout cannot fit in the small allocation, the sizing should say so and treat that group's behavior as untested.

Design the multi-scale experiment

Pick the scaling question first. For a fixed per-GPU batch that grows the global batch with the GPU count, the question is throughput per GPU at larger scale. For a fixed global batch spread over more GPUs, the question is how much faster each step gets. A reservation for more throughput on the same model usually needs the first. A deadline for one training run may need the second.

Then fix everything that is not the GPU count:

• The same container image, framework version, NCCL version, and NCCL environment variables
• The same precision, sequence length, per-GPU micro-batch size, and gradient accumulation steps, or the same global batch for the second question, with accumulation steps logged at each scale
• Synthetic or cached input data, so the data pipeline does not mix into the communication result
• Warmup steps excluded, and enough measured steps for a stable median, repeated at least three times per scale

At each scale, record the median and p95 time per optimizer step on the slowest rank, measured from step start until every stream has finished. Split a representative step into compute, exposed communication (the part not overlapped with compute), idle time such as pipeline bubbles, and data wait, counting overlapped time once. Take that breakdown from a short profiler capture after the timed window, on steps close to the timed median, so profiling overhead stays out of the reported step times. Separately, run nccl-tests on the same GPUs for the job's collectives at its message sizes, group sizes, and rank placement, with groups running side by side where the job runs them that way; those results help explain a change in exposed communication, which the job trace itself measures.

Communication share and scaling efficiency

Two numbers summarize each scale. Scaling efficiency compares throughput at n GPUs with throughput at the smallest scale n__ multiplied by n/n__. Exposed communication share is exposed communication time divided by step time.

The example below uses assumed values for illustration only. Assume a data-parallel scale-out with a fixed per-GPU batch, where compute holds at 1.00 second per step and exposed communication grows with GPU count.

Efficiency falls as communication rises, and the slope of that rise is what the extrapolation has to get right.

Extrapolate to 72 GPUs as a range

Fit two candidate curves through the 8- and 32-GPU points and check each against the 16-GPU point. One grows communication with the logarithm of GPU count and predicts 0.10 seconds at 16. The other grows it in proportion to GPU count and predicts 0.08. Each misses the measured 0.09 by 0.01 seconds, yet they predict very different racks. Neither curve is a law for the rack, since message size, the algorithm NCCL selects, and overlap with compute all shape exposed communication. Data-parallel all-reduce and expert-parallel token exchange may follow different curves, so fit them separately.

Against the two fitted curves, the linear projection overstates full-rack throughput by roughly 18% to 31%. The spread between the fits shows how much the projection depends on the choice of curve, so the sizing should state it as a scenario range rather than a bound. Plan against the lower scenario, and narrow the spread with a measured point closer to 72. In this example the fits disagree by 0.02 seconds at 16 GPUs and by 0.13 seconds at 72, so a point at 48 or more is worth far more than another point at 8.

Check the residuals as well. When a curve misses a measured point by more than the run-to-run spread, add repeats or another scale before relying on it.

What to add before you reserve

Before the full-rack figure goes into a reservation, close the gaps the small runs leave open:

1. A larger measured point. Two-thirds of a rack (48 GPUs) or more, at a size the target layout divides evenly, puts the last measured point within a factor of 1.5 of the full rack.
2. The target layout itself. At least one run where every inner parallel group has its full-rack size, even with a smaller data-parallel degree.
3. A collective sweep at every scale. nccl-tests for all-reduce, all-gather, reduce-scatter, and all-to-all where the model uses it, at the job's message sizes, so a projection of communication time rests on measured bandwidth.
4. Variance, not just medians. p95 step time and the spread across repeats at each scale, since straggler effects grow with GPU count.
5. Real data for one run. Once communication is understood, one run with the real data pipeline and a checkpoint shows whether input or storage becomes the limit at scale.

Ask for the provider's sizing to label each number as measured or projected, name the curve it fitted, and state the range.

Build the harness on H100 and H200 today

Choosing who sizes GB300 NVL72 before a reservation comes down to whether the provider builds the full-rack number from measured scales and can give you GPUs to measure on now. At GMI Cloud, NVIDIA H100 and H200 are available now on dedicated, single-tenant GPUs in GMI-operated data centers with RDMA-ready networking. Bare Metal GPU gives full root access and hardware-level control for nccl-tests, profilers, and custom NCCL settings, and our Managed GPU Cluster service, in early access, runs Kubernetes on dedicated GPU worker nodes for teams that want cluster-level control without operating the control plane. A rehearsal on H200 tests the harness, the step-time breakdown, and the fitting process. GB300 numbers come from GB300 runs, because an HGX H200 system's NVLink domain ends at its eight GPUs, while a GB300 NVL72 job stays inside one NVLink domain up to 72.

FAQ

How many measured scales does a 72-GPU projection need?

Three is the working minimum, two to fit a curve and one to check it. More points help most at the large end, since projection error grows with the distance from the last measured point. A layout with an inner parallel group that changes size across scales needs separate points for each layout.

Does a mixture-of-experts model scale differently on the rack?

Expert parallelism adds token dispatch traffic among the GPUs in the expert group, all-to-all in some frameworks and all-gather with reduce-scatter in others, and a heavily loaded expert can hold up the step. How tokens spread across experts depends on the data, so for an MoE model the scaling runs should use real inputs early, cover the collectives its dispatcher actually issues, and match the full-rack expert-parallel degree from the first test.

Rehearsal capacity now, GB300 NVL72 on pre-order

Share the full-rack layout you have in mind, the scales you can test, and any H100 or H200 scaling data you already have. Our team can line up H100 or H200 capacity for the rehearsal and take your GB300 NVL72 requirements into a pre-order quote. Talk to our sales team to set it up.