How Can a GPU Provider Tell Whether Your Workload Is Memory-Bound or Communication-Bound Before Recommending B200, B300, or NVL72?

A memory-bound job shrinks itself to fit: recomputation left on, micro-batches cut to one, or a serving engine preempting requests because its KV cache is full. A communication-bound job fits, yet its GPUs spend part of every step waiting for an all-reduce or all-to-all. A provider separates the two by reading measurements from your real job, three controlled sweeps and one profiler pass, then matches the result to hardware. More memory per GPU points toward B300 or GB300 NVL72, a parallel group wider than eight GPUs points toward an NVL72 rack, and a job held back by its input pipeline or host overhead needs a software fix first. On GMI Cloud, your engineers take those measurements on H100 and H200 bare metal, available now with full root access, and we use the results to size a Blackwell pre-order with you.

The two are easy to confuse. Recomputing activations to save memory lengthens every step, which hides network waits, and NCCL kernels count as busy time in nvidia-smi, so a communication-heavy job can look fully utilized.

Signs that memory is the limit

Capacity limits show up as work your team does only to make the model fit:

• Out-of-memory errors when you raise micro-batch size or sequence length
• Full activation checkpointing that stays on although it adds close to one extra forward pass per step (selective schemes cost less)
• Extra ZeRO or FSDP sharding, or pipeline stages, added for capacity rather than speed
• Optimizer states offloaded to CPU memory
• In serving, concurrency capped below target because the KV cache runs out, with preempted requests or a shortened context limit

Bandwidth limits look different. In small-batch decode on a large dense model, each decode iteration reads the weights from HBM to produce one token per sequence, so step time tends to track memory throughput rather than arithmetic. Confirm it from achieved memory throughput in the decode kernels, not from DCGM's DRAM activity alone. More capacity helps only indirectly, through larger batches within the latency target.

For capacity, reset peak statistics with torch.cuda.reset_peak_memory_stats after warm-up, then record two peaks per GPU over the same measured steps: torch.cuda.max_memory_reserved for PyTorch's allocator, and the sampled device-level peak from DCGM at a short interval, which also captures NCCL and CUDA runtime memory outside the allocator. Note which memory-saving settings are active.

Signs that communication is the limit

Communication limits show up in the timeline rather than in error messages:

• An Nsight Systems trace shows NCCL kernels running with no compute kernels overlapping them
• Throughput per GPU drops when the same job moves from one node to two at the same micro-batch
• A tensor-, expert-, or context-parallel group spans nodes, so per-layer collectives cross the scale-out network
• Mixture-of-experts all-to-all time grows as the expert-parallel width grows
• DCGM profiling metrics show NVLink or network traffic peaking while SM activity dips

A single slow rank makes every other rank wait inside collectives, which looks like a slow fabric; per-rank timing finds the rank that arrives last, and its clocks, link counters, and load separate a GPU fault from a network or load problem.

Three sweeps and a profile that separate the two

Each sweep changes one setting with the rest held fixed. Report throughput as tokens per second per GPU.

Read the four together. Capacity is the limit when the headroom sweep hits the memory ceiling while still gaining and recompute takes a real share of the step. Communication is the limit when exposed NCCL time is a large share of the step and the ladder loses efficiency with each node. Otherwise, look at compute, input, or host overhead. Set in advance the share of step time that would justify new hardware at your budget.

Scaling efficiency at N nodes is throughput at N nodes divided by N times the one-node throughput. If one node processes 100 samples per second and four process 300, efficiency is 300 ÷ (4 _ 100), or 75%: a quarter below linear scaling. The trace shows where that gap goes.

Pair the ladder with NVIDIA's nccl-tests, running your job's collectives at your real message sizes and rank layout, and compare large-message bus bandwidth with a known-good baseline of the same topology. A clear shortfall points to a link or configuration to investigate. Bandwidth at baseline with exposed communication still high sends you back to the trace: tune overlap and bucket sizes first, then count only the remaining time spent moving large messages, not per-call latency or waits on dependent operations, toward a larger NVLink domain or faster scale-out networking.

For serving, a tensor-parallel ladder (TP 2, 4, and 8) at a fixed request mix, concurrency, and latency target plays a similar role, though changing TP width also changes per-GPU compute and weight footprint.

From diagnosis to the configuration you ask for

B200 and B300 are GPUs, normally bought as eight-GPU HGX nodes; NVIDIA rates NVLink on both at 1.8 TB/s per GPU, and B300 carries more HBM per GPU. NVL72 is a rack: in NVIDIA's GB300 version, 72 Blackwell Ultra GPUs share one NVLink domain rated at 130 TB/s total, so a parallel group of up to 72 GPUs placed in one rack stays on NVLink.

When a bigger GPU or a bigger domain won't help

These causes sit outside the GPU and its links:

• The input pipeline. GPUs sit idle between steps with no NCCL activity in the trace. Data loader workers, storage throughput, or CPU-side decoding set the pace.
• A straggler. One throttled GPU or degraded link holds every collective back; repairing or replacing that node is the fix.
• Expert imbalance. A few experts receive most tokens, so all-to-all waits grow. Load-balancing loss, routing, and capacity-factor settings are the first lever; if dispatch traffic stays heavy after that, interconnect bandwidth matters again.
• Host overhead. Many small kernels leave the GPU waiting on CPU launches. CUDA Graphs or larger fused kernels address it.
• Checkpoint stalls. Step time spikes at every save interval. Asynchronous checkpointing or faster storage addresses it.

Hardware for the test runs and for the answer

H100 and H200 run in GMI-operated data centers, and bare metal access lets your engineers install Nsight Systems, DCGM, and nccl-tests and run the sweeps themselves. Container Service provides Kubernetes-based GPU environments for serving sweeps, and Cluster Engine orchestrates multi-node clusters with RDMA-ready networking for the scaling ladder.

For the answer, B200 nodes, B300 on Prime Inference, and GB200 and GB300 NVL72 racks, all on pre-order, give each row a candidate to size against your measured requirement. When the diagnosis lands on KV cache capacity or decode bandwidth, Prime Inference adds GMI's engineering team, who work with you from prototype to production SLA, and vLLM, TensorRT-LLM, and SGLang runtimes tuned for each GPU class.

FAQ

What can we share if our model weights can't leave our environment?

The sizing discussion runs on measurements rather than weights: model size and architecture, precision, parallel layout, peak memory per GPU, and the sweep results. On GMI hardware, randomly initialized weights in the same architecture give a sound structural estimate of memory use and communication volume when precision, optimizer, sharding, batch, sequence lengths, and cache settings match production. Data-dependent behavior, such as expert routing or serving output lengths, should come from your own runs.

How long does each experiment need to run?

Discard the first steps, where compilation and allocator warm-up distort timing, then profile 20 to 50 steady-state steps as a starting point and run each configuration at least twice, widening the window if repeats disagree. Multi-node setup usually takes longer than the runs, so prepare launch scripts before the nodes are allocated.

Turn the measurements into a pre-order request

Send the four results, with model size, precision, and target scale, to our sales and engineering team to work out whether B200, B300, or an NVL72 rack fits the job in a Blackwell pre-order. Numbers still missing can be measured first on H100 or H200 from the GMI Cloud console.