InfiniBand and NVLink: When Interconnect Becomes Your Bottleneck

Adding GPUs only speeds up training if the GPUs can exchange data fast enough to keep each other busy. NVLink connects GPUs inside a server or rack, InfiniBand connects servers to each other, and the roughly ninefold bandwidth gap between the two decides where your parallelism strategy should place each kind of communication.

• Two networks, two jobs: NVLink is the scale-up fabric, with 900 GB/s of total bandwidth per Hopper GPU and 1.8 TB/s per Blackwell GPU, while InfiniBand is the scale-out fabric, typically one 400 Gb/s or 800 Gb/s adapter per GPU.

• The gap is about 9× per direction: A Hopper GPU moves about 450 GB/s each way over NVLink but about 50 GB/s each way over a 400 Gb/s NDR InfiniBand link, so communication that crosses servers is far more expensive than communication that stays inside them.

• Parallelism has to follow topology: Tensor parallelism and MoE all-to-all traffic belong inside the NVLink domain, while data and pipeline parallelism tolerate the slower links between nodes.

• Tuning can recover large gaps: NVIDIA traced a 31% training slowdown at 512 GPUs to a single NCCL queue-pair setting on a 1.6 Tbps fabric, and in-network reduction with SHARP has doubled all-reduce bandwidth in NVIDIA’s published results.

• Rack-scale NVLink changes the boundary: GMI Cloud offers H200 servers with one 400 Gb/s InfiniBand adapter per GPU and GB200 NVL72 systems with a 72-GPU NVLink domain, so teams can choose where the fast fabric ends. Knowing that boundary is the first step to scaling efficiently.

Distributed training splits a model, its data, or both across many GPUs, then keeps those GPUs in sync. Every training step ends with collective operations such as all-reduce, all-gather, and reduce-scatter, and no GPU can start the next step until those collectives finish. When the interconnect cannot keep up, expensive GPUs sit idle waiting for data, and adding more of them makes the problem worse rather than better.

Interconnect problems are also easy to miss. Single-node benchmarks look healthy, GPU specifications look identical across providers, and the slowdown only appears at scale. In its Exemplar Cloud lessons, NVIDIA described a GB300 NVL72 deployment whose single-node throughput looked fine but whose training ran 31% slower than reference at 512 GPUs because of how collectives were using the network.

This guide explains how NVLink and InfiniBand differ, how each parallelism strategy loads them, how to tell when interconnect has become your bottleneck, and what to do about it, from mapping parallelism to topology to choosing rack-scale systems.

Two Networks Inside Every GPU Cluster

NVLink and NVSwitch: Scaling Up

NVLink is NVIDIA’s direct GPU-to-GPU interconnect, and NVSwitch chips connect every GPU in a domain to every other GPU at full NVLink speed. In an eight-GPU HGX H100 or H200 server, fourth-generation NVLink gives each GPU 900 GB/s of total bandwidth, about 450 GB/s in each direction, which NVIDIA describes as more than seven times the bandwidth of PCIe Gen5. Blackwell’s fifth-generation NVLink doubles that to 1.8 TB/s per GPU. The GB200 NVL72 extends the NVLink domain from one server to a full rack, joining 72 Blackwell GPUs with 130 TB/s of aggregate bandwidth so that all of them can communicate at NVLink speed.

InfiniBand: Scaling Out

Once a job needs more GPUs than one NVLink domain holds, traffic has to cross the scale-out network. In AI clusters, that is usually InfiniBand, with RDMA over Converged Ethernet as the main alternative. The common design gives each GPU its own network adapter, connected through GPUDirect RDMA so data moves between GPU memory and the network without passing through the CPU. With NDR InfiniBand, each adapter runs at 400 Gb/s, so an eight-GPU server has 3.2 Tb/s of total scale-out bandwidth. Newer XDR adapters such as ConnectX-8 run at 800 Gb/s. Clusters are usually built in a rail-optimized layout, where GPU 0 of every server connects to the same switch group, GPU 1 to another, and so on, which keeps most collective traffic to a single switch hop.

The numbers that matter most are the ratios. Within each generation, a GPU’s NVLink connection carries about nine times as much data per direction as its InfiniBand connection. Any communication pattern that ignores that boundary will run at the speed of the slower fabric.

How Parallelism Strategies Load the Interconnect

Data Parallelism

In data parallelism, every GPU holds a copy of the model and processes a different slice of the batch. After the backward pass, gradients are averaged across all replicas with an all-reduce. Sharded variants such as ZeRO and FSDP replace that with reduce-scatter and all-gather operations. The traffic is large but happens once per step, and frameworks overlap much of it with the backward pass, so data parallelism is the strategy that tolerates InfiniBand best.

Tensor Parallelism

Tensor parallelism splits individual layers across GPUs, which means collectives inside every transformer layer, in both the forward and backward pass. That traffic is frequent, latency-sensitive, and hard to overlap with compute. It is why tensor parallelism is almost always kept within a single NVLink domain, typically eight GPUs on an HGX server, and why rack-scale NVLink makes larger tensor-parallel groups practical.

Pipeline Parallelism

Pipeline parallelism assigns groups of layers to different GPUs and passes activations from one stage to the next. Communication is point-to-point and comparatively small, which makes pipeline stages a good fit for splitting a model across nodes. The cost is pipeline bubbles, idle time while stages wait for work, which scheduling techniques reduce but do not eliminate.

Expert Parallelism in MoE Models

Mixture-of-experts models route each token to a few of many experts, often placed on different GPUs. That creates all-to-all communication, where every GPU sends data to every other GPU, twice per MoE layer. All-to-all traffic is among the hardest patterns for a scale-out network, which is a major reason MoE models benefit so much from large NVLink domains. NVIDIA rates the GB200 NVL72 at 10× greater performance than Hopper for MoE architectures, much of it because expert traffic can stay on NVLink across 72 GPUs.

Inference Uses the Interconnect Too

Interconnect is not only a training concern. Models too large for one GPU are served with tensor parallelism, which puts the same per-layer collectives on the critical path of every token. Large MoE models served across many GPUs add all-to-all traffic, and serving architectures that separate prefill from decode move KV cache data between GPUs. For latency-sensitive serving, keeping a model inside one NVLink domain is usually worth more than adding servers.

Signs That Interconnect Is Your Bottleneck

Interconnect bottlenecks rarely announce themselves. The clearest signal is poor scaling efficiency: If doubling the number of nodes gives you much less than double the throughput, communication is likely consuming the difference. A profiler timeline from a tool such as Nsight Systems shows the same thing directly as exposed all-gather, reduce-scatter, or all-reduce time that is not hidden behind compute. GPU utilization that dips in a regular rhythm at synchronization points tells the same story.

Collective benchmarks confirm it. NCCL tests report bus bandwidth for each collective and message size, which you can compare with what the hardware should deliver. NVIDIA’s GB300 NVL72 case study is a good example. All-gather bus bandwidth measured about 28 GB/s, against about 61 GB/s once the number of queue pairs per connection, set with NCCL_IB_QPS_PER_CONNECTION, was raised from 1 to 4. On the reference cluster, that change cut iteration time from about 1.09 seconds to about 0.83 seconds. NVIDIA also cautions that the right value depends on the fabric and message sizes, so the setting should be swept and verified rather than copied.

Estimating Communication Cost Before You Scale

A rough estimate helps decide where each kind of traffic should run. In a ring all-reduce, each of N GPUs sends and receives about 2(N − 1)/N times the size of the data being reduced.

all-reduce time ≈ 2 × (N − 1) ÷ N × data size ÷ effective bandwidth per GPU

The short Python helper below applies the formula to the gradients of a 7B-parameter model in BF16, about 14 GB per step, and compares an all-reduce inside one NVLink server with a flat ring across 16 servers on 400 Gb/s InfiniBand, assuming 80% of link bandwidth is achieved.

def ring_allreduce_seconds(data_gb: float, num_gpus: int,
                          link_gb_per_s: float, efficiency: float = 0.8) -> float:
    """Approximate time for one ring all-reduce."""
    traffic_gb = 2 (num_gpus - 1) / num_gpus data_gb
    return traffic_gb / (link_gb_per_s efficiency)

grads_gb = 7e9 2 / 1e9   # 7B parameters with BF16 gradients, ~14 GB

# Eight GPUs in one HGX H100 or H200 server, NVLink at ~450 GB/s per direction
print(round(ring_allreduce_seconds(grads_gb, 8, 450), 3))     # ~0.068 s

# 128 GPUs across 16 servers, 400 Gb/s InfiniBand at ~50 GB/s per GPU
print(round(ring_allreduce_seconds(grads_gb, 128, 50), 3))    # ~0.694 s

The cross-node all-reduce takes roughly ten times longer for the same data. In practice, NCCL uses hierarchical and tree algorithms that reduce inside each server first, and frameworks overlap gradient communication with the backward pass, so real step times are better than this flat-ring estimate. The estimate is still useful as a ceiling. If communication time approaches your compute time per step, the interconnect will limit scaling no matter how many GPUs you add.

How to Fix an Interconnect Bottleneck

Map Parallelism to Topology

The most effective fix costs nothing. Keep tensor parallelism and expert all-to-all traffic inside the NVLink domain, use pipeline parallelism to span servers, and let data parallelism run across the scale-out fabric. Make sure the job launcher places ranks so that each tensor-parallel group sits on one server, and bind processes to GPUs and network adapters with topology-aware affinity rather than by rank order. Running nvidia-smi topo -m on each node shows which GPU shares a PCIe switch with which adapter.

Overlap and Shrink Communication

Frameworks can hide much of the data-parallel traffic behind the backward pass by bucketing gradients and starting all-reduces as soon as each bucket is ready. Sharded data parallelism can prefetch parameters for the next layer while the current one computes. You can also send fewer bytes, by communicating gradients in BF16 rather than FP32, or send less often, by accumulating gradients over several micro-batches before synchronizing.

Use In-Network Reduction

NVIDIA Quantum InfiniBand switches support SHARP, which performs reductions inside the switches as data passes through instead of sending every gradient to every GPU. In NVIDIA’s published SHARP results, all-reduce bandwidth doubled, which translated into 17% higher BERT training performance. SHARP requires supporting switches and the NCCL SHARP plugin, and NCCL uses it when collective network support is enabled, for example with NCCL_COLLNET_ENABLE=1. Ask your provider whether it is available on your fabric.

Tune NCCL for the Fabric

NCCL’s defaults are reasonable but not universal. Queue pairs per connection, the list of adapters NCCL uses, and the topology description it reads all affect collective performance. Set NCCL_DEBUG=INFO on a first run to see which transports and paths NCCL chooses, verify that any topology file is visible inside the training container, and test collectives at the message sizes your workload actually uses before and after each change.

Move to a Larger NVLink Domain

When a model’s communication pattern simply does not fit eight-GPU servers, the remaining option is a bigger scale-up domain. Rack-scale systems such as the GB200 NVL72 keep 72 GPUs on NVLink, which suits trillion-parameter models, large tensor-parallel groups, and MoE models with heavy all-to-all traffic. In MLPerf Training v5.0, GB200 NVL72 systems trained Llama 3.1 405B 2.2× faster per GPU than Hopper at 512-GPU scale, with fifth-generation NVLink among the contributing factors NVIDIA cited.

Choosing Interconnect on GMI Cloud

GMI Cloud offers both sides of the scale-up and scale-out trade-off. Its H200 bare metal configurations pair eight H200 GPUs with eight ConnectX-7 400 Gb/s NDR InfiniBand compute adapters, one per GPU, which gives each server 3.2 Tb/s of scale-out bandwidth for multi-node training with GPUDirect RDMA. Bare metal provides the root access needed to verify topology, run NCCL tests, and tune collectives yourself, and it is enabled per organization. GMI’s guide to comparing container, bare metal, and managed cluster quotes explains how the options differ.

For workloads that need a larger NVLink domain, GMI Cloud lists the GB200 as available now from $8.00 per GPU-hour as of October 2026, with the GB300 on pre-order. GMI Cloud has also been recognized as an NVIDIA Exemplar Cloud for training on GB300 NVL72 systems, meeting NVIDIA’s threshold of 95% of reference throughput, and its guide to the GB300 NVL72 for rack-scale inference covers the serving side. H100 and H200 capacity is available from $2.60 and $3.20 per GPU-hour. Rates vary by region and term and can change with promotions, so check the GMI Cloud pricing page and confirm the rate for your account in the GMI Cloud console.

Conclusion

Interconnect becomes your bottleneck the moment your communication pattern crosses from the fast fabric to the slower one. NVLink gives GPUs inside a server or rack roughly nine times the per-direction bandwidth of a typical InfiniBand link, so the core rule is simple: Keep tensor-parallel and expert traffic on NVLink, send data-parallel and pipeline traffic over InfiniBand, and overlap communication with compute wherever you can. Watch scaling efficiency and profiler timelines for exposed collectives, verify bus bandwidth with NCCL tests, and tune NCCL for the actual fabric. When a model’s communication no longer fits an eight-GPU server, a rack-scale NVLink domain is the next step. GMI Cloud offers InfiniBand-connected H200 bare metal and GB200 NVL72 rack-scale systems, so you can place that boundary where your workload needs it.