H100 vs H200 vs B200: Picking a GPU for Training vs Inference Workloads

Training and inference stress a GPU in different ways, so the best choice between the H100, H200, and B200 depends less on which card is newest and more on which bottleneck your workload hits first. Compute and interconnect decide training speed, while memory capacity and bandwidth decide most inference economics.

• Training rewards compute and interconnect: In MLPerf Training, Blackwell delivered 2× the per-GPU performance of H100 on GPT-3 pretraining and 2.2× on Llama 2 70B LoRA fine-tuning, while the H200 was only about 16% faster than the H100 on the same fine-tuning test.

• Inference rewards memory: The H200 keeps H100 compute but adds 76% more memory (141 GB vs 80 GB) and 43% more bandwidth, which is why NVIDIA reports up to 2× faster LLM inference without any code changes.

• The B200 leads on both, at a higher rate: With up to 180 GB of HBM3e, around 8 TB/s of bandwidth, native FP4, and 1.8 TB/s NVLink, an eight-GPU B200 system delivered about 3.1× the throughput of eight H200s on MLPerf Inference v5.0’s Llama 2 70B Interactive test.

• Judge cost per run and per token, not per hour: On GMI Cloud, on-demand rates start at $2.60 per GPU-hour for H100, $3.20 for H200, and $5.00 for B200 (as of October 2026), so the H200 must beat the H100 by 1.23× and the B200 by 1.92× on your workload to lower unit cost.

• Many teams split the fleet: Fine-tune or pretrain on the fastest GPU they can justify, then serve on the card whose memory fits the model at the lowest cost per token. Getting that split right matters more than any single spec, because inference now accounts for most AI compute.

Picking a GPU used to be a single decision made once per project. In 2026, most AI teams run at least two very different workloads on the same budget: Training or fine-tuning jobs that run for hours or days and then stop, and inference services that run around the clock and scale with users. The H100, H200, and B200 each fit those workloads differently.

The H100 is still the most widely deployed AI accelerator and the cheapest per hour. The H200 is the same Hopper chip with far more memory. The B200 is the first Blackwell GPU for standard eight-GPU servers, with roughly double the per-GPU training performance of Hopper and new low-precision formats that change inference costs. Comparing them on hourly price alone misses most of what matters.

This guide breaks down the H100 vs H200 vs B200 decision by workload. It explains why training and inference stress GPUs differently, which GPU fits pretraining, fine-tuning, and serving at different model sizes, and how to calculate cost per training run and cost per token using current GMI Cloud prices.

Why Training and Inference Stress GPUs Differently

Training Is Bound by Compute and Interconnect

Training runs a forward pass, a backward pass, and an optimizer update for every batch, which keeps the Tensor Cores busy with large matrix multiplications. When a model is split across several GPUs, those GPUs also exchange gradients and activations after every step, so the speed of NVLink and the cluster network directly sets how much of the compute you actually use. That makes Tensor Core throughput, support for FP8 training, and GPU-to-GPU bandwidth the specifications that matter most. Memory still counts, but mainly because more of it lets you use larger batches or less model parallelism.

Inference Is Bound by Memory

LLM inference has two phases. Prefill processes the prompt in parallel and behaves much like training. Decode then generates one token at a time and has to read the model weights from memory at every step, which makes it limited by memory bandwidth rather than compute. On top of that, every active request keeps a KV cache that grows with context length and batch size. The more memory a GPU has, the more concurrent users and longer contexts it can serve before you need another card. For serving, capacity and bandwidth usually decide cost per token more than peak FLOPS do.

The Three GPUs Side by Side

Specifications come from NVIDIA and server vendor data, and prices from the GMI Cloud pricing page as of October 2026. NVIDIA’s HGX B200 specification quotes 1,440 GB per eight-GPU server, or 180 GB per GPU, while GMI Cloud’s console lists its B200 servers with 192 GB per GPU, the physical HBM3e capacity. The B200’s higher power draw, rated at up to 1,000 W per GPU in vendor guides such as Lenovo’s HGX B200 product guide, is part of why it costs more per hour.

Picking a GPU for Training

Pretraining Large Models

For pretraining, the B200 is the clear performance leader. In MLPerf Training v4.1, Blackwell delivered twice the per-GPU performance of H100 on GPT-3 175B pretraining. In MLPerf Training v5.0, NVIDIA reported up to 2.6× higher per-GPU performance than Hopper across all seven benchmarks. NVIDIA also rates an HGX B200 system at 3× faster training than HGX H100 for very large models such as GPT-MoE-1.8T, thanks to the second-generation Transformer Engine, higher FP8 throughput, and fifth-generation NVLink.

The H200 adds little for compute-bound pretraining, since its Tensor Cores are the same as the H100’s. Its extra memory helps when it lets you cut tensor or pipeline parallelism, or fit a larger micro-batch per GPU, but it does not change raw training throughput much. If your pretraining budget is fixed and time to market matters, the B200 usually finishes the same run in about half the time.

Fine-Tuning: Full, LoRA, and QLoRA

Fine-tuning has a wider range of right answers because memory needs vary so much by method. Full fine-tuning with mixed precision and the Adam optimizer needs roughly 16 bytes per parameter for weights, gradients, and optimizer states, before activations. That is about 112 GB for a 7B model and about 1.1 TB for a 70B model, which means full 70B fine-tuning needs optimizer sharding across at least one full eight-GPU server.

Parameter-efficient methods change the math. LoRA keeps the base weights frozen at about 2 bytes per parameter in BF16 and trains small adapter matrices, so a 70B model needs around 140 GB plus activations, which fits on two H100s or two H200s, or on a single B200 with some headroom. QLoRA stores the base model in 4-bit precision, bringing a 70B model down to roughly 35 to 40 GB, which fits on a single H100.

On speed, MLPerf Training v4.1 measured Llama 2 70B LoRA fine-tuning on eight-GPU servers. The H200 system was about 16% faster than the H100, while a single HGX B200 server delivered 2.2× the performance of an HGX H100 server. For fine-tuning, the H100 remains the value choice for smaller models and adapters, and the B200 is the speed choice for large models and frequent retraining.

Cost per Training Run, Not per Hour

Training jobs end, so the number that matters is the cost of a finished run. A simple example shows why. Suppose a LoRA fine-tuning job takes 100 hours on one eight-GPU H100 server at $2.60 per GPU-hour, which costs $2,080. If the same job runs 16% faster on H200, it takes about 86 hours at $3.20 per GPU-hour, or about $2,207. If it runs 2.2× faster on B200, it takes about 45 hours at $5.00 per GPU-hour, or about $1,818.

In this example, which uses October 2026 GMI Cloud rates and the MLPerf speedups above, the H200 costs slightly more per run than the H100 because its training gain is smaller than its 23% price premium. The B200 costs less per run and finishes in less than half the time. Your own numbers will differ, so measure throughput on a short benchmark run before committing to a long job.

Picking a GPU for Inference

Small and Mid-Size Models: H100

Models up to roughly 34B parameters fit comfortably on a single 80 GB H100 at FP8, with room left for KV cache. At that size, memory is rarely the bottleneck, so the H100’s lower hourly rate usually gives the lowest cost per token. It is also the safest choice for mature serving stacks, since every major inference engine has been tuned on Hopper for years.

70B to 400B-Class Models: H200

The H200 is the inference upgrade within the Hopper family. A 70B model at FP8 needs about 70 GB for weights alone, which leaves only about 10 GB on an H100 for KV cache and batching. On an H200, the same model has more than 70 GB of headroom, which translates into larger batches, longer contexts, and more concurrent users per GPU. NVIDIA reports up to 2× faster inference than the H100 on LLMs such as Llama 2, and because the H200 runs the same software stack, the upgrade needs no code changes. Eight H200s also hold about 1.1 TB of memory, enough for a 671B-parameter MoE model at FP8 within one server.

High-Throughput and Low-Latency Serving: B200

The B200 combines more memory, roughly 1.7× the H200’s bandwidth, and native FP4 support, which can raise throughput substantially over FP8 when output quality holds up. In MLPerf Inference v5.0, an eight-GPU B200 system delivered about 3.1× the throughput of eight H200s on the Llama 2 70B Interactive benchmark, which measures serving under tight latency targets. NVIDIA’s headline figure of 15× faster real-time inference than H100 applies to very large MoE models such as GPT-MoE-1.8T running at FP4, so treat it as an upper bound rather than a typical gain.

The B200 makes the most sense for interactive products with strict latency targets, for large models serving many concurrent users, and for teams ready to validate FP4 quantization. For a small model with modest traffic, its extra capacity often goes unused.

Cost per Token at October 2026 Prices

For inference, convert hourly price into cost per token using measured throughput. GMI Cloud’s price-to-performance formula does exactly that.

cost per 1M tokens = (GPU price per hour ÷ tokens per second) × (1,000,000 ÷ 3,600)

The short Python helper below applies the formula, and the same approach to training runs, using GMI Cloud’s October 2026 on-demand rates. The throughput values are placeholders, so replace them with numbers measured on your own model, precision, and batch size.

# GMI Cloud on-demand "from" rates per GPU-hour, as of October 2026
PRICE_PER_GPU_HOUR = {"H100": 2.60, "H200": 3.20, "B200": 5.00}

def cost_per_million_tokens(gpu: str, tokens_per_second: float) -> float:
    """Serving cost per 1M tokens for one fully utilized GPU."""
    return PRICE_PER_GPU_HOUR[gpu] / tokens_per_second 1_000_000 / 3_600

def cost_per_training_run(gpu: str, num_gpus: int, hours: float) -> float:
    """Total cost of a training or fine-tuning job."""
    return PRICE_PER_GPU_HOUR[gpu] num_gpus * hours

# Placeholder throughput per GPU for the same model; replace with your benchmarks
for gpu, tps in [("H100", 1_000), ("H200", 1_500), ("B200", 3_000)]:
    print(gpu, round(cost_per_million_tokens(gpu, tps), 3))

With those placeholder values, the script returns about $0.72 per million tokens on H100, $0.59 on H200, and $0.46 on B200. The rule behind the result is simple. At October 2026 rates, the H200 lowers cost per token once it serves at least 1.23× the H100’s throughput, and the B200 once it serves at least 1.92×. Memory-bound serving of large models usually clears both bars, while small models often do not.

If you would rather not manage GPUs for serving at all, GMI Cloud also runs models as per-token serverless APIs and as dedicated endpoints. In the GMI Cloud console, a dedicated endpoint runs on one H200 GPU at $4.50 per hour per replica as of October 2026, which includes the managed inference layer on top of the raw GPU.

Training on One GPU, Serving on Another

The training vs inference split often leads to a mixed fleet. A common pattern is to fine-tune or pretrain on B200 servers, where faster runs cut both cost and iteration time, and then serve the result on H100s or H200s, depending on whether the model fits comfortably in 80 GB. Models trained in BF16 deploy cleanly to Hopper at FP8, while FP4 serving requires Blackwell, so decide on your serving precision before you lock in the inference fleet.

The reverse also happens. Teams with steady, high-volume inference traffic increasingly put serving on B200s to cut cost per token, while running smaller fine-tuning jobs on H100s that would otherwise sit idle. The right mix depends on how many hours each workload runs per month, which is why it helps to model training runs and serving traffic separately.

A Quick Decision Guide

Running H100, H200, and B200 on GMI Cloud

GMI Cloud runs all three GPUs on dedicated NVIDIA infrastructure inside GMI-operated data centers built on NVIDIA reference architectures. As of October 2026, H100 and H200 capacity is available now, from $2.60 and $3.20 per GPU-hour, while the B200 is listed from $5.00 per GPU-hour with limited availability. Reserved capacity and sustained deployments lower unit costs through commitment-based pricing, and promotions can change rates, so confirm the price that applies to you in the GMI Cloud console before you plan a large run.

For large training jobs, provider configuration matters as much as the GPU model. GMI Cloud has been recognized as an NVIDIA Exemplar Cloud for training on GB300 NVL72 systems, which means its clusters met NVIDIA’s performance thresholds against reference architecture. For serving, the same platform offers serverless APIs and dedicated endpoints through its inference engine, so you can move a fine-tuned model from training to production without changing providers.

Conclusion

The H100 vs H200 vs B200 decision gets easier once you separate training from inference. For training, compute and interconnect rule, so the B200 usually wins on both speed and cost per run, while the H100 stays the value choice for smaller fine-tuning jobs and the H200 helps mainly when memory limits your parallelism. For inference, memory rules, so the H100 fits small and mid-size models, the H200 is the natural home for 70B to 400B-class models, and the B200 leads on high-concurrency, low-latency, and FP4 serving. Measure throughput on your own workload, divide by the current hourly rate, and pick the GPU with the lowest cost per run or per token. GMI Cloud offers all three, along with serverless and dedicated inference, so you can match each workload to the right card.