H100 vs H200 vs B200: Choosing the Right GPU for Your Inference Workload

The instinct when a new GPU generation arrives is that newer is better and the upgrade decision is about budget. For inference workloads, that instinct produces expensive mistakes in both directions. A team that stays on H100 for a 70B model at FP16 is paying for two GPUs and tensor parallelism overhead where one H200 would do the job. A team that moves to B200 without adopting FP4 quantization is paying roughly 19 percent more per token than they would on H100, because the B200's price premium only pays off when its native FP4 hardware is actually used. The right GPU depends on the model size, the quantization you can accept, and whether your workload is memory-bandwidth-bound or compute-bound.

• The three GPUs differ most in memory capacity and bandwidth, which is what determines LLM inference throughput. H100 SXM: 80 GB HBM3 at 3.35 TB/s. H200 SXM: 141 GB HBM3e at 4.8 TB/s. B200 SXM: 192 GB HBM3e at 8 TB/s.
• The B200's economic case rests entirely on FP4. At FP4, B200 inference costs roughly 42 percent less per token than H100. At FP8, the same B200 costs roughly 19 percent more per token than H100. If your model or quality requirements cannot accept FP4, the B200 upgrade does not pay for itself on inference.
• GMI Prime Inference offers all three with per-model runtime tuning configured per GPU class: H100 at $2.00/hr, H200 at $2.60/hr, and B200 for workloads where FP4 throughput or 192 GB capacity is the requirement.
• The H200 is the single most practical upgrade for 70B class models. Llama 3.3 70B at FP8 needs roughly 70 GB of weights. On H100 (80 GB), that leaves 10 GB for KV cache, which caps concurrency severely. On H200 (141 GB), it leaves 71 GB, supporting meaningfully higher batch sizes and longer contexts on a single GPU.
• Software compatibility is not a migration obstacle. All three use the same CUDA toolchain (CUDA 12.x, cuDNN 9.x). Code running on H100 runs on H200 and B200 without modification. The migration work is in quantization configuration and serving parameter tuning, not in rewriting the application.
• Match the GPU to the memory math, not to the generation number. Compute the weight footprint at your target precision, add realistic KV cache for your context length distribution and target concurrency, and pick the smallest GPU that fits with headroom.

The Specifications That Actually Matter for Inference

LLM inference decode is memory-bandwidth-bound, not compute-bound. Generating each token requires reading the model weights from VRAM once. The rate at which weights can move from memory into compute units determines token generation speed. This makes memory bandwidth the primary performance metric for inference and memory capacity the primary constraint on what you can serve.

Three numbers drive the decision.

Memory capacity determines what fits.

A model's weight footprint at a given precision either fits in VRAM with room for KV cache, or it does not. If it does not, you need multiple GPUs and tensor parallelism, which adds inter-GPU communication overhead to every forward pass.

Memory bandwidth determines throughput.

The H200's 4.8 TB/s is 1.43 times the H100's 3.35 TB/s, which translates to roughly 1.4 times higher token generation throughput on the same model at the same precision. The B200's 8 TB/s is 2.4 times the H100's.

Native FP4 changes the memory math entirely.

A 70B model requires approximately 140 GB at FP16, 70 GB at FP8, and 35 GB at FP4. Only the B200 has hardware FP4 support. On B200, a 70B model at FP4 occupies 35 GB of the 192 GB, leaving 157 GB for KV cache and batching, or allowing multiple model instances on a single GPU.

Model Size to GPU Mapping

The practical decision starts with the weight footprint of the model you intend to serve, at the precision your quality requirements allow.

The pattern that emerges: for models in the 30B to 70B range, which covers the majority of production inference workloads, the choice between H100 and H200 is about KV cache headroom rather than whether the model fits at all. For models above 100B, the H200's capacity advantage becomes a hard requirement rather than a convenience.

Where the H100 Is Still the Right Choice

The H100 remains the correct default for a significant share of production inference workloads.

Models up to 32B at FP8.

Qwen3-32B at FP8 occupies 32 GB, leaving 48 GB of KV cache headroom on an H100. This supports high concurrency at typical context lengths without needing the H200's additional capacity. At $2.00/hr versus $2.60/hr, the H100 delivers the same workload at 23 percent lower cost.

Workloads where the model fits comfortably and bandwidth is not the bottleneck.

For applications with modest concurrency where the GPU is not saturated, the H200's bandwidth advantage does not translate into proportional throughput gains, because the bottleneck is request volume rather than memory bandwidth.

Cost-sensitive workloads with acceptable quality at smaller models.

‍Routing a task to a 32B model on an H100 rather than a 70B model on an H200 is often the larger cost lever than the GPU choice itself.

Broad availability and mature tooling.

H100 has the widest cloud availability, the most mature serving framework support, and the largest body of production deployment experience. For teams that value operational predictability, this matters.

Where the H200 Is the Clear Upgrade

The H200's advantage is specific and concentrated in one range: models between 70B and 141B at FP8.

Single-GPU serving for 70B models.

Llama 3.3 70B at FP8 needs roughly 70 GB. On H100 with 80 GB, this leaves 10 GB for KV cache, which at approximately 0.32 MB per token supports roughly 31,000 tokens of total KV cache across all concurrent requests. At a 4,000-token average context, that is 7 to 8 concurrent sessions. On H200 with 141 GB, the same model leaves 71 GB of KV cache headroom, supporting roughly 55 concurrent sessions at the same context length.

The alternative on H100 is two GPUs in tensor parallel configuration, which costs $4.00/hr and adds inter-GPU communication overhead on every forward pass. A single H200 at $2.60/hr eliminates both the cost premium and the coordination overhead.

Long-context workloads.

‍KV cache consumption scales linearly with context length. An application serving 32K to 128K contexts needs substantially more KV cache headroom than one serving 4K contexts. The H200's additional 61 GB over the H100 is entirely available for KV cache when serving a model that fits on both.

Throughput-bound production serving.

The 1.43 times bandwidth advantage produces roughly 1.4 times higher token generation throughput. For a GPU running at high utilization, this is a direct throughput increase that improves effective cost per token despite the higher hourly rate: at $2.60/hr for 1.4 times the throughput of a $2.00/hr H100, the effective cost per token is approximately 7 percent lower on H200.

Models between 80 GB and 141 GB at FP8.

Mistral Large 2 at FP8 (123 GB) does not fit on an H100 at all. On H200 it fits with 18 GB of headroom, enabling single-GPU serving where H100 requires two GPUs.

Where the B200 Wins, and Where It Does Not

The B200's case is more conditional than the H200's, and the condition is FP4.

The FP4 economics.

Independent pricing analysis puts the B200 at roughly $9.36/hr on-demand versus $3.94/hr for H100 SXM5 at comparable providers as of mid-2026. At those rates:

FP4 inference on B200 is approximately 42 percent cheaper per token than H100, because the throughput gain from FP4 plus the bandwidth advantage outpaces the price premium by a wide margin.

FP8 inference on B200 is approximately 19 percent more expensive per token than H100. The B200's FP8 throughput advantage does not compensate for its hourly rate premium.

This is the decision criterion most GPU comparison articles miss. The B200 is not a general upgrade over H100 for inference. It is specifically an FP4 machine, and its economics depend on running FP4.

When FP4 is acceptable.

FP4 quantization introduces quality degradation relative to FP8. The magnitude depends on the model and the task. For conversational applications, summarization, classification, and general instruction following, the degradation is often imperceptible. For precise mathematical reasoning, complex code generation, and tasks where small errors compound, FP4 quality loss is more likely to be measurable.

Models trained with quantization-aware training at FP4 or MXFP4 (Kimi K3 uses MXFP4 from the SFT stage) preserve quality significantly better than models post-training quantized to FP4. For these models, FP4 on B200 is the natural deployment target.

Where the B200 wins regardless of precision:

Models above 141 GB that would otherwise need multiple GPUs.

Llama 3.3 70B at full FP16 (140 GB) fits on a single B200 with 52 GB to spare. On H200, it fits with 1 GB of headroom, which is impractical for production. On H100 it requires two GPUs.

Multi-model serving on a single GPU.

‍At 192 GB, a B200 can hold multiple model instances simultaneously. Four FP4 copies of a 70B model, or a 70B model plus several smaller specialized models, fit on one GPU. For multi-model applications, this consolidation reduces the number of GPUs required.

Multi-GPU training and inference where NVLink bandwidth matters.

NVLink doubles from 900 GB/s on H100 and H200 to 1.8 TB/s on B200. For MoE models with heavy expert routing traffic across GPUs, and for tensor-parallel serving of very large models, this bandwidth improvement reduces the inter-GPU communication overhead that limits multi-GPU scaling efficiency.

Confidential computing requirements.

Blackwell introduces Trusted Execution Environment I/O, allowing compute and model weights to be encrypted with minimal throughput penalty. For regulated industries that cannot run inference on unencrypted data, this is a capability that Hopper generation GPUs do not have.

The Decision Framework

Four questions determine the right GPU for a given inference workload.

Question 1: What is the weight footprint at your target precision?

Compute it: parameters × bytes per parameter. FP16 is 2 bytes, FP8 is 1 byte, FP4 is 0.5 bytes. A 70B model at FP8 is 70 GB.

Question 2: How much KV cache do you need?

KV cache per token varies by model architecture. For Llama 3.3 70B with grouped query attention, it is approximately 0.32 MB per token at FP16, or 0.16 MB at FP8 KV cache quantization. Multiply by your average context length and target concurrent sessions.

At 4,000 average context and 30 concurrent sessions: 4,000 × 30 × 0.32 MB = 38 GB of KV cache required.

Question 3: Does the total fit on one GPU with headroom?

Weight footprint plus KV cache plus roughly 10 percent overhead for activations and framework memory. If the total fits on the smaller GPU, use it. If it does not, either move to the larger GPU or reduce precision.

Continuing the example: 70 GB weights plus 38 GB KV cache plus 11 GB overhead equals 119 GB. This does not fit on H100 (80 GB). It fits on H200 (141 GB) with 22 GB spare. H200 is the answer.

Question 4: Can your quality requirements accept FP4?

If yes, and the workload is large or high-volume enough that the 42 percent per-token cost advantage is meaningful, evaluate B200. If no, B200's FP8 economics are worse than H100 and the choice is between H100 and H200 based on the memory math from questions 1 through 3.

Migration Between GPU Generations

Moving between H100, H200, and B200 is operationally simpler than most hardware migrations because the software stack is shared.

CUDA compatibility is complete.

All three use CUDA 12.x and cuDNN 9.x. Code that runs on H100 runs on H200 and B200 without modification. There is no application rewrite, no framework version change, no driver-level incompatibility.

The actual migration work is quantization and serving configuration.

Moving H100 to H200: typically no changes required. The additional VRAM is available immediately for larger KV cache allocation or higher batch sizes. Tuning the serving framework's memory allocation parameters (--gpu-memory-utilization in vLLM, KV cache block configuration) captures the benefit.

Moving H200 to B200: to realize the economic benefit, the model must be quantized to FP4 and the serving framework configured to use native FP4 kernels. This requires a quantization pass (or obtaining an FP4-quantized variant of the model) and quality validation at the new precision. Without this work, B200 delivers FP8 inference at a higher hourly rate than H100.

Quality validation at the new precision is the real gate.

Before committing production traffic to FP4 on B200, run the evaluation set from your model selection process at FP4 and compare against the FP8 baseline. If quality holds, the 42 percent cost advantage is available. If it does not, the workload stays on H100 or H200.

GPU Selection Within GMI Prime Inference

GMI Prime Inference provides all three GPU classes with per-model runtime tuning configured per class, which means the serving stack (vLLM, TensorRT-LLM, or SGLang) is tuned for each model on each hardware target rather than applying generic settings.

H100 at $2.00/hr for models up to 70B at FP8 where KV cache requirements fit within the 10 GB headroom, and for models up to 32B where the headroom is comfortable. This is the price-performance default for the majority of production inference workloads.

H200 at $2.60/hr for 70B class models needing meaningful KV cache headroom, for models between 80 and 141 GB at FP8, and for long-context workloads where KV cache dominates memory consumption. The 1.4 times bandwidth advantage makes effective cost per token approximately 7 percent lower than H100 at high utilization despite the higher hourly rate.

B200 for FP4-quantized workloads where the 42 percent per-token cost advantage applies, for models requiring more than 141 GB, for multi-model consolidation on a single GPU, and for MoE workloads where NVLink 5's doubled bandwidth reduces expert routing overhead.

Because Prime Inference reserves dedicated single-tenant capacity with pre-loaded model weights, the GPU choice determines the sustained performance characteristics of the endpoint rather than a shared-pool average. The per-model runtime tuning includes KV cache sizing calibrated to the actual context length distribution of the workload, quantization precision selected for the quality and throughput target, and kernel selection appropriate to the GPU generation.

For teams that want to benchmark across GPU classes before committing, GMI Cloud's on-demand infrastructure provides hourly billing with no minimum commitment, making it practical to measure the same workload on H100, H200, and B200 and compare effective cost per token directly.

Conclusion

GPU selection for inference is memory arithmetic, not generation-number arithmetic. Compute the weight footprint at your target precision, add realistic KV cache for your context distribution and concurrency target, and pick the smallest GPU that fits with headroom.

For models up to 32B at FP8, H100 at $2.00/hr is the price-performance default. For 70B class models needing real KV cache headroom, or models between 80 and 141 GB, H200 at $2.60/hr eliminates the two-GPU tensor parallel configuration that H100 would require and delivers roughly 7 percent lower effective cost per token at high utilization.

For B200, the decision hinges on FP4. At FP4, B200 inference costs approximately 42 percent less per token than H100. At FP8, it costs approximately 19 percent more. The B200 is an FP4 machine, and treating it as a general upgrade over Hopper without adopting FP4 produces a more expensive deployment than the hardware it replaced.

Deploy on the right GPU with GMI Prime Inference

FAQs

When is the H200 worth the price premium over the H100 for inference? The H200 is worth it in three situations. First, when serving a 70B class model at FP8: the model needs roughly 70 GB, leaving only 10 GB of KV cache headroom on H100 (supporting 7 to 8 concurrent sessions at 4K context) versus 71 GB on H200 (supporting roughly 55 sessions). Second, when the model is between 80 and 141 GB at FP8, where H100 requires two GPUs and H200 needs one. Third, for high-utilization throughput-bound serving, where the 1.43 times bandwidth advantage produces roughly 7 percent lower effective cost per token despite the higher hourly rate.

Why is the B200 more expensive per token than the H100 at FP8? The B200's hourly rate is roughly 2.4 times the H100's at comparable providers, while its FP8 throughput advantage does not scale proportionally to that premium. The result is approximately 19 percent higher cost per token at FP8. The B200's economic advantage comes specifically from native FP4 hardware, which the H100 and H200 do not have. At FP4, throughput roughly doubles relative to FP8 on the same GPU, and the per-token cost drops to approximately 42 percent below H100. Deploying B200 without FP4 forfeits the reason to use it.

How do I calculate whether a model fits on a specific GPU? Three components: weight footprint (parameters × bytes per parameter, where FP16 is 2 bytes, FP8 is 1, FP4 is 0.5), KV cache (per-token cache size × average context length × target concurrent sessions), and roughly 10 percent overhead for activations and framework memory. For Llama 3.3 70B at FP8 serving 30 concurrent sessions at 4,000 average context: 70 GB weights plus 38 GB KV cache plus 11 GB overhead equals 119 GB. This exceeds H100's 80 GB and fits H200's 141 GB with 22 GB spare.

Is migrating between H100, H200, and B200 difficult? The software migration is minimal because all three use the same CUDA 12.x and cuDNN 9.x toolchain. Code running on H100 runs on H200 and B200 without modification. The real work is elsewhere: moving H100 to H200 typically requires only tuning the serving framework's memory allocation to use the additional VRAM. Moving to B200 requires quantizing the model to FP4 and validating quality at the new precision, without which the B200 delivers FP8 inference at a higher hourly rate than H100.

Which GPU is right for MoE models like Kimi K3, GLM-5.1, or Qwen3-235B? MoE models require the full weight matrix in VRAM regardless of the active parameter count per token, which pushes them into multi-GPU territory. Qwen3-235B at FP8 (235 GB) needs 2x H200 or 2x B200. GLM-5.1 at FP8 (approximately 800 GB) needs 8x H200 minimum. Kimi K3 at MXFP4 (1.56 TB) requires a 64-plus accelerator cluster. For all multi-GPU MoE serving, NVLink bandwidth matters more than for dense models because expert routing creates all-to-all inter-GPU traffic on every forward pass. The B200's 1.8 TB/s NVLink versus 900 GB/s on H100 and H200 reduces that routing overhead meaningfully at scale.