For serving a 70B-parameter model, how many tokens per second can a single B200 push versus an H200?

Teams sizing a single-GPU deployment for a 70B model usually want a straight tokens-per-second comparison between B200 and H200. The honest answer starts one step earlier, because for a model this size the first question is whether it fits on one GPU at all, and that depends on precision. For a 70B model, single-GPU tokens per second on B200 versus H200 is decided first by whether the model fits in each GPU's memory at your chosen precision, and only then by raw throughput, so the gap can be large when memory capacity is the constraint and modest when both chips fit the model comfortably. This guide explains how memory fit shapes the comparison, why precision changes the answer, and how to measure the real number for your deployment.

Memory fit decides the comparison before speed does

A 70B model's memory footprint is dominated by its weights plus the key-value cache for active requests. At FP16, 70B weights alone are on the order of 140GB, which is why precision and memory capacity, not clock speed, set the terms of a single-GPU comparison at this size.

The two chips sit on different sides of that line at high precision. A single B200 carries more high-bandwidth memory than a single H200, so a 70B model that is tight or does not fit on one H200 at higher precision can fit more comfortably on one B200, and where H200 must offload or quantize harder to fit, B200's headroom translates into a larger throughput advantage. When both chips hold the model comfortably, for example at a lower quantization, the comparison narrows to raw memory bandwidth and compute, and the gap shrinks. So the same B200-versus-H200 question has a different answer depending on the precision you serve at, which is why a single tokens-per-second figure without a precision attached is incomplete.

Why precision changes the answer

The quantization you choose moves both whether the model fits and how fast it runs. Three cases cover most 70B single-GPU deployments:

• FP16 or BF16: Largest footprint. A 70B model is tight or over budget on one H200 and fits with more room on one B200, so this is where B200's capacity advantage is widest.
• FP8: Roughly halves the weight footprint, letting 70B fit on both chips more easily, which narrows the gap toward a raw-bandwidth comparison.
• FP4 or lower: Smallest footprint and highest throughput, but only worth it if calibrated quantization holds output quality for your task.

The takeaway is that "how many tokens per second" is inseparable from "at what precision, with how much KV cache, at what concurrency." A 70B benchmark at batch 1 in FP8 and one at high concurrency in FP16 describe different worlds, and only the one matching your serving setup predicts your result.

Reading the single-GPU 70B choice

Use the frame below to judge which chip fits your 70B deployment before you benchmark. The throughput column is directional, not a quoted number.

Serving condition	Single H200	Single B200
70B at FP16/BF16	Tight on memory; may not fit with KV cache	Fits with more headroom
70B at FP8	Fits more easily	Fits comfortably, higher bandwidth
High concurrency (large KV cache)	Memory pressure limits batch	More room for larger batches
Quality-critical, no quantization	Constrained by capacity	Capacity advantage matters most

The pattern is consistent: B200's edge on a single-GPU 70B model grows with precision and concurrency, because both raise the memory footprint that its larger capacity relieves. Where you can quantize aggressively and keep the model small, the two chips converge and the decision shifts to bandwidth and price.

Measuring single-GPU 70B throughput on GMI

Since the real number depends on your precision, KV cache, and concurrency, the practical step is measuring it on both chips rather than assuming a multiple. We are an AI-native GPU and inference cloud that publishes dedicated NVIDIA GPU list pricing and carries both H200 and B200, so you can serve your 70B model on each and record tokens per second directly.

We currently list H200 at from $2.60 per GPU-hour and B200 at from $4.00 per GPU-hour under Limited Availability, so you can deploy your 70B model on a single H200 and a single B200 at your production precision and measure the real tokens-per-second gap rather than relying on a benchmark run at a different setup. Verify current rates and B200 availability on our GPU infrastructure (https://www.gmicloud.ai/en/gpus) and pricing (https://www.gmicloud.ai/en/pricing), since Blackwell pricing and stock move quickly. Benchmark at the precision, KV cache size, and concurrency you will run in production, because those decide whether the model fits and how fast it serves. When the workload is sustained 70B serving, our Prime Inference (https://www.gmicloud.ai/en/models/prime-inference) provides reserved GPU capacity with single-tenant isolation and warm, weights-preloaded serving, so your measured throughput reflects steady traffic rather than cold-start overhead. Start a comparison in our console (https://console.gmicloud.ai).

Check the fit first, then the tokens per second

If you compare B200 and H200 on a 70B model by tokens per second alone, you can miss that the two chips may not even hold the model at the same precision, which is what actually drives the gap. Ask the fit question first: decide the precision and KV cache your deployment needs, confirm whether 70B fits on each single GPU under that setup, then measure tokens per second on both. B200's larger memory gives it the widest advantage exactly when a 70B model strains an H200, and the two converge when aggressive quantization lets both hold the model, so the number that matters is the one you measure at your real serving precision.