What are the real-world tokens-per-second benchmarks for serving large language models on a B200, and for which model sizes?

Teams evaluating B200 usually want a clean table: this model, this many tokens per second. Published numbers exist, but they rarely transfer, because tokens per second on a B200 is set by the model size, precision, batch size, and serving stack, not by the chip alone. Real-world B200 tokens-per-second varies widely by model size and serving configuration, so a single published benchmark is only valid for the exact setup it was measured on, and the number that matters is the one you measure on your own model and traffic. This guide shows how model size shapes B200 throughput, why published figures drift, and how to run a benchmark you can actually trust.

Why a single tokens-per-second number rarely transfers

A tokens-per-second figure is only meaningful with its full configuration attached. The same B200 will report very different throughput depending on how the model is served, so a bare number copied from a blog post is a starting hypothesis, not a spec.

Four inputs move the result enough to change it several-fold. A B200 tokens-per-second benchmark is only comparable when the model, quantization or precision, batch size, and sequence length all match your workload, because changing any one of them can move throughput by a large margin. Precision format alone, for example, can shift tokens per second dramatically on the same hardware. This is why credible benchmarks always publish their configuration, and why a number without one should not drive a capacity decision.

How model size shapes B200 throughput

Model size is the biggest single driver of the pattern, even though it does not give you a fixed number. The relationship is directional and reliable:

• Small models (roughly 7B to 13B): Highest tokens per second, often fit comfortably in B200 memory, batch well, and serve many concurrent requests. This is where raw throughput numbers look largest.
• Mid-size models (roughly 30B to 70B): Lower tokens per second than small models, but B200's larger high-bandwidth memory helps them batch and serve without the spillover that hurts throughput on smaller-memory chips.
• Large models (100B and up, or MoE): Lowest per-token throughput and most sensitive to memory bandwidth and interconnect. This is where B200's generational advantage over older chips tends to show up most.

The takeaway is a shape, not a table: throughput falls as model size rises, but B200's advantage over previous chips often grows with size, because larger models are more memory-bound. So the right question is not "what is B200's tokens per second" but "what is it for my model at my batch size and precision."

Reading a B200 benchmark before you trust it

Use the checklist below to judge whether a published B200 number applies to you. At least one column should be quantifiable before you rely on it.

Benchmark attribute	Why it matters	Confirm before trusting
Model and size	Throughput scales inversely with size	Same model family and parameter count
Precision / quantization	Can shift tokens/sec severalfold	Same format as your deployment
Batch size and concurrency	Throughput compounds at high batch	Matches your production load
Input and output length	Long context lowers tokens/sec	Same sequence profile as your traffic

If a published B200 figure omits any of these, treat it as directional. A number measured at batch 1 on a 7B model in one precision tells you almost nothing about serving a 70B model at high concurrency in another. Match the configuration or measure your own.

Measuring real B200 tokens-per-second on GMI

Since the only trustworthy number is your own, the practical step is using our platform, which lets you run the benchmark on real B200 hardware. We are an AI-native GPU and inference cloud that publishes dedicated NVIDIA GPU list pricing and carries B200, so you can measure tokens per second for your exact model instead of borrowing someone else's figure.

We currently list B200 at from $4.00 per GPU-hour under Limited Availability, so you can serve your own model on B200 and record tokens per second at your production batch size, precision, and sequence length rather than relying on a published benchmark measured on a different setup. Verify the current rate and availability on our GPU infrastructure (https://www.gmicloud.ai/en/gpus) and pricing (https://www.gmicloud.ai/en/pricing), since Blackwell pricing and stock move quickly. Run the benchmark at the concurrency you actually expect, not a synthetic peak, because batch size and sequence length decide the real throughput. When the goal is sustained production serving rather than a one-off test, our Prime Inference (https://www.gmicloud.ai/en/models/prime-inference) provides reserved GPU capacity with single-tenant isolation and warm, weights-preloaded serving, so your measured tokens per second reflect steady traffic instead of cold-start overhead. Start a benchmark run in our console (https://console.gmicloud.ai).

Benchmark your model, not a headline number

If you plan B200 capacity around a tokens-per-second figure you found rather than one you measured, you risk sizing for a model, precision, or batch size that is not yours. The reliable path is to treat published numbers as directional, hold model size, precision, batch, and sequence length constant with your production setup, and run the benchmark on real B200 hardware. Throughput falls as models grow and B200's edge tends to rise with size, but the only number worth a capacity plan is the one your own workload produces.