How Should You Benchmark B200 and B300 Against a p95 Inference Latency Target and Decide Whether You Need NVL72?

The figure worth comparing between B200 and B300 is the highest request rate each one sustains while p95 latency stays under your target and errors stay inside budget. Peak throughput at saturation does not show the load you can deliver inside that target. Find the sustained rate per replica with a stepped load ramp on each GPU, convert it into replicas and GPU-hours for your peak traffic, and bring an NVL72 system into the test once no replica inside one eight-GPU node holds p95 at a useful load. B200 and B300 are both hardware options on GMI Cloud's Prime Inference reserved endpoints, on pre-order and coming soon, so the harness is built on H200, available today, and the same ramp becomes the test Blackwell capacity is planned against.

B300 comes out ahead when its sustained rate per GPU outweighs its hourly rate difference against B200. NVL72 joins the comparison when the model or its latency target needs a parallel group larger than eight GPUs, and the final call rests on latency, throughput, and cost measured at the allocation you would reserve.

Pin down the target before the first request

A p95 target needs five companions before a ramp can test it:

• Which latency. Time to first token for streaming chat, end-to-end time for agent steps and tool calls that wait on the whole response. When both matter, set two targets and pass both.
• The measurement window. Compute p95 per one-minute window during steady state, assigning each request to the window in which it was sent. Give every request one client timeout, count expiry as failure, and close a window only after all its requests finish or time out. A single p95 over a long run lets a calm stretch average away a bad minute.
• The request mix. Input and output lengths sampled from production logs, with the longest inputs kept at their real frequency rather than trimmed as outliers.
• The arrival pattern. Requests arriving at a set rate with random spacing, plus a burst profile taken from your own traffic.
• The error budget and quality gate. Timeouts, 5xx responses, and truncated outputs count as failures. Any precision change has to pass your evaluation set before its latency counts.

Step the load until p95 breaks

Drive the endpoint with open-loop arrivals. A closed-loop client waits for each response before sending the next, so when the server slows, offered load drops with it and p95 looks better than users would see. vLLM's benchmark client, SGLang's serving benchmark, and NVIDIA AIPerf all offer a request-rate mode; keep their concurrency caps well above the expected in-flight count, since a capped client slides back into closed-loop behavior.

1. Warm up. Deploy, send traffic for several minutes, and discard those results.
2. Start low. Begin well under your estimate of one replica's capacity and raise the rate in equal steps, each about a tenth of that estimate.
3. Hold each step. Run five to ten minutes per step so several measurement windows land at a stable rate.
4. Record per step. Offered rate, actual send rate, achieved rate, p50, p95, and p99 for each latency you set a target on, error rate, server queue depth, and KV cache usage.
5. Find the break. A step qualifies when every window passes and the achieved rate keeps pace with the offered rate, with no backlog building across the step. End the ramp once two consecutive windows miss. Rerun the last qualifying step; its rate is the qualified rate for one replica.
6. Run a burst check. For example, hold traffic at 70% of the qualified rate, raise it to 140% for 60 seconds, and record errors, queue depth, and how long p95 takes to return under target. Use the size and length of your own bursts, and repeat at a few headroom levels to find one that keeps recovery acceptable.

The ramp also suggests why p95 broke:

What the ramp shows	Where to look first
Achieved rate falls below offered rate	Server saturation and queueing, once the actual send rate and error counts rule out a client-side cause
p95 TTFT rises while inter-token latency stays flat	Queueing or prefill
Inter-token latency rises step by step	Decode batches growing past what the target allows, or prefills interrupting decode; batch size and prefill metrics tell which
KV cache usage near full, with preempted requests	Cache pressure; more memory per GPU, as on B300, is one fix to test alongside cache and batch settings
Errors appear before latency breaks	Error codes, timeout records, and server logs, which separate client timeouts, rejected requests, and server faults

Give B200 and B300 the same test, not the same configuration

Keep the checkpoint, request mix, arrival pattern, quantization recipe (weights, activations, and KV cache), and engine version identical on both GPUs. Then tune each GPU to its own best layout. B300 carries more HBM per GPU, so it may fit a replica in fewer GPUs or hold more concurrent sequences at the same tensor-parallel width. Forcing both onto one layout hides the very difference you are paying for.

Compare the results per GPU: qualified requests per second divided by GPUs per replica. Run an FP8 pair first, then an FP4 pair as a separate comparison that must pass the quality gate. Report the long-input slice separately, where extra memory usually shows first.

Convert qualified throughput into replicas and cost

Three formulas turn ramp results into a deployment plan:

• Replicas = peak arrival rate _ (1 + headroom) ÷ qualified rate per replica, rounded up
• GPUs = replicas _ GPUs per replica
• Monthly GPU cost = GPUs _ billed hours _ the per-GPU-hour rate in your quote

The replica formula assumes identical replicas, balanced routing, and no shared bottleneck, so rerun the ramp at the full replica count before acceptance.

A worked example with illustrative inputs, not measurements: assume a peak of 40 requests per second, 25% headroom, a four-GPU B200 replica qualifying at 6 requests per second, and a four-GPU B300 replica qualifying at 9. B200 needs 50 ÷ 6, rounded up to 9 replicas, or 36 GPUs. B300 needs 50 ÷ 9, rounded up to 6 replicas, or 24 GPUs. If both are billed per GPU-hour for the same hours, B300 has the lower GPU cost whenever its rate is below 1.5 times B200's (36 ÷ 24). Rounding to whole replicas, or a per-node quote that bills B200's 36 GPUs as five nodes, can move that threshold noticeably at small scale.

Add every other line item in each quote to the total. Reserved capacity also serves quiet hours, so divide monthly cost by requests served for cost per request, and by requests that met the latency target and quality gate for cost per qualified request. On Prime Inference, part of the headroom can come from burst capacity during spikes rather than permanently reserved replicas, and the split between the two belongs in the reservation terms.

When an NVL72 system enters the decision

In an NVL72 rack, all 72 Blackwell GPUs share one NVLink domain, which NVIDIA's GB300 NVL72 page rates at 130 TB/s of total NVLink bandwidth. On eight-GPU nodes, a parallel group wider than one node sends its traffic over the scale-out network instead, and the per-token collectives of tensor or expert parallelism feel that hop directly in inter-token latency. The ramp shows when that counts:

Ramp result	Decision
A replica on one B200 node holds p95 with headroom	Size replicas on B200; any rack comparison runs the same ramp at your reserved scale
B200 misses p95 only on long inputs or at the concurrency you need, and B300 holds it	Size replicas on B300
Both hold p95 per replica, but peak volume needs many replicas	Add replicas; volume alone rarely calls for a rack
p95 breaks only during bursts and recovers slowly	Add headroom or burst capacity; a faster GPU helps only when the replica's own service rate is the bottleneck
No layout within eight GPUs fits weights plus KV cache at target concurrency, even on B300	Evaluate NVL72
p95 is met only when tensor- or expert-parallel width exceeds eight GPUs, and a replica spread across two nodes shows worse inter-token latency than one inside a node	Evaluate NVL72

When NVL72 is in scope, often for very large mixture-of-experts models with wide expert parallelism or for prefill and decode split across many GPUs, run the same ramp at the allocation size you would reserve and compare qualified requests per second per GPU, and cost, against the node result. Prime Inference's per-model runtimes include multi-node serving on GB200 NVL72, and GB200 NVL72 and GB300 NVL72 are both on pre-order.

What Prime Inference adds to the ramp

On GMI Cloud, the ramp runs on the platform that will serve the result:

• A harness on H200 today. Deploy the model with the replica layout you plan to test from the console, CLI, or API. The Dedicated Endpoint guide shows billing applies only while a deployment is Running, so stop it between ramp sessions.
• A clean measurement path. Run the load generator in the endpoint's region so wide-area jitter stays out of the serving numbers, then use region-pinned endpoints across Asia-Pacific, North America, and Europe to check the round trip users see.
• Engineers on the runtime. Our inference engineers handle per-model kernel, scheduling, and routing optimization on vLLM, TensorRT-LLM, or SGLang, with configurable quantization. Reserved GPUs stay warm with weights loaded, so no ramp step includes a cold start.

The reservation terms can then carry the p95 target and window, the request mix, the ramp method, and the qualified rate per replica the B200 or B300 deployment is expected to reach at delivery.

Take the ramp from H200 to a Blackwell reservation

Deploy your model on H200 from the Deployments page in the GMI Cloud console to build and validate the ramp. Then share the qualified rate, request mix, and latency target with our sales team to plan B200 or B300 capacity in the Blackwell pre-order, along with an NVL72 evaluation if the ramp points there.