How to Size and Tune a 70B Model Deployment on B200 or B300

Size a 70B deployment from weight bytes, KV-cache bytes per token × context × concurrency, and runtime reserve. GMI Cloud Prime Inference combines dedicated B200/B300 serving with model-specific sizing and runtime tuning.

The worked example covers memory-limited concurrency, the throughput crossover, monthly cost per session, and workload validation.

Who should size and tune a 70B deployment on B200 or B300?

What that covers for a 70B deployment, all from the Prime Inference page:

• The hardware you are choosing between. The page lists B200 at 192 GB HBM3e and B300 at 288 GB HBM3e, and GMI Cloud states it benchmarked "tuned per-model runtimes on B200, B300, and GB200 NVL72."
• The engine work. "vLLM, TensorRT-LLM, and SGLang pre-tuned per GPU class. Quantization configurable. Multi-GPU orchestration handled." Weight precision is the lever that moves every number in this article, and on Prime Inference it is a configuration choice rather than a research project on your side.
• The shape you pick. Step 2 of the deployment flow is "GPU type, GPU count per replica, replica count, and target region." Those four fields are the output of the sizing math below.
• Weights stay resident. "Reserved GPUs stay warm with weights pre-loaded," which is what makes a fixed GPU count behave predictably under load.
• Your checkpoint. "Any open-source, fine-tuned, or proprietary weights. Load from Hugging Face, S3, or your own storage."

Tuning here means the serving runtime, not retraining. A 70B checkpoint you have already fine-tuned loads onto the same stack; the Prime Inference tier lists "Custom and fine-tuned model deployment" among its capabilities.

The three numbers that decide a 70B deployment

Everything on the GPU is one of three things: weights, KV cache, or working memory. Compute them in that order.

Number 1: weight bytes. Multiply parameters by bytes per parameter. BF16 is 2, FP8 is 1, and NVFP4 stores "one 4-bit value plus minor overhead of one FP8 scale per 16 values (4.5 bits per value)" per NVIDIA's NVFP4 introduction, which is 0.5625 bytes per parameter.

Number 2: KV cache bytes per token, times concurrency, times context. The per-token figure comes from the model's own architecture:

KV bytes per token = 2 x layers x KV heads x head dimension x bytes per element

For the 70B configuration in Meta's Llama 3 paper (80 layers, 64 attention heads, 8 key/value heads, model dimension 8,192, so a head dimension of 128, per Table 3 of The Llama 3 Herd of Models), an FP8 KV cache costs 2 x 80 x 8 x 128 x 1 = 163,840 bytes per token. That is 0.164 MB per token, or:

One sequence at 128K with a BF16 KV cache costs 42.9 GB, which is 54% of an entire 80 GB H100 spent on a single user. Halving that to FP8 is the single largest lever in the whole exercise, which makes KV precision a sizing decision rather than a tuning afterthought. Check your own model's config file for layer count and KV head count before reusing these figures; models with compressed or hybrid attention come out lower.

Number 3: the runtime reserve. Activations, CUDA graphs, engine workspace, and allocator fragmentation all sit outside the two numbers above. Every table in this article holds back 15% of board memory for them, which is the planning figure this guide uses throughout. Your engine, batch shape, and parallelism setting move the real number, so confirm it with GMI Cloud before you lock a GPU count.

How many concurrent sequences fit on one B200 or B300?

One B200 holds 61 concurrent 8K sequences at FP8 weights, and one B300 holds 114. Those numbers come from subtracting numbers 1 and 3 from board memory, dividing what is left by 0.164 MB per token to get the token pool, then dividing by context length.

Two per-GPU memory figures are in circulation for these boards, and the gap is 7% on B200 and 10% on B300, so state which one you are planning against. GMI Cloud's Prime Inference page lists 192 GB per B200 and 288 GB per B300. NVIDIA's system specifications work out lower per GPU: DGX B200 ships "8x NVIDIA Blackwell GPUs" with "1,440 GB total" of GPU memory, which is 180 GB each (NVIDIA DGX B200), and HGX B300 lists 2.1 TB across 8 GPUs, about 262.5 GB each (NVIDIA HGX). The table below uses the NVIDIA values as the conservative planning floor.

Concurrent sequences per GPU, each holding its full context, FP8 KV cache:

What that table says about the hardware decision:

1. B300 buys you exactly one precision step. B300 at BF16 and B200 at FP8 land on the same 83 GB KV budget and the same 61 sequences at 8K. If your team has already validated an FP8 checkpoint, B200 reaches the capacity that B300 gives you for free at BF16.
2. BF16 weights consume 78% of a B200 board. With 140 GB of weights and a 15% reserve, 13 GB of KV budget buys nine sequences at 8K and not even one full 128K sequence. The two ways out are quantizing to FP8 or sharding the replica across two GPUs, and the first one is free.
3. Context length is the expensive variable, not concurrency. Going from 8K to 32K costs you about three quarters of your sessions on identical hardware. Cap the context your application actually sends rather than paying for the model's maximum.

Sharding a replica across two GPUs does more than double the KV budget, because the weights are split rather than copied. Two B200s at FP8 give 2 x 180 GB, less the 15% reserve, less one 70 GB copy of the weights, which is 236 GB of KV budget and 175 sequences at 8K. That is about 87 sequences per GPU against 61 on a single-GPU replica, roughly a 43% gain in seats per GPU from amortizing one weight copy across two boards. On Prime Inference this is the "GPU count per replica" field, and "Multi-GPU orchestration handled" means the parallelism setup is not your team's build.

Does memory or throughput bind first?

For a 70B model at normal context lengths, the memory budget is what sets the GPU count, and the published throughput ceiling acts as a check on that number rather than a replacement for it. The ceiling is still worth computing, because it tells you when adding KV memory stops buying anything, and because a measured throughput number from a benchmark is what ultimately confirms or overrides a memory-based GPU count.

GMI Cloud publishes "Per-model tuned runtimes: up to 500K TPM (tokens per minute) per GPU" for Prime Inference. Two things to settle before you plan against it: it is an upper bound across tuned runtimes rather than a figure measured on a specific 70B checkpoint, and the token accounting behind it, output only or input plus output, is worth confirming with GMI Cloud for your own model. The table below reads it as output tokens, which is the optimistic end; if input counts against the same budget, every stream count drops and the crossover below moves up.

Now put that next to the memory table. One B200 running FP8 weights holds 61 sequences at 8K context, well below the 278 streams the TPM ceiling would allow at 30 tokens per second. The crossover, where a configuration's KV budget runs out at exactly the same point as its share of the ceiling, is specific to each row. Dividing the KV budget by 278 streams and then by 0.164 MB per token:

Above your row's crossover, KV memory runs out first, and the levers are more GPUs, lower KV precision, or shorter prompts. Below it, the throughput ceiling arrives first, adding KV memory changes nothing, and the lever is more GPUs or replicas.

Which side you land on is a question about your own traffic rather than your product category, so pull the median and p95 prompt length from a week of production logs before you size anything. Those two numbers are also what a GMI Cloud engineer needs from you to turn this table into a GPU type and a replica count, and they sit in the six-line sizing brief at the end of this article. The cost of skipping that step comes straight off the seats table: 200 sessions sized against a configured 32K window need 14 B200s at FP8, at 15 sessions per GPU, while the same 200 sessions sized against a measured 8K average need 4, at 61 per GPU.

B200 or B300 for a 70B model?

For a 70B dense model, B200 is the default and B300 is the context-length upgrade. The compute difference does not change the shape of the deployment: NVIDIA's HGX specifications give an 8-GPU B300 system 144 PFLOPS sparse and 108 PFLOPS dense FP4, against 144 and 72 for B200, with networking bandwidth of 1.6 TB/s versus 0.8 TB/s (NVIDIA HGX). A 70B model on a single GPU does not use that interconnect headroom, and the FP4 uplift only materializes if the runtime actually serves an NVFP4 checkpoint.

Reserve B300 on Prime Inference when one of these is true:

• Your average context is 32K or above and you want to stay at one GPU per replica. B300 at FP8 holds 28 sequences at 32K against B200's 15.
• You are serving BF16 weights and will not requantize. B300 at BF16 equals B200 at FP8 on capacity.
• A single request needs more than the roughly 500K-token KV pool a B200 gives it at FP8, which is where long-document workloads land.

Otherwise reserve B200, listed on GMI Cloud's pricing page at from $5.00 per GPU-hour with a Limited Availability tag, as of October 2026. B300 carries no published hourly rate; it is offered through Prime Inference and quoted by GMI Cloud sales, so bring your model, precision, context target, and token volume to that conversation. For the spec-by-spec comparison of the two cards and the reservation terms around them, GMI Cloud's guide to reserving B200 or B300 capacity covers that ground.

One boundary from the Prime Inference FAQ is worth keeping in view: "Most production workloads on GMI Cloud run on H100 or H200, with Blackwell reserved for performance-critical frontier use cases." A 70B model at FP8 and 8K context runs 37 sequences per H200, so a deployment under that number has a cheaper home on GMI Cloud's H200 capacity, on the same Prime Inference tier with engines pre-tuned for that GPU class. H100 is the one step that does not work for this model at FP8: 70 GB of weights leaves nothing usable on an 80 GB board.

What does a sized 70B deployment cost per month?

A 70B model at FP8 serving 200 concurrent sessions at 8K average context needs four B200 GPUs, which is from $14,600 a month at the October 2026 list rate. Here is that target worked backwards.

The throughput row is there to show what produced the GPU count: 360K tokens per minute sits under the published per-GPU ceiling, so nothing in the published numbers pushes the count above what the KV math already requires. Whether the measured throughput of your tuned runtime also carries 200 sessions on four GPUs is the question a benchmark answers.

Divide the hourly rate by sequences per GPU and you get a unit that makes precision and context decisions concrete, the cost of one concurrent session:

Based on the B200 list rate of $5.00 per GPU-hour as of October 2026, GPU compute only. Two decisions show up in that table with real money attached: quantizing from FP8 to NVFP4 cuts the per-session cost by 27% at 8K and 29% at 32K, and raising the context cap from 8K to 32K multiplies it by about four. For the token-level view of the same spend, GMI Cloud's explainer on cost per million tokens converts GPU-hours into the unit most teams budget in.

These figures are a deliberate worst case: they assume every in-flight sequence holds its full context at once. Serving engines allocate KV cache in blocks, and a sequence only reaches its full context at the end of generation, so treat the table as a planning floor and measure your own prompt distribution against it before committing to a GPU count.

What "tuning" covers on Prime Inference

Tuning a 70B deployment on Prime Inference covers weight quantization, KV cache handling, engine selection across vLLM, TensorRT-LLM, and SGLang, multi-GPU orchestration, and region placement. Those five decisions turn the sizing table into a running endpoint, and the split of work on each one is clear.

The per-model work behind that list is what separates a tuned runtime from a container: GMI Cloud describes it as "Per-model kernel, scheduling, and routing optimization, not a generic stack," and states that its inference engineers continuously tune the runtimes behind the most-deployed open-source models.

On accuracy after quantization, there is a published reference point: NVIDIA reports "1% or less accuracy degradation on key language modeling tasks for DeepSeek-R1-0528, when quantized from its original FP8 format to NVFP4" (NVIDIA, Introducing NVFP4). Run the same check on your own eval set before NVFP4 reaches production, because the quantization saving above is only real if quality holds.

How to validate the sizing before you commit

Validating a 70B sizing means registering the checkpoint on GMI Cloud, benchmarking it against your own traffic, then deploying against the number the benchmark returns. Every figure above is arithmetic, which makes it a starting hypothesis; these steps turn it into a measured GPU count:

1. Register the checkpoint. GMI Cloud's Console has a My Models page that, per the documentation, lets you "manage user-owned, fine-tuned, or custom models that your dedicated deployments can target." That is where a 70B checkpoint becomes a deployable target.
2. Benchmark instead of assuming. The same page carries a PerfLab action, documented as "opens GMI's external benchmarking tool in a new tab." The measurement that settles a GPU count is sequences sustained at your p95 latency target, not sequences that fit in memory, so ask GMI Cloud how to run your own prompt profile and which numbers come back.
3. Deploy against the measured number. Prime Inference goes live in four steps (pick a model, choose the setup, deploy, operate and scale), with the endpoint "live in minutes, not days." Prime Inference also notes that "Qualified prospects also receive free GPU-hour trial credits to validate performance against their own workload," so ask GMI Cloud sales about eligibility when you send the sizing brief.

A complete sizing brief fits in six lines: parameter count and architecture, target weight precision, target KV precision, p50 and p95 context length, peak concurrent sessions, and target output tokens per second per user. Those six lines are enough for a GMI Cloud engineer to return a GPU type, a GPU count per replica, and a replica count.

FAQ

How many B200 GPUs does a 70B model need? In the example, one 180 GB B200 with FP8 weights and a 15% runtime reserve leaves 83 GB for KV cache, about 61 sequences at 8K. BF16 weights occupy 140 GB and leave room for nine; two B200s provide 166 GB of KV budget and 123 sequences. One B300 at BF16 matches the B200 FP8 example's 61 sequences. Set GPU count per replica and replica count accordingly; these are memory calculations, not throughput guarantees.

Is B300 worth it over B200 for a 70B model? B300 helps when context drives memory demand: the cited 262.5 GB per GPU supports 28 sequences at 32K versus B200's 15, and BF16 capacity matches B200 at FP8. For 8K FP8 workloads, B200's October 2026 starting rate is $5.00/hour. Obtain a B300 quote and benchmark both options.

How do you calculate KV cache size for a 70B model? KV bytes/token = 2 × layers × KV heads × head dimension × bytes/element. Multiply by context and concurrency. For the cited Llama 3 70B configuration (80 layers, eight KV heads, dimension 128), FP8 KV uses 163,840 bytes/token: 5.37 GB at 32K and 21.5 GB at 128K.

Can GMI Cloud deploy our own fine-tuned 70B weights? Yes. Register custom weights from Hugging Face, S3, or private storage in My Models, then target them from a dedicated Prime Inference deployment.

What throughput should we expect per GPU? GMI's up-to-500K TPM/GPU figure is a tuned-runtime upper bound, not a 70B measurement. It equals about 8,333 tokens/second or 278 streams at 30 output tokens/second each. In the example, FP8 B200 becomes memory-limited above roughly 1,800 context tokens, and B300 near 3,400. Size memory first and confirm achievable throughput on actual traffic.

Does Prime Inference handle the tuning, or do we configure the engine ourselves? GMI manages vLLM, TensorRT-LLM, or SGLang tuning, configurable quantization, and multi-GPU orchestration. Supply the checkpoint, precision target, and traffic profile.

Size your 70B deployment with GMI Cloud

Send the six-line sizing brief, and we will size the B200 or B300 replica against the tables above, validate it against your own traffic, and quote on-demand or reserved rates with no minimum contract. Start on the Prime Inference page, or check current GPU list prices on the pricing page.