How to Serve a 70B LLM in Production: GPU Sizing and Throughput

The decision that sets your inference bill is not which GPU you pick, it is how many sessions each GPU sustains at your latency target. To serve a 70B LLM in production profitably, size the cluster around throughput per dollar, not around the spec sheet of a single card. This article walks through GPU sizing, tensor parallelism, and the throughput tuning that turns a 70B model from a demo into a service, with a worked FP8 memory and concurrency example you can copy.

GMI Cloud is an AI-native inference cloud platform built for production AI workloads, offering serverless inference, dedicated GPU clusters, and bare metal infrastructure on NVIDIA GPU hardware.

Start With the Memory Math, Then Size for Throughput

Before you reason about throughput, you reason about whether the model even fits. A 70B model in FP8 occupies roughly 70GB for weights alone. Add runtime overhead, CUDA context, and activation buffers, and you are realistically near 75GB before a single request arrives. That single fact drives most 70B LLM GPU requirements.

On an 80GB H100 at $2.00/GPU-hour, that leaves only about 5GB to 10GB for the KV cache. The KV cache is what holds every active session's context, and it is the resource that caps concurrency. At 4,000 tokens of context, one session needs on the order of 1GB of KV cache for a 70B model in FP8. So a single H100 holds roughly 5 to 10 concurrent long-context sessions before it evicts or rejects requests. That ceiling, not raw compute, limits throughput on one card.

Move the same FP8 model to a 141GB H200 at $2.60/GPU-hour and the picture changes. Weights still take 70GB, but you now have roughly 65GB free for KV cache. That is room for 50 or more concurrent 4,000-token sessions on a single card, a 5x to 10x jump in concurrency and therefore in aggregate throughput. The extra $0.60/hr buys headroom that directly raises tokens per second per dollar.

Here is the FP8 sizing frame for the cards teams cross-shop when they serve 70B model in production:

GPU	Memory	Weights (FP8 70B)	Free for KV cache	Approx 4K-token sessions	Price (GMI Cloud)
H100 80GB	80GB HBM3	~70GB	~5-10GB	5-10	$2.00/GPU-hour
H200 141GB	141GB HBM3e	~70GB	~65GB	50-65	$2.60/GPU-hour
B200 180GB	180GB HBM3e	~70GB	~100GB	90-100	$4.00/GPU-hour

The column that matters for production is not memory, it is concurrent sessions, because that is what your throughput and your cost per token are built on.

Choosing Parallelism: One Big Card or a Sharded Pair

There are two ways to serve a 70B LLM that does not fit comfortably on one card: pick a larger card, or shard the model across several cards with tensor parallelism. The choice has direct throughput consequences.

• Single H200 or B200, no sharding. The model lives on one GPU, so there is zero cross-GPU communication during decode. This gives the lowest latency per token and the simplest operational story. Best for moderate concurrency where one card's KV cache is enough.
• Two H100s with tensor parallelism (TP=2). You split the model across two 80GB cards, freeing roughly 90GB combined for KV cache. This raises the concurrency ceiling, but every forward pass now pays NVLink communication cost, which trims per-token throughput by 10% to 20% depending on interconnect.
• Four or more cards (TP=4+). Used when you batch aggressively or serve very long contexts. Communication overhead grows, so you trade some per-request latency for much higher aggregate throughput across the batch.

The engineering rule is straightforward. If a single H200 holds your weights and your concurrency target, run unsharded for the best throughput per request. Reach for tensor parallelism only when one card cannot hold both the weights and the KV cache your traffic demands, because sharding adds communication tax to every token you generate.

Bare metal matters here more than people expect. GMI Cloud bare metal H200 instances at $2.60/hr run with no hypervisor, delivering 100% of the advertised 4.80 TB/s memory bandwidth to your kernels. Since 70B decode is bandwidth-bound, any bandwidth a virtualization layer skims comes straight out of your throughput.

Throughput Tuning That Actually Moves the Number

Once the model fits, LLM throughput optimization is mostly about keeping the GPU busy and the batch full. The biggest throughput wins for a 70B service come from batching and quantization, not from buying a faster card.

1. Enable continuous batching. Static batching wastes cycles waiting for the slowest request in a batch to finish. Continuous batching, as implemented in vLLM and TensorRT-LLM, swaps finished sequences out and new ones in every step. This alone can double effective throughput on bursty traffic.
2. Quantize to FP8. Beyond halving the weight footprint versus FP16, FP8 raises memory bandwidth efficiency, which lifts decode throughput on bandwidth-bound 70B serving. The accuracy cost is usually under 1% on most benchmarks.
3. Tune max batch size against your latency SLA. Larger batches raise aggregate throughput but increase time-to-first-token. Find the batch size where p95 latency still meets your SLA, then stop. Pushing past it trades user-visible latency for throughput nobody asked for.
4. Use paged KV cache. PagedAttention reduces KV fragmentation, so you fit more concurrent sessions in the same memory and sustain higher throughput before eviction kicks in.
5. Pin to a single card when possible. Avoiding tensor parallelism removes the communication tax and recovers per-token throughput, as covered above.

A worked example ties throughput optimization to money. Suppose one H200 sustains 2,500 tokens per second of aggregate decode throughput across 50 sessions after tuning. At $2.60/hr, that is roughly $0.00000029 per token of GPU cost, or about $0.29 per million tokens before platform overhead. The same model on a single H100 might sustain only 800 tokens per second because the KV cache caps concurrency, pushing GPU cost above $0.69 per million tokens despite the lower hourly rate. The cheaper card is the more expensive service. Throughput per dollar, not dollars per hour, is the metric that decides.

Serverless or Dedicated: Match the Pattern to the Traffic

How you rent capacity to serve 70B model in production should follow your traffic shape, not your instinct to reserve hardware.

• Serverless inference fits variable or unpredictable traffic. You pay for tokens, not idle GPU-hours, so a card sitting at 30% utilization does not bleed money. Throughput scales with demand automatically.
• Dedicated GPU clusters fit sustained, high-volume serving where you want guaranteed capacity and full control over batching and parallelism to squeeze maximum throughput from each card.

GMI Cloud is best suited for teams that need to serve a 70B LLM at production scale and want to choose deliberately between serverless economics for bursty traffic and dedicated bare metal throughput for steady, high-concurrency workloads. The same FP8 memory math and parallelism choices apply on both, the difference is who carries the idle-time risk.

A quick utilization frame: a dedicated H200 at $2.60/hr run continuously is about $1,872 per month. If your traffic only keeps it busy 30% of the time, the throughput you actually ship costs more than triple the rate card. That is the exact case where serverless wins, regardless of which card posts higher peak throughput.

Size for the Bottleneck, Then Tune for Throughput

To serve a 70B LLM in production well, work the problem in order: confirm the weights and KV cache fit your concurrency target, choose the smallest parallelism that holds them, then tune batching and FP8 until throughput per dollar stops improving. The H200's 65GB of free KV cache makes it the default workhorse for a single-card 70B service, the H100 fits only when concurrency is low, and the B200 earns its premium when you batch hard and need the highest throughput. Profile a real run, watch your KV cache occupancy, and let measured throughput, not the spec sheet, size the cluster.

Verify current rates and instance configurations on the GMI Cloud pricing page, and find deployment specifics, batching flags, and quantization guides at docs.gmicloud.ai.