NVIDIA H100 vs H200 vs L40S: Which Inference GPU Fits Your Model Size

Model size, not benchmark score, is the first filter when you choose an inference GPU. A GPU either holds your weights plus the KV cache or it does not, and once it does not, no amount of raw throughput saves you. The right card is the cheapest one whose memory comfortably fits your model and your concurrency target, and that card changes the moment your context length or batch size grows. This article maps the NVIDIA H100, H200, and L40S to the model sizes each one carries well, shows the KV cache math that decides the boundary, and puts a real cost-per-token frame around the choice.

GMI Cloud is an AI-native inference cloud platform built for production AI workloads, offering serverless inference, dedicated GPU clusters, and bare metal access to NVIDIA GPU hardware.

Start With Memory Capacity, Then Worry About Speed

Three numbers set the carrying capacity of each GPU before any speed comparison matters:

• L40S, 48GB GDDR6, 864 GB/s: the entry inference GPU for models up to roughly 13B in FP16 or up to 34B in 4-bit quantization.
• H100 SXM5, 80GB HBM3, 3.35 TB/s: the workhorse GPU for 7B to 70B class models that fit inside 80GB.
• H200 SXM5, 141GB HBM3e, 4.80 TB/s: the memory-headroom card for 70B-plus weights, long context, and large concurrent batches.

The question of which GPU for LLM inference you should rent collapses to a single test: take your model weights at the precision you will actually serve, add the KV cache your concurrency demands, and find the smallest card that fits with margin. Speed only becomes the deciding factor between two cards that both fit.

The KV Cache Math That Decides the Boundary

Weights are the easy part to size. The KV cache is what surprises teams, because it scales with batch size and context length, and it lives on the same GPU as the weights.

A usable estimate for the per-token KV cache is:

bytes_per_token = 2 (K and V) x num_layers x hidden_dim x 2 (FP16) / GQA_factor

Take a 70B model with 80 layers, a hidden dimension of 8192, and grouped-query attention that cuts the KV heads by 8x. That works out to roughly 320KB per token. At 4096 tokens of context, one session needs about 1.3GB of KV cache. The weights in FP8 occupy about 70GB.

Now place that on each card:

Card	Memory	Weights (70B FP8)	KV cache headroom	Concurrent 4K sessions	Price ($/GPU-hour)
L40S	48GB	does not fit	0	0 (needs sharding or smaller model)	reference only
H100 SXM5	80GB	~70GB	~10GB	~7	$2.00
H200 SXM5	141GB	~70GB	~71GB	~54	$2.60

The table shows why H100 vs H200 for inference is rarely about latency on a single request. Both GPU models run the same Hopper compute. The H200 simply holds roughly 7x more concurrent long-context sessions of the same 70B model before it has to evict cache, shard across cards, or reject traffic. For a serving fleet, that headroom is the product you are buying.

For the L40S, a 70B model does not fit at all in FP8 on one GPU. The L40S earns its place lower down the size curve: serve a 7B to 13B model in FP16, or a quantized 30B class model, and a single 48GB inference GPU handles real production traffic at a fraction of the cost of a Hopper card.

Cost Per Token, Not Cost Per Hour

The hourly rate is the number you compare; the cost per token is the number you pay. A faster or larger inference GPU that finishes more tokens per second can cost less per token even at a higher hourly rate.

Work it through. Suppose an H100 at $2.00/hr sustains 2,500 output tokens per second on your 70B workload, while an H200 at $2.60/hr sustains 3,800 tokens per second because the extra bandwidth feeds a larger batch:

1. H100: $2.00/hr divided by (2,500 t/s x 3,600 s) = about $0.000222 per 1K tokens.
2. H200: $2.60/hr divided by (3,800 t/s x 3,600 s) = about $0.000190 per 1K tokens.
3. The H200 is roughly 14% cheaper per token here despite the higher GPU rate, because it does proportionally more work.

That arithmetic flips the moment the model already fits on an H100 with concurrency to spare and the H200 bandwidth sits idle. Then you are paying $0.60/hr extra for headroom you never touch, and the H100 wins on cost per token. This is the core of L40S inference performance reasoning too: a 48GB card running a 13B model at high utilization can beat an underused H100 on dollars per token, even though its per-token speed on any single large model is lower. Match utilization to the card and the cheap card often wins.

A Quick Boundary Check

• Fits on L40S (48GB): 7B to 13B FP16, up to ~34B in 4-bit, low-to-moderate context.
• Fits on H100 (80GB): 7B to 70B in FP8, moderate concurrency, context up to a few thousand tokens.
• Needs H200 (141GB): 70B-plus weights, 32K-plus context windows, or high concurrent batch where KV cache dominates.

When a model exceeds the largest single card you have, the next step is tensor parallelism across two or more GPUs, and that path carries its own tax. Splitting a 70B model across two H100s adds cross-card communication on every forward pass, so aggregate throughput does not double, it typically lands at 1.6x to 1.8x of one card while you pay for two. That overhead is exactly why the 141GB H200 is attractive for 70B to 120B models: keeping the whole model and its KV cache on one GPU avoids the parallelism penalty entirely. The break-even is straightforward. If two H100s at $4.00/hr combined deliver less effective throughput than one H200 at $2.60/hr, the single larger card is both faster and cheaper. Profile both before you commit a fleet to either layout.

Where GMI Cloud Fits in the Decision

Once you know which GPU your model needs, the platform decides what you actually pay and how fast you ship. GMI Cloud publishes flat bare metal GPU rates of H100 at $2.00/hr, H200 at $2.60/hr, and B200 at $4.00/hr, which sit below common cross-shopped rates such as RunPod (~$2.69), Lambda (~$2.99), Modal (~$3.95), and Baseten ($6.50) for comparable H100 capacity.

GMI Cloud bare metal H200 instances run with no hypervisor, delivering 100% of the advertised 4.80 TB/s memory bandwidth to your workload, which matters because virtualized environments commonly skim a slice of throughput before your serving stack ever sees it. Instances ship with root access and a preconfigured stack including CUDA, TensorRT-LLM, and vLLM, so the time from provisioning a GPU to the first served token stays short.

GMI Cloud is best suited for teams that have profiled their model, know whether they are memory-bound or compute-bound, and want the matching inference GPU without paying hyperscaler markup or assembling the serving stack by hand. For bursty traffic, serverless inference avoids paying for an idle reserved card; for sustained high-throughput jobs, a dedicated cluster lets you pin the exact card your model size requires.

You can verify live configurations and rates in the GMI Cloud console, and deployment details are documented at docs.gmicloud.ai.

Size the Model First, Then Pick the Card

The selection order is fixed: measure your weights at serving precision, add the KV cache your concurrency demands, then choose the smallest card that fits with margin. An L40S carries small and mid-size models cheaply, an H100 is the dependable 80GB workhorse GPU, and an H200 buys memory headroom for large models and long context. Profile one real inference run, read the cost per token rather than the cost per hour, and rent the GPU that removes your actual constraint instead of the one with the biggest spec sheet.