What Makes the Best LLM GPU? Memory Bandwidth, FP8, and KV Cache Explained

Ask most engineers what the best LLM GPU is and they'll point to whatever has the most TFLOPS. That answer is wrong for inference. LLM inference is memory-bound, not compute-bound.

Why LLM inference is memory-bound, not compute-bound

A transformer generates tokens one at a time. At each step it loads the full model weights from GPU memory into the compute cores, does a small amount of matrix math, and writes the output back. The math takes microseconds. The memory transfer takes longer.

The bottleneck shows up clearly in the math. A 70B parameter model in FP16 is roughly 140 GB of weights. At each token step, the GPU reads all 140 GB. If your card has 3.35 TB/s of memory bandwidth (H100 SXM), the theoretical ceiling is about 24 tokens per second per request, and real-world numbers sit lower after accounting for KV cache reads, attention computation, and framework overhead. Doubling the FLOPS changes nothing here. Doubling the memory bandwidth doubles it.

This is the first filter for any LLM GPU: memory bandwidth, not peak compute, sets the inference speed limit.

The four properties that define a good LLM GPU

A card becomes a strong LLM GPU when it excels across four dimensions. Each one maps to a specific cost in the inference pipeline.

• Memory bandwidth: How fast weights and KV cache move from HBM to compute. This is the primary driver of tokens per second for single-request decoding and the single most important spec on any LLM GPU.
• Total memory capacity: How large a model fits on one card and how much room is left for KV cache. A 141 GB HBM3e card (H200) can hold a 70B FP8 model with room for long context windows. An 80 GB card forces tensor parallelism across two GPUs earlier, which adds latency and cost.
• Low-precision support (FP8, FP4): FP8 halves the weight footprint and memory traffic compared to FP16, which effectively doubles decode throughput for the same bandwidth. FP4 goes further but demands hardware-native support to avoid accuracy collapse.
• KV cache and long context capability: Every active request holds a growing KV cache proportional to context length. Serving 32K or 128K context windows requires either large memory headroom or paged KV cache management that the GPU's memory architecture can sustain.

These four properties interact. A card with high bandwidth but small memory runs out of KV cache headroom before it runs out of speed. A card with huge memory but no FP8 support wastes bandwidth moving FP16 weights it didn't need to move. The best LLM GPU balances all four for your specific model size and context length.

Memory bandwidth: the real speed limit

Here's how current NVIDIA data center cards compare on the specs that matter for LLM inference:

GPU	HBM capacity	Memory bandwidth	FP8 support	Max context (single 70B FP8)
H100 SXM	80 GB HBM3	3.35 TB/s	Yes	~64K tokens
H200 SXM	141 GB HBM3e	4.8 TB/s	Yes	~128K tokens
B200	192 GB HBM3e	8.0 TB/s	Yes + FP4	~256K tokens
GB200 NVL72	13.5 TB HBM3e (72 GPUs)	8.0 TB/s per GPU	Yes + FP4	Multi-million token pool

The jump from H100 to H200 is telling. The H200 increases bandwidth by 43 percent and nearly doubles memory capacity, which is why it handles 128K context windows on a 70B model in FP8 where the H100 runs out of room. For teams serving long-context reasoning models or large RAG pipelines, that memory capacity difference is the deciding factor, not FLOPS.

FP8 and FP4: cutting the cost per token

FP8 inference is the biggest single cost lever in LLM deployment today. Storing and moving weights in FP8 instead of FP16 halves the memory footprint and the memory traffic per token. On a memory-bound workload, that translates to roughly double the tokens per second at the same bandwidth.

The catch is that FP8 requires hardware-native tensor core support. Cards before Hopper (H100, H200) don't have it. Running FP8 emulation on older hardware is slower than native FP16 because you pay conversion overhead on every operation.

FP4 goes a step further, halving memory traffic again. The Blackwell architecture (B200, GB200) supports FP4 natively, and early benchmarks show 2x to 2.5x throughput improvements over FP8 on inference workloads where accuracy holds. The trade-off is model-specific: not every model maintains quality at FP4, and you need calibration or quantization-aware training to get there safely.

If you're evaluating an LLM GPU for production inference, FP8 support is now table stakes. FP4 support is the differentiator for the next generation of cost-sensitive deployments.

KV cache: the hidden scaling cost

KV cache is the memory cost of keeping a conversation alive. For every active request, the GPU stores key and value tensors for every token in the context window. The cache grows linearly with sequence length and with the number of concurrent requests.

For a 70B model in FP8, each token of KV cache takes roughly 40 KB per layer per request. At 32K context across 32 concurrent requests, that's over 40 GB of KV cache alone, before you count the 70 GB of model weights. This is why memory capacity on an LLM GPU isn't just about fitting the model. It's about fitting the working set of active conversations.

1. Calculate your KV cache headroom: Subtract model weight footprint from total HBM to get available KV cache space. For a 70B FP8 model on H200 (141 GB), that's roughly 70 GB free for KV cache after weights.
2. Map headroom to concurrency: Divide available KV cache by per-request cache size at your target context length. 70 GB free / 1.3 GB per 32K request gives you roughly 50 concurrent long-context requests per GPU.
3. Decide on paged KV cache: Frameworks like vLLM and TensorRT-LLM support paged attention, which reduces KV cache fragmentation and can raise effective concurrency by 20 to 40 percent without more memory.
4. Plan for context growth: If your product roadmap includes 128K or 1M context windows, the KV cache math dominates. Pick a card with headroom or plan for tensor parallelism across multiple GPUs.

KV cache is why two cards with identical FLOPS can serve wildly different numbers of concurrent users. The one with more memory capacity and better paged attention support wins.

Long context windows change the hardware bar

Context length is the spec most likely to break your deployment. A year ago, 4K context was the default. Today, production workloads routinely run 32K to 128K, and frontier models push past 1M tokens. Each doubling of context length roughly doubles KV cache memory per request and increases attention computation quadratically unless you're using sparse or linear attention.

This changes the LLM GPU requirements in two ways. First, memory capacity becomes a hard ceiling, not a nice-to-have. A card that comfortably serves a 70B model at 8K context may hit out-of-memory errors at 128K. Second, memory bandwidth matters more at long context because each decode step reads a larger KV cache.

If your roadmap includes long-context reasoning, retrieval, or agentic workflows with long conversation memory, prioritize memory capacity and bandwidth over raw compute. The best LLM GPU for long context is the one that holds the model, the KV cache, and the attention intermediates without spilling to slower memory tiers or splitting across GPUs unnecessarily.

How GMI Cloud approaches the LLM GPU question

GMI Cloud is an AI-native inference cloud built for production AI. The platform runs on NVIDIA H100 and H200 GPUs, with B200 and GB200 NVL72 available, and the full stack is configured for LLM inference from the networking layer up.

GMI Cloud's GPU infrastructure is backed by 30,000-plus GPUs deployed, 99.99 percent platform availability, and sub-200ms average cross-region latency across North America, Europe, and Asia-Pacific. Current rates start at $2.00 per GPU-hour for H100 and $2.60 for H200, and you can review the full catalog on the GMI Cloud GPU catalog or check current pricing on the pricing page.

The reason this matters for the LLM GPU question is simple. A card's specs only translate to real performance if the platform around it avoids stealing bandwidth through virtualization, gives you access to FP8 and paged KV cache frameworks, and lets you scale from a single serverless endpoint to a multi-GPU bare metal cluster without re-architecting.

Pick the card that fits your model, not the spec sheet

The best LLM GPU is the one that matches your model size, context length, and concurrency target without wasting memory or bandwidth. Here's the short version of how to decide:

1. Start with your model. A 7B model in FP8 fits comfortably on a single H100 with room for KV cache. A 70B model in FP8 wants an H200 or larger to avoid tensor parallelism overhead.
2. Add up KV cache for your target context length and concurrency. If the total exceeds available HBM after weights, you need a larger card or multiple GPUs.
3. Require FP8 support. It's the single biggest cost lever for inference today, and cards without it will underperform on cost per token regardless of their FLOPS.
4. Check long context headroom. If your roadmap includes 128K or longer contexts, prioritize memory capacity and bandwidth. Don't assume a card that works at 8K will work at 128K.
5. Compare on delivered cost per token, not per GPU-hour. A higher hourly rate on a card with FP8 and double the bandwidth can deliver lower cost per token than a cheaper card that's memory-starved.

When you're ready to test these numbers against real hardware, the GMI Cloud models page lists the 100-plus models available on the Inference Engine, and the console lets you provision an H100 or H200 endpoint and measure tokens per second on your actual workload. That's the only benchmark that matters.