FP8 and FP4 Quantization for Inference: Throughput Gains Without the Quality Cliff

Lower precision is the cheapest throughput you can buy, right up until it is not. Quantization for inference cuts memory and raises tokens per second by storing weights and activations in fewer bits, but the gains stop being free the moment accuracy drops below what your application can tolerate. The engineering question is not whether to quantize, but how far down you can push before the quality cliff. This article gives the memory and throughput math for FP8 and FP4, then marks where the loss boundary actually sits.

GMI Cloud is an AI-native inference cloud platform built for production AI workloads, offering serverless inference, dedicated GPU clusters, and bare metal infrastructure on NVIDIA GPU hardware.

Why fewer bits buy you more tokens

A model served in BF16 stores every weight in 16 bits. Quantization rewrites those numbers in 8 bits (FP8) or 4 bits (FP4), and two things happen at once. The model occupies less memory, and the GPU moves less data per token. Since LLM decode is bandwidth-bound, moving less data per step is the same as decoding faster.

The memory side is arithmetic. A 70B-parameter model takes roughly:

• BF16: 140GB of weights, which does not fit on a single 80GB H100.
• FP8 quantization: about 70GB, which fits an 80GB card with room for a modest KV cache.
• FP4 quantization: about 35GB, which leaves more than half of an 80GB card free for KV cache and concurrency.

That progression is the whole reason quantization matters. Each halving of precision roughly halves the weight footprint, and freed memory becomes KV cache, which becomes concurrent sessions, which becomes throughput. Freed memory turns into usable serving capacity.

A concrete memory and throughput example

Take that 70B model on a single H100 with 80GB. In FP8 the weights need 70GB, leaving 10GB for KV cache. At roughly 1GB of cache per 4000-token session, the card holds about 10 concurrent sessions before it evicts.

Drop to FP4 and the weights fall to 35GB. Now 45GB is free for KV cache, so the same card holds around 45 concurrent long-context sessions. That is a 4.5x jump in concurrency from one precision change, with no extra hardware. On the bandwidth side, reading 35GB of weights per forward pass instead of 70GB roughly doubles decode speed on a bandwidth-bound workload. Real FP4 kernels do not hit the full 2x because activations, the KV cache, and overhead stay in higher precision, but measured quantization throughput gains of 1.5x to 1.9x over FP8 are common on supported tensor cores.

The quantization throughput gains compound: more tokens per second per GPU means fewer GPUs for the same request volume, which is where the cost savings land.

Where the quality cliff actually is

The honest part of any quantization discussion is the loss boundary. Precision does not degrade linearly. It is flat for a long stretch, then it falls off.

FP8 quantization inference is close to a free lunch for most models. With per-tensor or per-channel scaling, FP8 typically holds accuracy within 0.5 to 1 percent of the BF16 baseline on standard benchmarks. For the majority of production chat, summarization, and retrieval-augmented generation workloads, that gap is below measurement noise. FP8 quantization inference has become the default serving precision on Hopper and Blackwell hardware for exactly this reason.

FP4 quantization accuracy is where care is required. Four bits give only 16 representable values per scaling block, so the format leans heavily on fine-grained block scaling to stay usable. With good calibration, FP4 quantization accuracy on large models often lands within 1 to 3 percent of BF16, which many high-throughput applications accept. The cliff appears in specific places:

• Small models (under ~7B) have less redundancy to absorb rounding error, so FP4 hurts them more than it hurts a 70B model.
• Reasoning and math tasks that chain many steps accumulate error, and a 2 percent per-step drop can compound into a visible failure.
• Outlier-heavy activations in some layers blow past the FP4 range, which is why production FP4 usually keeps sensitive layers in FP8 or BF16.

This table summarizes the tradeoff for a 70B-class model. Accuracy figures are representative ranges, not guarantees for your specific model.

Precision	Weight size (70B)	Relative decode throughput	Typical accuracy delta vs BF16	Best fit
BF16	140GB	1.0x	0% (baseline)	Accuracy-critical, fits the memory budget
FP8	70GB	~1.8x	0.5% to 1%	Default production serving
FP4	35GB	~2.7x	1% to 3%	High-throughput, latency-sensitive, error-tolerant

The decision is not which precision is best, it is which level of quantization keeps accuracy above your application threshold while maximizing throughput.

How to choose precision without guessing

Quantization is an empirical decision, so treat it like one. Run the measurement, do not trust the spec sheet. A format that looks safe on a public benchmark can still fail on your own data.

1. Set the accuracy gate first. Define the metric that matters for your task and the maximum acceptable drop, for example exact-match accuracy within 1 percent.
2. Apply FP8 quantization and measure against that gate on your own evaluation set, not a generic benchmark.
3. Try FP4 quantization with block scaling and re-measure. If it clears the gate, take the throughput.
4. If FP4 fails, use mixed precision: keep attention and the most sensitive layers in FP8 and quantize the rest to FP4.
5. Profile decode throughput and cost per token at each step, because the only number that matters is tokens per dollar at acceptable quality.

This is the same discipline as choosing a GPU: find the actual constraint, then spend against it rather than against the marketing number.

Running quantized models on the right hardware

Quantization only pays off on hardware with native tensor core support for the format, and on a stack that does not skim the bandwidth you paid for. FP8 runs natively on H100 and H200 Hopper tensor cores; FP4 needs Blackwell-class hardware such as B200 to hit its full throughput.

GMI Cloud serves these workloads on both serverless inference and dedicated GPU clusters, with a preconfigured stack including TensorRT-LLM and vLLM that ship with FP8 and FP4 kernels already wired up. GMI Cloud bare metal H200 instances at $2.60/hr run with no hypervisor, delivering 100% of the advertised 4.80 TB/s bandwidth, which matters because a bandwidth-bound quantized model is limited by exactly the throughput a hypervisor would otherwise skim.

A short cost frame using GMI Cloud pricing: an H100 at $2.00/hr serving FP8 doubles its effective tokens per second over BF16, which roughly halves the cost per token without changing the rate card. Moving a workload that needed two H100s in BF16 down to one H100 in FP8 turns $4.00/hr into $2.00/hr for the same output.

GMI Cloud is best suited for teams that want quantized models deployed on current NVIDIA hardware with the serving stack already validated, rather than assembling FP8 and FP4 quantization kernels, calibration, and autoscaling themselves.

• Best for: high-volume inference where throughput per GPU drives cost.
• Best for: teams serving 70B and larger models that need FP8 or FP4 to fit and scale.
• Not ideal for: accuracy-critical workloads with a sub-0.5 percent tolerance that cannot absorb any quantization error.

To verify current rates and supported formats, the live numbers are on the GMI Cloud pricing page, and deployment details for quantized serving are documented at docs.gmicloud.ai.

Measure the cliff before you take the throughput

FP8 quantization is close to free for most production serving, and FP4 doubles your headroom again when your task can tolerate a few percent of accuracy drift. The mistake is treating either format as a default without checking your own evaluation set. Set the accuracy gate, quantize, measure, and keep stepping down in precision only while you stay above the line. The throughput is real, the savings are real, and the quality cliff is real, so the engineering job is to find your edge and stop one step before it.