AI Inference Speed Explained: How to Measure It and How to Make It Faster

Ask two teams how fast their model serves and you'll often get one number: tokens per second. That single figure hides most of what users actually feel. AI inference speed is not one metric, it's at least three: time to first token, inter-token latency, and sustained throughput, and a system can look fast on one while feeling slow on another. A chatbot that streams 80 tokens per second but takes two seconds to start feels sluggish, even though the throughput number looks fine. This guide breaks down how to measure inference speed properly, what slows it down, and the levers that reliably make it faster.

The three numbers that define inference speed

Latency and throughput are different things, and mixing them up is the most common reason speed discussions go nowhere. Latency is how long one request takes. Throughput is how much work the system does across all requests at once. You tune them separately, and pushing one often costs you the other.

For a language model, break latency into two parts. Time to first token (TTFT) is how long from request arrival until the first output token appears. It covers queue wait, the prefill pass over your prompt, and any network hop. Inter-token latency (ITL), sometimes called time per output token, is the gap between each streamed token after the first. Multiply ITL by output length and add TTFT, and you get end-to-end latency for a single response.

Here's how the core metrics line up:

Metric	What it measures	Unit	What it drives
TTFT	Request arrival to first token	ms	Perceived responsiveness, "does it feel alive"
ITL	Gap between streamed tokens	ms/token	How fast text appears to scroll
Throughput	Tokens served across all requests	tokens/sec	Cost efficiency, capacity per GPU
P99 latency	Slowest 1% of requests	ms	Worst-case user experience, SLA risk

The last row matters more than teams expect. Averages hide tail behavior. If your median TTFT is 200ms but your p99 is 4 seconds, one in a hundred users waits four seconds, and at scale that's thousands of frustrated sessions. Quote inference speed as a distribution, not a single average, and always report p99 alongside the median.

A worked end-to-end latency breakdown

Numbers make this concrete. Suppose you're serving a 70B model and a user sends a 1,000-token prompt expecting a 500-token answer. Here's a realistic decomposition:

1. Queue wait: The request sits behind others in the batch. Say 30ms under moderate load.
2. Prefill: The model processes all 1,000 prompt tokens in parallel. On a well-configured GPU this might run at 10,000 tokens/sec, so 1,000 tokens takes about 100ms.
3. First token emission: A few milliseconds of overhead. TTFT so far is roughly 30 + 100 + 10 = 140ms, plus network.
4. Decode: The model generates 500 output tokens one at a time. At an ITL of 20ms/token, that's 500 x 20 = 10,000ms, or 10 seconds.

End-to-end latency is about 140ms + 10s = 10.14 seconds, and the decode phase dominates completely. That's the key lesson: for long outputs, ITL times output length is the whole ballgame, and shaving 5ms off ITL saves 2.5 seconds here while shaving 50ms off TTFT is barely noticed. Where you invest depends on your workload. Short-answer, high-turn chat lives and dies on TTFT. Long-form generation lives and dies on ITL.

What actually slows inference down

Once you can measure speed, the next question is what governs it. Four factors explain most of the variance between a fast serving stack and a slow one.

• Model size: Every generated token requires reading the full set of model weights from GPU memory. A 70B model in FP16 moves roughly 140GB per token step. This makes decoding memory-bandwidth-bound, not compute-bound, which is why memory bandwidth often predicts token speed better than raw FLOPS.
• Batching: Serving requests one at a time wastes the GPU. Continuous batching packs many in-flight requests together so the expensive weight read is amortized across all of them, which raises throughput sharply. The tradeoff is that a very large batch can nudge up per-request ITL.
• GPU memory bandwidth and interconnect: Because decode is bandwidth-bound, the card's memory bandwidth sets a ceiling on token speed. For models too large to fit on one GPU, the interconnect between GPUs (and between nodes) becomes the bottleneck, since every token step exchanges activations across devices.
• Quantization and precision: Running weights in a lower precision such as FP8 or INT4 shrinks how many bytes move per token step, which directly lifts token throughput and cuts ITL, usually with a small and measurable accuracy cost.

Network sits underneath all of this and is easy to forget. If your inference endpoint is in one region and your users or your retrieval database are in another, cross-region round trips add straight to TTFT before the model does any work at all.

How to speed up AI inference

Speeding up inference is a stack of choices, not a single switch. Work through them in the order that gives the most return for the least risk.

1. Match precision to tolerance: Quantize to FP8 or INT4 where accuracy holds. This is often the largest single win for token speed because it reduces bytes moved per step.
2. Turn on continuous batching: A modern serving runtime that batches in flight will lift throughput several times over naive one-request-at-a-time serving.
3. Use KV caching and prefix reuse: Caching the attention state means you never recompute past tokens, and reusing shared prompt prefixes cuts prefill for repeated system prompts.
4. Right-size the model: A distilled or smaller model that meets quality bar will always decode faster than a larger one, because fewer weight bytes move per token.
5. Fix the placement: Put inference close to users and to your data, and use a high-bandwidth interconnect so multi-GPU models don't stall on activation exchange.
6. Remove virtualization overhead: A hypervisor layer between your workload and the GPU can quietly skim memory bandwidth, and bandwidth is exactly what decode speed depends on.

Speculative decoding is worth a mention as a further lever: a small draft model proposes several tokens that the large model verifies in one pass, which can cut effective ITL when the draft is accurate. It adds complexity, so treat it as a tuning step after the basics are in place.

Where the serving platform decides your ceiling

You can optimize the model perfectly and still lose speed to the infrastructure underneath it. Because decode is memory-bandwidth-bound and multi-GPU models depend on the interconnect, the platform sets a hard ceiling on how fast you can go.

GMI Cloud is an AI-native inference cloud built for production AI, and the stack is built around the two things that govern inference speed: bandwidth and placement. Bare metal GPU access runs with no hypervisor, so you get 100 percent of the advertised memory and network bandwidth rather than a virtualized slice, and decode speed tracks that bandwidth directly. RDMA-ready networking keeps activation exchange fast for models split across multiple GPUs and nodes, which protects ITL on large models. For placement, GMI Cloud runs GPU regions across North America, Europe, and Asia-Pacific with under 200ms average cross-region latency, so TTFT stays low for a global user base. Real-time inference workloads see this in practice: Higgsfield cut p95 latency by 65 percent while lowering compute cost by 45 percent on the platform.

The serving side matters as much as the hardware. GMI Cloud's Inference Engine offers Model-as-a-Service with continuous batching and scale-to-zero, so you get high throughput without paying for idle GPUs, and dedicated endpoints when you need predictable low latency. You can run the same model on serverless for bursty traffic and on bare metal for steady, latency-sensitive load without re-architecting. Current GPU options and rates are on the GMI Cloud pricing page, and the model catalog is at GMI Cloud models.

Measure first, then tune

Inference speed only improves when you know which number is hurting you. Instrument TTFT, ITL, throughput, and p99 separately before touching anything, then decide whether your workload is TTFT-bound or ITL-bound. Short-turn chat needs fast first tokens; long-form generation needs fast per-token decode. Match your precision, batching, and placement to that shape, run inference on hardware that gives you full bandwidth, and re-measure. Do it in that order and inference speed becomes a number you control instead of one you hope for.