How to Calculate the Cost of AI Inference: Formulas, Throughput, and a Worked Example

If you're trying to pin down the cost of AI inference, the GPU hourly rate is the wrong place to stop. An hourly rate tells you what an hour of hardware costs. It says nothing about how many requests or tokens that hour produces, and that ratio is where the real number lives. The cost of AI inference is calculated by dividing your all-in hourly cost by the useful work the GPU produces in that hour, then expressing it per token or per request. This guide walks through the two formulas that matter, the three inputs each one needs, and a full worked example you can copy for your own model.

The two units that matter: per request and per token

Almost every inference cost question resolves to one of two units. Pick the one that matches how your workload behaves before you calculate anything.

• Cost per request: Best for workloads where each call does a similar, bounded amount of work, such as an image classification, a short embedding, or a fixed-size completion. You care about the price of one call.
• Cost per token: Best for generative language models, where the work per call varies with how much text goes in and comes out. Cost per million tokens is the standard way this gets quoted.

The distinction matters because a per-request average hides variance. A chatbot that answers in 40 tokens and one that writes 800-token essays have wildly different costs per call on the same model. Tokens normalize that, which is why cost per million tokens is the honest metric for anything generative.

The core formula for the cost of AI inference

Whether you land on per request or per token, the shape of the calculation is the same. You need three inputs.

1. All-in hourly cost: The GPU rate plus a fair share of storage, data egress, and networking. Not just the headline number.
2. Throughput: How many units of work the GPU completes per hour at your batch size, sequence length, and model, measured as tokens per second or requests per second scaled to an hour.
3. Utilization: The fraction of paid hours the GPU is actually serving traffic rather than sitting idle.

From those three, the formula is:

Cost per unit = All-in hourly cost / (Throughput per hour x Utilization)

The utilization term is the one teams forget, and it's usually the biggest driver of the gap between the number you planned and the number you paid. A GPU rented by the hour bills whether it's working or idle, so effective throughput is always raw throughput multiplied by how busy the GPU stays.

Why throughput and utilization decide the real number

Two teams can rent the same GPU at the same rate and report inference costs that differ by 5x. The rate is identical, so the difference is entirely throughput and utilization.

Input	Team A	Team B	Effect on cost
GPU rate	$2.00/hr	$2.00/hr	None; identical
Throughput (tokens/sec)	2,500	1,200	Higher throughput lowers cost
Utilization	80%	25%	Higher utilization lowers cost
Effective tokens/hour	7.2M	1.08M	6.7x difference
Cost per million tokens	$0.28	$1.85	Same rate, very different cost

Team A and Team B pay the identical hourly rate and run the identical model class, yet Team A's cost per million tokens is roughly a sixth of Team B's. Throughput came from a larger batch size and a well-tuned serving stack; utilization came from steady traffic instead of a mostly idle reserved GPU. This is the single most important thing to understand about the cost of AI inference: the hourly rate is a minor variable next to how much useful work you extract from each paid hour.

A worked example: cost per million tokens on an H100

Numbers make this concrete. Assume you're serving a mid-sized language model on a single H100 and you want the cost per million output tokens.

Start with the inputs:

• GPU rate: $2.00 per GPU-hour.
• Add-ons: storage and egress allocated at roughly $0.20 per hour for this workload, so all-in hourly cost is $2.20.
• Throughput: at your batch size and sequence length, the server sustains 2,000 tokens per second. Over an hour that's 2,000 x 3,600 = 7,200,000 tokens per hour at full load.
• Utilization: traffic keeps the GPU busy 60 percent of the time it's reserved.

Now apply the formula:

1. Effective tokens per hour = 7,200,000 x 0.60 = 4,320,000 tokens.
2. Cost per token = $2.20 / 4,320,000 = $0.000000509.
3. Cost per million tokens = $0.000000509 x 1,000,000 = $0.51.

So this setup delivers roughly $0.51 per million tokens. Now watch what happens if utilization drops to 25 percent because traffic is bursty and the reserved GPU sits idle most of the day:

1. Effective tokens per hour = 7,200,000 x 0.25 = 1,800,000 tokens.
2. Cost per million tokens = $2.20 / 1,800,000 x 1,000,000 = $1.22.

Same GPU, same rate, same model. The cost per million tokens more than doubled purely because of idle time. If you'd quoted the cost of AI inference from the hourly rate alone, you'd have missed this entirely.

The per-request version of the calculation

For non-generative workloads, swap tokens for requests and the method is unchanged. Suppose the same H100 processes 40 image embeddings per second at your batch size:

• Requests per hour at full load = 40 x 3,600 = 144,000.
• At 60 percent utilization, effective requests per hour = 86,400.
• All-in hourly cost $2.20 / 86,400 = $0.0000255 per request, or about $2.55 per 100,000 requests.

The reason per-request math is cleaner here is that each embedding does a fixed amount of work. When work per call varies, go back to tokens.

Where serverless changes the calculation

Everything above assumes you rent a GPU by the hour and manage utilization yourself. There's a second path that removes the utilization variable from your side of the equation. With per-request serverless inference billed per token, the provider absorbs idle time by scaling to zero when no one calls your endpoint. You pay a published price per token or per request and skip the throughput and utilization math entirely, because the provider has already done it and priced it in.

This is why teams comparing options should calculate both ways:

• Rent by the hour when traffic is steady enough to keep utilization high. High utilization makes the per-hour cost per token very low, as the worked example showed at 60 percent.
• Pay per token serverless when traffic is bursty or uneven. A published per-token price beats a reserved GPU that bills through the idle hours.

The break-even point is the utilization level where your calculated per-hour cost per token equals the serverless per-token price. Below that utilization, serverless is cheaper; above it, renting wins.

Calculating the cost of AI inference with transparent pricing

The formulas only work if the inputs are real. A rate card with hidden egress or throttling breaks the all-in hourly cost term, and a per-token price you can't see makes the break-even comparison impossible. GMI Cloud is an AI-native inference cloud built for production AI, and it publishes transparent per-GPU-hour rates with no hidden fees, so the all-in hourly cost you plug into the formula matches the invoice.

NVIDIA GPU	GMI Cloud rate	Use in the calculation
H100	from $2.00/GPU-hour	Baseline for the worked example
H200	from $2.60/GPU-hour	Higher throughput can lower cost per token
B200	from $4.00/GPU-hour	Compare on delivered cost, not rate
GB200 NVL72	from $8.00/GPU-hour	Multi-GPU throughput for large models

GMI Cloud's two engines map onto the two calculation paths. The Cluster Engine covers per-hour rental through container, bare metal, and managed cluster options, with no hypervisor overhead so you get the full advertised throughput that goes into your tokens-per-hour figure. The Inference Engine covers Model-as-a-Service, where per-token billing does the utilization math for you and scale-to-zero means idle time costs nothing. Because MaaS bills per token, it's the fastest way to get a real cost of AI inference for a given model without running your own throughput benchmarks first. You can check current rates on the GMI Cloud pricing page or browse available models on the models page.

Run the numbers before you pick a rate

The cost of AI inference is a calculation, not a rate card lookup. Get your all-in hourly cost, measure real throughput at your batch size, be honest about utilization, and divide. Do that per token for generative models and per request for fixed-work calls. Then compare that delivered number against a published per-token price to decide whether to rent or go serverless. Quoted the raw hourly rate way, inference cost is a guess; quoted per million tokens with utilization counted, it's a number you can plan a budget around.