Serverless GPU Inference vs Dedicated Endpoints: When to Use Which

Serverless inference and dedicated endpoints run the same models on the same GPUs, but they bill, scale, and behave very differently under load. The right choice depends on how steady your traffic is, how strict your latency targets are, and how much control you need over the model.

• Billing model sets the baseline: Serverless inference charges per token with no idle cost, while a dedicated endpoint bills for reserved GPU time whether requests arrive or not.

• Utilization decides the crossover: Once a dedicated GPU stays busy roughly 40 to 50% of the time or more, per-GPU-hour pricing usually beats per-token pricing for the same model.

• Cold starts are the hidden serverless tax: Loading a 70B-class model into GPUs can take tens of seconds or more, so scale-to-zero designs need warm capacity for latency-sensitive traffic.

• Control favors dedicated endpoints: Fine-tuned or proprietary weights, no rate limits, workload isolation, and stable p95 latency are the main reasons teams move off shared serverless APIs.

• Most production stacks use both: GMI Cloud runs serverless and dedicated endpoints on one OpenAI-compatible inference engine, so teams can start serverless and move steady traffic to dedicated GPUs without re-architecting. With inference expected to make up about two-thirds of all AI compute in 2026, this choice now drives a large share of the AI infrastructure bill.

Deloitte predicts that inference will account for roughly two-thirds of all AI compute in 2026, up from about half in 2025 and a third in 2023. As running models overtakes training as the main AI workload, the way a team serves those models has a direct effect on cost, latency, and reliability.

The first serving decision most teams face is whether to call a serverless inference API or to deploy a dedicated inference endpoint. Serverless gets a model into production in minutes and charges only for what you use. Dedicated endpoints reserve GPUs for your model and trade some flexibility for predictable performance and control. Neither is better in general. Each fits a different traffic shape.

This guide compares dedicated vs serverless inference across cost, latency, cold starts, isolation, and operational effort, then gives a practical framework for choosing between them and for running both together on GMI Cloud.

What Serverless Inference and Dedicated Endpoints Mean

Serverless Inference: Shared Models, Per-Token Billing

Serverless inference gives you access to a model through an API without provisioning any GPUs. The provider deploys the model, manages capacity, and scales it with demand across many customers. You send requests and pay per token or per request, with no charge for idle time.

On GMI Cloud, serverless endpoints are pre-configured, OpenAI-compatible APIs for popular models. They need zero setup beyond an API key, autoscale with demand, and bill per token with no idle compute charges. GMI recommends them for prototypes, small apps, variable workloads, and any team that would rather not run infrastructure.

Dedicated Endpoints: Reserved GPUs for Your Model

A dedicated inference endpoint runs a model on GPU capacity reserved for you alone. You choose the model, the GPU type, the region, and the scaling policy, and you pay for the GPU time the endpoint is running, regardless of how many requests it serves.

GMI Cloud’s dedicated endpoints let teams deploy their own fine-tuned or proprietary weights, or an open-source model from the catalog, on dedicated GPUs. They offer full control over hardware, scaling, and isolation, predictable throughput and latency, and no API rate limits. Billing applies only while a deployment is in the Running state, and a stopped deployment can be restarted later.

Two Kinds of Serverless GPU

The term serverless GPU covers two different products, and the difference matters for latency. The first is a managed model API, where the provider keeps popular models deployed and you only send tokens. The second is a serverless container platform, where you bring your own model and the platform starts GPU workers on demand, then scales them to zero when traffic stops. The second kind gives you more control over the model, but it exposes you directly to cold starts, because every new worker has to load your weights before it can answer.

Cost: Where the Crossover Sits

Per-Token Versus Per-GPU-Hour Pricing

Serverless pricing tracks usage exactly. If your application processes no tokens overnight, the bill for those hours is zero. Per-token rates also carry time-limited promotions in the console. In October 2026, for example, DeepSeek V4.1 Flash was listed at 40% off, at $0.18 per million input tokens instead of $0.30. Dedicated pricing tracks time. On GMI Cloud, a dedicated endpoint created from the console runs on one H200 GPU with 10 CPUs and 100 GB of memory at $4.50 per hour per replica (GMI Cloud console, as of October 2026), so a single replica running around the clock for a 730-hour month costs about $3,285. Raw GPU compute without the managed inference layer is listed separately, from $2.60 per GPU-hour for an H100 and $3.20 for an H200 on the GMI Cloud pricing page. Check the GMI Cloud pricing page for current rates before modeling your own numbers. Rates can change with promotions, and GMI Cloud also offers commitment-based pricing for reserved capacity, so confirm the figure that applies to you in the GMI Cloud console before you budget.

To compare the two models fairly, convert the GPU-hour rate into a cost per million tokens. GMI Cloud’s price-to-performance formula does this by dividing the hourly price by sustained tokens per second.

cost per 1M tokens = (GPU price per hour ÷ tokens per second) × (1,000,000 ÷ 3,600)

Utilization Is the Variable That Decides

The formula assumes the GPU is busy. In practice, a dedicated endpoint is only as cheap as its average utilization. The table below is an illustrative worksheet, not a quoted price. It assumes a dedicated H200 endpoint at $4.50 per hour (the October 2026 console rate) serving a model that reaches 3,000 tokens per second at full load with batching, and shows how the effective cost per million tokens changes as average utilization falls.

If a serverless API offered the same model at a blended $0.80 per million tokens, the dedicated endpoint would become cheaper once average utilization passed about 52%. Below that line, you are paying for idle GPU time that serverless pricing would never charge. This matches the guidance in GMI Cloud’s GPU pricing comparison: Above roughly 50% GPU utilization, per-GPU-hour pricing is typically cheaper, and below it, per-token pricing avoids paying for idle capacity. Plug in your own throughput, model price, and traffic profile, since the crossover moves with every variable.

Costs That Sit Outside the Rate Card

Rate cards leave out several costs that change the comparison. A dedicated endpoint sized for peak traffic sits partly idle at every other hour, so a spiky workload can run at low average utilization even when its peaks are heavy. Running your own serving stack also takes engineering time for capacity planning, upgrades, and monitoring, which per-token pricing absorbs into the rate.

Serverless has hidden costs too. On container-based serverless platforms that bill per GPU-second, a cold start is metered time. If a request needs 5 seconds of inference but waits 30 seconds for the model to load, about 86% of the billed GPU time goes to loading rather than serving. Managed model APIs avoid that particular charge, but they can add rate limit handling and retries to your application logic at high volume.

Latency and Cold Starts

Why Cold Starts Hit LLMs Harder

A cold start happens when a request arrives and no warm GPU worker holds the model. For a normal serverless function, that costs a few hundred milliseconds. For an LLM, the worker has to fetch and load gigabytes of weights before it can produce a single token.

Research on ServerlessLLM measured the scale of the problem. Downloading a 130GB checkpoint such as LLaMA-2-70B from object storage takes at least 26 seconds even on a fast 5GB/s network, and loading LLaMA-2-70B into 8 GPUs with PyTorch took 84 seconds, against less than 100 milliseconds to generate a token once the model is warm. Platform tuning helps but does not erase the gap. In an August 2026 test on two H200 GPUs, RunPod cut the median cold start for a 32B FP8 Qwen model from 324 seconds to about 91 seconds by caching compiled artifacts, prefetching weights, and limiting CUDA graph capture.

This is why scale-to-zero suits batch jobs, internal tools, and background tasks, but not chat interfaces or agent loops where a user is waiting. For catalog models on a managed serverless inference API, the provider keeps the model deployed, so you skip the weight loading step. For your own model on serverless containers, plan for cold starts explicitly.

Tail Latency and Noisy Neighbors

Average latency rarely tells the full story. Shared serverless pools serve many customers at once, and when another tenant’s traffic surges, queueing and batching decisions can push your p95 and p99 latency up even when your own traffic is flat. GMI Cloud’s serverless layer uses built-in batching, latency-aware scheduling, and multi-tenant isolation to keep this predictable, but a dedicated endpoint removes the shared factor entirely.

That difference shows up in production results. Higgsfield, which runs real-time generative video workloads on GMI Cloud, reports 65% lower p95 inference latency, 45% lower compute cost, and a 99.9% request success rate under peak traffic. For user-facing products with strict latency targets, predictable tail latency is often worth more than the lowest average price.

Keeping Dedicated Endpoints Warm

Dedicated endpoints are not immune to cold starts. When an autoscaling policy adds a replica, that replica still has to load the model. Set a minimum replica count that covers your baseline traffic so the first requests of the day never hit an empty endpoint, and scale up ahead of predictable peaks rather than in reaction to them. On GMI Cloud, new deployments move through Queued and Deploying states before Running, and billing starts only once the endpoint is Running, so you can budget warm capacity precisely.

Control, Isolation, and Compliance

Custom and Fine-Tuned Models

Serverless inference APIs serve what is in the provider’s catalog. If your product depends on a fine-tuned checkpoint, a distilled model, or proprietary weights, a dedicated endpoint is usually the only way to serve it with production-grade performance. It also lets you pin a model version, so an upstream update cannot change your outputs without warning.

Rate Limits and Throughput Guarantees

Shared APIs protect every tenant by applying rate limits. That is fine for moderate traffic, but high-volume pipelines, large batch jobs, and agents that fan out into many parallel calls can hit those limits at the worst moment. GMI Cloud’s dedicated endpoints carry no API rate limits, so throughput is bounded by the GPUs you provision rather than by a shared quota.

Data Isolation and Workload Separation

Some workloads carry data that compliance teams want kept on single-tenant infrastructure, in a chosen region, with a clear record of which hardware processed it. A dedicated endpoint gives you that separation and lets you pick the deployment region. Serverless APIs on reputable platforms isolate tenants logically, but if your policies require physical separation of compute, dedicated capacity is the simpler answer to defend.

A Decision Framework: When to Use Which

Choose Serverless Inference When

Serverless inference is the right default when traffic is unpredictable, low in volume, or still being discovered. It fits prototypes, new features searching for product-market fit, internal tools used a few hours a day, and workloads that spike for short periods and then go quiet. It also suits teams that want to test several models before committing, since switching models means changing one field in the request rather than redeploying anything.

Choose a Dedicated Endpoint When

A dedicated endpoint earns its cost when traffic is steady enough to keep GPUs busy, when latency targets are strict at the tail, or when you need control the shared tier cannot give. The clearest signals are sustained utilization above roughly 50%, a fine-tuned or proprietary model, rate limits that start shaping your architecture, a p95 target your users notice, and compliance requirements for single-tenant compute. Eigen AI, for example, runs dedicated endpoints on GMI Cloud for its production workloads, alongside flexible model access for evaluation.

Run Both: The Hybrid Pattern

Many production systems settle on a hybrid. Steady baseline traffic runs on a dedicated endpoint sized for average load, and overflow above that baseline spills to a serverless inference endpoint for the same or an equivalent model. The dedicated tier keeps utilization high and latency stable, while the serverless tier absorbs peaks without a team having to provision for the worst hour of the month.

The hybrid pattern also works across model sizes. A team might serve its main fine-tuned model on dedicated GPUs and route classification, summarization, or long-tail requests to serverless models. A routing layer such as GMI Router can choose the most cost-efficient serverless model for each of those requests automatically.

Moving Between the Two on GMI Cloud

Because GMI Cloud runs serverless and dedicated endpoints on one inference engine with OpenAI-compatible APIs, moving traffic between them is mostly a configuration change. You start on a serverless model, measure real traffic, and when the numbers justify it, create a dedicated deployment from the console by selecting the model, region, GPU type, and autoscaling policy. The client code stays the same, apart from the base_url and model values.

from openai import OpenAI

# Serverless inference endpoint
serverless = OpenAI(api_key="YOUR_GMI_API_KEY", base_url="https://api.gmi-serving.com/v1")

# Dedicated endpoint, using the URL shown for your deployment in the console
dedicated = OpenAI(api_key="YOUR_GMI_API_KEY", base_url="YOUR_DEDICATED_ENDPOINT_URL")

response = serverless.chat.completions.create(
    model="deepseek-ai/DeepSeek-V4.1-Flash",
    messages=[{"role": "user", "content": "Summarize this support ticket in one sentence."}]
)

The inference documentation covers both endpoint types, and the model catalog lists the models available as serverless APIs and for dedicated deployment. When workloads outgrow a single endpoint, the same platform offers dedicated GPU clusters and bare metal built on NVIDIA H100, H200, and Blackwell systems.

Conclusion

The choice between serverless GPU inference and dedicated endpoints comes down to traffic shape. Serverless inference wins when demand is bursty, low, or still uncertain, because you never pay for idle GPUs and you can switch models freely. Dedicated endpoints win when utilization is steady, tail latency matters, rate limits get in the way, or the model is your own. Most mature systems end up with both, using dedicated capacity for the baseline and serverless for peaks and long-tail tasks. GMI Cloud supports that full path on one platform, from per-token serverless APIs to dedicated endpoints and GPU clusters, so teams can start small and move each workload to the right serving model as the numbers come in.