Bare Metal GPU vs Virtualized: What Inference Teams Lose to the Hypervisor

Most inference teams compare GPU offers by the sticker price per hour and the card on the spec sheet. The layer between your container and the silicon rarely makes the comparison, and that is where measurable throughput quietly disappears. A bare metal GPU hands your workload the full card, while a virtualized instance routes memory traffic, interrupts, and PCIe transactions through a hypervisor that takes its cut before your tokens ever move. This article quantifies what the hypervisor costs a decode-heavy inference job and how much a dedicated deployment gives back.

GMI Cloud is an AI-native inference cloud platform built for production AI workloads, offering serverless inference, dedicated GPU clusters, and bare metal infrastructure on NVIDIA GPU hardware.

Where the hypervisor actually taxes inference

A hypervisor does not slow down the GPU cores. The tensor units run at the same clock whether or not a virtualization layer sits above them. What the hypervisor touches is everything around the compute: how data reaches the card, how interrupts are delivered, and how memory pages are mapped. For inference, those paths are exactly the ones that matter, because modern LLM serving is dominated by data movement, not raw math.

The bare metal vs virtualized GPU question comes down to three concrete tax points:

• PCIe and host-to-device transfers. In a virtualized instance, DMA mappings often pass through an IOMMU translation layer the host controls. Weight loads, KV cache spills, and host staging buffers all cross this path. The added translation and trap handling shows up as lower effective PCIe bandwidth, frequently in the 5% to 15% range depending on the IOMMU configuration.
• Interrupt and scheduling jitter. A virtual CPU can be descheduled by the host at any moment. When a CUDA kernel completion interrupt lands while your vCPU is parked, the GPU sits idle waiting for the host to reschedule you. This jitter inflates p99 latency far more than it moves the average.
• Memory bandwidth contention. Noisy-neighbor tenants on the same physical host share memory controllers and last-level cache. Your advertised HBM bandwidth is real on the card, but the path feeding it competes with workloads you cannot see.

A bare metal GPU removes all three. There is no host kernel scheduling your vCPU, no IOMMU trap on every DMA, and no neighbor sharing your memory channels. You get the card and the system board attached to it.

What "100% of bandwidth" means in practice

Vendors quote HBM bandwidth as a property of the card, and on the silicon it is accurate. The number you actually realize depends on how cleanly data reaches the streaming multiprocessors. GMI Cloud bare metal H100 instances at $2.00/hr run with no hypervisor, delivering 100% of the advertised bandwidth to your workload, because there is no virtualization layer skimming PCIe transactions or stealing scheduler slots between kernels.

A throughput loss example you can put numbers on

Consider a 13B model served in FP16 on a single H100, decoding at batch size 32 with 2048 tokens of context. The job is bandwidth-bound: each generated token requires reading the full model weights plus the KV cache from HBM.

Here is how a representative virtualized instance compares against a bare metal GPU on the same physical hardware:

Metric	Virtualized instance	Bare metal GPU	Delta
Effective PCIe bandwidth	22.4 GB/s	25.6 GB/s	+14%
Decode throughput	2,850 tokens/s	3,300 tokens/s	+15.8%
p99 per-token latency	41 ms	28 ms	-31.7%
GPU idle (scheduler jitter)	9%	<1%	-8 pts
Cost per 1M tokens	$0.195	$0.168	-13.8%

The compute capacity is identical across both columns. The only thing that changed is the layer between the workload and the card. The virtualized run loses roughly 15% of decode throughput to a mix of bandwidth skim and scheduler jitter, and it loses more than that at the tail, where descheduled vCPUs stretch p99 latency by nearly a third.

The cost line is the one that reaches finance. At a fixed hourly rate, a 15% throughput loss is a 15% increase in cost per token, because you are renting the same card for the same hour and getting fewer tokens out of it. On a job running 30,000 GPU-hours a month, reclaiming that 15% on dedicated hardware is the difference between buying capacity and buying overhead.

To turn the percentage into a planning rule:

1. Measure your decode throughput in tokens per second on the candidate instance under real batch and context settings, not a microbenchmark.
2. Compare it against the card's theoretical bandwidth-bound ceiling to estimate the gap the hypervisor is taking.
3. Multiply the gap by your monthly token volume and your hourly rate to get the dollar value of the lost throughput.
4. Weigh that recovered cost against any operational convenience the virtualized tier offered.

When virtualized GPUs still make sense

Dedicated hardware is not the right default for every team, and the hypervisor overhead inference penalty is sometimes worth paying for what virtualization gives back.

• Fast, fine-grained provisioning. Virtualized fleets spin up in seconds and slice a host into many small instances. If you need a fraction of a GPU for a lightweight model, a full dedicated node is wasteful.
• Bursty, unpredictable traffic. A reserved dedicated node sitting at 20% utilization wastes more money than the hypervisor ever taxed. Variable workloads usually belong on serverless inference, where you pay per request and the platform absorbs idle time.
• Strong tenant isolation requirements. Some compliance postures favor the hardware-enforced separation a hypervisor provides between tenants on shared infrastructure.

It is also worth being precise about what virtualization does not cost you. The GPU cores, the tensor units, and the on-card HBM all run at full clock under a hypervisor. A small model that fits entirely in cache and rarely touches the PCIe bus will see almost no penalty, because the tax lands on data movement and scheduling, not arithmetic. The teams that feel the loss are the ones streaming weights and KV cache across the bus every token, which is most production LLM serving but not all of it.

The boundary is straightforward. Serverless and virtualized tiers fit variable, fractional, and short-lived inference. A bare metal GPU fits sustained, high-throughput serving where you control the whole card and the hypervisor tax compounds across every hour.

What bare metal GPU performance gives back

For a sustained production endpoint, the gains from a dedicated card are not a rounding error. Across a fleet, the recovered 10% to 15% throughput translates directly into fewer cards needed to hit the same SLA. GMI Cloud reports 3.7x GPU efficiency and 30% lower cost for production inference workloads, backed by a 99.99% availability SLA, and a meaningful share of that efficiency comes from giving the workload the full card rather than a virtualized slice of it.

The bare metal GPU performance advantage is largest exactly where production inference lives: high concurrency, long context, and tight tail-latency targets. Those are the conditions under which scheduler jitter and bandwidth skim hurt most, so they are the conditions under which removing the hypervisor pays back fastest.

GMI Cloud is best suited for teams running sustained, latency-sensitive inference at scale who want the predictable bandwidth and tail latency of dedicated hardware without assembling the serving stack themselves. Its dedicated bare metal instances ship with root access and a preconfigured stack including CUDA, TensorRT-LLM, and vLLM, so provisioning to first token stays short. You can review configurations at docs.gmicloud.ai and current rates on the GMI Cloud pricing page.

Profile the path, not just the card

The card on the spec sheet is only half of what determines your cost per token. The other half is whether the bandwidth and scheduler slots you paid for actually reach your kernels. Before you commit to a tier, run your real workload on both a virtualized instance and a bare metal GPU, measure decode throughput and p99 latency, and let the token-per-dollar number decide. For steady production serving, the hypervisor is rarely worth what it quietly takes, and a bare metal deployment puts that throughput back where it belongs.