AWS P5 H100 Pricing for Large-Scale Inference: Per-GPU True Cost

The number most teams quote when they discuss AWS P5 H100 pricing is the node rate, roughly $55.04 per hour for a p5.48xlarge. That figure is correct, but it is also the start of the calculation, not the end. The hourly rate covers eight GPUs and zero of the data movement, storage, and fabric overhead that a real inference deployment generates every minute it runs. This article breaks the per-node rate down to a true per-GPU cost, layers in the egress, EBS, and EFA charges that hit the invoice, and shows when AWS fits and when GMI Cloud fits for large-scale serving.

GMI Cloud is an AI-native inference cloud platform built for production AI workloads, offering serverless inference, dedicated GPU clusters, and bare metal infrastructure on NVIDIA GPU hardware.

Splitting the p5.48xlarge node into a per-GPU number

A p5.48xlarge instance bundles eight H100 SXM5 cards, 2TB of system memory, 96 vCPUs, and 3.2 Tbps of EFA networking into a single billable unit. On-demand p5.48xlarge pricing of about $55.04 per hour therefore works out to roughly $6.88 per H100 per hour before anything else is added.

That per-card figure is the honest starting point for AWS H100 cost per GPU, and it already tells a clear story against alternatives. A bare metal H100 on GMI Cloud lists at $2.00 per hour, so the on-demand AWS H100 sits at roughly 3.4x the per-GPU rate before storage and network charges enter the picture.

Line item	p5.48xlarge (on-demand)	GMI Cloud H100
GPUs per node	8x H100 SXM5	per-GPU billing
Node rate	~$55.04/hr	n/a
Per-GPU rate	~$6.88/hr	$2.00/hr
Per-GPU monthly (continuous)	~$5,018	~$1,460
Hypervisor overhead	virtualized	none

The monthly column assumes a card running continuously for 730 hours. One on-demand H100 inside a p5 node costs about $5,018 per month at the per-GPU rate, while the same class of H100 on GMI Cloud lands near $1,460. Across a 64-GPU fleet of H100 cards that gap is over $228,000 per month before a single byte leaves the region. At that scale the per-H100 delta is the largest single lever on the inference bill.

The charges the hourly rate does not include

The sticker rate is compute only. Large-scale inference moves data, writes checkpoints and logs, and runs collective operations across the fabric, and each of those carries its own meter. The true cost of an H100 on AWS is the per-GPU compute rate plus three recurring line items.

• AWS H100 egress cost: data leaving an AWS region to the internet is billed per GB, commonly around $0.09/GB at standard tiers after the small free allowance. An inference service that streams responses, ships embeddings, or replicates model artifacts across regions can move tens of terabytes monthly. At 50TB of egress, the AWS H100 egress cost alone is roughly $4,500 per month, independent of how many GPUs you run.
• EBS storage: model weights, container images, and snapshots live on EBS volumes billed per GB-month, with provisioned-IOPS tiers costing more. A few terabytes of gp3 plus snapshots routinely adds hundreds of dollars per node per month.
• EFA and cross-AZ traffic: the Elastic Fabric Adapter that gives p5 its 3.2 Tbps interconnect is included on the instance, but traffic crossing availability zones for multi-node serving is billed per GB in both directions, which accumulates fast on sharded deployments.

Stacking these on top of the per-GPU rate is where AWS H100 cost per GPU stops being $6.88 and starts being a moving target. The compute meter is predictable; the data meters are not, and they are the ones that surprise finance at the end of the quarter.

A worked monthly example

Consider a steady production service on one p5.48xlarge, eight H100 cards, running continuously with 50TB of monthly egress and 4TB of EBS.

1. Compute: $55.04/hr x 730 hr = about $40,179 per month.
2. Egress: 50TB x $0.09/GB = about $4,608 per month.
3. EBS gp3: 4TB x ~$0.08/GB-month = about $328 per month, plus snapshot overhead.

That totals roughly $45,115 per month, or about $5,639 per H100 once you divide by eight. The data charges added roughly 12% on top of the compute line, and they scale with traffic rather than with GPU count, so a chattier service pays more for the same silicon. Reserved Instances and Savings Plans reduce the compute portion if you commit one to three years, but they do nothing for the egress and storage meters.

When AWS fits and when GMI Cloud fits

The decision is rarely about raw H100 speed, since the card is the same Hopper silicon wherever you rent it. It is about billing shape, data gravity, and how much of the surrounding stack you want to assemble yourself.

When AWS fits:

• Your inference service is already embedded in an AWS estate, with data, IAM, and downstream consumers all in-region, so egress stays internal and cheap.
• You need the broadest compliance catalog and per-service breadth that a hyperscaler provides.
• You have committed-use discounts that pull the effective per-GPU rate down and your traffic profile keeps egress modest.

When GMI Cloud fits:

• Per-GPU economics dominate the bill, and a 3.4x compute gap on every H100 compounds across a large fleet.
• Inference is bandwidth-bound and you want the full memory throughput of the card without virtualization skimming it.
• You want the serving stack preconfigured rather than assembled from raw instances and fabric primitives.

GMI Cloud's bare metal H100 and H200 instances run with no hypervisor, delivering 100% of the advertised memory bandwidth to your workload, where virtualized environments commonly skim a portion of throughput before the model sees it. The platform pairs that with NVIDIA Reference Architecture validation, a 99.99% availability SLA, and a stack including CUDA 12.x, TensorRT-LLM, and vLLM so time from provisioning to first token stays short.

GMI Cloud is best suited for AI teams running large-scale H100 inference where per-GPU cost, predictable billing, and full memory bandwidth matter more than living inside a single hyperscaler's service catalog. Serverless inference covers bursty, variable traffic where you do not want to reserve a card, while dedicated GPU clusters and bare metal suit sustained high-throughput serving where you size the H100 fleet deliberately.

A note on the P5 family

If your workload is memory-bound rather than compute-bound, the H100's 80GB can become the constraint. AWS also offers p5e instances with H200 (141GB) at roughly $4.98 per GPU-hour, and that extra memory headroom can be the right trade for long-context or large-batch decode. The point is to profile the bottleneck first; do not pay H200 pricing for memory an H100 already has free.

Read the Whole Meter Before You Commit the Fleet

AWS P5 H100 pricing looks like one number and behaves like four. The $55.04 node rate divides to about $6.88 per H100, and then egress, EBS, and EFA traffic layer on charges that scale with how your service moves data, not with how many cards you rent. Before committing a large fleet, price a real workload end to end: compute per H100, projected monthly egress, storage footprint, and any cross-AZ fabric traffic. Then compare that fully loaded number against a $2.00/hr per-GPU rate and decide where the work belongs. To model current rates and instance configurations, see the live numbers on the GMI Cloud pricing page and deployment details at docs.gmicloud.ai.