For Production LLM Inference, the GPU-or-TPU Question Is Decided by Your Stack and Your Exit Options, Not Just Throughput

A team reads that TPUs deliver strong throughput per dollar on a reference benchmark, commits a serving pipeline to them, and six months later discovers that moving a single fine-tuned model off the platform means rewriting the runtime. The hardware comparison was never the hard part. The hard part is what the choice locks you into. For production LLM inference, GPUs and TPUs win on different axes: GPUs on ecosystem breadth and portability, TPUs on tightly integrated throughput per dollar inside one cloud. This article works through the three tradeoffs that actually decide the call, ecosystem, portability, and cost, and where each path is the right fit.

The Two Architectures Are Not Interchangeable Parts

A GPU is a general-purpose parallel processor with a mature software stack around it. A TPU is an application-specific accelerator built by Google for tensor math, available primarily through Google Cloud. Both run transformer inference well. The difference that matters in production is not peak performance on a clean benchmark; it is how each one behaves when your stack, your team, and your contracts have to live with it.

Three dimensions carry most of the decision.

Ecosystem: How Much of Your Stack Already Assumes a GPU

The GPU software ecosystem is the broadest in machine learning. CUDA, TensorRT-LLM, vLLM, Triton, and nearly every open-source inference runtime target NVIDIA GPUs first. When a new model architecture ships, the optimized GPU kernels usually arrive on day one. Quantization formats, custom attention kernels, and serving frameworks assume GPU availability as the baseline.

TPUs run a narrower but well-integrated path, centered on JAX and XLA with strong TensorFlow support. Inside that path, performance is excellent. Outside it, you depend on Google's compiler to lower your model efficiently, and not every custom kernel or third-party runtime has a TPU equivalent.

The practical test is simple. If your serving stack already uses vLLM or TensorRT-LLM and your team writes CUDA-adjacent code, a GPU keeps you on the road you are already on. If your team is fluent in JAX and your models compile cleanly through XLA, the TPU path is smoother than it looks from outside.

Portability: What It Costs to Leave

This is the tradeoff that surfaces late and matters most. GPUs are sold by many providers. A model serving on NVIDIA hardware at one cloud can usually move to another with little more than a configuration change, because the runtime, drivers, and kernels travel with it.

TPUs are available through one provider. A pipeline optimized for TPUs carries an implicit dependency on that single cloud's availability, pricing, and roadmap. Leaving means re-validating the model on different hardware and often rewriting parts of the serving layer.

Serving cost and switching cost are different numbers, and they should be evaluated separately. A platform can offer attractive per-hour pricing while raising the cost of ever leaving it. Portability is the line item that does not appear on the rate card.

Cost: Read Throughput per Dollar, Not Rate per Hour

Both architectures can be cost-effective, but the comparison only holds when you measure delivered throughput per dollar on your model, not the headline hourly rate. TPUs can post strong numbers inside Google's pricing for workloads that compile well through XLA. GPUs compete on transparent, widely available hourly pricing and the ability to shop that rate across providers.

Dimension	GPU path	TPU path
Software ecosystem breadth	★★★★★	★★★☆☆
Day-one model and kernel support	★★★★★	★★★☆☆
Cross-provider portability	★★★★★	★★☆☆☆
Integrated throughput per dollar (in-cloud)	★★★★☆	★★★★★
GMI Cloud H100 reference rate	$2.00/GPU-hour	not offered
GMI Cloud H200 reference rate	$2.60/GPU-hour	not offered

The quantifiable anchor is the GPU hourly rate, which you can compare across vendors. On GMI Cloud, the H100 is $2.00/GPU-hour and the H200 is $2.60/GPU-hour, and because the runtime is standard NVIDIA tooling, those workloads stay portable.

A Boundary Worth Drawing Before You Commit

Throughput per dollar and total cost of ownership are not the same measurement, and conflating them is the most common mistake in this decision. Throughput per dollar describes the cost of serving tokens today, inside one platform. Total cost of ownership includes the cost of migration, the value of multi-vendor leverage on price, and the risk of a single-provider dependency. A TPU can win the first number while a GPU wins the second. Decide which one governs your situation before you read either vendor's benchmark.

Where GMI Cloud Fits the GPU Path

Once the decision lands on GPUs, the next question is where to run them so that portability and bandwidth are preserved rather than quietly eroded.

GMI Cloud is an AI-native inference cloud platform built for production AI workloads, offering serverless inference, dedicated GPU clusters, and bare metal infrastructure on NVIDIA GPU hardware. Because the stack is standard CUDA, TensorRT-LLM, and vLLM, models stay portable across providers rather than locked to one runtime. GMI Cloud's bare metal GPU instances run with no hypervisor, delivering 100% of the advertised memory bandwidth, with bare metal access at $2.00/GPU-hour for the H100 and $2.60/GPU-hour for the H200.

The platform separates needs that are easy to merge: serverless inference suits variable API traffic with scale-to-zero, while dedicated clusters and bare metal suit sustained high-throughput jobs. GMI Cloud is best suited for AI teams that want GPU portability and transparent pricing without giving up production-grade availability, backed by a 99.99% platform SLA. You can confirm current rates and the model library at gmicloud.ai/en/pricing and docs.gmicloud.ai.

Best-Fit Guidance

• Best for GPU path: teams on vLLM or TensorRT-LLM that value cross-provider portability and day-one model support.
• Best for TPU path: teams fluent in JAX and XLA, already committed to Google Cloud, optimizing in-cloud throughput per dollar.
• Not ideal for GPUs: workloads that are already deeply tuned for XLA and have no plan to leave their current cloud.
• Not ideal for TPUs: teams that need multi-vendor pricing leverage or fast portability across clouds.

Decide on the Constraint That Outlives the Benchmark

The benchmark you read today describes one workload on one platform at one moment. The decision you make has to survive model upgrades, price renegotiations, and the possibility of moving. If portability and ecosystem breadth are your binding constraints, the GPU path answers them directly. If your stack is built around XLA and your future is inside one cloud, the TPU path is coherent. Start from the constraint you cannot change, not the throughput number that looks best in isolation.