What Is AI Inference and Why Does It Matter for AI Applications?

GMI Cloud Blog | AI Infrastructure Guide | gmicloud.ai

AI inference is the process where a trained model takes new input and produces an output. Every time you prompt a chatbot, generate an image, or get a real-time translation, that's inference at work.

It's the moment AI stops learning and starts delivering value, and it's also where most of your compute budget goes.

That's exactly why platforms like GMI Cloud (gmicloud.ai) exist: to give you optimized GPU infrastructure purpose-built for inference workloads, so you can focus on building your application instead of managing hardware.

But knowing the definition isn't enough. To build AI apps that are fast, affordable, and scalable, you also need to understand what hardware powers inference and how to match your workload to the right GPU. This guide covers all of that, focusing on NVIDIA's data center lineup (H100, H200, A100, L4).

AMD MI300X, Google TPUs, and AWS Trainium are outside scope.

How AI Inference Actually Works

During training, a model adjusts millions (or billions) of parameters by processing massive datasets. That's the learning phase. Inference flips the script: the parameters are frozen, and the model uses them to process new, unseen inputs.

For a large language model, that means reading your prompt, running it through dozens of transformer layers, and predicting the next token, one at a time, until the response is complete.

Here's the thing: that token-by-token generation is what makes inference tricky from a hardware perspective. Each token requires reading the model's entire weight matrix from GPU memory.

For a 70B parameter model at FP8 precision, that's 70 GB of data the GPU needs to move through its memory bus on every single forward pass. This is why memory bandwidth, not raw compute, is usually the bottleneck for LLM inference. And it's why picking the right GPU matters so much.

Training vs. Inference: Different Jobs, Different Hardware Needs

Inference and training place completely different demands on hardware. The table below breaks down the key differences, showing why you can't just reuse your training setup for inference.

(Training / Inference)

• Goal - Training: Learn patterns from data - Inference: Apply learned patterns to new inputs
• Compute Pattern - Training: Massive parallel, batch-heavy - Inference: Latency-sensitive, often real-time
• Duration - Training: Days to weeks - Inference: Milliseconds to seconds per request
• Key Bottleneck - Training: FLOPS (raw compute power) - Inference: Memory bandwidth + VRAM capacity
• Cost Profile - Training: One-time (or periodic retraining) - Inference: Ongoing, scales with user traffic

The bottom line: training is a sprint you run once (or occasionally). Inference is a marathon that runs every day, for every user, as long as your application is live. For most production AI applications, inference accounts for 80-90% of total compute spend over the model's lifetime.

That's why optimizing your inference infrastructure isn't optional. It's the single biggest lever you have for controlling costs.

The Three Constraints That Define Inference Performance

So if inference is where your budget goes, what should you optimize? It comes down to three constraints that are always in tension. Every GPU choice, every precision format, every serving framework is a tradeoff among these three.

Latency. Users don't wait. If your chatbot takes 5 seconds to start generating a response, you'll lose them. For LLM inference, the metrics that matter are time-to-first-token (TTFT) and tokens-per-second (TPS).

Both depend heavily on memory bandwidth, which controls how fast the GPU feeds data to its compute cores. The H200's 4.8 TB/s bandwidth (source: NVIDIA H200 Product Brief, 2024), for example, is why it generates tokens faster than the H100 at 3.35 TB/s on the same model.

Throughput. Latency tells you how fast one user gets a response. Throughput tells you how many users you can serve at once. If you're handling 1,000 concurrent requests, you need enough VRAM to batch them together. Larger VRAM means higher concurrency, better utilization, and lower cost per request.

This is where the H200's 141 GB advantage over the H100's 80 GB compounds.

Cost. Inference runs 24/7. A 10% efficiency improvement compounds into significant savings over months. Choosing the right GPU, the right precision format (FP8 vs. FP16), and the right serving framework can cut your per-token cost in half.

On GMI Cloud, H100 instances start at approximately $2.10/GPU-hour and H200 at approximately $2.50/GPU-hour (check gmicloud.ai/pricing for current rates), so even small efficiency gains translate to real dollar savings.

The GPU Lineup for AI Inference: Specs Compared

Now that you know what to optimize, let's look at the hardware. Here's how NVIDIA's data center GPUs stack up, starting with the H100 and H200 that dominate production inference.

Architecture

• H100 SXM: Hopper
• H200 SXM: Hopper
• A100 80GB: Ampere
• L4: Ada Lovelace

VRAM

• H100 SXM: 80 GB HBM3
• H200 SXM: 141 GB HBM3e
• A100 80GB: 80 GB HBM2e
• L4: 24 GB GDDR6

Memory BW

• H100 SXM: 3.35 TB/s
• H200 SXM: 4.8 TB/s
• A100 80GB: 2.0 TB/s
• L4: 300 GB/s

FP8

• H100 SXM: 1,979 TFLOPS
• H200 SXM: 1,979 TFLOPS
• A100 80GB: N/A
• L4: 242 TOPS

INT8

• H100 SXM: 3,958 TOPS
• H200 SXM: 3,958 TOPS
• A100 80GB: 624 TOPS
• L4: 485 TOPS

NVLink

• H100 SXM: 900 GB/s*
• H200 SXM: 900 GB/s*
• A100 80GB: 600 GB/s
• L4: None (PCIe)

TDP

• H100 SXM: 700W
• H200 SXM: 700W
• A100 80GB: 400W
• L4: 72W

*NVLink 4.0: 900 GB/s bidirectional aggregate per GPU on HGX/DGX platforms. Sources: NVIDIA H100 Datasheet (2023), H200 Product Brief (2024), A100 Datasheet, L4 Datasheet.

H100 SXM is the production inference workhorse. In MLPerf Inference v3.1, H100-based systems were the most widely submitted data center platform for LLM and image generation tasks (source: mlcommons.org/benchmarks/inference-datacenter).

With 80 GB HBM3, native FP8 via the Transformer Engine, and MIG support (up to 7 isolated instances), it handles 7B to 70B models on a single card. If you're deploying your first production model, the H100 is the proven starting point.

H200 SXM shares the same Hopper compute cores as the H100, so raw FLOPS are identical. The upgrade is all about memory: 141 GB HBM3e at 4.8 TB/s, giving you 76% more VRAM and 43% more bandwidth. Per NVIDIA's official H200 Product Brief (2024), this translates to up to 1.9x inference speedup on Llama 2 70B vs.

the H100 (tested with TensorRT-LLM, FP8, batch 64, 128/2048 tokens). Independent cloud tests confirm 1.4-1.6x in typical workloads. If you've outgrown the H100's 80 GB or you're running 70B+ models with long context, the H200 is the natural next step.

A100 80GB is the budget-friendly option for teams on Ampere hardware. It handles 7B-34B models at FP16/INT8, though it lacks native FP8. At 2.0 TB/s bandwidth, decode-heavy workloads run slower than on Hopper. Still, if cost is your primary constraint and your models fit, it's a solid choice.

L4 is NVIDIA's entry-level inference card: 24 GB GDDR6, 72W TDP, PCIe only. It runs 7B models at INT8/INT4, making it a fit for students or early-stage builders on a budget. But 24 GB VRAM limits what you can serve, and there's no NVLink for multi-GPU scaling.

To know whether your model fits on an L4 or needs something bigger, you'll want to understand VRAM sizing.

How to Size Your GPU: A Quick VRAM Framework

Before you commit to a GPU, run this quick VRAM budget. It takes two minutes and saves you from over-provisioning (wasting money) or under-provisioning (OOM errors in production).

Step 1: Model weights. A 70B parameter model at FP16 needs roughly 140 GB (2 bytes per parameter). At FP8, that drops to 70 GB. This tells you whether a given GPU can even load the model.

Step 2: KV-cache. On top of weights, the model stores key-value pairs for every token in the sequence during inference. The formula: KV per request = 2 x num_layers x num_kv_heads x head_dim x seq_len x bytes_per_element.

For Llama 2 70B (80 layers, 8 KV heads, 128 head_dim) at FP16 with 4K context, that's roughly 0.4 GB per concurrent request. Multiply by your target concurrency to get total KV-cache demand.

Step 3: Check the fit. Add weights + total KV-cache + 10-15% overhead for the serving framework. If it doesn't fit, quantize further (FP8, INT4) or upgrade to more VRAM.

This is why the H200's 141 GB is so valuable: Llama 2 70B at FP8 (70 GB weights) leaves 60+ GB for KV-cache, meaning higher concurrency and better cost-per-request. Now that you can size your workload, here's how to pick your GPU.

Which GPU Should You Start With?

Use this table to match your situation to a starting point.

Your Situation (Start Here)

• 7B-70B models, FP8, latency-sensitive, need MIG isolation - Start Here: H100 SXM
• 70B+ models, long context (8K-128K), decode-bound - Start Here: H200 SXM
• 7B-34B models, existing Ampere fleet, cost priority - Start Here: A100 80GB
• 7B INT8/INT4, student projects, budget exploration - Start Here: L4

One thing to watch out for: consumer GPUs like the RTX 4090/5090 are fine for development, but NVIDIA's GeForce EULA contains data center use restrictions (see nvidia.com/en-us/drivers/geforce-license). Using them in production creates compliance risk.

Once you've picked a data center GPU, the next question is where to run it.

Running Inference on GMI Cloud

GMI Cloud (gmicloud.ai) delivers H100 SXM and H200 SXM instances on-demand, so you don't need to buy, rack, or manage hardware. Each node comes with 8 GPUs connected via NVLink 4.0 (900 GB/s bidirectional aggregate per GPU on HGX/DGX platforms) and 3.2 Tbps InfiniBand for inter-node communication.

The stack comes pre-configured: CUDA 12.x, cuDNN, NCCL (tuned for cluster topology), TensorRT-LLM, vLLM, and Triton Inference Server. That means you can go from zero to running optimized inference in minutes, not days.

Pricing starts at approximately $2.10/GPU-hour for H100 and $2.50/GPU-hour for H200 (check gmicloud.ai/pricing for current rates).

On the flip side, if you're a student running a 7B model for a class project, GMI Cloud's on-demand model means you can spin up a single GPU for a few hours and pay under $10. If you're a startup scaling to production, reserved instances lower the cost further.

And if you're a researcher who needs H200 access without a six-figure capex commitment, cloud GPUs are the practical path. Whatever your starting point, the right inference infrastructure turns your model from a prototype into a product.

FAQ

Can I run AI inference on a CPU?

Technically, yes. But CPUs lack the parallelism and memory bandwidth GPUs provide. For anything beyond toy models or very low-traffic demos, a GPU is essential.

What's the difference between FP16, FP8, and INT8 for inference?

These are precision formats that control how many bytes each parameter uses. FP16 is the default (2 bytes). FP8 halves memory with minimal quality loss on H100/H200/L4. INT8 is widely supported including on A100. Lower precision means less VRAM and faster inference, but always validate quality on your specific task.

How much does inference cost in production?

It depends on model size, traffic, and GPU choice. A practical framework: take your $/GPU-hour rate, multiply by GPUs needed for your targets, and project monthly. On GMI Cloud, H100s start at around $2.10/GPU-hour. Check gmicloud.ai/pricing for current rates.

Tab 2