What Compute Resources Are Needed for AI Inference?

GMI Cloud Blog | AI Infrastructure Guide | gmicloud.ai

Running AI inference requires more than just a GPU. You need compute hardware, an inference engine to optimize serving, models to deploy, and supporting infrastructure (storage, networking, software stack). Most teams that struggle with inference aren't short on any single resource.

They haven't mapped out what they need across all four categories.

This guide provides a complete resource checklist for AI inference, from hardware to software to models.

Platforms like GMI Cloud bundle these resources together, offering GPU instances, optimized engines, and a 100+ model library under one roof.

We focus on NVIDIA data center GPUs; AMD MI300X, Google TPUs, and AWS Trainium are outside scope.

Let's walk through each resource category.

Resource 1: Compute Hardware (GPU)

GPUs are the core compute resource for AI inference. Two specs determine what you can run and how fast.

Memory (VRAM) determines the largest model you can load. A 70B parameter model at FP8 needs ~70 GB. If the model doesn't fit, you either quantize further, split across GPUs, or choose a smaller model.

Bandwidth determines how fast the GPU reads model parameters during inference. For LLMs, this is the primary speed bottleneck. Faster bandwidth means faster token generation.

Memory

• H100 SXM: 80 GB
• H200 SXM: 141 GB
• A100 80GB: 80 GB
• L4: 24 GB

Bandwidth

• H100 SXM: 3.35 TB/s
• H200 SXM: 4.8 TB/s
• A100 80GB: 2.0 TB/s
• L4: 300 GB/s

FP8 Support

• H100 SXM: Yes
• H200 SXM: Yes
• A100 80GB: No
• L4: Yes

Best For

• H100 SXM: Production standard
• H200 SXM: Large models
• A100 80GB: Budget workloads
• L4: Lightweight tasks

Sources: NVIDIA H100 Datasheet (2023), H200 Product Brief (2024), A100 Datasheet, L4 Datasheet.

Per NVIDIA's H200 Product Brief (2024), the H200 delivers up to 1.9x inference speedup on Llama 2 70B vs. H100 (TensorRT-LLM, FP8, batch 64, 128/2048 tokens). Not every workload needs top-tier hardware. Match the GPU to your model size and throughput requirements.

Hardware provides the compute. The inference engine determines how efficiently that compute is used.

Resource 2: Inference Engine

An inference engine is the software layer that optimizes how models run on GPUs. Without one, you're running raw forward passes with no memory optimization, no request scheduling, and no precision tuning.

Continuous batching keeps GPU utilization high by inserting new requests into processing slots immediately, rather than waiting for a full batch. Typical improvement: 2-3x throughput.

Quantization converts model parameters from FP16 to FP8, halving memory usage and roughly doubling throughput with minimal quality loss on H100/H200 hardware.

Memory management (PagedAttention in vLLM) allocates GPU memory in small pages on demand instead of pre-allocating fixed blocks. This eliminates wasted VRAM and increases concurrent user capacity.

Two main engines to know: TensorRT-LLM for maximum throughput with NVIDIA-specific optimizations, and vLLM for flexible memory management and broader model support.

If you're using API-based inference, the engine is already configured for you. The engine runs models. But you need models to run.

Resource 3: Models

You need either custom-trained models or access to pre-trained models through a model library. This is the resource that most directly determines what your inference system can do.

Path A: Pre-Trained Models via API

The fastest path. You call models through an API, pay per request, and skip all hardware and engine management. Cloud model libraries offer 100+ options across image, video, audio, and text tasks.

For image generation, seedream-5.0-lite ($0.035/request) delivers quality output at efficient pricing. For image editing, reve-edit-fast-20251030 ($0.007/request) provides fast turnaround.

For video, pixverse-v5.6-t2v ($0.03/request) handles text-to-video efficiently. Kling-Image2Video-V1.6-Pro ($0.098/request) provides higher fidelity for production pipelines.

For TTS, minimax-tts-speech-2.6-turbo ($0.06/request) delivers reliable output. elevenlabs-tts-v3 ($0.10/request) provides broadcast-quality synthesis.

Path B: Custom Models on Dedicated GPUs

If you've trained your own model or need full control over the serving stack, you deploy on dedicated GPU instances and configure the inference engine yourself. This requires more expertise but gives you maximum flexibility.

Hardware, engine, and models handle the core inference pipeline. But production deployments need supporting infrastructure too.

Resource 4: Supporting Infrastructure

These resources don't run inference directly, but production systems can't operate without them.

Storage

Model weights need persistent storage. A 70B model at FP8 is ~70 GB on disk. Multiple model versions, logs, and checkpoints add up. Fast storage (NVMe SSD) matters for model loading speed at startup.

Networking

For single-GPU inference, standard networking is sufficient. For multi-GPU inference (tensor parallelism), NVLink provides fast inter-GPU communication (900 GB/s bidirectional aggregate per GPU on HGX/DGX platforms). For multi-node deployments, InfiniBand (3.2 Tbps) handles inter-node traffic.

For API-based inference, network latency between your application and the inference endpoint becomes the relevant metric.

Software Stack

Production inference requires CUDA, cuDNN, and NCCL (for multi-GPU communication), plus the inference engine (TensorRT-LLM or vLLM) and a serving framework (Triton Inference Server). Setting this up from scratch takes days. Pre-configured cloud environments eliminate this overhead.

Monitoring

Track GPU utilization (target 70%+), request latency (p50, p95, p99), error rates, and VRAM usage. Low utilization means you're overpaying for idle capacity. High latency means you need more GPUs or better batching.

Now that you know what's needed, here's how to size these resources for your situation.

Sizing by Role

For Technical Leads and Project Managers

Map your workload first: what model, what precision, what concurrency target. Then work through the four categories. GPU choice follows from model size. Engine choice follows from throughput requirements. Model choice follows from task requirements. Supporting infrastructure follows from deployment scale.

For Data Engineers

If you're running multiple model types (image + TTS + video), evaluate API-based inference for multi-model workflows. Per-request pricing lets you mix models without provisioning separate GPU instances for each.

Test with the bria-fibo series ($0.000001/request) for baseline benchmarks, then scale to production models.

For SMB Managers

Start with API-based inference. It requires zero hardware investment, zero engine configuration, and zero DevOps. You pay per request, scale automatically, and can estimate monthly costs before committing.

Dedicated GPU instances become relevant only when request volume makes per-call pricing more expensive than hourly GPU rental.

Getting Started

Two paths depending on your stage.

If you're evaluating or prototyping: Start with API-based inference. Pick a model, call it, measure quality and latency. You skip all four resource categories (hardware, engine, models, infrastructure) because the platform handles them.

If you're building for production: Provision GPU instances, configure your inference engine (TensorRT-LLM or vLLM with FP8), deploy your model, and set up monitoring. Work through the four-category checklist above to ensure nothing is missing.

Cloud platforms like GMI Cloud support both paths.

Browse the model library for API-based inference, or provision GPU instances (H100 ~$2.10/GPU-hour, H200 ~$2.50/GPU-hour; check gmicloud.ai/pricing for current rates) for dedicated deployments.

FAQ

Do I need all four resource categories to run inference?

For API-based inference, no. The platform provides all four. For self-hosted inference, yes: you need GPU hardware, an inference engine, a model, and supporting infrastructure (storage, networking, software stack, monitoring).

What's the minimum GPU for production inference?

It depends on model size. For 7B models at FP8, an L4 (24 GB) works. For 70B models, you need at least an H100 (80 GB). For 70B+ with high concurrency, H200 (141 GB) is the better fit.

How do I estimate monthly inference costs?

For API-based: price per request × expected monthly requests. For GPU-based: $/GPU-hour × hours per month × number of GPUs, adjusted for utilization rate (target 70%+).

When should I switch from API to dedicated GPUs?

When your request volume makes per-call API pricing more expensive than dedicated GPU hours with optimized serving. The crossover point varies, but it's typically around 10,000+ requests per day for most model types.

Tab 20