Cluster Capacity Planning for AI Teams: How Much Compute You Really Need

Cluster capacity planning for AI teams comes down to three numbers: The GPU-hours your training runs need, the GPUs your inference traffic needs at peak, and the headroom you add for failures and growth. Each can be estimated from a few inputs before you sign anything, which is far cheaper than discovering the answer in a half-idle cluster.

• Training compute follows a simple formula: Total training FLOPs are about 6 × parameters × tokens, which predicts the 3.8 × 10²⁵ FLOPs Meta reported for Llama 3 405B on 15.6 trillion tokens to within about 1%.

• Efficiency turns FLOPs into GPU-hours: At 40% model FLOPs utilization on H100s rated at 989 dense BF16 TFLOPS, pretraining an 8B model on 1 trillion tokens needs about 33,700 GPU-hours, or about 5.5 days on 256 GPUs.

• Inference is sized by memory first, then throughput: Llama 3 70B needs about 0.33 MB of BF16 KV cache per token, so 32 concurrent requests at 8,192 tokens add roughly 86 GB on top of 140 GB of weights.

• Most clusters are oversized, not undersized: Cast AI measured average GPU utilization of 5% across roughly 23,000 Kubernetes clusters in 2026, while optimized environments reach about 50%. A plan sized to measured demand avoids paying for capacity that never runs.

• Price the plan at today’s rates: On GMI Cloud, H100 capacity starts at $2.60 and H200 at $3.20 per GPU-hour (as of October 2026), which puts the 8B pretraining example at about $87,600 before reliability overhead. Capacity planning is how a team spends that budget on results instead of idle GPUs.

AI teams rarely run short of GPUs because they bought too few. More often they bought the wrong shape, at the wrong time, for a workload they had not measured. Cast AI’s 2026 State of Kubernetes Optimization Report found GPU utilization averaging just 5% across the clusters it analysed, and its authors point to about 50% as a realistic level for optimized production systems. The gap between those two numbers is mostly a planning problem.

Capacity planning for AI clusters is different from planning for web services. Training is a batch job measured in GPU-hours with a deadline. Inference is a service measured in tokens per second with a latency target and a daily traffic curve. Both depend on GPU memory before they depend on compute, and both need headroom for failures that become routine at scale.

This guide gives a working method for each part: Estimating training compute from model size and tokens, sizing inference from memory and measured throughput, adding the right amount of headroom, and turning the result into a mix of reserved and on-demand capacity. Every formula comes with a worked example you can rerun with your own numbers.

Start with the Workload Inventory

A capacity plan starts with a list, not a GPU count. Write down every workload the cluster must serve over the planning period, usually the next one or two quarters, and classify each one. Pretraining and continued pretraining are long, synchronous jobs where the whole cluster works on one model. Fine-tuning and evaluation are shorter jobs that come in bursts. Inference is continuous and follows user traffic. Development and experimentation are small, interactive, and hard to predict.

For each workload, record the model size, the precision you will run it in, the data volume or traffic level, the deadline or latency target, and how certain each number is. That last column matters. A committed product launch with known traffic deserves reserved capacity; an exploratory research idea deserves on-demand capacity you can release.

Sizing Training Compute

Estimate Total FLOPs

For dense transformer models, training compute is well approximated by six floating point operations per parameter per token, covering the forward and backward passes. The approximation comes from OpenAI’s scaling laws paper, which writes training compute as about 6 × parameters × batch size × steps, and batch size × steps is simply the number of training tokens.

training FLOPs ≈ 6 × N (parameters) × D (training tokens)

It holds up at frontier scale. Meta’s Llama 3 paper reports a pretraining budget of 3.8 × 10²⁵ FLOPs for the 405B model on 15.6 trillion tokens. The formula gives 6 × 405 × 10⁹ × 15.6 × 10¹² ≈ 3.79 × 10²⁵, within about 1%. Mixture-of-experts models are the main exception: Use the number of active parameters per token rather than the total.

Convert FLOPs to GPU-Hours

GPUs never run at their peak rating, so divide by the throughput you can actually sustain. Model FLOPs utilization (MFU) captures this. NVIDIA’s DGX Cloud benchmarking recipes rate the H100 at 989 TFLOPS for dense BF16 and report about 43 to 45% MFU for Llama 3.1 405B in FP8 across 32 to 2,304 GPUs, while Meta reported 38 to 43% BF16 MFU at up to 16,384 H100s. For planning, 35 to 40% is a reasonable assumption for a well-tuned run on H100-class hardware; use your own measured MFU as soon as you have it.

GPU-hours = training FLOPs ÷ (peak FLOPS per GPU × MFU × 3,600)

The table below applies both formulas at 40% MFU on H100s. The cost column uses GMI Cloud’s H100 starting rate of $2.60 per GPU-hour from the pricing page, as of October 2026, and excludes the reliability overhead covered later.

The days column is where deadlines meet budgets. Doubling the cluster roughly halves wall-clock time for the same GPU-hours, as long as the job scales efficiently, so the cluster size is set by the deadline and the GPU-hours by the model and data.

Check the Memory Floor

Before choosing a cluster size, confirm that the model fits. The ZeRO paper shows that mixed-precision training with the Adam optimizer needs about 16 bytes per parameter for model states alone: 2 for BF16 weights, 2 for gradients, and 12 for the FP32 master weights, momentum, and variance. A 70B model therefore needs about 1.12 TB for model states before activations, which already spans 14 H100s at 80 GB each. Sharding methods such as ZeRO and FSDP spread that state across GPUs, but the total must still fit in the cluster’s combined memory with room left for activations.

Sizing Inference Capacity

Memory: Weights Plus KV Cache

Inference capacity starts with the memory one replica needs. Weights take parameters × bytes per parameter: 2 bytes in BF16, 1 in FP8. The KV cache grows with every token of every active request, and Red Hat’s explanation of inference gives the per-token size as 2 × layers × KV heads × head dimension × bytes per value.

replica memory ≈ weights + (KV bytes per token × context length × concurrent requests) + runtime overhead

For Llama 3 70B, the model configuration lists 80 layers, 8 key-value heads, and a head dimension of 128 (8,192 hidden size across 64 attention heads). That gives 2 × 80 × 8 × 128 × 2 bytes, about 0.33 MB per token in BF16. Thirty-two concurrent requests at 8,192 tokens each need about 86 GB of KV cache on top of about 140 GB of BF16 weights, so one replica needs roughly 226 GB plus runtime overhead. Two H200 GPUs at 141 GB each fit that with room to spare; four H100s at 80 GB each also fit, with more interconnect traffic. GMI’s guide to KV cache optimization covers ways to shrink the cache line.

Throughput: Replicas at Peak

Memory tells you the smallest replica; throughput tells you how many replicas you need. The only reliable throughput number is one you measure: Run a load test on one replica with your model, your prompt and output lengths, and your latency target, and record the output tokens per second it sustains while staying within that target. Then size for the peak hour, not the daily average.

replicas = peak output tokens per second ÷ (measured tokens per second per replica × target utilization)

As an illustration, suppose peak demand is 20,000 output tokens per second and a load test shows one 2× H200 replica sustains 2,500 tokens per second within the latency target. Eight replicas would cover the peak at 100% load, which leaves no room for spikes or a failed node. Planning to run at 70% of measured capacity gives 20,000 ÷ (2,500 × 0.7) ≈ 11.4, so 12 replicas, or 24 H200 GPUs. Running all month at GMI Cloud’s H200 starting rate of $3.20 per GPU-hour (as of October 2026), that is about $56,064.

Once the service is live, keep the sizing honest with serving metrics. vLLM’s production metrics expose vllm:num_requests_waiting and vllm:kv_cache_usage_perc. A queue that grows while the KV cache is full means you need more memory per replica or more replicas; an empty queue and a mostly empty cache for hours means you have more than you need.

Planning Headroom for Failures and Growth

Reliability Overhead for Training

Large training jobs lose time to failures, restarts, and checkpoints. Meta reported more than 90% effective training time for Llama 3 despite frequent interruptions over a 54-day snapshot, which is a strong result at 16,000 GPUs. Plan for 90% goodput unless you have measured better: Divide the GPU-hour estimate by 0.9. For the 8B example, that turns about 33,700 GPU-hours into about 37,400, and about $87,600 into about $97,400 at $2.60 per GPU-hour (as of October 2026).

Spare Capacity for Inference

For inference, headroom has three parts. Target utilization (70% in the example above) absorbs short spikes. N+1 replicas let you lose a node without breaching the latency target. Growth allowance covers traffic increases before the next planning cycle, and it is better handled by an elastic layer than by reserving capacity months early.

Development and Experimentation

Engineers need GPUs to debug, profile, and try ideas. A common pattern is a small shared pool sized to the number of engineers who work on GPUs at the same time, with automatic shutdown for idle sessions. Without that guard, development capacity becomes the quiet source of the low utilization numbers above.

Turning the Estimate into a Capacity Mix

Once the numbers are in, split them by certainty. The capacity you will use every hour of the planning period, such as the inference base load and committed training runs, is a candidate for reserved or committed pricing. Capacity that depends on traffic peaks, experiment outcomes, or uncertain launch dates belongs on demand or on serverless inference, where you pay only while it runs.

Revisit the plan every month with measured data. Replace assumed MFU with measured MFU, assumed replica throughput with load-test and production numbers, and assumed traffic with the actual daily curve. The first plan is a forecast; the second and third should be accounting.

Planning Capacity on GMI Cloud

GMI Cloud offers capacity in the shapes this method produces. For training, the Cluster Engine provides dedicated, single-tenant GPUs priced per GPU-hour, as a Kubernetes-based Container Service, as bare metal servers with root access and RDMA-ready networking, or as managed multi-node GPU clusters. GMI’s guide to comparing container, bare metal, and managed cluster quotes explains how to request comparable options. GMI has also been recognized as an NVIDIA Exemplar Cloud on GB300 NVL72 systems for training, a program that benchmarks deployments against NVIDIA’s reference architecture.

For inference, serverless inference scales to zero with no idle cost and covers the uncertain layer of demand, while dedicated endpoints and reserved GPUs cover the base. As of October 2026, the pricing page lists H100 from $2.60 per GPU-hour, H200 from $3.20, B200 from $5.00 with limited availability, and GB200 from $8.00, with commitment-based pricing for reserved capacity quoted per customer. See the GPU page for hardware details.

Conclusion

Cluster capacity planning for AI teams works best as arithmetic, not instinct. Estimate training compute as 6 × parameters × tokens, convert it to GPU-hours with a realistic MFU of 35 to 40%, check that model states fit at about 16 bytes per parameter, and add about 10% for reliability. Size inference from memory first, using weights plus KV cache, then from measured throughput at the peak hour with a target utilization near 70% and an N+1 spare. Split the result into an always-on base for reserved capacity and a variable layer for on-demand or serverless, and replace every assumption with measurements as soon as the cluster runs. Teams that plan this way avoid both missed deadlines and the 5% utilization that many clusters run at today. To price your own plan with dedicated GPUs and serverless inference in one place, start with GMI Cloud.