How Should a Provider Size GB300 NVL72 Capacity for Long-Context Inference at Your Target Concurrency and Latency Before You Reserve It?

KV cache grows linearly with context length and with the number of sequences in flight for a full-attention model at a fixed cache precision, so when requests share no prefix, doubling either one doubles the cache a long-context deployment budgets, while the weights stay fixed for a given model, precision, and layout. A provider sizing GB300 NVL72 for this workload should therefore start from your real distribution of prompt lengths at your target concurrency, test where latency breaks before memory runs out, and add a margin it names before turning the result into replicas, compute trays, and racks. GB300 NVL72 pre-orders at GMI Cloud go through our sales team while the system is coming soon, and the inputs this sizing rests on can be measured today on H100 and H200 GPUs, both available now.

Long context changes which limit binds first. With short prompts, capacity is usually set by throughput, and memory has room to spare. Once a meaningful share of requests carries 64K or 128K tokens, two limits move together. The KV cache can fill the GPUs' memory before compute is busy, and long prefills can push time to first token past your target even when memory is fine. A sizing for this workload has to show both limits and say which one sets the count.

Describe the workload as a distribution

An average context length hides the requests that decide the sizing. Give the provider these inputs as numbers:

• Concurrency target. Peak sequences in flight per region, plus the growth you expect over the reservation term.
• Latency targets. Time to first token (TTFT) at p95 and inter-token latency at p95, each with the load level it applies to.
• Input length distribution. The share of requests in each length band, plus the p95 and the maximum you will accept.
• Output length. Typical and maximum generated tokens, since output tokens also occupy cache until the request finishes.
• Prefix reuse. How much of each prompt is a shared system prompt or document that a prefix cache could hold once.
• Precision. Weight precision and KV-cache precision, since each changes the memory per replica.

Weights and KV-cache budget per replica

Per-token KV size follows from the model's layer count, KV heads, head dimension, and cache precision. For sizing, use the figure your serving engine actually allocates per token, which includes block granularity and padding that a formula leaves out.

The worked example below uses assumed inputs only. Assume a model whose FP8 weights take 400 GB, served on 8-GPU replicas. NVIDIA's reference architecture for GB300 NVL72 lists four GPUs and 1,152 GB of HBM per compute tray, so one replica spans two trays linked by NVLink and holds 2,304 GB in total. Assume weights and cache shard evenly across the eight GPUs, 10% of memory is held back for activations and runtime buffers, and the engine reports 120 KiB of FP8 cache per token. That leaves about 1,674 GB for KV cache, or roughly 13.6 million tokens. Assume the deployment caps cache fill at 85% as headroom against preemption, which gives about 11.6 million usable tokens per replica. The table's percentages describe sequences resident at one moment, each budgeted for its full input plus 2K output tokens. Long requests stay resident longer than arrival logs suggest, so a real sizing takes its mix from a replay.

The median understates the need, and the maximum overstates it by more than double. Reserve against the real mix and check the margin against the long-heavy hour, since that shift in request mix raises memory demand by nearly 30% with no change in concurrency.

Run a length-by-concurrency test matrix

Memory arithmetic gives an upper bound on sequences per replica. Latency sets the real one, so the next step is a test grid run on one replica at a time. Load counts sequences the engine has admitted, and each column is a share of your peak target.

For each cell, record peak KV-cache use, TTFT at p50 and p95, inter-token latency at p95, the number of preempted or requeued sequences, and goodput, meaning the requests per second that finished inside both latency targets. Run each cell long enough for the cache to reach a steady state, since a short burst finishes before long sequences pile up.

The replayed mix, sent at its recorded arrival times, is the row the sizing rests on. It reproduces a 128K prefill landing among many short decodes, and its timestamps keep the queue visible, which a fixed-concurrency client hides by sending less as responses slow. Prefill work for attention grows faster than linearly with prompt length, and in many engines a long prefill delays the decode steps of every other sequence on the replica. Chunked prefill and limits on tokens per batch reduce that effect, so record the engine settings with the results and keep them identical across cells.

Read peak memory and latency together

The sustainable concurrency for one replica is the highest load that meets four conditions at once: cache fill stays under the fill target, client-side TTFT p95 meets its target, inter-token p95 meets its target, and no sequences are preempted. Whichever condition fails first sets the number.

In the example, memory allows about 226 sequences per replica on the long-heavy mix. Assume the grid shows TTFT p95 passing a 3-second target only up to 150 sequences, because long prefills queue behind each other. The replica is latency-bound at 150, and memory still has room left. With the model layout and engine settings unchanged, more memory per GPU does not shorten that queue. More replicas, a separate pool for the longest requests, or a looser TTFT target for the 128K band are the levers that move the result.

When cache fill hits its limit while latency is still inside target, the replica is memory-bound. A larger replica or a lower-precision cache raises the token budget, and the changed setup goes back through the grid.

From replicas to a reservable GB300 NVL72 quantity

With sustainable concurrency per replica measured, the count follows:

1. Replicas for peak load. Divide peak concurrency by sustainable concurrency per replica and round up. In the example, 400 ÷ 150 gives 3.
2. Failover. Add enough replicas to keep the latency target with one replica out of service. One spare makes 4.
3. Growth over the term. Apply the expected growth to peak concurrency, then repeat steps 1 and 2. Doubling to 800 gives 6 plus 1 spare, or 7.
4. GPUs, trays, racks. Multiply by GPUs per replica, then divide by four GPUs per tray and 72 GPUs per rack. Four 8-GPU replicas use 32 GPUs on eight trays, and seven use 56 GPUs. A full rack holds nine such replicas.

These are physical counts; the reservable unit and replica placement are settled in the quote, and covering a rack-level outage takes replicas spread across racks. State each margin once and say what it covers. The 85% fill cap gives headroom against preemption, the spare replica covers a single replica failure, and the growth factor covers the term. Stacking an extra percentage on top of all three without naming it inflates the reservation and hides which assumption it protects.

Inputs that H200 runs can supply today

Some inputs above belong to your model and carry over between GPU generations while the model, cache format, and parallel layout stay fixed: the logical KV bytes per token, how much of each prompt repeats a shared prefix, and the lengths and arrival times in your request trace. Runtime overhead, the cache the engine actually allocates, and every latency figure depend on the hardware and are measured again on GB300 NVL72. On Prime Inference, H200 with 141 GB of HBM3e is positioned for memory-heavy workloads, long context, and large KV caches. Container workloads can launch from templates that include vLLM and SGLang with JupyterLab.

Run the same grid on H200 at a context range that fits, then hand the results to the sizing. For GB300 NVL72, the sizing should write down its speedup assumption over H200, separately for prefill and decode, along with the per-replica throughput it implies, so the first runs on delivered capacity can be checked against it cell by cell.

At GMI Cloud, those measurements can start today on dedicated, single-tenant H200 GPUs, and Prime Inference adds inference engineers who tune kernels, scheduling, and routing per model, with optimized KV-cache handling and bounded P95 and P99 latency listed for long-context RAG and chat workloads.

FAQ

How much does prefix caching change the capacity we should reserve?

Requests that start with the same system prompt or document share the cached KV blocks for that prefix rather than each holding a copy, which can move a replica from memory-bound toward latency-bound. Across several replicas the hit rate also depends on the router sending matching prefixes to the same replica, so measure it on replayed traffic across all replicas and size from that rate.

Should 128K requests run on separate replicas from short requests?

A separate pool for the longest band is worth testing when the grid shows long prefills pushing TTFT for short requests past target. Compare it with a shared pool on the same total GPUs and the same replayed traffic, and keep the split if goodput inside both targets comes out higher.

Bring your length mix and latency targets to the GB300 NVL72 pre-order

Have your context-length distribution, peak concurrency, TTFT and inter-token targets, and any H100 or H200 grid results ready when you contact our sales team about a GB300 NVL72 pre-order.