Which Inference Providers Can Tune a Continuously Running Workload on GB200 NVL72?

For a model that serves traffic around the clock on GB200 NVL72, choose a managed dedicated inference provider that operates the serving stack, not just the rack. The rack pays off through choices made above the hardware: how the model is laid out across 72 NVLink-connected GPUs, how prefill and decode are split, how requests reach cached context, and which precision holds up on your evals. Those choices drift as traffic, prompts, and engine versions change, so the provider should tune the runtime for your model, accept the work against latency targets users feel, and keep measuring those targets after launch.

GMI Cloud Prime Inference is built for this: dedicated single-tenant GPUs with weights kept warm, runtimes tuned per model, and our inference engineers working with you from prototype to production SLA. GB200 NVL72 for Prime Inference is in pre-order and coming soon, and our Prime Inference benchmarks on it use multi-node serving with NVIDIA Dynamo and SGLang.

What the provider needs to own on an always-on NVL72 deployment

GB200 NVL72 offerings fall into two broad groups. Infrastructure offerings deliver the rack or its nodes as compute. CoreWeave's NVL72-powered instances, for example, are deployed as full 18-node racks on CoreWeave Kubernetes Service (CKS). We plan to offer that path too: GB200 NVL72 on our GPU infrastructure is in pre-order and coming soon, for teams that want to operate the serving stack themselves. Managed dedicated inference, by contrast, includes the engine, parallel layout, and routing on reserved hardware, tuned to your model.

For a continuously running workload, managed inference usually fits better, because the tuning is never finished. These questions separate providers:

Put the answers into the deployment plan.

The tuning levers a 72-GPU NVLink domain adds

A GB200 NVL72 rack combines 36 Grace CPUs and 72 Blackwell GPUs in a single NVLink domain, with 130 TB/s of GPU-to-GPU bandwidth and 13.4 TB of HBM3E across the rack, according to NVIDIA. A model that fits comfortably on one 8-GPU node usually gains less from it; a large mixture-of-experts (MoE) model or a long-context workload can gain a lot.

Parallel layout across the rack

The NVLink domain lets a large MoE model spread its experts across many GPUs, known as wide expert parallelism, without falling back to slower networking between nodes. It also lets prefill and decode use different layouts. In the SGLang team's published GB200 NVL72 experiments with DeepSeek V3/R1, decode ran expert parallelism across 48 ranks while each prefill instance used two to four ranks. Those were research settings, not a production workload, but the point holds: the right layout depends on model and phase, and someone has to choose and revisit it.

Prefill/decode disaggregation and SLO-driven scaling

NVIDIA's Dynamo documentation describes prefill as compute-bound, growing with input length, and decode as memory-bound, growing with concurrent sequences and KV-cache use. A single replica count cannot describe both. Disaggregated serving runs them as separate pools that scale independently, with TTFT as the target for prefill capacity and ITL as the target for decode. NVIDIA's published Dynamo results on GB200 NVL72 show the largest gains when disaggregated serving is combined with wide expert parallelism on a large MoE model.

KV-cache-aware routing

When many requests share context, such as a system prompt, a retrieved document set, or an agent's running conversation, sending each request to a worker that already holds that context avoids recomputing prefill for the shared portion. NVIDIA Dynamo's KV-aware router scores workers on both cache overlap and current load. On a steady workload, measure the cache hit rate and its effect on TTFT with your real prompt mix.

Precision with a quality gate

Blackwell supports NVFP4, which needs roughly 1/1.8 the memory of FP8. NVIDIA reports that DeepSeek-R1-0528, quantized from its native FP8 to NVFP4, lost 1% accuracy or less on the key language tasks NVIDIA evaluated. That is one model on NVIDIA's evals; whether FP4 holds for yours is a question for your own evaluation set. On Prime Inference, quantization is configurable, so precision can follow your eval results.

Keeping a workload tuned when traffic never stops

A one-time benchmark shows what the rack can do on a good day. An always-on workload needs repeat testing as traffic and software change: measure a baseline, change one setting, and compare the result before rolling it out.

Measure the metrics users feel

TTFT includes queueing, prefill, and network time, and it grows with prompt length. ITL is the average gap between consecutive tokens after the first one, also reported as time per output token (TPOT). End-to-end latency is TTFT plus generation time. Track p95 and p99, not averages; the tail is what breaks SLOs. Throughput and ITL trade off against each other: larger batches raise tokens per second per GPU while slowing each individual stream. Fix the ITL target first, then push batching as far as that target allows.

For cost, track cost per million tokens at your measured utilization. Our own Prime Inference benchmarks show what a well-tuned rack can reach. On GB200 NVL72, running Dynamo with SGLang across nodes, DeepSeek V4 Pro (1.6T) came to $0.20 per million output tokens against a $1.13 GMI Cloud serverless list price for the same model, and GLM-5.1 (744B) came to $0.20 against $0.98. Those figures assume 8K input and 1K output tokens per request, FP4 precision, fully loaded reserved GPUs at sustained full utilization, and idle time excluded. They are benchmark results, not a quote, and GB200 NVL72 for Prime Inference is still in pre-order. Your own cost depends on your traffic shape and how busy the hardware stays.

Change one thing, then regress on real traffic

Hold the model version, prompt set, and hardware fixed, and change one variable at a time: batch limits, parallel layout, precision, or engine version. Replay a representative trace of production traffic rather than a uniform synthetic load, since the mix of input lengths and concurrency drives both TTFT and ITL. Roll out engine and model upgrades gradually, and keep the previous configuration ready to restore if p95 moves the wrong way.

Plan for failures inside the rack

A rack with 72 GPUs (built from 18 compute trays in NVIDIA's DGX GB200 design) should be planned for component failures over months of continuous operation. Decide in advance what happens to capacity and to in-flight requests when one occurs. NVIDIA Dynamo offers one approach: health checks flag unhealthy workers and remove them from routing, and requests on a failed worker can migrate and continue elsewhere. Whatever the mechanism, the plan should state the capacity target that applies during recovery.

How GMI Prime Inference will run always-on GB200 NVL72 workloads

Prime Inference is our dedicated, single-tenant inference service on reserved NVIDIA GPUs, with GB200 NVL72 in pre-order and coming soon. For any always-on workload, Prime Inference provides:

• Reserved, warm capacity. GPUs are reserved only for your workload and weights stay loaded, so calls avoid cold-start delays and do not contend for GPUs with other tenants' traffic.
• Tuned runtimes. vLLM, TensorRT-LLM, and SGLang come pre-tuned per GPU class, with per-model kernel, scheduling, and routing optimization. Our inference engineers continuously tune the runtimes behind the most-deployed open-source models, so when you pick one of them, the kernel work is already done. Multi-GPU orchestration is handled, and quantization is configurable.
• Your own weights. Import open-source, fine-tuned, or proprietary model weights from Hugging Face, S3, or your own storage, and serve them on the same inference stack.
• Hardware-bounded throughput. Performance is bounded by the hardware you provision, and rate limits are set to your dedicated capacity rather than a shared pool. In the standard Prime Inference workflow, you choose the GPU type, GPUs per replica, replica count, and region, then monitor latency and throughput as the endpoint runs. The dedicated endpoint documentation covers the deployment wizard, scaling settings, and endpoint status.
• Placement and uptime. Prime Inference endpoints can be region-pinned for first-token latency or region-locked for data residency across our Asia-Pacific, North America, and Europe regions. Because GB200 NVL72 is in pre-order, the regions that will host it are confirmed during deployment planning. Uptime SLAs are determined by GPU series, deployment configuration, and contract.

Any custom tuning beyond the pre-tuned per-model runtimes is agreed as part of the deployment scope.

FAQ

Our latency keeps missing its SLO. Where do we start?

Split the miss into TTFT and ITL. A TTFT miss with healthy ITL points to prefill or queueing: long inputs, too little prefill capacity, or poor cache reuse. An ITL miss points to decode: batches that are too large, KV-cache pressure, or too little decode capacity for current concurrency. Before changing the deployment, check the measurement itself: concurrency, maximum batch size, and request rate all shift measured latency.

Who monitors the rack and recovers failed nodes?

On Prime Inference, the platform handles model loading, resource orchestration, and routing, and our engineering team works with you from prototype through to a production SLA. Monitoring, response, and escalation for your deployment are scoped with our engineers alongside that SLA.

Scope your GB200 NVL72 deployment

Share a representative traffic trace and your latency targets with our inference engineers to define the configuration, tuning work, and acceptance tests for a pre-order GB200 NVL72 Prime Inference deployment.

Qualified Prime Inference prospects also receive free GPU-hour trial credits to validate performance against their own workload. Talk to an engineer.