For an AI agent platform serving thousands of concurrent users, is a GB200 NVL72 the right backbone, and how do I size it?

Platform teams often assume that thousands of concurrent agent users automatically call for the biggest system available, and reach for a GB200 NVL72 on scale alone. But concurrency and pooled-domain hardware solve different problems, so high user counts do not by themselves make an NVL72 the right backbone. A GB200 NVL72 is the right backbone for an agent platform when your models are large enough that its 72-GPU pooled NVLink domain is actually used, not simply because you serve many users, and you size it by concurrent sessions times calls per agent loop times per-call footprint, not by user count alone. This guide covers when the NVL72 fits and how to size it for real concurrency.

Concurrency and pooled hardware are two different questions

Serving thousands of users is a concurrency problem: many independent sessions hitting the platform at once. A GB200 NVL72 solves a different problem: it links 72 Blackwell GPUs and 36 Grace CPUs into a single NVLink domain so one large model can span many GPUs as if they were one accelerator. Those two things are easy to conflate and are not the same.

That distinction decides the backbone. High concurrency alone can be served by independent B200 nodes running many sessions in parallel, so the NVL72's pooled domain earns its place only when a single model or routing path is large enough to need GPUs coupled tightly together, not when you simply have many users. If your agent's models fit comfortably on a few GPUs, thousands of users are a horizontal-scaling problem you solve by adding nodes. If your models are large enough that they must span nodes, the NVL72 removes the cross-node network penalty that would otherwise throttle every call.

When the NVL72 is the right backbone, and when it is overkill

The deciding factor is whether the pooled domain gets used, judged by your largest single job rather than your largest total traffic. Buying a 72-GPU coupled domain to run models that only need a handful of GPUs pays for interconnect you never exercise.

The signals are concrete. The NVL72 is the right backbone when your platform serves large or mixture-of-experts models whose weights and routing span many GPUs, when consistent low latency across that span matters for every agent call, and when pooled memory lets a model run that independent nodes could not hold; it is overkill when your largest model fits on a few B200s and concurrency is better met by adding independent nodes. Size the backbone to the largest single workload it must run coupled, then scale user concurrency across that unit. Getting this backwards, sizing to total users, is how teams buy a pooled domain and then split it into work that never needed pooling.

Sizing for concurrency, loop, and footprint

Sizing an agent platform is not a single number; it is driven by three multipliers that user count alone hides. Use the frame below.

Sizing input	What drives it	Why it matters for agents
Concurrent sessions	Peak simultaneous users, not total users	Only active sessions consume GPU memory at once
Calls per agent loop	Steps in the reasoning-plus-tool loop	Each user request is many inference calls, not one
Per-call footprint	Model weights plus KV cache at your context length	KV cache under high concurrency often dominates, not weights

The pattern is that an agent platform's real load is concurrent sessions multiplied by the calls in each loop, and the memory pressure at scale comes as much from KV cache across many active sessions as from the weights themselves. So you size by measuring peak concurrent sessions, multiplying by your loop length, and provisioning capacity for the combined weight-plus-KV-cache footprint at that concurrency, then verify it under a real concurrent load test rather than a single-session benchmark.

Building the backbone on GMI

Once you know whether the pooled domain fits and how large to size it, the practical step is using our platform, which offers the NVL72 as a managed backbone and meters agent workloads. We are an AI-native inference cloud that carries GB200 NVL72 and B200, and exposes agent-oriented tooling alongside them.

Our [AgentBox](https://www.gmicloud.ai/en/models/agentbox), in early access, meters agentic workloads per request so the true load of a multi-step agent loop is visible when you size for thousands of concurrent sessions, while our [Prime Inference](https://www.gmicloud.ai/en/models/prime-inference) keeps your models warm on reserved GB200 or B200 capacity so per-call latency stays consistent across the loop. We currently list GB200 NVL72 at from $8.00 per GPU-hour and B200 at from $4.00 under Limited Availability, so confirm current capacity and rates on our GPU infrastructure (https://www.gmicloud.ai/en/gpus) and pricing (https://www.gmicloud.ai/en/pricing) before committing, since Blackwell availability moves quickly. Because the NVL72 is a 72-GPU rack-scale unit and a B200 is a single GPU, decide first whether your models need the pooled domain, then load-test at your real peak concurrency before locking size. Start in our console (https://console.gmicloud.ai).

Size to the Largest Model First, the User Count Second

If you pick a backbone by user count, you will buy a pooled domain for a concurrency problem that independent nodes could have solved. Decide the backbone by your largest single workload: reach for the GB200 NVL72 when a model genuinely needs its coupled 72-GPU domain, and stay on independent B200 nodes when concurrency is the only thing that is large. Then size by concurrent sessions times loop length times per-call footprint, and prove it under a concurrent load test. So let the largest model you must run coupled, not the number of users you serve, decide whether the NVL72 is your backbone.