Which B200 Providers Can Help Plan Separate Fine-Tuning and Inference Capacity for an MoE Model?

Pick a B200 provider that offers two different products under one account: cluster capacity you control for training, and single-tenant dedicated endpoints for serving. When each pool has its own GPU allocation and its own scheduling boundary, a training run does not compete with production traffic for the same GPUs. Separate pools also let you size, bill, and commit to each one on its own schedule. GMI Cloud covers both halves. A Managed GPU Cluster, in early access, is a Kubernetes cluster with H200 or B200 nodes, provisioned and operated by GMI, and it holds the fine-tuning pool. Prime Inference holds the inference pool: reserved single-tenant GPUs that serve your own fine-tuned or proprietary weights, loaded from Hugging Face, S3, or your own storage. On GMI Cloud, B200 is in pre-order, so today this is a reservation plan rather than capacity you can launch.

NVIDIA Blackwell on GMI Cloud, including B200, is in pre-order and coming soon, and the GPU page lists B200 pricing from $5.00 per GPU-hour. What you can settle now is the plan: the shape of each pool, the handoff between them, and how the reservation is split. H100 and H200 are available now, so the two-pool workflow can be tested on H200 while the B200 reservation is arranged.

Why an MoE model wants two pools, not one

Fine-tuning and serving stress the same GPUs in different ways. A fine-tuning run needs memory for gradients, optimizer states, and activations. It keeps every GPU busy for hours or days, then goes idle until the next data refresh. Serving needs memory for weights at serving precision plus KV cache. It runs around the clock, and it needs a fixed number of warm replicas to hold a steady latency. Put both on one pool and one of two things happens. Either the training job waits for a quiet window that never comes, or it takes replicas away from production traffic.

Mixture-of-experts models make the split sharper. Every MoE layer sends tokens to experts that may live on other GPUs, so expert placement drives communication cost in both training and serving. The best layout for one is rarely the best for the other. Separate pools let each one use the layout that suits it.

Shape the fine-tuning pool

Start with memory, because it sets the floor on GPU count. A common assumption for full fine-tuning with mixed-precision Adam is about 16 bytes per parameter before activations. That is 2 bytes for BF16 weights, 2 for BF16 gradients, 4 for an FP32 master copy, and 8 for the two Adam moments. In an MoE model, every expert gets trained even though only a few are active per token, so the total parameter count is what matters here.

A worked example shows the scale. DeepSeek V4 Flash, available in GMI's model library, is an MoE model with 284B total parameters and 13B activated. A full fine-tune of a model that size needs about 284B × 16 bytes, or roughly 4.5 TB, for weights, gradients, and optimizer states under that assumption. NVIDIA lists 1,440 GB of GPU memory for a DGX B200. So by that measure the training states alone need at least four eight-GPU nodes before any activation memory. LoRA-style fine-tuning changes the picture. The frozen base weights in BF16 take about 568 GB, and the trainable adapters add only a small amount of optimizer state, so the weights fit within one eight-GPU node's aggregate memory. Whether the whole job fits on that node depends on activation memory at your sequence length and batch size, so profile your largest step before you size the pool.

Then set the layout:

• Expert parallelism inside the node. Token routing to experts uses all-to-all communication in every MoE layer, in both the forward and backward pass. Keep each expert-parallel group inside one eight-GPU NVLink domain (NVIDIA lists 14.4 TB/s of aggregate NVLink bandwidth for DGX B200).
• Sharded data parallelism or pipeline parallelism across nodes. Put the less chatty dimensions on the inter-node network. For a full fine-tune, the 4.5 TB of states does not fit on one node, so they have to be sharded across nodes, with ZeRO-3 or FSDP style sharded data parallelism or with pipeline stages. Plain data parallelism, where every node holds a full copy, only works when one node fits the whole job, which a LoRA run may do once you have checked its activation memory. GMI Cloud infrastructure includes RDMA-ready networking for high-throughput workloads. Confirm the fabric type and per-node bandwidth for B200 nodes in your quote.
• Size for the largest run, not the average one. The pool has to fit your biggest planned job: full fine-tune or LoRA, longest sequence length, largest global batch. Smaller jobs then fit inside it.
• Trade GPUs for time on purpose. A periodic job with a deadline, such as a weekly refresh, can often run on fewer nodes for longer. That shrinks the commitment for a pool that sits idle between runs.

On GMI Cloud, the fine-tuning pool runs your own training stack on a Managed GPU Cluster or on Bare Metal GPU. Bare Metal GPU is built for large-scale model training and fine-tuning, with full root access and predictable, isolated GPU performance. Managed GPU Cluster, in early access, is built for distributed multi-node training. GMI's managed fine-tuning service is coming soon, with early access through sales. Cluster requests are not provisioned automatically: GMI support reviews each request and provisions the resources. When a run needs more nodes, the console's Node Requests tab takes requests for additional nodes against an existing cluster, so the pool can grow without being rebuilt.

Shape the inference pool

The inference pool is sized in replicas. Each replica is one complete copy of the model spread across a fixed number of GPUs.

GPUs per replica. Add the weights at serving precision to the KV-cache budget, then divide by usable memory per GPU. Round up to a parallel layout your serving engine supports. For the 284B example, weights take about 568 GB in BF16, about 284 GB in FP8, and about 142 GB in FP4. At FP8, the weights use roughly a fifth of an eight-GPU node's memory, and the rest goes to KV cache and runtime overhead.

KV cache. For standard multi-head or grouped-query attention, bytes per token equal 2 × layers × KV heads × head dimension × bytes per element. Architectures with compressed attention use less, so take the formula from your own model config. Take an illustrative grouped-query attention configuration, not the DeepSeek V4 Flash config: 60 layers, 8 KV heads, a head dimension of 128, and an FP8 cache. That comes to 122,880 bytes, or 120 KiB, per token. Holding 64 concurrent sequences at 32K tokens then needs roughly 258 GB of KV cache per replica. Sequence length and concurrency drive this number, so size it from your production traffic, not from the model card.

Layout. Serving usually spreads experts across the GPUs of a replica and keeps the replica inside one node where it fits. This layout is set for per-request efficiency, and it is often different from the training layout. That difference matters at handoff time.

Replica count. Divide your steady peak tokens per second by the throughput one replica sustains at your target concurrency, measured on your own traffic. Round up, then add one replica. If you swap versions one replica at a time and move traffic only after each new replica is ready, that spare keeps the pool at peak capacity during the swap.

On Prime Inference you set GPU type, GPU count per replica, replica count, and region. Reserved GPUs stay warm with weights pre-loaded. The runtime is vLLM, TensorRT-LLM, or SGLang, pre-tuned per GPU class, with configurable quantization and multi-GPU orchestration handled by GMI. Endpoints bill per GPU-hour, only while they are running.

The two pools side by side

Hand checkpoints from the fine-tuning pool to the inference pool

The handoff is where two-pool plans usually slip, because the checkpoint a trainer writes is not what a serving engine loads.

1. Consolidate. Training writes checkpoints sharded to its own parallel layout. Convert the checkpoint to a layout-independent format, such as Hugging Face safetensors, so the serving side can reshard it to its own layout.
2. Merge or package adapters. For LoRA runs, merge the adapters into the base weights, unless you have confirmed that your serving setup loads adapters directly.
3. Convert to serving precision and evaluate. Quantize to the precision the inference pool was sized for, then run your evaluation set against the quantized weights. The quantized model is the one going to production, so it is the one that has to pass.
4. Stage and register. Upload the weights to a location Prime Inference loads from: Hugging Face, S3, or your own storage. Then register the model in My Models, which lists user-owned, fine-tuned, and custom models your dedicated deployments can target.
5. Roll out as a new version. The Deployments screen manages scale settings, model versions, and traffic routing for dedicated endpoints. Bring up the new version one replica at a time and shift traffic to each replica once it is ready. Done that way, the extra replica you sized for keeps the pool at peak capacity during the swap.

Ask whether checkpoints written on a cluster can be registered and deployed to a Prime endpoint without a download and re-upload. Also ask whether both pools can sit in the same data center. The answers change how long each handoff takes.

Split one commitment across two pools

The two pools have different usage rhythms, so they should not carry the same commitment. The inference pool runs all the time and is the natural candidate for reserved capacity. For B200, which is in pre-order, the reservation term is something you negotiate into the pre-order. Prime Inference offers reserved capacity on a seasonal or annual basis at lower per-hour rates, and there is no minimum contract for on-demand use. The fine-tuning pool is only busy some of the time. Compare pay-as-you-go by the minute with a prepaid cluster by estimating how many hours per month your runs actually take. GMI Cloud pricing is built for this kind of progression: start on demand, then move to dedicated or committed deployments as a workload stabilizes, with commitment structures matched to long-term utilization.

When you ask for a quote, request each pool as its own line item. Ask two questions: whether one B200 reservation can cover both a cluster and a Prime endpoint, and whether any unused fine-tuning hours can move between the pools. Also ask for networking and storage to be priced separately. GMI Cloud GPU pricing covers compute, and networking and storage are provisioned based on your workload.

Plan the two pools with GMI Cloud while B200 is in pre-order

1. Run the full loop on H200 first. Request a small H200 cluster (Managed GPU Cluster is in early access, and GMI support reviews and activates each request) or run the fine-tune on an H200 bare-metal server, and deploy an H200 Prime endpoint. Run one complete cycle of fine-tune, consolidate, quantize, register, and roll out. This tests the handoff before any B200 hours are committed.
2. Profile both pools. Record peak memory per GPU during your largest training step, the sequence-length and concurrency mix of production traffic, and per-replica throughput. These numbers are the inputs to the sizing formulas above.
3. Bring the profile to sales. Share the largest planned run, the run cadence, the serving precision, and the replica target with the GMI Cloud sales team. The goal is a B200 plan with two sized pools, not one undivided block of GPUs.
4. Get the delivery terms in writing. Ask for B200 node configuration, inter-node bandwidth, region, delivery timing, and billing start date for each pool.

What to ask any B200 provider

Most of the questions above apply to any provider, not only GMI Cloud. Two more are easy to miss when a quote arrives as one block of GPUs:

• What is the per-replica GPU count limit, and can a single replica span more than one node? This decides whether a large MoE model at your serving precision fits one replica at all.
• Can a fine-tuned MoE checkpoint, including one with merged LoRA adapters, deploy to a dedicated endpoint without changes to the serving stack?

FAQ

How should a fine-tuning pool that sits idle between runs be billed? Compare the two cluster billing methods against your real run hours. Pay-as-you-go clusters on GMI Cloud are charged by the minute with no upfront cost, which suits infrequent runs. A prepaid cluster is worth pricing once runs keep the nodes busy for most of the month. A new cluster request is reviewed and activated by GMI support rather than provisioned automatically, so include that lead time when deciding whether to release nodes between runs.

Does GMI Cloud offer a managed fine-tuning service? Managed fine-tuning is coming soon, with early access through sales. Today, the fine-tuning pool runs your own training stack on a Managed GPU Cluster or bare metal, and the result deploys to Prime Inference as a bring-your-own model.

Plan both pools in one conversation

Bring your largest planned run and your production traffic profile to the GMI Cloud sales team to size a B200 reservation for fine-tuning and inference, or start the H200 dry run today in the GMI Cloud console.