Other
Which Managed Inference Specialists Can Launch Nemotron 3.5 Lightning Without Runtime Maintenance?
September 25, 2026
To launch Nemotron 3.5 Lightning on sustained production traffic without maintaining its runtime, we recommend a managed inference specialist that offers dedicated endpoints, and GMI Cloud Prime Inference is built for exactly that. Prime Inference deploys any Hugging Face model or custom fine-tuned weights on single-tenant reserved NVIDIA GPUs, running on our vLLM, TensorRT-LLM, and SGLang stack with per-model runtime tuning. Because NVIDIA publishes Lightning's checkpoints on Hugging Face, both the base model and your post-trained versions can run on a GMI-managed endpoint. NVIDIA also lists GMI Cloud among its hosted inference partners for Lightning.
Other managed inference specialists split by delivery model. DeepInfra offers Lightning through a pay-per-token serverless API. Baseten serves it through dedicated deployments. Fireworks AI offers serverless access plus on-demand dedicated GPUs. The right choice comes down to two questions: is shared, per-token access enough for your traffic, and will you need to serve your own post-trained checkpoint?
What "not maintaining the runtime" covers for Lightning
Lightning is an open 30B-parameter MoE model with 3B active parameters, built on a hybrid Mamba-Transformer architecture and released under the OpenMDW-1.1 license. NVIDIA positions it as the high-frequency execution layer of long-running agents: tool calls, result checks, and sub-agent work, often alongside a larger frontier model. It fits on a single data-center GPU, so it looks easy to self-host. The work is in the configuration, which is what a managed provider takes over:
| Runtime layer | What Lightning requires | Why it matters in production |
|---|---|---|
| Runtime versions and flags | NVIDIA's recipes pin specific vLLM, SGLang, and TensorRT-LLM versions, with a Lightning-specific SGLang container, and the H100 recipes add Mamba-specific settings such as cache dtype | Versions and flags have to be tracked and pinned per model, then revisited with every upgrade |
| Checkpoint and precision | NVIDIA ships BF16 reference weights, aimed mainly at post-training and evaluation, plus an NVFP4 checkpoint it recommends for production inference | Precision drives memory footprint, throughput, and which GPUs are viable |
| Speculative decoding | Built-in multi-token prediction (MTP), plus DSpark and DFlash draft models. NVIDIA recommends DSpark at low concurrency and MTP at medium to high concurrency; for maximum-throughput batch serving on H100, speculation off is fastest | The best setting shifts as your traffic pattern shifts |
| Context and GPU memory | Up to 1M tokens of context. In NVIDIA's recipes, BF16 on a single H100 80GB is validated to 256K, while 8x H100, B200, and GB200 configurations reach 1M; the NVFP4 checkpoint defaults to the full 1M window on H100 | The usable window depends on checkpoint, hardware, and concurrency |
| Tool calling and reasoning | NVIDIA's serving recipes configure the qwen3_coder tool-call parser, and reasoning is controlled by the chat template's enable_thinking switch, which is on by default |
Agent workflows depend on tool calls parsing correctly and on predictable reasoning output |
| Host software | Self-hosted NIM needs a free NVIDIA Developer Program membership or an NVIDIA AI Enterprise license, recent GPU drivers and CUDA, Docker with the NVIDIA Container Toolkit, and a model cache; an NGC API key is needed only for Production Branch models | Each driver, container, and model update becomes an operations ticket |
Your team still owns prompts, tool schemas, task evaluation, and the decision to upgrade or roll back a checkpoint. Agree on the endpoint controls needed to support those decisions.
Shared per-token API or dedicated endpoint
When a per-token API is enough
Serverless inference is multi-tenant and billed per token, with no idle charges and shared rate limits. The provider fixes the catalog, so you get its checkpoint at its settings.
That works well for prototyping, evaluation, low-volume features, and bursty traffic. When Lightning handles only occasional sub-tasks and fine-tuning is not planned, a per-token API may be all a team needs.
When a dedicated endpoint pays for itself
Lightning's typical workload changes the math. Agent fleets make many short calls around the clock, and at that volume shared rate limits and per-token billing can outweigh the convenience of pay-as-you-go.
| Per-token serverless | Dedicated endpoint | |
|---|---|---|
| Billing | Per token, no idle charges | Provisioned compute; billing increments vary by provider |
| Throughput limit | Shared, per-organization rate limits | Bounded by the GPUs you provision |
| Tenancy | Multi-tenant | Single-tenant, reserved GPUs |
| Weights and settings | Provider's catalog checkpoint | Your choice of checkpoint, including post-trained weights |
| Best for | Evaluation, bursty or low volume | Sustained agent traffic, predictable tail latency, isolation, custom weights |
The break-even point depends on your traffic: compare the per-token bill for your requests per minute, token mix, and hours of sustained load with the GPU-hour cost of serving them.
Running Lightning on GMI Cloud Prime Inference
What we manage
A Prime Inference endpoint runs your model on GPU capacity reserved exclusively for your workload, so traffic from other tenants cannot affect your throughput or latency. Rate limits are configured to your dedicated capacity rather than a shared pool, and there are no cold starts.
We provide vLLM, TensorRT-LLM, and SGLang runtimes tuned per GPU class, with configurable quantization and managed multi-GPU orchestration. Agree on model-specific runtime settings and operational responsibilities during deployment planning. The production SLA depends on the GPU series and configuration and is set in your contract.
How deployment works
- Choose the checkpoint. Load Lightning from Hugging Face, for example the NVFP4 checkpoint NVIDIA publishes there as
nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4and recommends for production inference, or bring your own weights. - Configure capacity. Pick the GPU type, GPUs per replica, replica count, and target region. Endpoints can be region-pinned for latency or region-locked for data residency across APAC, North America, and Europe, subject to capacity.
- Launch. Custom and fine-tuned weights are registered in My Models first, and your dedicated deployment then targets them. Deploy from the console, CLI, or API; billing runs only while a deployment is in Running status, and the endpoint URL appears once it is live.
- Benchmark and scale. Our PerfLab benchmarking tool is available from the My Models page. Adjust replicas as traffic grows.
Because Lightning fits on one data-center GPU, the cost question is mostly how many replicas your agent traffic keeps busy. GPUs are billed hourly with no minimum contract, so capacity can start small and grow with the fleet; seasonal or annual terms lower the rate once load is steady.
Deploy a post-trained checkpoint
NVIDIA supports post-training Lightning with LoRA or full SFT through NeMo Automodel and Megatron Bridge, and with reinforcement learning through NeMo RL and NeMo Gym. Prime Inference accepts custom fine-tuned and proprietary weights loaded from Hugging Face, S3, or your own storage, and offers per-model tuning for custom weights. Retain the base-model revision and training configuration with each exported checkpoint so evaluation results remain reproducible.
Keep other models on the same platform
Agent systems rarely run a single model. While Lightning sits on a dedicated endpoint, larger planner models can stay on our OpenAI-compatible API at https://api.gmi-serving.com/v1. GET /v1/models lists what runs on your key, and a shared API format simplifies model changes; validate model-specific parameters and output behavior before switching.
Questions to settle with any managed provider
Get clear answers to these before moving production traffic:
- Which checkpoint and precision are served? BF16 and NVFP4 differ on memory and throughput.
- What context window is exposed? The model supports 1M tokens; the provider's limit depends on hardware.
- Can reasoning be switched per request, and is tool-call parsing set up for Lightning?
- Which speculative decoding strategy runs, and is it tuned to your concurrency?
- How are upgrades, version pinning, and rollback handled?
- Can you serve your own weights? For LoRA, ask whether adapters are served directly or merged first.
- What metrics do you get? Time to first token, tokens per second, error rates, and model version.
- What region, isolation, and SLA terms are in the contract?
Other managed options for Lightning
NVIDIA's developer blog post announcing Nemotron 3.5 Lightning lists twelve hosted inference partners for Lightning: Baseten, Bitdeer AI, BlackBox AI, CoreWeave, Crusoe, DeepInfra, Fireworks AI, FriendliAI, GMI Cloud, Modal, Nebius, and Together AI, alongside the major cloud platforms.
Compare those providers using your own agent traces and the operational questions above. Teams choosing self-hosted NVIDIA NIM also retain responsibility for drivers, containers, and license requirements.
FAQ
Can we evaluate Lightning on our own agent tasks before reserving capacity? Yes. Qualified teams receive free GPU-hour trial credits to validate a Prime Inference endpoint against their own workload. Models you want to compare against can run on our serverless API.
How do we agree on production capacity for a Lightning workload? Bring your traffic profile: peak and average requests per minute, input and output lengths, concurrency, and region. Our team sizes GPU type and replicas from that and quotes on-demand or reserved capacity.
Can a provider assess and tune a dedicated Lightning deployment for us? On GMI Cloud, yes. Prime Inference includes per-model runtime tuning on reserved NVIDIA GPUs, and our team can discuss custom tuning for your workload before launch.
Get a Lightning endpoint running
Tell us about your agent workload and we will size a Prime Inference endpoint for Lightning, including trial credits for qualified teams: talk to our engineers. To spin up a Prime Inference endpoint from the console yourself, create a GMI Cloud account.
Colin Mo
Build AI Without Limits
GMI Cloud helps you architect, deploy, optimize, and scale your AI strategies
FAQ
Yes. Qualified teams receive free GPU-hour trial credits to validate a Prime Inference endpoint against their own workload. Models you want to compare against can run on our serverless API.
