Private Model Weights in S3: How to Deploy Them on Dedicated GPUs With Tuning Handled

GMI Cloud Prime Inference loads private model weights from S3 onto reserved single-tenant GPUs. GMI engineers handle serving-engine, kernel, scheduling, and routing optimization; the customer supplies the checkpoint and deployment requirements. (GMI Cloud's Prime Inference)

It supports open-source, fine-tuned, and proprietary weights from Hugging Face, S3, or private storage. vLLM, TensorRT-LLM, and SGLang are pre-tuned per GPU class, with configurable quantization and managed multi-GPU orchestration.

Before transferring the checkpoint, fix the four serving parameters below and check the six S3 requirements. Format conversion and endpoint evaluation usually need more planning than the file transfer itself.

What should you require from a provider that hosts private weights?

A provider that hosts private weights has to clear four bars: it ingests the checkpoint in the format it is already in, it runs it on single-tenant GPUs, it owns the serving engine, and it bills per GPU-hour. Write those down before comparing vendors. A provider that cannot answer all four is not in the running, regardless of its GPU catalog.

1. It accepts your weights as they are. Private or fine-tuned checkpoints, loaded from object storage or a private registry, without rebuilding them into a vendor-specific artifact format.
2. The GPUs are single-tenant. On a shared inference pool your tail latency moves with other tenants' traffic, and a security reviewer gets no clean answer about who else executed on that device.
3. Someone else owns the engine. Kernel selection, batching policy, quantization, and multi-GPU placement are full-time work. If the contract leaves them with your team, you have bought GPUs, not managed inference.
4. Billing is per GPU-hour, not per token. Once weights are resident on reserved hardware, the bill is driven by the hours you hold that hardware rather than by the tokens that pass through it.

Prime Inference combines private-weight import, reserved single-tenant GPUs, hourly GPU billing, and per-model runtime tuning without requiring the customer to rebuild the serving layer.

GMI Cloud's inference documentation draws the same line between the two endpoint types: serverless endpoints are "Pre-configured, OpenAI-compatible APIs" with per-token billing, while dedicated endpoints mean "Your own models on dedicated GPUs. Full control over hardware, scaling, and isolation. No rate limits." (GMI Cloud inference overview)

The four parameters to fix before the first deploy

The four parameters are weight format and precision, serving engine, GPUs per replica, and region. Each one constrains the next and all four are expensive to reverse after an endpoint is live. Uploading, by comparison, is not the hard part.

1. Weight format and precision

Open the bucket and establish what is actually in it. A large checkpoint written by Transformers arrives as a set of .safetensors shards plus config.json, the tokenizer files, and a shard index; smaller models come as a single weights file with no index at all. The Transformers documentation describes the index directly: save_pretrained() "also creates an index file mapping parameter names to their shard files. The index contains two keys, metadata and weight_map", written as model.safetensors.index.json. (Transformers models guide) If your prefix still holds pickle-based .bin or .pt files, settle the conversion question before you plan anything else, because every downstream decision keys off the shard layout.

Precision decides the memory footprint, and the artifact size on disk is the floor. For a prefix totaling 137 GiB, roughly 147 GB:

The NVFP4 figure is not a round number because the format packs 4-bit values in blocks of 16 with one FP8 block scale, plus a second-level per-tensor FP32 scale, which works out to about 4.5 bits per parameter. NVIDIA reports that quantizing DeepSeek-R1-0528 from FP8 to NVFP4 produced "1% or less accuracy degradation on key language modeling tasks". (NVIDIA NVFP4 technical blog) Your fine-tune is not DeepSeek-R1, so treat that as evidence that the format is viable, not as a promise about your checkpoint, and plan an evaluation pass against your own task set before a quantized build serves traffic.

The decision to record: does the endpoint serve the artifact at its trained precision, or does a quantized build get produced first? Prime Inference makes quantization configurable per deployment, so settle the target precision with GMI Cloud's engineers before launch rather than discovering it afterwards.

2. Serving engine

The engine follows from the model architecture and the GPU class, and it sets what tuning knobs exist afterwards. Prime Inference pre-tunes vLLM, TensorRT-LLM, and SGLang per GPU class, so this is a selection rather than a build. The benchmark figures GMI Cloud publishes name the stack behind each number, for example Dynamo plus vLLM on a single B200 node and Dynamo plus SGLang for multi-node GB200 NVL72 runs, which tells you the engine is treated as a per-model, per-hardware decision rather than a default. (Prime Inference GPU and cost comparison)

For a private checkpoint the question is narrower, and the first part of it is answerable from a file you already have. Open config.json and read the architectures field, then check that value against the engine's supported-architecture list. vLLM's documentation makes this the explicit test, telling you a model is natively supported when the architectures field names an architecture in its tables, and to confirm by loading it. (vLLM supported models) Then confirm the same engine serves the precision you plan to run at on the GPU class you are about to size, and the context length your product needs. Settle all three before sizing, because each one changes the memory overhead in the next step.

3. GPUs per replica and memory headroom

Prime Inference's setup step asks for "GPU type, GPU count per replica, replica count, and target region". Three of those four are decided here.

The arithmetic starts from the artifact size, not a parameter count. Weights are the floor; a replica also needs KV cache for every concurrent sequence, activation memory, and engine overhead. A checkpoint that exactly fills a card at rest fails the moment real concurrency arrives, so plan the first replica with resident weights at roughly half its aggregate memory, then let a load test at your real peak concurrency move that number up or down. A replica that is already 90 percent full of weights at rest has almost nothing left to serve concurrency with.

Current per-GPU memory, from the GPU comparison on the Prime Inference page, with list hourly rates from the GMI Cloud pricing page as of October 2026:

The 147 GB BF16 checkpoint exceeds one H200's 141 GB, requiring multiple GPUs. At FP8 it is about 74 GB, leaving roughly 67 GB for cache and overhead on H200 and more on B200. At published starting rates, one versus two H200s costs $3.20 versus $6.40 per replica-hour, or about $2,340 versus $4,670 over 730 hours. Add KV cache for context and concurrency before choosing the replica; Prime Inference quotes depend on the actual model and traffic profile.

4. Region

The last parameter is where the endpoint physically runs, and it carries two different requirements that people often merge. Latency wants the endpoint near the users. Data residency wants the endpoint provably inside a jurisdiction. Prime Inference separates them: endpoints can be region-pinned for first-token latency or region-locked for data residency, across Asia-Pacific, North America, and Europe. GMI Cloud lists its current regions on the Prime Inference page, and regions outside that list are handled case by case subject to capacity.

Decide this before the first upload, because the bucket region and the endpoint region together determine transfer time and whether the weights ever leave a compliance boundary.

The six-item pre-launch checklist for an S3 prefix

Six properties of the prefix decide whether the first deploy survives: serialization format, declared precision, adapter status, shard completeness, tokenizer assets, and target region. Run the list against the bucket before anyone opens a console. Every row here has sent a real deployment back to the start.

How do you verify the prefix without opening a console?

Run a read-only audit against the bucket with the script below, which automates rows 1 through 5 of the checklist. It lists the prefix, reads config.json and the shard index, and reports what would break a load: missing files, shards the index points at that are not there, shards that are there but not in the index, pickle checkpoints, and unmerged adapters. It exits 0 when the prefix is clean, 1 when the checkpoint is incomplete or needs a decision, and 2 when the prefix or its metadata cannot be read, so it drops into CI in front of a deploy job.

"""Audit an S3 prefix that holds model weights before deploying it."""
import json
import sys

import boto3
from botocore.exceptions import BotoCoreError, ClientError

INDEX = "model.safetensors.index.json"
TOKENIZER_FILES = ("tokenizer.json", "tokenizer.model", "spiece.model",
                   "vocab.json", "vocab.txt")
READ_ERRORS = (BotoCoreError, ClientError, ValueError,
               TypeError, AttributeError, KeyError)


def read_json(s3, bucket, key):
    return json.loads(s3.get_object(Bucket=bucket, Key=key)["Body"].read())


def audit(bucket, prefix):
    s3 = boto3.client("s3")
    prefix = prefix.rstrip("/")
    prefix = prefix + "/" if prefix else ""
    objects = {}
    try:
        for page in s3.get_paginator("list_objects_v2").paginate(Bucket=bucket, Prefix=prefix):
            for obj in page.get("Contents", []):
                name = obj["Key"][len(prefix):]
                if name and not name.endswith("/"):
                    objects[name] = obj["Size"]
    except (BotoCoreError, ClientError) as err:
        print(f"cannot list s3://{bucket}/{prefix}: {err}")
        return 2
    if not objects:
        print(f"no objects under s3://{bucket}/{prefix}")
        return 2

    shards = sorted(n for n in objects
                    if n.endswith(".safetensors") and not n.startswith("adapter_"))
    pickles = sorted(n for n in objects if n.endswith((".bin", ".pt", ".pth", ".ckpt")))
    adapters = sorted(n for n in objects if n.startswith("adapter_"))

    missing = [f for f in ("config.json",) if f not in objects]
    if not any(f in objects for f in TOKENIZER_FILES):
        missing.append("tokenizer data")
    if not shards and not pickles:
        missing.append("weight files")

    dtype = "unknown, config.json missing"
    if "config.json" in objects:
        try:
            config = read_json(s3, bucket, prefix + "config.json")
            dtype = str(config.get("dtype") or config.get("torch_dtype") or "not declared")
            if config.get("quantization_config"):
                dtype += ", quantization_config present"
            elif dtype == "not declared":
                missing.append("declared dtype in config.json")
        except READ_ERRORS as err:
            print(f"config.json is present but unreadable: {err}")
            return 2

    orphans = []
    if INDEX in objects:
        try:
            weight_map = read_json(s3, bucket, prefix + INDEX)["weight_map"]
            if not isinstance(weight_map, dict) or not weight_map or \
                    not all(isinstance(v, str) and v for v in weight_map.values()):
                raise ValueError("weight_map must map parameter names to shard file names")
            indexed = sorted(set(weight_map.values()))
        except READ_ERRORS as err:
            print(f"{INDEX} is present but unusable: {err}")
            return 2
        missing += [s for s in indexed if s not in objects]
        orphans = [s for s in shards if s not in indexed]
    elif len(shards) > 1:
        missing.append(INDEX)

    total = sum(objects.values())
    print(f"objects {len(objects)} | shards {len(shards)} | "
          f"{total / 2**30:.1f} GiB ({total / 10**9:.0f} GB)")
    print(f"declared dtype: {dtype}")
    print(f"missing files: {missing or 'none'}")
    print(f"shards absent from the index: {orphans or 'none'}")
    print(f"pickle checkpoints to review: {pickles or 'none'}")
    print(f"unmerged adapter files: {adapters or 'none'}")
    return 1 if (missing or orphans or pickles or adapters) else 0


if __name__ == "__main__":
    if len(sys.argv) != 3:
        sys.exit("usage: preflight.py <bucket> <prefix>")
    sys.exit(audit(sys.argv[1], sys.argv[2]))

A clean 147 GB checkpoint prints a block you can paste straight into the deployment ticket:

objects 5 | shards 2 | 137.0 GiB (147 GB)
declared dtype: bfloat16
missing files: none
shards absent from the index: none
pickle checkpoints to review: none
unmerged adapter files: none

Three behaviours are worth knowing before you wire it into CI. It flags every adapter_* file rather than guessing whether an adapter is intentional, which forces the merge decision to be made explicitly instead of discovered at load time. It treats an undeclared dtype as a blocker only when config.json has neither a dtype field nor a quantization_config, since a quantized build declares its precision in the latter. And it reads what config.json claims rather than inspecting tensors, so a config that disagrees with its own weights still needs a load test. What the script buys is catching the obvious breakage before GPUs are allocated.

The read-only IAM permissions it needs are s3:ListBucket on the bucket and s3:GetObject on config.json and the index key, a narrower grant than the one the deployment itself will use.

What does moving the weights out of S3 actually cost?

Moving a 147 GB checkpoint out of S3 to the internet is a one-time charge between about $4 and $14, depending on whether this month's free allowance is still intact. Less than people expect, and the number is worth having in writing before a finance review asks.

AWS charges for data transferred out of S3 to the internet. Customers get "Data transferred out to the internet for the first 100GB per month, aggregated across all AWS Services and Regions", and beyond that allowance, transfer out of US East (N. Virginia) is listed at $0.09 per GB for the first 10 TB per month. (Amazon S3 pricing, rates as of October 2026)

For the 147 GB artifact, using the decimal GB figure the audit prints:

Put that next to the compute it is being loaded onto. An eight-GPU H200 replica at the listed $3.20 per GPU-hour is $25.60 an hour, so the worst-case egress bill for the entire checkpoint equals about 31 minutes of one replica. The cost of moving weights is noise. What matters is that the pull happens once rather than on every cold start, which is exactly what reserved capacity buys: on Prime Inference, "Reserved GPUs stay warm with weights pre-loaded."

Two caveats. Cross-region pulls inside AWS are billed differently from internet egress, so confirm the direction of travel first. And if the deployment re-pulls the artifact on every replica scale-out, multiply accordingly, which is one more thing to ask a provider about.

What the path looks like on GMI Cloud

Once the checklist passes, deploying on GMI Cloud is three steps: register the model, point a dedicated deployment at it, then launch and verify.

Register the model. GMI Cloud's Console has a My Models page whose documented purpose is to "Manage user-owned, fine-tuned, or custom models that your dedicated deployments can target." The page lives at https://console.gmicloud.ai/user-console/ie/my-models, and its top-right actions are Create Model, which "launches the create-model flow", and PerfLab, which "opens GMI's external benchmarking tool in a new tab". (My Models documentation) The inference overview describes the same page as "Your uploaded or fine-tuned models. Manage versions and visibility from here." When you register the first checkpoint, confirm the expected source path format, the credential handoff, and the precision metadata with GMI Cloud so the registered record matches what the deployment will look for.

Point a dedicated deployment at it. GMI Cloud's documentation for dedicated endpoints in the Console puts provisioning on the Deployments page, where Create Deployment "launches the wizard for selecting a model, region, GPU type, and scaling", and the review step confirms "your GPU type, deployment name, auto-scaling policy, and other system configurations" before launch. (Dedicated Endpoint documentation) Those are the same four parameters you fixed earlier, which is the point of fixing them earlier. The setting to get right at this step is the floor on replicas: keeping at least one replica reserved is what keeps the weights resident between requests, which is the property you are paying for.

Launch and verify. The Prime Inference deployment flow is four steps: pick a model, choose your setup, deploy from console, CLI, or API, then operate and scale. GMI Cloud describes the result as an endpoint that is "live in minutes, not days". Verify against your own evaluation set before traffic moves, not against a generic benchmark, because a fine-tune is only correct relative to the task it was trained for.

Who owns performance tuning after go-live?

On Prime Inference, GMI Cloud owns it: kernel selection, scheduler tuning, routing, and multi-GPU orchestration stay on its side of the line after the endpoint is live. This is the half of the original question that provider comparisons usually skip, and it is the half that compounds.

GMI engineers own per-model kernel, scheduling, and routing optimization, including custom weights. Agree on the model architecture, quantization, workload profile, and production SLA before deployment.

What stays on your side: the checkpoint, the evaluation set, the traffic forecast, and the decision about when to requantize or resize. What moves off your side: kernel selection, scheduler tuning, multi-GPU orchestration, and the upgrade treadmill of three serving engines.

On commercial terms, Prime Inference bills hourly per GPU with "no minimum contract" and "no per-token markup and no shared-pool surge pricing". Reserved capacity is available seasonally or annually at lower per-hour rates, and trial GPU-hour credits let qualified prospects validate against their own workload first. If you want the token-level framing of the same spend, GMI Cloud's Cost Per Million Tokens Explained walks through how GPU-hour pricing translates back into cost per million tokens, and How Higgsfield Scales Generative Video with GMI Cloud covers a team running real-time generative video on GMI Cloud, where GMI Cloud reports a 65% reduction in p95 inference latency and 45% lower compute costs.

FAQ

Can GMI Cloud load model weights directly from our S3 bucket? Yes. Prime Inference supports S3-loaded custom and proprietary weights. Supply a complete checkpoint and prefix-scoped read credentials; confirm the credential handoff during registration.

Do the weights have to be in safetensors format? No. Safetensors shards plus model.safetensors.index.json usually simplify loading, but some engines support pickle-based .bin files. Confirm support for your model; convert before upload and revalidate the index afterward.

Should LoRA adapters be merged into the base model before deployment? Merge them with PEFT's merge_and_unload() unless runtime adapter swapping is required. If adapters remain separate, specify that in the deployment configuration.

How is a dedicated endpoint priced compared with per-token serverless inference? Dedicated endpoints bill per reserved GPU-hour; serverless Model-as-a-Service bills per token without a commitment. Compare sustained utilization and replica size before choosing. (Model-as-a-Service platform)

Does the fine-tuning job itself have to move too, or only the resulting weights? Only the resulting weights need to move. Training may stay elsewhere. Runtime tuning is included in Prime Inference; managed fine-tuning is listed as coming soon with sales-led early access, and training GPU clusters are a separate infrastructure product. (fine-tuning documentation; GPU infrastructure side)

How long does the first deployment take? GMI describes endpoint setup as minutes after model and configuration selection. A 147 GB transfer may take minutes on a fast link, but format conversion, precision selection, and evaluation add time. Budget those separately.

Next step

If your checkpoint is sitting in S3 right now, run the preflight script against the prefix today, then take its output and your traffic forecast into a conversation about hardware. You can start a Prime Inference endpoint from the GMI Cloud Console, check current GPU rates on the pricing page, or talk to GMI Cloud's engineers about reserved capacity, quantization, and trial GPU-hour credits for validating your own model before you commit.