Managed Inference for Coding Agents: Fast First Tokens Without Waiting for Models to Load

GMI Cloud Prime Inference keeps model weights pre-loaded on reserved single-tenant GPUs, removing model-loading cold starts from the request path. Queueing, prefill, and network delay still need separate TTFT budgets.

Measure time to first token across the four segments below rather than comparing a single latency claim.

GMI Cloud operates NVIDIA GPU infrastructure and provides serverless APIs and dedicated inference endpoints. (company overview)

What actually happens between the keystroke and the first token?

Between the keystroke and the first token, latency accumulates in four phases: admission and queueing, weight loading, prefill, and first-token delivery. Each has a different owner, a different order of magnitude, and a different fix, which is why a single averaged latency number keeps teams tuning the wrong one.

Segments 1, 3, and 4 are engineering problems with engineering answers. Segment 2 is different: it is a consequence of a billing decision. If your inference runs on capacity that scales to zero when idle, somebody's request has to pay the reload, and in a coding agent that somebody is a developer watching a cursor blink.

NVIDIA splits the phases the same way in its disaggregated serving work, describing prefill as "compute-intensive" and benefiting "from high floating point operations (FLOPS)," and decode as "memory-bandwidth-bound because of the autoregressive nature of LLMs" (NVIDIA Technical Blog). Because the two phases hit different hardware limits, TTFT and tokens-per-second respond to different optimizations, and blending them into a single latency metric hides the one you are trying to find.

How long does loading weights actually take?

Long enough that it can never sit on the request path of an interactive tool: on NVIDIA's published measurements, a 15 GB checkpoint takes between 4.88 and 47.99 seconds to load depending on storage and loader, which projects to roughly 23 to 224 seconds for a 70 GB checkpoint. Those numbers come from NVIDIA loading Llama 3 8B, about 15 GB in Safetensors format, from three storage back ends (NVIDIA Run:ai Model Streamer post):

Converting each row to an effective transfer rate lets you project it onto the checkpoint sizes a coding agent actually serves. Weight bytes are just parameter count times bytes per parameter: FP8 stores one byte per parameter and BF16 two (NVIDIA PTX ISA, alternate floating-point formats), so a 30B model at FP8 is about 30 GB, a 70B at FP8 about 70 GB, and that same 70B at BF16 about 140 GB. Those are weight payloads, not total GPU memory: scales, metadata, and the KV cache sit on top.

Two caveats on this projection. It assumes load time scales linearly with bytes from a single published setup, and NVIDIA's own results show the effective rate moves with storage tier and read concurrency, so a different stack will land somewhere else on the table. It also counts weight transfer only: real engine readiness adds process start, parallel rank setup, and CUDA graph capture on top. Read the cells as order-of-magnitude estimates under a constant-rate assumption, not as a measurement of your stack.

The conclusion survives both caveats. The best case in the table is about 23 seconds for a 70 GB checkpoint, roughly 45 times the entire budget of an agent that wants its first token inside half a second. No loader, no storage tier, and no caching trick closes that gap on the request path. The only way to spend zero seconds on segment 2 is to never do it during a request.

What "warm by default" means on a Prime Inference endpoint

On a Prime Inference endpoint, "warm by default" means the checkpoint is loaded once at deploy time and stays in GPU memory, so the weight-loading phase is removed by construction rather than optimized away. The product page states it directly: "Reserved GPUs stay warm with weights pre-loaded. Every call lands hot, no cold-start delay, no first-token jitter." The GPUs are reserved for your deployment, so nothing evicts the weights between requests.

That same reservation removes the cross-tenant half of segment 1. Your own concurrency can still queue behind itself, which is a replica-count question, but no other customer's traffic shape reaches your tail. Prime Inference is single-tenant: "GPUs reserved only for your workload. No noisy neighbors, no contention under load, no shared-tier surprises." The same page describes the shared serverless tier, where capacity is shared and "throughput and latency vary with platform load." That tier's "Automatic scaling to zero with no idle cost" (GMI Cloud homepage) is the right property for batch jobs and for traffic that genuinely goes quiet. An IDE plugin is the opposite case: there, scale-to-zero is the mechanism generating your worst TTFT samples, and a Prime Inference endpoint with resident weights is what takes that mechanism out of the path.

Segment 3 is where the engine work lives, and a coding agent stresses it harder than a chatbot does because it resends a large, mostly unchanged prefix every turn. Prime Inference ships "vLLM, TensorRT-LLM, and SGLang pre-tuned per GPU class" with quantization configurable and multi-GPU orchestration handled, and lists "Optimized KV-cache" as part of the high-throughput chat and RAG profile. Prefix reuse comes with its own memory budget, though. A September 2026 H100 serving benchmark identifies two regimes, one "where prefix reuse provides substantial first-token latency reductions" and one "where cache pressure erodes them," and reports that the remaining differences between runtimes "arise above the cache, in the scheduling layer" (PrefixBench-H100). In other words, which regime you land in is a memory-pressure and scheduling question, which is an argument for a runtime somebody tunes per model rather than a generic stack.

Segment 4 is a placement decision. The Prime Inference page names the lever directly: "Region-pin endpoints for first-token latency". If your developers are in Berlin and your endpoint is in Oregon, no amount of engine tuning recovers the round trip. Pick the region where the keystrokes originate, not where the rest of your stack happens to live. (Region locking for data residency is a separate requirement, covered in the companion piece on zero-retention dedicated hosting.)

Coding agents are a named target for the product line: the use-case list on the Prime Inference page leads with agents and copilots, specifically "coding agents & developer tools." The tail-latency argument is not theoretical for GMI Cloud either. The GMI Cloud homepage publishes "65% lower p95 inference latency" and "45% lower compute cost" for Higgsfield, which runs real-time generative video on the platform (full Higgsfield case study). Different workload, but the same metric worth copying: p95, not average.

How fast is fast enough for a coding agent?

As working targets, aim for p95 TTFT under 200 ms for inline completion, under 500 ms for a chat panel, and under 1 second per call inside an agent loop. GMI Cloud publishes one number that makes the upper end of that range concrete. In its cost comparison for Kimi K2.7 Code on a single dedicated H200 node, the footnote reads "720K TPM · 43 tok/s per user" (Prime Inference). Those are two different measurements doing two different jobs: 720K tokens per minute is what the node sustains in aggregate, and 43 tokens per second is what one user's stream feels like.

The per-user figure is the one your product manager cares about. At 43 tok/s, each token after the first arrives about 23 ms apart, which is faster than a developer reads. A 400-token code block finishes streaming in roughly 9.3 seconds of steady output. Now add a 2-second cold start to the front of that interaction: the response takes about 22% longer, and all of the added delay lands in the one window where the screen is blank and the user has nothing to look at.

That asymmetry is why TTFT deserves its own SLO. Treat the targets below as starting points to tune against your own telemetry, not as industry constants:

• Inline completion (ghost text as you type): p95 TTFT under 200 ms. This is the regime where segment 4 alone can break you, because a cross-region round trip can consume the whole budget.
• Chat panel and single-file edits: p95 TTFT under 500 ms, with 1 s as the point where the interaction starts to feel like a page load.
• Multi-step agent loops (plan, read files, patch, test): per-call p95 TTFT under 1 s. A ten-call loop multiplies every stray second you leave on the table, so one extra second per call is ten seconds the user waits.
• Any of the above on a scale-to-zero backend: the weight-load table above moves into your tail. How far into the tail depends on how often a request arrives after an idle window, which is a property of your traffic pattern, not of your model.

A TTFT budget worksheet

A TTFT budget assigns a measured number to each of the four phases, then checks whether what is left over is enough to run prefill in. Filling it in takes an afternoon and it changes the provider conversation from "is your API fast" to "which of these four phases are you responsible for."

1. Segment 4 first, because it is the cheapest to fix. Measure round-trip time from your users' network to each candidate region. Anything you cannot remove here is a fixed tax on every target above.
2. Segment 3 next. Take your agent's real prompts, not a toy prompt. Record prompt token counts at the 50th and 95th percentile across a working day. Once repo context, diffs, and tool output get attached, the p95 runs well above the median, and prefill tracks the p95.
3. Segment 1 by observation. Send the same request every 60 seconds for 24 hours against a shared tier and plot TTFT by hour. Queueing shows up as a time-of-day shape that your own traffic does not explain.
4. Segment 2 by provocation. Leave the deployment idle longer than its scale-down window, then send one request and time it. That single sample is not a percentile, it is one observed cold reading, and it is the number a developer will quote back to you after lunch.

Subtract 1, 2, and 4 from your target. Whatever is left is the prefill budget you are actually buying. If that remainder is negative, no model choice fixes it.

Measuring TTFT against an OpenAI-compatible endpoint

The measurement that matters is the time from request send to the first chunk that carries non-empty content, not the first chunk of any kind, since the opening delta usually carries only a role. This script measures that against GMI Cloud's OpenAI-compatible endpoint and handles the failure paths you will hit in practice.

#!/usr/bin/env python3
"""Measure p50/p95 time-to-first-token against an OpenAI-compatible endpoint."""

import json
import math
import os
import sys
import time
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime

import requests
from urllib3.exceptions import ReadTimeoutError

BASE_URL = os.environ.get("GMI_BASE_URL", "https://api.gmi-serving.com/v1")
# Copy the exact model id from the Console model library, or list them
# with GET /v1/models.
MODEL = os.environ.get("GMI_MODEL", "deepseek-ai/DeepSeek-V4-Pro")
API_KEY = os.environ.get("GMI_API_KEY")

CONNECT_TIMEOUT = 5.0      # seconds to establish the connection
READ_TIMEOUT = 30.0        # seconds to wait for the next chunk
MAX_ATTEMPTS = 3           # total attempts per sample, so two retries
MAX_BACKOFF = 60.0         # stop retrying if the server asks for longer
RETRY_STATUS = (429, 500, 502, 503, 504)


def first_token_latency(prompt, max_tokens=64):
    """Return seconds to the first non-empty content delta, or None on failure."""
    if not isinstance(prompt, str) or not prompt.strip():
        raise ValueError("prompt must be a non-empty string")
    if not API_KEY:
        raise RuntimeError("GMI_API_KEY is not set")

    payload = {
        "model": MODEL,
        "messages": [{"role": "user", "content": prompt}],
        "max_tokens": max_tokens,
        "stream": True,
    }
    headers = {
        "Authorization": "Bearer " + API_KEY,
        "Content-Type": "application/json",
    }

    for attempt in range(MAX_ATTEMPTS):
        last_attempt = attempt == MAX_ATTEMPTS - 1
        response = None
        started = time.perf_counter()
        try:
            response = requests.post(
                BASE_URL.rstrip("/") + "/chat/completions",
                headers=headers,
                json=payload,
                stream=True,
                timeout=(CONNECT_TIMEOUT, READ_TIMEOUT),
            )

            if response.status_code in RETRY_STATUS:
                status = response.status_code
                wait = _retry_after(response, attempt)
                if last_attempt or wait > MAX_BACKOFF:
                    print("giving up (HTTP %d, next retry would wait %.1fs)"
                          % (status, wait), file=sys.stderr)
                    return None
                time.sleep(wait)
                continue

            if response.status_code != 200:
                print("HTTP %d: %s" % (response.status_code, response.text[:200]),
                      file=sys.stderr)
                return None

            # chunk_size=1 matters here. The default 512-byte buffer makes
            # iter_lines wait for the buffer to fill, which would be counted
            # as first-token latency that the server never spent.
            response.encoding = "utf-8"
            for line in response.iter_lines(chunk_size=1, decode_unicode=True):
                content = _content_of(line)
                if content:
                    return time.perf_counter() - started
                if content is False:          # explicit [DONE] sentinel
                    break

            print("stream ended with no content delta", file=sys.stderr)
            return None

        except requests.exceptions.RequestException as exc:
            if not _is_timeout(exc):
                print("request failed: %s" % exc, file=sys.stderr)
                return None
            print("timeout on attempt %d" % (attempt + 1), file=sys.stderr)
            if last_attempt:
                return None
            time.sleep(2 ** attempt)
        finally:
            if response is not None:
                response.close()

    return None


def _content_of(line):
    """Content string of an SSE line, False for [DONE], None for anything else."""
    if not line or not line.startswith("data:"):
        return None
    data = line[len("data:"):].strip()
    if data == "[DONE]":
        return False
    try:
        chunk = json.loads(data)
    except json.JSONDecodeError:
        return None
    if not isinstance(chunk, dict):
        return None
    choices = chunk.get("choices")
    if not isinstance(choices, list) or not choices:
        return None                 # usage-only events land here
    if not isinstance(choices[0], dict):
        return None
    delta = choices[0].get("delta")
    if not isinstance(delta, dict):
        return None
    content = delta.get("content")
    return content if isinstance(content, str) and content else None


def _is_timeout(exc):
    """requests wraps a mid-stream read timeout in ConnectionError, not Timeout."""
    if isinstance(exc, requests.exceptions.Timeout):
        return True
    if isinstance(exc, requests.exceptions.ConnectionError):
        return any(isinstance(arg, ReadTimeoutError) for arg in exc.args)
    return False


def _retry_after(response, attempt):
    """Honor Retry-After in either RFC 9110 form, otherwise exponential backoff."""
    header = response.headers.get("Retry-After", "").strip()
    if header.isascii() and header.isdigit():
        try:
            seconds = float(header)
        except ValueError:
            seconds = float("nan")
        if math.isfinite(seconds):
            return seconds
    elif header:
        try:
            when = parsedate_to_datetime(header)
            if when.tzinfo is None:
                when = when.replace(tzinfo=timezone.utc)
            return max(0.0, (when - datetime.now(timezone.utc)).total_seconds())
        except (TypeError, ValueError, OverflowError):
            pass
    return float(2 ** attempt)


def percentile(sorted_values, fraction):
    """Nearest-rank percentile. sorted_values must be non-empty and sorted."""
    rank = max(1, math.ceil(fraction * len(sorted_values)))
    return sorted_values[rank - 1]


def main():
    prompt = (
        "Refactor this function to return early on invalid input:\n\n"
        "def parse(cfg):\n"
        "    if cfg is not None:\n"
        "        return cfg.get('value')\n"
    )
    samples = []
    failures = 0
    for _ in range(20):
        latency = first_token_latency(prompt)
        if latency is None:
            failures += 1
        else:
            samples.append(latency)
        time.sleep(1.0)

    if not samples:
        print("no successful samples; %d failures" % failures)
        return 1

    samples.sort()
    print("n=%d  failures=%d" % (len(samples), failures))
    print("p50 TTFT: %.3f s" % percentile(samples, 0.50))
    print("p95 TTFT: %.3f s" % percentile(samples, 0.95))
    print("max TTFT: %.3f s" % samples[-1])
    return 0


if __name__ == "__main__":
    sys.exit(main())

Run it twice. First during working hours against a warm endpoint, recording p50 and p95. Then let the deployment sit idle past its scale-down window and take one sample on its own. The separate cold reading matters because with 20 sequential requests the nearest-rank p95 is the 19th sorted value, so a single cold outlier never appears in it. The gap between that cold reading and the warm p50 is your segment 2 estimate, not an isolation: queueing and prefill can differ between the two runs, so repeat it on a few different days before the number goes into a design document.

Which GMI Cloud product line matches which traffic shape

GMI Cloud maps sustained interactive traffic with a hard TTFT target to Prime Inference, and bursty or early-stage traffic to serverless MaaS. The dividing line is utilization, not preference.

• Sustained interactive traffic with a hard TTFT target. Prime Inference. Single-tenant reserved GPUs, weights pre-loaded, runtimes tuned per model, region-pinned endpoints. The published break-even marker is "~35% Sustained utilization where dedicated breaks even with serverless" (the comparison panel on the same page phrases it as "~35-45%"), and the methodology note adds that your actual break-even depends on traffic shape, so model it with your own token mix. Billing is hourly per GPU with no minimum contract and no per-token markup; H200 capacity is listed from $3.20 per GPU-hour on the pricing page as of October 2026, and Prime Inference quotes are built per workload with sales.
• Spiky or early-stage traffic where a one-second first token is fine. Start on the serverless Model-as-a-Service tier, pay per token, and move later. The MaaS page FAQ notes that teams "transition between these deployment models without changing APIs," and the full serverless-to-dedicated cost comparison is covered in moving a serverless inference bill to dedicated GPUs.
• Multiple model tiers inside one agent. Keep the latency-critical path on a warm dedicated endpoint and route the rest. The mechanics of routing a coding agent without burning an extra call or breaking the cached prefix are covered separately in routing a coding agent's cached context, and GMI Cloud's own harness experiment on escalation is in Dynamic Model Escalation for Coding Agents.

Deployment is four steps: pick a model (any open-source model, anything from Hugging Face, or your own weights), choose GPU type, GPU count per replica, replica count and target region, deploy from console, CLI or API, then operate and scale. The page states the endpoint is "live in minutes, not days."

The GPU-count part of step two follows from the weight arithmetic above. A 70B coding model at FP8 is about 70 GB of weights, which fits inside the 141 GB of HBM3e on a single H200 (NVIDIA H200 specifications) and leaves headroom for the KV cache that a long repo context consumes. The same 70B at BF16 is about 140 GB and leaves effectively none, which is what pushes one replica onto several GPUs. GMI Cloud's own published Prime Inference measurement for Kimi K2.7 Code runs FP8 on H200, and H200 capacity starts at $3.20 per GPU-hour on the pricing page, so the quantization decision and the per-replica bill move together.

FAQ

Does keeping weights warm fix first-token latency, or only cold starts? It removes model loading, not queueing, prefill, or network latency. Prime Inference pre-loads weights on reserved GPUs; the remaining segments still govern warm-request tails.

How much of TTFT can KV-cache reuse save for a coding agent? Reuse can cut prefill substantially when the prefix remains stable. Cache pressure can erode that benefit, and published H100 benchmarking attributes remaining runtime differences to scheduling. Validate both memory pressure and scheduling on the target workload.

Our traffic is bursty and quiet overnight. Do we have to pay for a warm GPU around the clock? Not necessarily. Hourly billing has no minimum contract, with graceful scale-down and burstable capacity. Agree on the minimum warm replica count and idle policy. GMI's roughly 35% sustained-utilization break-even is a workload-dependent benchmark; low-utilization traffic may cost less on MaaS, with later API-compatible migration.

Can we run our own fine-tuned coding model on a warm endpoint? Yes. Supply open-source, fine-tuned, or proprietary weights through Hugging Face, S3, or private storage. GMI manages runtime tuning, quantization, multi-GPU serving, and the endpoint. Managed fine-tuning is a separate service listed as coming soon with early access.

What TTFT number should we put in our internal SLO? Set targets by interaction type and specify the measurement method. Keep the full-traffic distribution for the user-facing SLO, plus warm-request p95 and a separate post-idle cold reading. With 20 samples, nearest-rank p95 excludes the single largest value.

Our p50 is fine but the p95 doubles during standups. Is that a cold start? Check queue/admission time, replica-start and model-load events, and prompt token counts in the same window. Rising queue time alone suggests replica capacity; a matching load event suggests cold start. Hourly TTFT cannot separate these causes.

Next step

If your coding agent's p99 TTFT looks nothing like its p50, run the two-measurement test above before changing models. When the gap turns out to be weight loading, the fix is a Prime Inference endpoint with your weights resident on reserved GPUs in the region your developers sit in. Check current GPU rates on the pricing page, or contact our sales team for a quote built around your model and traffic shape, including trial GPU-hour credits to validate TTFT against your own workload before you commit.