Model Routers for Coding Agents: Routing Without a Judge Call or a Cold Cache

GMI Cloud's GMI Router selects a model without a judge-model call and accounts for cache state when routing. It switches only when the expected quality gain outweighs the cache-miss cost.

In the worked example below, a judge call costs about $0.0008, while a cold 40,000-token prefix costs 25 to 245 times as much. Cache misses can therefore outweigh the fee saved by avoiding a judge.

What does an extra LLM call actually cost per turn?

A judge-based router adds a model round trip; switching models can also create a cold prefix. These are separate costs.

The judge queues, prefills, and generates before the main request starts, adding its latency to TTFT. GMI Router instead uses a precomputed benchmark and a lightweight prediction, as described in the Router launch post.

The judge call itself does not invalidate the main model's cache. Switching models does: the new model has not seen the conversation prefix and must prefill it at full input price.

Assume a judge receives a 900-token rubric and 4,000 tokens of recent trajectory, then returns a 30-token verdict. Including recent tool output lets it distinguish a typo from a deeper failure. These are worked assumptions, not GMI measurements; substitute your own prompt sizes.

That is $0.032 for a 40-turn session, or about $319 for 10,000 sessions. Rates use Qwen/Qwen3.8-Flash list prices in the GMI Cloud model library on October 15, 2026, without discounts. Check current rates in the Console.

Why does one model switch cost more than the judge call?

A cache miss costs the difference between fresh-input and cached-input prices for every unmatched prefix token. Compare that difference, not just headline input rates.

At $0.000798 per judge call, the following prefix lengths make one missed cache hit more expensive than the judge:

Prices read from the GMI Cloud model library on October 15, 2026, all with no catalog discount on that date.

These are base-tier rates. Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 has further tiers at 32,000 and 128,000 tokens; openai/gpt-5.5 has one at 272,000. Use the tier that applies to the whole request.

For the frontier row, the threshold is about 177 tokens, often less than the system prompt and tool schema alone.

At turn 12 with a 40,000-token prefix, both models below remain in their base pricing tier:

Frontier escalation costs about 20 judge calls with a warm prefix, versus 245 with a cold one: roughly 225 calls' worth is the cache miss. Prefilling those 40,000 tokens also adds TTFT; output tokens are excluded.

In Dynamic Model Escalation across two harnesses, GMI tested 89 Terminal-Bench 2.1 tasks on Terminus 2 and Claude Code. Cheap-model trials cost $0.04 and $0.07, versus $1.15 and $1.97 on a frontier model. On Terminus 2, 57 tasks needed only the cheap model. Escalation policy must account for both model quality and the cost of losing a warm prefix. (Dynamic Model Escalation experiment across two harnesses)

How does GMI Router decide without a judge model?

The GMI Router documentation describes five steps within one routing request:

1. Understand. "The router reads your latest message and classifies the task and its requirements."
2. Compare. "It scores the allowed model pool across quality, cost, and latency."
3. Validate. "The top candidate is checked before it is committed. A high score alone does not win the request."
4. Select. "The router locks in the winning model."
5. Respond. "The selected model generates the completion, returned inline or streamed token by token."

Three properties of that sequence matter for a coding agent:

• It is one call, not two. The /autoroute endpoint "picks the model and returns the completion in one call." The agent sends one request per turn and selection happens on the way through. The product page puts routing time under 200ms, and the launch post reports roughly under 200ms of routing-backend latency in GMI's current measurements.
• Validation is a separate step from scoring. Step 3 exists so that a candidate with the highest score still has to pass a check before it gets the request. A score-only policy is the kind that flips models on small deltas, which is the behavior that costs you the prefix.
• Scoring is bounded by your pool. Routing runs inside the Allowed Model Pool your organization's owner approves, so the set of models a session can move to is a configuration you set rather than an open catalog.

The GMI Router product page lists recommendation and routing as free for a limited preview period. Standard token charges for the selected model still apply.

How does the router price a cache hit when it ranks models?

GMI's launch post defines the cache-adjusted ranking formula:

"Routing ranks on effective cost per request, which differs from list price. A matched prefix bills at the cached-input rate, so the same model carries two different prices depending on how warm the session is."

effective cost = hit × cached input price + (1 − hit) × full input price + output

For 40,000 input tokens, a warm zai-org/GLM-4.7-FP8 turn costs $0.0044; a cold openai/gpt-5.5 turn costs $0.2000. Their list input rates, $0.60 versus $5.00, suggest an 8x gap; actual cache-adjusted input cost differs by about 45x.

Cost ranks by cache-adjusted request price; Balanced optimizes quality per cache-adjusted dollar; Quality uses cache hits to break ties between models with equal quality and price.

The serving layer hashes incoming prefixes against a global KV index synced across clusters and sends requests to the cluster with the longest match. Matched tokens skip prefill. The product page limits this to supported workloads, so confirm coverage for your model pool.

If the router is stateless, who owns the prefix?

The client owns the conversation. These statements refer to different layers:

• Product page: "KV-cache Preserved across turns".
• Documentation: "The router is stateless. Resend the full conversation on every request."

The stateless router stores no session transcript. The serving layer reuses matching prefixes supplied by the client, and routing prices that reuse.

How do you call /autoroute from a coding agent?

Send one POST per turn with the full messages array. Optional mode accepts cost, balanced, or quality. Streaming is the default; set stream to false for one-shot JSON. Routing considers the full conversation, not just its latest message.

import os
import time

import requests

ROUTER_URL = "https://console.gmicloud.ai/api/v1/ie/recommendation/autoroute"
VALID_MODES = ("cost", "balanced", "quality")

# The router's documented non-streaming budget is a combined 10 minutes for the
# primary model plus one backup, so the client allowance sits above it. Note that
# the read timeout in requests is an inactivity window between received bytes,
# not a total wall-clock deadline.
CONNECT_TIMEOUT_S = 10
READ_TIMEOUT_S = 630


class RouterError(RuntimeError):
    def __init__(self, message, status=None, recommendation_id=None):
        super().__init__(message)
        self.status = status
        self.recommendation_id = recommendation_id


def route_turn(messages, mode="balanced", session=None, api_key=None):
    """Send one coding-agent turn through /autoroute and return the reply."""
    if not messages:
        raise ValueError("messages must contain at least one message")
    for m in messages:
        if not isinstance(m, dict) or not m.get("role"):
            raise ValueError("every message needs a role")
        # An assistant turn that only calls tools carries no content, which is
        # normal in a coding agent. Everything else needs a content value.
        if m.get("content") is None and not m.get("tool_calls"):
            raise ValueError(f"{m['role']} message needs content or tool_calls")
    if not any(m["role"] == "user" for m in messages):
        raise ValueError("messages must contain at least one user message")
    if mode not in VALID_MODES:
        raise ValueError(f"mode must be one of {VALID_MODES}, got {mode!r}")

    key = api_key or os.environ.get("GMI_API_KEY")
    if not key:
        raise ValueError("set GMI_API_KEY before routing a turn")

    http = session or requests.Session()
    started = time.perf_counter()
    try:
        response = http.post(
            ROUTER_URL,
            headers={
                "Authorization": f"Bearer {key}",
                "Content-Type": "application/json",
            },
            json={"messages": messages, "mode": mode, "stream": False},
            timeout=(CONNECT_TIMEOUT_S, READ_TIMEOUT_S),
        )
    except requests.Timeout as exc:
        raise RouterError("autoroute did not respond inside the client budget") from exc
    except requests.RequestException as exc:
        raise RouterError(f"autoroute transport failure: {exc}") from exc

    if response.status_code != 200:
        # 401 and the non-429 4xx codes are request problems that a retry will
        # not fix. For 429 and 5xx, the documented fallback is a single backup
        # attempt inside the router, so an immediate client-side retry starts a
        # second full routing cycle instead of resuming the first. Back off.
        raise RouterError(
            f"autoroute returned {response.status_code}: {response.text[:200]}",
            status=response.status_code,
        )

    try:
        payload = response.json()
    except ValueError as exc:
        raise RouterError("autoroute returned a non-JSON body", status=200) from exc
    if not isinstance(payload, dict):
        raise RouterError("autoroute returned a JSON value that is not an object", status=200)

    # Autoroute returns model, message, and routing_metadata at the top level.
    # There is no choices array to index into.
    message = payload.get("message")
    message = message if isinstance(message, dict) else {}
    content = message.get("content")
    content = content if isinstance(content, str) else ""
    tool_calls = message.get("tool_calls")
    tool_calls = tool_calls if isinstance(tool_calls, list) else []
    metadata = payload.get("routing_metadata")
    metadata = metadata if isinstance(metadata, dict) else {}
    rec_id = metadata.get("recommendation_id")

    if not content.strip() and not tool_calls:
        # A reply with neither text nor tool calls is a failed turn, not an
        # assistant message, and writing it into the transcript poisons the
        # prefix for every turn after it. A tool-calling turn legitimately
        # carries no text, so it has to be admitted here rather than rejected
        # alongside the genuinely empty ones.
        raise RouterError(
            "autoroute returned neither content nor tool calls",
            status=200,
            recommendation_id=rec_id,
        )

    return {
        "content": content,
        "tool_calls": tool_calls,
        "selected_model": metadata.get("selected_model") or payload.get("model"),
        "task_type": metadata.get("task_type"),
        "selected_mode": metadata.get("selected_mode"),
        "fallback_models": metadata.get("fallback_models") or [],
        "recommendation_id": rec_id,
        "elapsed_ms": round((time.perf_counter() - started) * 1000, 1),
    }

The non-streaming parser expects this response shape:

{
  "model": "provider/model-name",
  "message": { "role": "assistant", "content": "..." },
  "routing_metadata": {
    "selected_model": "provider/model-name",
    "task_type": "summarization",
    "selected_mode": "balanced",
    "fallback_models": ["provider/backup-a"],
    "recommendation_id": "bab0a8bb-195f-49c5-8f9c-016da0d89cc8"
  }
}

Two integration paths, two response shapes

• OpenAI-compatible. The launch post shows an OpenAI client pointed at https://console.gmicloud.ai/api/v1/ie/recommendation with model="auto" and the mode passed in extra_body. That path returns an ordinary chat completion, so response.choices[0].message.content works and existing client code migrates with a base URL and key change.
• Native /autoroute. Documented above, with model, message, and routing_metadata at the top level and no choices array, which is why the parser reads payload["message"]["content"]. Code that assumes choices against this endpoint raises a KeyError on the first turn.

The launch post's /recommendations endpoint inspects task type and model selection without generating a completion. Use it for offline reports; calling it before every turn would add another round trip.

What happens when the selected model fails mid-turn?

One backup may be tried before the first token. Once streaming starts, failures retain partial output, emit an error event, and close the stream without further fallback. Client retries must account for that boundary.

Streaming chunks follow the selected provider's schema, so validate the extractor against a live response. OpenAI-style text uses choices[0].delta.content; tool-only turns require accumulating delta.tool_calls and must not be treated as failures merely because text is empty. The documented sequence is chunks, routing_metadata, then [DONE], or an error event. The parser below recognizes control events by event name and payload key. It accumulates data: lines and dispatches only on an empty line: event: lines and space-only lines are not boundaries. At EOF it deliberately flushes an unterminated final event to preserve partial output. Capture a live response before deployment to confirm the actual envelope.

import json


def consume_stream(lines, extract_text):
    """Turn an SSE line iterator into (text, metadata, status).

    extract_text(payload) -> str is supplied by the caller, because chunk bodies
    are proxied from the selected model's provider rather than normalized by the
    router. Framing follows the SSE rules: data lines accumulate, an event line
    names the event in progress, and an empty line dispatches it. Control events
    are matched on the event name and on the payload key, so either wire shape
    works. Nothing here lets a bad byte, a bad chunk, or a dropped connection
    cost the caller tokens already received.
    """
    state = {"text": [], "metadata": {}, "status": "incomplete", "dropped": 0}
    event_name = None
    data_parts = []

    def handle(body, name):
        """Process one dispatched event. Returns True when the stream stops."""
        if body.strip() == "[DONE]":
            state["status"] = "complete"
            return True
        try:
            payload = json.loads(body)
        except ValueError:
            state["dropped"] += 1
            return False
        if not isinstance(payload, dict):
            state["dropped"] += 1
            return False
        # Error is checked first, so a payload carrying both an error and
        # routing metadata cannot be swallowed by the metadata branch.
        if name == "error" or "error" in payload:
            state["status"] = "stream_error"
            state["metadata"].setdefault("error", payload.get("error", payload))
            return True
        if name == "routing_metadata" or "routing_metadata" in payload:
            found = payload.get("routing_metadata", payload)
            if isinstance(found, dict):
                state["metadata"].update(found)
            else:
                state["dropped"] += 1
            return False
        try:
            chunk_text = extract_text(payload)
        except Exception:
            state["dropped"] += 1
            return False
        if chunk_text:
            state["text"].append(chunk_text)
        return False

    def dispatch():
        """Emit the buffered event, if any, and reset the buffers."""
        nonlocal event_name, data_parts
        name, parts = event_name, data_parts
        event_name, data_parts = None, []
        if not parts:
            return False
        return handle("\n".join(parts), name)

    first_line = True
    try:
        for raw in lines:
            try:
                line = raw.decode("utf-8") if isinstance(raw, bytes) else str(raw)
            except UnicodeDecodeError:
                state["dropped"] += 1
                continue
            if first_line:
                first_line = False
                if line.startswith("\ufeff"):
                    line = line[1:]  # one leading BOM belongs to the stream
            # Strip the line terminator only. A line holding a single space is a
            # field line with an unrecognized name, not an event boundary, and
            # treating it as one would cut a multi-line payload in half.
            line = line.rstrip("\r\n")

            if line == "":
                if dispatch():
                    break
                continue
            if line.startswith(":"):
                continue
            field, _, value = line.partition(":")
            # One space after the colon belongs to the delimiter. Everything
            # else, trailing spaces included, is part of the value.
            if value.startswith(" "):
                value = value[1:]
            if field == "event":
                # An event field names the event in progress. It does not end
                # it, so metadata that arrives before its own event line is
                # still attributed correctly. A bare "event" line clears the
                # name again, which is what the spec says an empty value means.
                event_name = value
            elif field == "data":
                data_parts.append(value)
        else:
            dispatch()  # flush an event that arrived without a trailing blank line
    except Exception as exc:
        # A dropped connection ends the stream without an error event. Flush
        # whatever was buffered before recording the failure.
        try:
            dispatch()
        except Exception:
            pass
        if state["status"] != "complete":
            state["status"] = "stream_error"
        state["metadata"].setdefault("error", repr(exc))

    text = "".join(state["text"])
    status = state["status"]
    if status == "complete" and not text.strip():
        status = "empty"
    elif status == "complete" and state["dropped"]:
        # Reaching [DONE] is not the same as receiving a complete turn.
        status = "complete_with_gaps"
    if state["dropped"]:
        state["metadata"]["dropped_chunks"] = state["dropped"]
    return text, state["metadata"], status

Mock the injectable session and iterator to test timeouts, connection drops, 429/5xx responses, empty or malformed completions, stream errors, truncated streams, missing user messages, and invalid modes.

What breaks a prefix cache in a coding agent?

Keep the prefix stable so cache-aware ranking can help:

1. A timestamp, request ID, or token budget counter in the system prompt. One changing character at position 40 invalidates everything after position 40. Move volatile values into the latest user message, at the tail of the prefix.
2. Re-summarizing or re-ordering earlier turns mid-session. Compaction is sometimes necessary, but it rewrites the prefix. Treat it as a deliberate, budgeted cache reset rather than something a background job does every few turns.
3. Non-deterministic serialization of tool results. Unsorted dictionary keys, platform-dependent float formatting, and file lists in filesystem order all produce a different prefix for identical state. Serialize tool output with sorted keys and a fixed float format.
4. Switching models for a reason unrelated to the cache. This is the row worth 225 judge calls in the table above. A quality score that moves by a point does not pay for a 40,000-token refill.

Which routing mode should a coding agent use?

Match the routing mode to the task:

GMI recommends starting in Cost mode, measuring against your quality threshold, and using Balanced or Quality where results justify the premium.

Treat model changes as explicit cost decisions once prefixes exceed roughly 177 tokens on openai/gpt-5.5 or 1,100 on Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8. Single-shot requests below those thresholds have little cache value to lose. Cost per million tokens explains the token accounting. (cost per million tokens)

FAQ

Does GMI Router add a second model call to every request? No. /autoroute selects the model and returns its completion in one call, using a lightweight prediction rather than a judge-model request.

Does the router store my conversation so the cache survives between turns? No. Resend the full conversation on every request. The serving layer matches that prefix against its KV index; the router does not store the transcript.

Does cache state actually change which model gets picked? Yes. Cost uses cache-adjusted price, Balanced uses quality per cache-adjusted dollar, and Quality uses cache hits to break equal-quality, equal-price ties. A switch must justify its cache-miss cost.

Can I see which model handled a turn? Yes. Non-streaming responses include routing_metadata with selected_model, task_type, selected_mode, fallback_models, and recommendation_id. Streaming sends the metadata before [DONE]. Log selected_model for each turn.

Does the API tell me whether a turn hit the cache? No documented cache-hit field appears in the LLM API usage object, which contains prompt_tokens, completion_tokens, and total_tokens. Use Console usage and invoices to measure savings.

Is there a routing fee on top of the tokens the selected model bills? Recommendation and routing are free for a limited preview period. The selected model's token charges still apply; rates are in the model library and billed through GMI Cloud's model API account.

How should I set a client timeout for /autoroute? Allow for the service's timeout budget: non-streaming permits up to 10 minutes across primary and backup; streaming can try a backup after a 10-second first-token timeout. Shorter client deadlines may end requests that remain within the service allowance.

Start routing

Start one session in cost mode, log selected_model and task_type per turn, and compare a week of billing with your fixed-model baseline.

• Read the mechanics: GMI Router documentation
• See the modes, pool controls, and preview pricing: GMI Router
• Check per-model input, output, and cached input rates: GMI Cloud model library

Limit the Allowed Model Pool to acceptable alternatives and agree on pool and mode settings before moving production traffic.