Automatic Model Selection for Every Prompt: GMI Router Is Live

GMI Router is now live on GMI Cloud: a benchmark-backed routing layer that reads each prompt and sends it to the model with the best measured quality for the cost. One endpoint, three optimization modes, KV-cache-aware infrastructure underneath, and lightweight routing without a judge model in the request loop. Here is why it exists and what it does with your traffic.

One Model for Every Prompt

A coding agent rarely does just one kind of work. In the same session, it may inspect a repository, summarize context, generate code, call tools, debug an error, and plan the next step. Yet most applications send every one of those requests to the same model. That is simple to operate, but it also means paying the same intelligence premium for very different levels of difficulty.

This is how most teams run inference today. Pick one strong model, point all traffic at it, ship. The choice feels safe and it ships fast. But a production application behaves like several tools in one. A single chatbot session summarizes, translates, debugs code, classifies, extracts, and reasons, and only a fraction of that work needs the strongest model money can buy. The rest rides along at premium prices, and spend grows faster than output. When a lighter model produces an equally good answer, that answer is the better buy.

The obvious fix fails for the opposite reason. Switch everything to a cheap model and the hard requests, the ones your users judge you on, get worse. Betting on any single model means overpaying on the easy work and underdelivering on the hard work at the same time. Router is engineered to break that trade-off.

The Prompt Is the Deciding Unit

Categories show why per-prompt routing pays off, and the prompt itself carries the detail that picks the model. "Write a regex to validate an email" and "untangle a race condition in an async job queue" are both coding prompts, and one is routine while the other deserves the strongest model available. The unit of decision is the individual prompt: its complexity, its task type, its domain. That decision arrives ten thousand times a day, so teams make it once at build time and hold it.

The benchmark shows the same thing at the category level. On GMI's internal benchmark, the leaderboard reshuffles from one task category to the next. A model that leads on coding may not lead on agentic planning, reasoning, or long-context work, since the ranking shifts with the task. The interactive leaderboard on the Router page makes this visible: switch tabs from Coding to Reasoning to Math and watch the ranking rearrange itself.

Router makes that call on every request.

Read the Prompt. Route the Model.

Router is a routing layer in front of GMI's inference cloud. You send every request to one endpoint, and a lightweight prediction picks the model before any tokens are generated. Your prompt then runs verbatim on the model that wins, and the response comes back through the same endpoint with the selected model in the routing metadata.

The prediction is grounded in measurement rather than popularity. Models are evaluated across more than 10 benchmarks and additional evaluation sources to build task-level performance profiles. At request time, four steps happen:

1. Analyze. The router reads your prompt on complexity, task type, and domain, and finds the closest matches in the evaluation set. The match is made on your specific prompt, so a routine request and a hard request from the same category can route to different models.

2. Score. Each model has already been graded on those reference prompts, so the router reads off its measured quality on prompts like yours.

3. Rank. Measured quality is weighed against each model's cost, with your chosen mode setting the balance, across the models in your allowed pool.

4. Route. The top model runs your prompt.

Two properties of this design matter in production.

Routers that ask a judge model to score your prompt add a full LLM call in front of yours. GMI Router reads a precomputed benchmark instead, so the decision costs one lightweight prediction. With no judge model in the request loop, the routing backend has roughly <200ms of latency in our current measurements.

Routing also runs on GMI's own KV-cache-aware inference infrastructure. Agents repeat their system prompt, tool definitions, and conversation history on every turn, so GMI Router hashes each incoming prefix against a global KV index synced across clusters and dispatches to the cluster holding the longest match. Matched tokens skip prefill entirely, which lowers time to first token and raises throughput per GPU. This is a core differentiator of GMI Router: model routing and KV-cache-aware serving are designed as connected layers for multi-turn and agent workloads.

The net effect is frontier intelligence at lower cost. Each prompt goes to the model that clears the quality bar for the least spend, so premium prices apply only where premium capability is doing the work.

GMI Router operates across a technically curated set of frontier and cost-efficient open- and closed-source models, with additional models under continuous evaluation for future launches. The pool includes Kimi K3, GPT-5.5, Gemini-3.1-Pro, Claude Opus 4.8, DeepSeek-V4-Pro, GLM-5.2, Qwen 3.7-Max, Hy-3, and more.

Key facts:

Choose What GMI Router Optimizes For

Routing always weighs quality against cost. The mode decides how the scale tips.

• Cost minimizes spend while keeping the highest practical intelligence, built for high-volume workloads like classification and summarization.

• Balanced targets strong quality at lower cost, landing on the everyday models most teams already daily-drive. This is the sensible default for most production traffic.

• Quality maximizes capability, delivering top-tier performance while staying the lowest-priced among the models that can handle the task.

Quality mode carries a design decision worth pausing on. When several models reach the same top quality for a prompt, it takes the cheapest of them. On an easy prompt that many models can ace, even Quality mode may pick an efficient model, and that is the point: you asked for the best answer, and Quality mode delivers it at the lowest price that reaches it.

Two workspace-level settings round out the controls: the Allowed Model Pool, set by the workspace owner, and Auto Mode, which turns automatic routing on or off per workspace. Both persist across sessions, while the mode itself is picked per request, so a single application can run its bulk traffic in Cost and its critical paths in Quality.

Cache Hits Shape the Choice

Routing ranks on effective cost per request, which differs from list price. A matched prefix bills at the cached-input rate, so the same model carries two different prices depending on how warm the session is.

effective cost = hit × cached input price + (1 − hit) × full input price + output

A warm cache is an asset. Staying on a model keeps its KV warm and a switch starts cold, so GMI Router moves when the quality gain beats the cache-miss cost. Each mode reads that signal differently: Cost ranks by cache-adjusted price per request, Balanced optimizes quality per cache-adjusted dollar, and Quality uses cache hits to break remaining ties between models of equal quality and equal price.

Measured, Head to Head

In GMI's internal evaluations across the ten task categories:

• Quality mode scored 2.4 points higher than routing everything to GPT-5.5, while also cutting cost per task by 28%.

• Balanced mode cut cost per task 84% versus Opus 4.8 while holding comparable output quality.

• On the quality-versus-cost curve, Quality mode reached 84.9% benchmark quality at 0.72x the cost of always running GPT-5.5 (82.5% quality at 1.00x). Balanced mode matched an always-Opus-4.8 setup on quality, 81.3% for both, at 0.22x the cost baseline versus 1.38x.

• In a multi-task chatbot session, dynamic routing reduced total session cost by above 90% compared to running every message on a flagship model.

Figures are based on GMI's internal benchmark comparing GMI-served models head-to-head, are directional, and may vary by workload, prompt length, allowed model pool, and routing settings.

GMI Router Quality mode scores above the best single model in the pool, and Balanced mode reaches near-frontier quality at roughly a fifth of the cost per task.

Head-to-head results across public benchmarks:

Quality mode leads every single-model baseline on all five public benchmarks.

The pattern worth noticing sits in the Cost column. Cost mode stays within a few points of the flagships on graduate-level science questions and math, and outscores Opus 4.8 edges past on LiveCodeBench and data analysis, while routing to models priced at a fraction of flagship rates.

The economics behind these numbers are straightforward. Compact models on GMI start under $0.10 per million tokens while flagship output runs $25 per million. A fixed model choice leaves that entire spread on the table. Routing collects it, request by request: routine prompts land on price-efficient models, and the hard ones stay on the strongest.

What We Learned Building on It

Before launch, our team built two working applications on Router to see how routing behaves under real traffic. Both pointed at the same conclusion: Cost mode is remarkably effective at keeping spend down while meeting quality expectations.

The first build is a multi-task chatbot that summarizes, translates, debugs code, classifies, extracts, and reasons within a single session. A live ledger compares every message against a flagship-only baseline, and it regularly showed savings above 90%. The compact models Cost mode routed to met the quality bar outright: summaries read cleanly, classifications landed correctly, and extractions came back structured, at prices under 10 cents per million tokens.

The chatbot demo's session ledger, comparing per-message cost and output speed against a single flagship model.

The second build is Autograder, an education agent that grades a full class set of papers through an eight-step pipeline spanning very different kinds of work, from structured extraction to open-ended evaluation. The graded papers held up side by side against a comparison lane running every step on Opus 4.8, at a fraction of the cost. Our advice after weeks of building: start in Cost mode, measure against your own quality bar, and reserve Balanced or Quality for the workloads where your data shows they earn the premium.

Autograder grading a batch of papers, with each pipeline step routed to its best-fit model.

Start Routing

Router works with any OpenAI-compatible client, so existing code migrates with a base URL and key change. Point your requests at the autoroute endpoint and the router handles model selection:

import openai
client = openai.OpenAI(
    api_key="YOUR_GMI_API_KEY",
    base_url="https://console.gmicloud.ai/api/v1/ie/recommendation"
)
response = client.chat.completions.create(
    model="auto",
 messages=[
        {"role": "user", "content": "Write a Python function that deduplicates a list of dicts by key."}
    ],
    extra_body={"mode": "balanced"}
)
print(response.choices[0].message.content)   

To inspect routing decisions without generating a completion, call the recommendations endpoint with a short intent string:

import requests
r = requests.post(
    "https://console.gmicloud.ai/api/v1/ie/recommendation/recommendations",
    headers={"Authorization": "Bearer YOUR_GMI_API_KEY"},
    json={"intent": "Debug a race condition in an async job queue", "mode": "quality"}
)
print(r.json())  # detected task type and selected model

You can also try routing interactively before writing any code on the test routing page in the console, which shows the selected model and an estimated cost saving per prompt. Check the full documentation on: https://docs.gmicloud.ai/inference-engine/gmi-router/gmi-router-overview