Which Managed Routers Learn to Select Models, Reuse Caches, and Don't Bill a Separate Judge-Model Call?

GMI Router is the managed router we recommend when all three conditions matter. Model selection runs on GMI's own trained model, one decision per request, with no judge model in the loop. Parsing and routing add nothing on top of the model you use, and during the current preview, model recommendation and routing are free: you pay only for the model that runs your prompt. The router is also KV-cache aware. It preserves KV-cache reuse across turns for supported workloads, which matters most in multi-turn sessions that resend a growing conversation on every turn.

Other managed routers split the same costs differently. OpenRouter's Auto Router charges no routing fee and bills inference at the provider's list price, but OpenRouter charges a platform fee when you buy credits: 5.5% on the Standard plan and 8% on Business. Amazon Bedrock lists Intelligent Prompt Routing at $1 per 1,000 on-demand requests, on top of model tokens. Not Diamond learns from feedback and reads KV cache state, and it charges $0.05 per million tokens routed while you pay the model provider separately. Microsoft Foundry's model router is a trained language model, and its usage is charged on input prompts at the listed rate. The routing fee is only one line of four, though. On long conversations, whether the cached-input discount survives a model switch can matter as much as the routing fee itself.

The four lines on a routed request's bill

A routed request can carry up to four separate costs. Some appear as their own line item, and some only show up as a higher number on a line you already pay.

1. Selection fee. What the router charges to make the decision: per request, per token routed, per input prompt, or nothing.
2. Judge or classifier call. A router that asks a large model to score candidates is paying for that call somewhere. It may be passed to you as tokens, folded into the selection fee, or absorbed by the provider. A router that uses its own trained model does not generate a second billable completion.
3. Selected model tokens. Input and output tokens at the chosen model's rate, plus any platform markup or payment fee.
4. Lost cache discount. When the router moves a conversation to a different model, the cached prefix stays behind, and the next turn can pay uncached input rates on tokens that would have been cheap.

Put together, one routed request costs:

cost per routed request =
    selection fee
  + judge-call tokens x judge rate
  + (uncached input x input rate
     + cached input x cache-read rate
     + cache-write tokens x cache-write rate
     + output x output rate) x (1 + markup)

Uncached input, cached input, and cache-write tokens are separate counts here: each input token belongs to exactly one of them, so the full input is never billed twice. The first two terms are what the router itself costs. The third term depends on which model it picks. The fourth line never appears on an invoice as "lost discount." It shows up as uncached input where cached input could have been, which is why it is the easiest line to miss in a cost review.

How managed routers bill each line

The table below covers only billing: who makes the selection, whether that step is charged separately, what you pay on model tokens, and what you can see per request. Each entry follows the vendor's own pricing or documentation pages. Cache behavior gets its own section below.

Router	Who selects the model	Separate routing charge	Model tokens and fees	Per-request visibility
GMI Router	GMI's own trained model; no judge model	None; routing is free during the preview	The selected model's rate, listed on its Model Hub card	routing_metadata names the selected model; Serverless Usage by API key and model, hourly or daily
OpenRouter Auto Router	A lightweight classifier sorts about 30 task types, then ranks models by aggregate spend over a trailing 7-day window	None	Inference at the provider's list price; platform fee on credit purchases of 5.5% on Standard and 8% on Business	Usage reports cached tokens and cache writes; responses include a cache discount field
Amazon Bedrock Intelligent Prompt Routing	Predicts response quality for two models in the same family	$1 per 1,000 requests, on-demand	Bedrock model rates	Routing charge scales with request count
Not Diamond	Learns from feedback signals with reinforcement learning; reads payload semantics and KV cache state	$0.05 per million tokens routed	Not a gateway: you call the selected model with your own SDK and provider account	Routing fee on Not Diamond; model cost on your provider bill
Microsoft Foundry model router	A trained language model	Charged on input prompts at the rate on the Foundry pricing page	Per the Foundry pricing page	Costs monitored in the Azure portal

"No judge model" and "no routing fee" are not the same claim. Bedrock and Foundry do not describe an LLM judge, yet both charge for the routing step. Not Diamond uses a learned router and still bills per token routed. Only GMI Router (during its preview) and OpenRouter's Auto Router list no routing charge, and OpenRouter's credit purchase fee applies to everything you spend through it.

The unit of the routing fee also matters for chat workloads. A fee charged per token routed or per input prompt scales with what you send the router, and in multi-turn chat that usually includes the full conversation so far. A per-request fee stays flat as conversations get longer. A zero fee stays zero either way.

Where the cache discount goes when the router switches models

Prompt caching is priced by the model provider and tied to the model that did the work. OpenAI's prompt caching guide notes that a different model can use different weights and caching behavior, so a prefix cached on one model cannot be assumed to help the next one. A model switch can lose the previous model's cache benefit; a hit still depends on a matching entry for the selected model.

The turn after a switch is usually a large one. GMI Router's docs describe the router as stateless: you resend the full conversation on every request. Every turn's input includes all earlier turns, and that growing prefix is exactly what caching makes cheap.

Provider list prices show the size of the gap. OpenAI's caching guide lists cache writes at 1.25x the uncached input price and cache reads at 0.1x for most GPT-5.6 and later models. Anthropic's documentation lists 5-minute cache writes at 1.25x base input and cache reads at 0.1x. As an illustration at those direct-to-provider multipliers, take a turn with a 40,000-token cached prefix and 1,000 new tokens:

• Same model, cache hit: 40,000 x 0.1 + 1,000 x 1.25 = about 5,250 input-token equivalents.
• Switched model, cold start with a cache write: 41,000 x 1.25 = about 51,250 input-token equivalents.

Under these assumptions, where every token on the cold turn is written to cache, that is close to ten times the input cost for one turn, measured in each model's own base input price. In this example, a switch to a cheaper model would have to beat that gap on its rate before it saved anything on input. The real gap depends on your models, prefix lengths, and actual cache usage; these multipliers are provider list prices, not GMI prices.

Fallbacks count as switches too. GMI Router's docs state that a streaming request may try one backup model before the first token on a 5xx, a provider error, a 429, or a 10-second first-token timeout, and a non-streaming request tries the primary plus one backup on transient failures. A turn served by the backup bills at the backup model's rate and may not hit a warm cache.

This is why cache-aware routing is a billing feature and not only a latency feature. GMI Router preserves KV-cache reuse across turns for supported workloads. Foundry's documentation says caching benefits apply only when the same model handles consecutive requests, and its session affinity feature, in preview, can ask the router to keep the same model across turns but does not guarantee a cache hit. With OpenRouter and Not Diamond, cache pricing is the underlying provider's.

What a GMI Router request costs today

Walk the four lines through GMI Router and three of them are straightforward.

Selection fee and judge call: zero. There is no judge model and no routing fee. Parsing and routing run on GMI's own model and add nothing on top. During the preview, recommendation and routing are free.

Selected model tokens: the model card rate. GMI's pricing docs point to the console for current rates: open Inference > Model Hub, and each model card shows current input and output rates and any region modifiers. You control which rates can apply. Cost, Balanced, and Quality modes shift the quality-cost trade-off, an Allowed Model Pool approved by the organization owner limits which models the router can pick, and Auto Mode is a workspace-level switch. GMI's internal benchmark across eight task categories reports 28% lower cost per task than GPT-5.5 in Quality Mode and 84% lower than Opus 4.8 in Balanced Mode. Those results are directional and not independently audited, and the leaderboard itself notes that cost does not imply a fixed price because it depends on your prompt. Your own bill is the number that counts.

Cache line: reuse is preserved for supported workloads. Model Hub cards show input and output rates, so price the cache term from your actual usage and billing records rather than from another provider's multipliers. If cache pricing matters for the models you expect the router to pick most often, bring it into the volume pricing conversation. Turns the router sends to a different model, including fallbacks, are the ones to watch in your logs.

Visibility. Each non-streaming response carries routing_metadata with the selected model, task type, selected mode, fallback models, and a recommendation ID. Streaming responses send the same metadata as an event after the text. Most routed responses also include an X-Recommendation-ID header. The GMI Router docs cover the full request and response format:

POST https://console.gmicloud.ai/api/v1/ie/recommendation/autoroute
Authorization: Bearer YOUR_TOKEN

{ "messages": [...full conversation...], "mode": "balanced", "stream": false }

Reconcile one week of routed traffic

Start with a representative week of traffic and compare its usage with the bill.

1. Give routed traffic its own API key. The Serverless Usage page filters by API key, so a dedicated key keeps routed spend separate from direct calls.
2. Log routing metadata per request. Store recommendation_id, selected_model, selected_mode, and fallback_models with a conversation ID. If you also count tokens on your side, treat those counts as a troubleshooting aid; the usage page is the billing record.
3. Pull hourly usage. In the console, go to Billing > Serverless Usage, set the timezone to UTC to match your logs, choose Hourly granularity, and filter by your routing key and by model name.
4. Match per model, per hour. For each selected model, compare the usage page's figures for that hour with the requests you logged, and check the charges against the model's Model Hub rate. After the first batch, confirm that routed requests appear under the calling key and the selected model's name. If usage shows a model you did not log as selected, compare it with the fallback_models you stored for that hour; a backup may have served those turns.
5. Count switches inside conversations. For each conversation, find the turns where the selected model differs from the previous turn, including turns a fallback served. Those are the turns where the previous model's cache cannot carry over. Check them against usage for the same hours; where your usage and billing records break out cached input, you can see what those turns cost, and where they do not, track the switch rate as the number to keep low.
6. Look for a routing line item. During the preview there should be none. Check the Router page for changes to the preview terms before you project the week forward.

At the end of the week you have a zero selection fee and zero judge cost confirmed, model tokens reconciled against Model Hub rates, and a count of the switches and fallbacks where cache reuse could not carry over. Divide the reconciled total by routed requests and you have your cost per routed request.

Price your routed traffic on GMI Router

Send a sample of your prompts through Test Routing in the GMI console to see which models the router selects in each mode and whose rates they carry, then move a week of traffic onto a dedicated key and reconcile it with the steps above. For volume pricing, custom commitments, or pricing questions on the models you route to most, contact our sales team.