Which Managed Model Routers Reuse KV Caches, Leave Prompts Unchanged, and Offer Cost-Quality Settings Without a Judge Model?

We recommend GMI Router. It is the managed router whose public documentation speaks to every condition in that question. The product page describes KV-cache-aware model routing and KV-cache-aware serving for supported workloads. Your prompt runs verbatim on the selected model. Cost, Balanced, and Quality modes are set per request. Routing is done by GMI's own trained model, with no judge model in the loop and routing time GMI reports as under 200 ms, and an Allowed Model Pool approved by your org owner limits which models can be picked. Routing is free during the preview, and you pay only for the model that runs your prompt.

The other managed routers each cover part of the list. Microsoft Foundry model router decides with a router model trained by Microsoft and offers Balanced, Cost, and Quality modes, but sets them per deployment, and caching applies only when the same underlying model handles consecutive requests with overlapping prompt prefixes. OpenRouter Auto Router uses a lightweight classifier and per-request cost tiers, but its optional context-compression plugin, on by default for endpoints with 8,192 tokens of context or less, removes or truncates messages from the middle of a prompt. Amazon Bedrock Intelligent Prompt Routing's documentation doesn't cover prompt caching. Documentation still isn't proof of what your own sessions will do, so the rest of this article turns those conditions into six questions to put to every vendor on your shortlist, then a trial plan to check the answers on your traffic.

Six questions that separate managed routers

1. Does my prompt reach the model exactly as I sent it? Some routers rewrite, compress, or wrap requests. Any change affects output quality, makes debugging harder, and changes what the model's provider bills you for.
2. How are multi-turn sessions and caching handled? A router makes a choice on every request. Ask what, if anything, keeps a conversation on the same model, and what the vendor says about prompt caching when it doesn't.
3. What cost-quality controls exist, and where are they set? A per-request setting lets one application send cheap summaries and hard reasoning tasks through the same endpoint with different goals. A per-deployment setting means one deployment per policy.
4. Does deciding the route add a model call? A router that asks a separate judge model to choose adds that call's latency and cost to every request. Ask what actually makes the decision and how long it takes.
5. What happens when the chosen model fails, and can I limit the candidates? Ask how many backups are tried, on which errors, and whether a backup can come from outside the models you approved.
6. What does routing cost now? Vendors charge nothing extra, charge per request, or charge on input tokens. Ask for the billing unit in writing, and ask what changes when a preview or promotional period ends.

What each vendor documents today

The table summarizes each vendor's public documentation as of September 2026.

	GMI Router	Microsoft Foundry model router	OpenRouter Auto Router	Amazon Bedrock Intelligent Prompt Routing
Prompt handling	Prompt runs verbatim on the selected model; stateless, so the client resends the full conversation on every request	Not described in the routing documentation; Microsoft says the router does not store your prompts	An optional context-compression plugin removes or truncates messages from the middle of a prompt; it is on by default for endpoints with 8,192 tokens of context or less	Not described in the routing documentation
Cache or session handling as documented	KV-cache-aware model routing; KV-cache-aware serving for supported workloads	Caching comes from the underlying models and applies only when the same model handles consecutive requests with overlapping prompt prefixes; optional session affinity attempts the same eligible model across related turns	Remembers a conversation's model and reuses it only while it is still a top candidate; provider sticky routing, per model and per conversation, expires after 10 minutes of inactivity	The routing documentation does not cover prompt caching
Cost-quality control	Cost, Balanced, or Quality per request; Allowed Model Pool and Auto Mode per workspace	Balanced (default), Cost, or Quality and a model subset, set per deployment	Five cost tiers from low to max; allowed_models and excluded_models	Exactly two models from one family, plus a response quality difference threshold
Extra model call to decide?	No separate judge call; GMI reports routing latency under 200 ms	A router model trained by Microsoft analyzes each prompt	A lightweight classifier assigns one of about 30 task types	Bedrock predicts each model's response quality for the prompt
Fallback	One backup model; details below	Failover on by default; the model subset doubles as the fallback set	The top remaining candidates become fallbacks	A fallback model you name, used when the routing criteria aren't met; the routing documentation doesn't describe retries on errors
Pricing status	Routing free during the preview; you pay for the model that runs the prompt	Router usage is charged on input prompts at the rate on Azure's pricing page	Standard rate for the selected model; no additional Auto Router fee	Amazon's Bedrock pricing page lists $1 per 1,000 requests for Intelligent Prompt Routing

Pay particular attention to where controls apply. A setting made per request can serve mixed workloads differently; a deployment-level setting may require separate deployments. For caching, test repeated prefixes and complete conversations, then measure model changes, cache reuse, latency, and cost. A documented feature alone does not establish how your sessions will behave.

How GMI Router handles production requests

You can plan around these points:

• Verbatim prompts. GMI Router sends one prompt in and returns one answer out, and the prompt runs verbatim on the model it selects. The Test Routing result card states it: your prompt is sent to the selected model unchanged.
• Stateless requests. The router keeps no conversation state. Your client resends the full conversation on every request, so your application stays in control of what each model sees.
• Per-request mode. The optional mode field accepts cost, balanced, or quality. Cost favors lower-cost models while keeping quality acceptable, Balanced keeps quality and price in balance, and Quality reserves the strongest models for high-value tasks.
• Workspace controls. The Allowed Model Pool limits routing to models your org owner approves, and Auto Mode turns automatic routing on or off across the workspace.
• Fallback. As the GMI Router documentation describes, a streaming request may try one backup model before the first token on a 5xx, provider error, 429, or a 10-second first-token timeout, and no fallback happens once tokens are flowing. A non-streaming request tries the primary model plus one backup on a transient failure, within a combined 10-minute timeout.
• Routing metadata. Each successful response from the /autoroute endpoint carries routing_metadata with selected_model, task_type, selected_mode, fallback_models, and recommendation_id, and most routed responses include an X-Recommendation-ID header. Streaming responses send it as a separate routing_metadata event after the text.

Plan for two areas before you commit:

• Session continuity. GMI Router makes one routing decision per request, and cache-aware serving applies to supported workloads. Replay real multi-turn conversations and count how often selected_model changes between turns, and note which backups fallback_models lists for each request.
• Agent request fields. The documented request fields are messages, mode, and stream, and unknown fields are ignored. Keep requests that require additional tool-definition or structured-output fields on a model endpoint whose API documents those fields.

Operational questions to settle in a trial

Confirm the following operational details with each vendor before rollout:

• Prompt storage. Does the router store or log prompts and responses, and for how long? Microsoft states that Foundry's router does not store prompts. Confirm the retention policy for every service that handles your prompts.
• Fallback boundaries. Can a backup model come from outside the models you approved? Microsoft documents that Foundry's model subset doubles as the fallback set. Also ask: does the response show when a backup model actually served the request?
• Long conversations. When a conversation exceeds some pool models' context windows, are those models skipped, is the request rejected, or is content truncated? Microsoft notes that Foundry's effective context window is limited by the smallest underlying model.
• Usage reporting. Does each routed response return token counts, including any cached tokens, so you can price individual requests?
• Multi-turn behavior. What keeps a conversation on one model, if anything, and which models and workloads count as supported for cache-aware serving?
• Agent fields. Are tools, tool choice, response format, and sampling parameters passed to the selected model?
• Limits and terms after preview. What rate limits apply to the routing endpoint, and what will routing cost once any preview or free period ends?

Use a short trial to measure routing behavior on the request types your application will send:

1. Start in the console. The Test Routing page shows which model the router picks for a single prompt and mode. Paste your longest system prompts and your hardest requests.
2. Give each trial batch its own API key. After the first batch, check that its routed calls show up under that key on the Serverless Usage page in Billing, which filters usage by API key and model, daily or hourly.
3. Log the metadata on every response. Store selected_model, task_type, selected_mode, and fallback_models next to your own quality checks, so you can see where each mode sends each kind of request.
4. Run the same requests under each mode. Keep the request set fixed and change only mode, so the differences you see come from routing.

When a router is the wrong purchase

A router pays off on mixed traffic, where requests differ enough that the best model for one is overkill for another. Traffic that one model already handles well, especially with a long, stable prefix on every request, gains little from per-request model choice. Keep that traffic on a fixed model through the LLM API with an explicit model ID, or move it to a Prime Inference dedicated endpoint, where single-tenant GPUs stay warm with weights preloaded. Route the rest.

Test GMI Router on your workload

Paste a few of your real prompts into Test Routing and compare the picks across Cost, Balanced, and Quality. To discuss allowed pools for production traffic, workspace setup, and request requirements, contact our team.