Which Model Routers Balance Cost and Quality While Reusing Multi-Turn Caches?

In the worked ten-turn conversation, an 8,000-token system prefix leads to 101,750 input tokens, of which 92,250 (90.7%) repeat earlier text. Switching models can force a fresh prefill. GMI Router accounts for cache state when choosing a model, with no judge-model call and a published routing time under 200 ms.

GMI Router provides Cost, Balanced, and Quality modes within an organization-approved model pool on GMI Cloud's inference platform. (About)

Compare cached-turn cost, switching cost, and the prefix-length crossover before setting routing policy.

Why does a multi-turn product's bill grow faster than its traffic?

A multi-turn bill grows faster than its traffic because every turn resends the whole conversation. The model is stateless, so turn 10 pays for turns 1 through 9 again.

Take a realistic assistant shape: an 8,000-token static prefix (system prompt, tool schemas, policy text), a 150-token user message per turn, and a 300-token reply.

Traffic grew by one user message per turn. Billable input grew by 450 tokens per turn, which is three times faster, and 90.7% of the total is repetition. This is the structural reason multi-turn products hit budget ceilings that single-shot products never see, and it is why the two numbers that decide your unit economics are the cache read price and the cache hit rate.

How much does cache reuse save on a 10-turn session?

On GPT-6 Astra, cache reuse cuts a 10-turn session by roughly 71%.

Run the same 10 turns through GPT-6 Astra at its list price of $10.00 per 1M input tokens, $50.00 per 1M output tokens, and $1.00 per 1M cache read tokens (model catalog, as of October 2026, console.gmicloud.ai/model-library). Column three bills every input token at full rate. Column four bills the repeated prefix at the cache read rate.

Each turn includes 300 output tokens at $50.00 per 1M, which is $0.0150 of fixed cost per turn. Prices are list prices from the catalog; GPT-6 Astra carried no catalog discount on October 15, 2026.

Three things fall out of that table:

1. Turn 1 is the expensive turn. Uncached, it is 8.3% of session spend; cached, it is 28.6%. Cold prefill is the cost you are protecting, and every model switch recreates it.
2. Cached per-turn cost is nearly flat. It drifts from $0.0250 to $0.0286 across eight turns, a 14% rise, while the uncached column rises 36%. Cache reuse turns a growing per-turn cost into a near-constant one, which is what makes long sessions financially predictable.
3. Output becomes the dominant input-side-independent cost. In the cached column, the 3,000 output tokens account for $0.1500 of $0.3373, or 44.5% of the session. Once caching works, rank candidate models by output price first, not by the headline input price.

At 100,000 sessions of this shape per month, the gap is $116,750 against $33,725, a difference of $83,025. Cost efficiency through "KVcache reuse, scheduling, load planning, and more" is listed as a platform feature of GMI Cloud's Model-as-a-Service platform, and the savings show up where they are authoritative: the Serverless Usage view and the invoice in Console (usage docs).

Which models give the biggest cache read discount?

DeepSeek-V4.1-Flash gives the biggest one of the five models compared here: it bills a repeated prefix at one fiftieth of its input price. The multiple between a model's input price and its cache read price varies by more than an order of magnitude across the catalog, so it belongs in your model selection criteria rather than in a footnote.

All figures below are list prices from the GMI Cloud model catalog as of October 15, 2026 (console.gmicloud.ai/model-library), per 1M tokens.

Read the multiple as "how much of the first pass you keep paying for." At 50x, a repeated prefix on DeepSeek-V4.1-Flash costs 2% of what it cost the first time. At 5x, the same prefix on Kimi K2.7-Code still costs 20% of its first pass, so a long conversation on that model stays meaningfully expensive even with a perfect hit rate.

Two of these models carried catalog discounts on October 15, 2026: in the model catalog, DeepSeek-V4.1-Flash was listed at $0.18 input and $0.0036 cache read under a 40% discount, and GLM-5.3 at $0.98 input and $0.182 cache read under a 30% discount. Every calculation in this article uses list prices, so that discounts cannot flatter the result.

What does a mid-conversation model switch actually cost?

A mid-conversation model switch costs one full cold prefill at the new model's input price, paid on the turn the switch happens. Whether that is a good trade depends entirely on how far apart the two models are priced.

Start at turn 6 of the session above: 10,250 tokens of shared prefix, 150 new tokens, 300 output tokens.

Case A: a modest downgrade that loses money. The session is on GLM-5.3 and a cost-seeking router moves it to Kimi K2.7-Code, whose input price is 32% lower.

The switch costs 2.6 times as much on the turn it happens. From turn 7 onward the cheaper model saves about $0.00098 per turn, so the cold prefill takes roughly 7 more turns to repay. A 10-turn session only has 4 turns left, and the full session lands at $0.05344 instead of $0.05049, which is 5.8% more expensive than never switching at all.

Case B: a large downgrade that pays immediately. Same turn, same prefix, but the session moves from GPT-6 Astra to GLM-5.3.

Here the switch is 41% cheaper on the switch turn itself, before any payback period, because the output price gap of $45.60 per 1M dwarfs the $0.40 per 1M premium on the cold prefix. Over the full session, switching at turn 6 costs $0.2328 against $0.3373, a 31% saving.

The lesson is not "never switch." It is that a router making this decision has to price the cache, and a router that only compares headline input prices will confidently pick Case A.

Where is the threshold? A rule you can implement

Switching models mid-conversation is cheaper than staying on the current model whenever the destination model's savings on new input and output tokens outweigh the cost of reloading the shared prefix at its full input price. Written out, the switch turn is cheaper when:

prefix_tokens x (input_price_new - cache_read_price_current)
    <  new_tokens x (input_price_current - input_price_new)
     +  output_tokens x (output_price_current - output_price_new)

The left side is the cold prefill premium. The right side is everything the cheaper model saves on that turn. Solving for the prefix length gives a crossover point: the conversation length past which a given downgrade stops being worth it. Using the same per-turn shape (150 new input tokens, 300 output tokens) and October 2026 list prices:

Two conditions sit behind every row. The thresholds use the October 15, 2026 list prices rather than the discounts two of these models carried that day, and they assume the destination model starts cold on that conversation. Recheck them whenever catalog prices move.

Three operating rules follow from this table, and they are simple enough to encode in your routing policy:

1. If the candidate's input price is at or below the current model's cache read price and its output price is no higher, switch freely. With both conditions met the left side of the inequality is zero or negative, so a longer conversation never makes the switch worse. GPT-6 Astra's $1.00 cache read price puts much of the catalog in that position.
2. If the candidate's input price is above the current model's cache read price, require a wide output price gap. The cold prefill is a one-time charge, but the output saving repeats every turn, so the switch has to survive the payback arithmetic within the session length you actually observe. Measure that length; do not assume it.
3. Escalate on task type; downgrade with the formula. An escalation is priced against the cost of a retry: when a turn has to be asked again, the session pays another cold or cached prefill plus another set of output tokens. Between neighboring price tiers a downgrade is worth fractions of a cent per turn, which is exactly the range a cold prefill swallows; the savings worth chasing come from wide gaps like the Astra case above.

These rules assume one cache lineage per conversation and no retrieval step that rewrites the front of the prompt. Prompt ordering decides whether that assumption holds, which the FAQ below covers.

On GMI Router the rules turn into two settings rather than code you maintain: the allowed model pool fixes which switches are available at all, and the mode fixes how hard the router pushes toward the cheap end of that pool.

How GMI Router handles cost, quality, and cache together

GMI Router applies the task-versus-cost decision per request, inside a model pool the organization approves, without adding a second model call in front of your traffic.

• Cache-aware by design. GMI Router is "KV-cache-aware model routing" and keeps "KV-cache Preserved across turns," which is exactly the property the arithmetic above demands (GMI Router).
• No judge model in the path. Routing runs on GMI Cloud's own trained model: "No judge model. No routing fees. Parsing and routing run on GMI's own model and add nothing on top." Routing time is given as "<200ms." A judge-model router would add an extra LLM call and its latency to every turn.
• Three modes, set at the workspace level. "Cost, lower-cost models, quality kept acceptable." "Balanced, quality and price in balance." "Quality, strongest models for high-value tasks." Multi-turn chat products generally belong in Balanced, with Quality reserved for the task types where a wrong answer is expensive.
• Published benchmark. GMI Cloud reports "−84% Lower cost per task than Opus 4.8, Balanced Mode" and "+2.4 pts Higher quality than only using GPT-5.5, Quality Mode," both on the GMI Router page. The qualifier is on the same page and belongs with the numbers: "Based on GMI's internal benchmark across 8 task categories, comparing models head-to-head. Routing runs within your allowed pool. Quality is a benchmark score; cost is relative to fixed-model baselines. Directional; not independently audited."
• Your prompt runs verbatim. GMI Router runs "YOUR PROMPT, RUN VERBATIM" and that matters for cache reuse: a router that rewrites or compresses your prompt changes the prefix and invalidates the cached copy.
• One account, one API key, one invoice. Routed models are billed through the same GMI Cloud MaaS account, with "Centralized billing with a single invoice across all models" (MaaS).
• Free during preview. GMI Router is "FREE DURING PREVIEW" and "You pay only for the model that runs your prompt." Model recommendation and routing are free for a limited time, so the line to budget is the model price, not the routing.

One design detail worth knowing before you integrate: the router itself does not store your conversation. The GMI Router documentation states "The router is stateless. Resend the full conversation on every request." Cache reuse happens in the serving layer, not in the router's memory, so your client keeps owning the message list exactly as it does today against an OpenAI-style chat endpoint. The call itself is one POST to https://console.gmicloud.ai/api/v1/ie/recommendation/autoroute with an OpenAI-style messages array and an optional mode field set to cost, balanced, or quality; the response carries a routing_metadata block naming the model that was selected.

Two companion pieces cover the neighboring ground. Automatic Model Selection for Every Prompt: GMI Router Is Live covers what routing does and how it was introduced; Cost Per Million Tokens Explained covers the per-token arithmetic underneath any model price. This article stays on the part neither one prices: what repetition and cache reuse do to a conversation that runs for ten turns.

A six-item checklist before you route production chat traffic

1. Measure your real session shape. Pull the median and p90 turn count, static prefix size, and reply length from a week of production traffic. Every threshold above moves when those three numbers move.
2. Record each candidate model's cache read price, not just its input price. Across the five models tabled above the multiple ranges from 5x to 50x, and that spread changes the ranking of "cheap" models for multi-turn work.
3. Rank candidates by output price for cached workloads. Output was 44.5% of the cached session cost in the worked example.
4. Set the allowed model pool before you turn routing on. Routing only picks from models the org owner approves, which is where you exclude anything whose context window is shorter than your p90 session.
5. Pick a GMI Router mode per surface, not per company. Interactive multi-turn chat in Balanced, one-shot classification and extraction in Cost, high-value reasoning in Quality. The dividing line is whether the surface carries a shared prefix worth protecting: traffic where every request stands alone has no multi-turn cache at stake, so there is nothing for a cheaper model to break.
6. Validate against the invoice. Compare one week of routed traffic to one week of fixed-model traffic using Console usage and billing, not estimates derived from your own token counts.

FAQ

Does GMI Router keep my conversation history so the cache stays warm? No. The router is stateless; resend the full messages array on every request. Prefix reuse occurs at the serving layer, not in a stored router conversation.

Which routing mode should a multi-turn chat product use? Start by testing Balanced for the main conversation, Quality for costly errors, and Cost for stateless classification or extraction. GMI's cited 84% lower cost versus Opus 4.8 is an internal directional result, not independently audited; validate the choice on actual tasks.

How do I confirm that cache reuse is actually happening on my traffic? Replay a fixed session cohort for a week, then reconcile Console Serverless Usage by model and key with the invoice. The example predicts roughly 80% lower input spend with unchanged output spend; that depends on realizing the modeled prefix reuse. (docs)

Does switching models mid-conversation always cost more? No. Compare the new model's cold-prefill cost against future savings using the crossover inequality. If its fresh-input rate is no higher than the current cached-input rate and output is no dearer, switching pays immediately. Close rates can reverse that result: the GLM-5.3/Kimi K2.7-Code example raises ten-turn cost by 5.8%.

Does retrieved context break the cache on every turn? New context invalidates the shared suffix when inserted before it. Put stable system text, tools, and policies first, then history, then new retrieved passages. If prompt ordering cannot retain a stable prefix, evaluate per-request routing without assuming durable cache reuse.

My traffic is steady and high volume. Is routing still the right layer? Routing can still choose by task. Move a model's sustained, predictable traffic to a warm single-tenant Prime Inference endpoint billed per GPU-hour while other workloads continue routing.

Run one week of your own sessions through it

Create a key in Console, set your allowed model pool, and send a week of your existing multi-turn conversations to GMI Router by posting them to https://console.gmicloud.ai/api/v1/ie/recommendation/autoroute with "mode": "balanced". The request body is the same OpenAI-style messages array you already build; what changes on the way back is that the response also names the model that was chosen. Compare the invoice against your fixed-model baseline. If you want help picking the pool for your session shape, or a quote for sustained production traffic, talk to the GMI Cloud team through the MaaS page.