How Cache-Aware Model Routing Reduces LLM Infrastructure Costs Without Sacrificing Quality

Model routing and prompt caching are usually treated as separate optimizations. Routing decides which model handles each request based on task type, quality requirements, and cost, typically reducing cost 40 to 60 percent. Caching reduces the cost of repeated context by reusing computed KV values, typically reducing input costs 60 to 90 percent for workloads with stable prefixes. Combining them produces a third effect that neither delivers alone: when the routing decision accounts for cache state, requests can be directed to the endpoint where their context is already cached, converting what would have been a cache miss into a cache hit.

• Cache-aware routing changes the routing objective from "which model is best for this task" to "which model and endpoint is best for this task given current cache state." A request whose 8,000-token context is already cached on endpoint A should route to endpoint A rather than to an equivalent endpoint B where the context would require full prefill.
• The savings from cache-aware routing are largest for workloads with high context reuse and multiple serving endpoints. RAG pipelines with a shared document corpus, multi-tenant systems with per-tenant system prompts, and agentic sessions with accumulated history all benefit.
• GMI Model Theorem provides the model-tier routing layer through prompt-aware task detection and benchmark-backed quality scoring. Combined with GMI Prime Inference dedicated endpoints where KV cache persists across requests, the two layers together produce cache-aware routing at the infrastructure level.
• Cache-aware routing does not sacrifice quality because it operates within the model tier that routing already selected. The routing decision first identifies which models are appropriate for the task and quality requirements. Cache-aware placement then selects among those appropriate models based on cache state. Quality is preserved because the candidate set is quality-filtered before cache considerations apply.
• Session affinity is the simplest form of cache-aware routing and delivers most of the available benefit for conversational and agentic workloads. Routing all turns from a session to the same endpoint preserves KV cache continuity, eliminating per-turn context rebuild that would otherwise consume 200 to 800ms per turn and the corresponding compute cost.
• The measurement that confirms cache-aware routing is working is the cache hit rate broken down by routing decision. If the cache hit rate for requests routed to their cache-affinity endpoint is significantly higher than for requests routed elsewhere, the routing layer is correctly incorporating cache state.

The Two Optimizations and Why They Interact

Model routing reduces cost by matching capability to requirement.

Production traffic is not homogeneous. A customer service application handles simple FAQ lookups, moderate policy questions, and complex multi-part troubleshooting requests. A coding assistant handles autocomplete requests, single-function generation, and multi-file refactoring. Routing each request to the smallest model that handles it adequately reduces average cost per request substantially compared to routing everything to a frontier model.

The savings scale with the fraction of traffic that does not require frontier capability. Analysis of production LLM traffic consistently shows 60 to 80 percent of requests fall into categories that mid-tier models handle at equivalent user-perceived quality. Routing those requests to appropriately sized models reduces total inference cost 40 to 85 percent depending on the traffic distribution.

Prompt caching reduces cost by eliminating redundant computation.

When a request shares a prefix with a prior request on the same endpoint, the KV cache for that prefix can be reused rather than recomputed. For workloads with large stable prefixes (system prompts, retrieved documents, conversation history), this eliminates the majority of input token compute.

The savings scale with the fraction of input tokens in stable prefixes. A RAG pipeline where 80 percent of input tokens are retrieved documents that repeat across queries achieves 60 to 70 percent input cost reduction with effective prefix caching.

Where they interact: endpoint placement.

Routing typically selects a model. But in production, a model is served by one or more endpoints, and cache state is endpoint-specific. The KV cache lives in a specific GPU's VRAM. A request routed to model X may land on endpoint X1 or endpoint X2, and the cache state on those two endpoints is different.

Standard routing is cache-blind: it selects the model and then load-balances across available endpoints for that model, typically round-robin or least-connections. This distribution actively works against cache efficiency: a conversation whose first turn cached context on endpoint X1 has its second turn routed to X2, forcing a full context rebuild.

Cache-aware routing adds cache state to the endpoint selection decision. Given that the routing layer has selected model X for this request, and given that this request's prefix is cached on endpoint X1, route to X1.

Four Levels of Cache-Aware Routing

Cache-aware routing exists at four levels of sophistication, each delivering additional benefit at additional implementation complexity.

Level 1: Session affinity (highest ROI, simplest implementation)

Route all requests from the same session to the same endpoint. Implemented through consistent hashing on the session identifier at the load balancer layer.

Benefit: preserves KV cache continuity across conversation turns and agentic steps. For a 10-turn conversation with 2,000 tokens of accumulated history, session affinity eliminates 2,000 tokens of prefill per turn after the first.

Implementation cost: minimal. Most load balancers support consistent hashing on a header value or cookie. The primary requirement is that the client passes a stable session identifier.

Failure mode to handle: when the affinity target endpoint is unavailable or at capacity, the request must route elsewhere and accept a cache miss. Configure overflow routing rather than failing the request.

Level 2: Tenant affinity (high ROI for multi-tenant applications)

Route all requests from the same tenant to the same endpoint or endpoint group. For multi-tenant applications where each tenant has a distinct system prompt or configuration, tenant affinity ensures that the tenant-specific prefix stays cached on the endpoint serving that tenant.

Benefit: for a SaaS application with 500 tenants each having a 3,000-token custom system prompt, tenant affinity means each tenant's system prompt is cached once per endpoint rather than being repeatedly evicted by other tenants' prompts competing for cache space.

Implementation: consistent hashing on tenant identifier, with endpoint group assignment to distribute tenant load across available capacity.

Level 3: Content-hash routing (high ROI for RAG and document workloads)

Route requests to the endpoint where their specific content prefix is cached, based on a hash of the prefix content rather than a session or tenant identifier.

For a RAG pipeline: hash the retrieved document set. Route all queries retrieving that same document set to the same endpoint, ensuring the document prefix cache is shared across all queries against those documents.

Benefit: converts what would be independent cache misses across endpoints into cache hits on a single endpoint. For a document corpus where popular documents dominate retrieval, this concentrates the cache benefit rather than fragmenting it across endpoints.

Implementation: requires computing a content hash of the cacheable prefix before routing, then maintaining a mapping from content hash to endpoint. More complex than session or tenant affinity but delivers benefit for workloads where sessions and tenants do not correlate with content reuse.

Level 4: Cache-state-aware routing (highest sophistication)

Query the actual cache state on each candidate endpoint before routing, and select the endpoint with the best cache match for this specific request.

This requires the routing layer to have visibility into each endpoint's current cache contents, which most serving frameworks do not expose directly. SGLang's RadixAttention maintains a trie of cached prefixes that could theoretically be queried, but exposing this to an external routing layer requires custom instrumentation.

Benefit: maximum cache hit rate because routing decisions use ground-truth cache state rather than heuristics.

Practical consideration: the added latency of querying cache state before routing may offset the benefit. For most production workloads, Levels 1 through 3 capture the available benefit at substantially lower implementation cost.

Where Model Routing and Caching Conflict

Cache-aware routing has a structural tension with model-tier routing that must be resolved explicitly rather than accidentally.

The conflict: model-tier routing wants to send each request to the cheapest adequate model. Cache-aware routing wants to send each request to the endpoint where its context is cached. These objectives can point in different directions.

Scenario: a conversation begins with a simple question that routes to a 32B model on endpoint A. The conversation accumulates 6,000 tokens of context cached on endpoint A. Turn 8 asks a complex question that would benefit from a 70B model on endpoint B, where the 6,000-token context is not cached.

Options:

• Route to the 70B model on endpoint B, paying full prefill cost for 6,000 tokens (approximately 400 to 800ms and the corresponding compute cost) in exchange for higher quality on this turn.
• Route to the 32B model on endpoint A, benefiting from the cached context but accepting lower quality on a turn that would benefit from more capability.

Resolution principle: quality requirements are a hard constraint, cache state is an optimization within that constraint.

The correct approach is to let model-tier routing determine the eligible model set based on quality requirements, then apply cache-aware endpoint selection within that eligible set. If the task genuinely requires a 70B model, route to the 70B model and accept the cache miss. If the task is adequately served by either model tier, prefer the endpoint with the cached context.

This is exactly the structure that GMI Model Theorem's recommendation model provides: the task-type detection and quality scoring identify which models are appropriate for the detected task type. Within that eligible model set, additional optimization criteria (cache state, current endpoint load, latency signals) select the specific endpoint.

Practical policy for mixed-complexity conversations:

Set a quality threshold below which cache affinity takes precedence and above which quality takes precedence. For a conversational application, most turns fall below the threshold and stay on the cache-affinity endpoint. Occasional complex turns exceed the threshold and route to a higher-capability model, accepting the cache miss for that turn. After the complex turn, subsequent turns can return to the cache-affinity endpoint if the context has been maintained there.

Quantifying the Combined Savings

The savings from combining routing and caching are multiplicative rather than additive, because they apply to different components of the cost.

Baseline: no routing, no caching. All requests to a frontier model at $3.00/M input, $15.00/M output. For 10,000 daily requests with 4,000 input tokens and 500 output tokens:

• Input cost: 10,000 × 4,000 × $3.00/M = $120/day
• Output cost: 10,000 × 500 × $15.00/M = $75/day
• Total: $195/day

With model routing only. Assume 70 percent of requests route to a mid-tier model at $0.60/M input, $1.20/M output; 30 percent stay on the frontier model.

• Mid-tier input: 7,000 × 4,000 × $0.60/M = $16.80/day
• Mid-tier output: 7,000 × 500 × $1.20/M = $4.20/day
• Frontier input: 3,000 × 4,000 × $3.00/M = $36/day
• Frontier output: 3,000 × 500 × $15.00/M = $22.50/day
• Total: $79.50/day (59 percent reduction)

With caching only (no routing). Assume 3,000 of the 4,000 input tokens are in a stable cacheable prefix with 85 percent cache hit rate.

• Cache-hit input (85%): 8,500 × (1,000 × $3.00/M + 3,000 × $0.30/M) = $33.15/day
• Cache-miss input (15%): 1,500 × 4,000 × $3.00/M = $18/day
• Output cost: unchanged at $75/day
• Total: $126.15/day (35 percent reduction)

With both routing and caching.

• Mid-tier requests (70%) with 85% cache hits:
  
  • Cache-hit: 5,950 × (1,000 × $0.60/M + 3,000 × $0.06/M) = $4.64/day
  • Cache-miss: 1,050 × 4,000 × $0.60/M = $2.52/day
  • Output: 7,000 × 500 × $1.20/M = $4.20/day
• Frontier requests (30%) with 85% cache hits:
  
  • Cache-hit: 2,550 × (1,000 × $3.00/M + 3,000 × $0.30/M) = $9.95/day
  • Cache-miss: 450 × 4,000 × $3.00/M = $5.40/day
  • Output: 3,000 × 500 × $15.00/M = $22.50/day
• Total: $49.21/day (75 percent reduction from baseline)

The combined reduction of 75 percent exceeds either optimization alone (59 percent routing, 35 percent caching) because they apply to different cost components. Routing reduces the per-token rate; caching reduces the number of tokens charged at that rate.

Implementation Sequence

Implement these optimizations in the order that delivers the most benefit per unit of engineering effort.

Step 1: Prompt structure for cacheability (1 day) Restructure prompts so stable content precedes dynamic content. This is a prerequisite for any caching benefit and typically requires only reordering the prompt construction logic. Verify by checking cache hit metrics in provider API responses.

Step 2: Session affinity routing (1 to 2 days) Configure the load balancer for consistent hashing on session identifiers. This delivers the majority of cache-aware routing benefit for conversational and agentic workloads at minimal implementation cost.

Step 3: Model-tier routing (1 week) Implement task-type-based model selection, either through custom classification logic or through GMI Model Theorem's model=auto routing. Model Theorem eliminates the classification and quality-scoring implementation work by handling task detection and benchmark-backed model selection at the infrastructure layer.

Step 4: Tenant affinity for multi-tenant applications (2 to 3 days) For applications with tenant-specific system prompts, add tenant-based endpoint affinity. This preserves per-tenant prompt caching without cache eviction from competing tenants.

Step 5: Content-hash routing for RAG workloads (1 to 2 weeks) For RAG pipelines where content reuse does not correlate with sessions or tenants, implement content-hash-based endpoint routing. This is the highest-effort optimization and should be prioritized only after Steps 1 through 4 are complete and measured.

Measuring Cache-Aware Routing Effectiveness

Four metrics confirm whether the combined optimization is working as intended.

Cache hit rate by routing decision. Break down cache hit rate for requests that were routed to their cache-affinity endpoint versus requests routed elsewhere (overflow, capacity-driven rerouting, or quality-driven model tier change). The affinity-routed requests should show substantially higher cache hit rates. If they do not, the affinity configuration is not working correctly.

Effective cost per request by model tier. Compute total cost divided by request count separately for each model tier. Compare against the theoretical cost without caching. A mid-tier model serving requests at 40 percent of its nominal per-token cost indicates that caching is delivering significant savings within that tier.

Cross-tier routing rate. The fraction of requests that route to a different model tier than the previous request in the same session. High cross-tier routing (above 20 percent of turns in multi-turn sessions) indicates that the quality threshold policy is causing frequent cache misses. Consider whether the quality threshold is set too aggressively.

Overflow rate. The fraction of requests that could not route to their cache-affinity endpoint because that endpoint was unavailable or at capacity. A high overflow rate (above 10 percent) indicates insufficient capacity on affinity endpoints, and expanding reserved capacity would improve both latency and cost through better cache hit rates.

GMI Cloud Infrastructure for Cache-Aware Routing

Cache-aware routing requires two infrastructure properties: a routing layer that makes quality-appropriate model selections, and endpoints where KV cache persists across requests. GMI Cloud provides both.

Model Theorem for the routing layer. Prompt-aware task detection maps each request to one of eight task types (Coding, Agent and Tool Use, Math, Reasoning, Knowledge, Long Context and RAG, Instruction Following, Data Analysis and Language). Benchmark-backed quality scoring identifies the appropriate model tier for the detected task type. Three mode preferences (Balanced, Cost, Quality) adjust the weighting between quality and cost in the selection. The Allowed Models setting constrains the candidate pool to organizationally approved models, ensuring that cache-aware optimization operates within governance boundaries.

Prime Inference for cache-persistent endpoints. Dedicated single-tenant GPU capacity with pre-loaded model weights means the KV cache persists between requests without eviction from other tenants' workloads. This is the property that makes cache affinity meaningful: on shared serverless infrastructure, cache state is unpredictable because other tenants' requests compete for the same cache space. On dedicated endpoints, the cache contains only your workload's prefixes.

Per-model runtime tuning includes cache configuration. SGLang's RadixAttention (trie-based prefix sharing) and vLLM's automatic prefix caching are configured per model based on the workload's actual context length distribution and prefix reuse patterns, rather than applying generic defaults.

Regional endpoints for cache locality. Multi-region deployment with region-pinned endpoints means cache affinity and network latency optimization align: routing a request to its cache-affinity endpoint in the user's region minimizes both prefill compute and network round-trip time.

For teams building on the managed inference path, GMI Cloud's on-demand model access provides both the 100-plus model library for routing across model tiers and bare metal GPU access for teams that need custom cache configuration on dedicated infrastructure.

Conclusion

Model routing and prompt caching are complementary optimizations that most teams implement in isolation. Routing reduces the per-token rate by matching model capability to task requirements. Caching reduces the number of tokens charged at that rate by eliminating redundant computation of stable prefixes. Combining them delivers 70 to 80 percent cost reduction for workloads with both heterogeneous task complexity and significant context reuse, which describes most production AI applications.

The key architectural principle is that quality requirements are a hard constraint and cache state is an optimization within that constraint. Model-tier routing determines which models are eligible for a request based on task type and quality requirements. Cache-aware endpoint selection then chooses among eligible endpoints based on cache state. Quality is never sacrificed because the candidate set is quality-filtered before cache considerations apply.

Session affinity is the highest-ROI implementation and should be the first step. Model-tier routing through GMI Model Theorem eliminates the classification and quality-scoring implementation work. Prime Inference dedicated endpoints provide the cache persistence that makes affinity routing meaningful, because cache state on shared infrastructure is unpredictable in ways that dedicated capacity is not.

FAQs

What is cache-aware model routing and how does it differ from standard model routing? Standard model routing selects which model handles each request based on task type, quality requirements, and cost, then load-balances across available endpoints for that model (typically round-robin). This distribution is cache-blind: a conversation whose context is cached on endpoint A may have its next turn routed to endpoint B, forcing full context rebuild. Cache-aware routing adds cache state to the endpoint selection decision, directing requests to the endpoint where their prefix is already cached. The model-tier selection remains quality-driven; cache awareness operates as an optimization within the quality-appropriate candidate set.

Does cache-aware routing ever compromise output quality? No, when implemented correctly. The architectural principle is that quality requirements are a hard constraint and cache state is an optimization within that constraint. Model-tier routing first identifies which models are appropriate for the detected task type and quality requirements. Cache-aware endpoint selection then chooses among those already-appropriate options based on cache state. If a task genuinely requires a higher-capability model, routing sends it there and accepts the cache miss. Quality degradation only occurs if cache affinity is allowed to override quality requirements, which is an implementation error rather than a property of the approach.

Which cache-aware routing level delivers the best return on implementation effort? Session affinity (Level 1) delivers the majority of available benefit at the lowest implementation cost, typically 1 to 2 days of work configuring consistent hashing on session identifiers at the load balancer. For conversational and agentic workloads, session affinity eliminates per-turn context rebuild that would otherwise consume 200 to 800ms and the corresponding compute cost per turn. Tenant affinity (Level 2) is the next highest ROI for multi-tenant applications with per-tenant system prompts. Content-hash routing (Level 3) and full cache-state-aware routing (Level 4) deliver additional benefit but at substantially higher implementation complexity, and should be prioritized only after Levels 1 and 2 are complete and measured.

How much cost reduction does combining routing and caching actually deliver? The reductions are multiplicative because they apply to different cost components. For a representative workload of 10,000 daily requests with 4,000 input tokens (3,000 in a stable cacheable prefix) and 500 output tokens: model routing alone (70 percent to mid-tier) reduces cost 59 percent, caching alone (85 percent hit rate) reduces cost 35 percent, and combining both reduces cost 75 percent from the frontier-model-only baseline. Routing reduces the per-token rate; caching reduces the number of tokens charged at that rate. The combined savings exceed either optimization alone.

Why does cache-aware routing require dedicated endpoints rather than shared serverless infrastructure? On shared serverless infrastructure, cache state is unpredictable because multiple tenants' requests compete for the same cache space. A prefix cached by your request may be evicted moments later by another tenant's workload, making cache affinity routing ineffective. On dedicated single-tenant endpoints, the KV cache contains only your workload's prefixes, so cache state is predictable and affinity routing reliably produces cache hits. GMI Prime Inference's dedicated capacity with pre-loaded model weights provides this cache persistence, which is the infrastructure property that makes cache-aware routing worthwhile.