Migrating from a Fixed Model to Routing: A Production Playbook

Every routing article explains why routing saves money. Almost none explain how to get there from a production system that already works. The gap matters because the migration is where teams get stuck: the application currently sends every request to one model, that model's behavior is known, and the prompts have been tuned against it. Switching to routing means different requests will now be handled by different models, each with its own output style, formatting tendencies, and edge case behavior. Done carelessly, this produces a week of quality regressions and a rollback. Done deliberately, it produces measured cost reduction with quality parity confirmed before any user traffic is affected.

• The migration is a measurement problem before it is an infrastructure problem. You cannot confirm routing preserves quality without a baseline of what quality looks like on the current fixed model, measured on your actual traffic distribution rather than published benchmarks.
• Shadow routing is the mechanism that de-risks the entire migration. Send production traffic to both the fixed model and the routing layer, compare outputs, and measure quality and cost differences on real requests without any user seeing routed output until the data supports it.
• GMI Router supports incremental migration through the Allowed Model Pool control (start with two models, expand as confidence grows), Auto Mode toggle (route some traffic, pin the rest), and three mode preferences that let teams begin in Quality mode and shift toward Balanced or Cost mode as quality parity is confirmed.
• Start with the narrowest viable model pool, not the widest. A pool of two models (the current fixed model plus one alternative) is enough to validate that routing works, produces a manageable comparison surface, and limits the number of output styles the application must handle.
• Prompt portability is the most common migration blocker. Prompts tuned to one model's quirks (specific formatting instructions, model-specific phrasing that works around known weaknesses, few-shot examples calibrated to one model's style) may perform worse on other models. Identifying and neutralizing these dependencies is prerequisite work.
• The rollback plan should be a configuration change, not a deployment. Auto Mode off, or a single-model Allowed Model Pool, returns traffic to the fixed model immediately without a code change or redeploy.

Phase 0: Establish the Baseline

Migration without a baseline produces an unfalsifiable claim. If quality drops after routing is enabled and there is no pre-migration measurement, the team cannot distinguish a routing problem from normal variance, a prompt change, or a model provider update that happened the same week.

Build an evaluation set from production traffic.

Sample 100 to 200 real requests from production logs, covering the full distribution of what the application handles: typical requests, edge cases, long-context requests, requests in each supported language, and the failure cases that generated support tickets. Redact any personal data before storing the evaluation set.

The evaluation set must reflect the actual traffic distribution, not a curated set of clean examples. A model that performs well on well-formed requests and poorly on the ambiguous ones that constitute 15 percent of production traffic will look fine on a clean evaluation set and produce complaints in production.

Measure the current fixed model on the evaluation set.

Run every evaluation request through the current production model with the current production prompts. Record: the output, time to first token, total generation time, input token count, output token count, and cost. This is the baseline.

Define quality criteria and score the baseline.

For each evaluation request, define what a good response looks like. For tasks with verifiable correct answers, automated checking works. For subjective quality, use an LLM-as-judge rubric calibrated against human review on a sample of 30 to 50 requests.

Score the baseline outputs. This produces the number that routing must match or beat: the current model's quality score on the actual production distribution.

Record the current cost structure.

Compute the actual monthly cost breakdown: total input tokens, total output tokens, effective rate paid, and any prompt caching benefit currently realized. The migration's cost benefit is measured against this number, not against list pricing.

Phase 1: Audit Prompt Portability

Prompts tuned against one model often contain dependencies on that model's specific behavior. These dependencies degrade performance when the same prompt reaches a different model.

Four categories of model-specific prompt dependency:

Workaround instructions.

Instructions added to compensate for a known weakness in the current model: "Do not include a preamble before the answer" (added because the model habitually preambles), "Always close all markdown code blocks" (added because the model sometimes forgets), "Answer in exactly three sentences" (added because the model over-explains). These instructions are harmless on models that do not have the weakness but consume tokens and occasionally produce over-correction.

Format calibration.

Few-shot examples showing exactly the output style desired, calibrated against how the current model naturally responds. On a different model, the same examples may push output toward a style that model produces less naturally, degrading quality relative to what the model would produce with a simpler instruction.

Model-specific feature usage.

Anthropic's cache_control markers, OpenAI's response_format parameter, provider-specific system message structures. These do not transfer across providers and must be handled at the routing layer or normalized in the application.

Implicit capability assumptions.‍

Prompts written assuming a specific context window size, a specific tool calling format, or a specific reasoning capability. A prompt that assumes 200K context breaks on a model with 128K.

The portability audit:

Read each production prompt and classify every instruction: does this exist because the task requires it, or because the current model needs it? Instructions in the second category are candidates for removal.

Test the simplified prompt against the current model first. If quality holds, the workaround was unnecessary and removing it improves portability. If quality drops, the workaround is load-bearing and the prompt is genuinely model-dependent, which means routing to other models will require prompt variants or the model should stay pinned for that use case.

The output of this phase is a set of prompts that express what the task requires without encoding one model's quirks, plus a list of use cases that remain genuinely model-dependent and should be excluded from routing initially.

Phase 2: Select the Initial Model Pool

The instinct is to enable routing across the full model catalog to maximize the cost benefit. The correct approach for a first migration is the narrowest pool that demonstrates the benefit.

Start with two models.

The current production model plus one alternative. The alternative should be selected for the task categories that dominate the application's traffic, and should be meaningfully cheaper than the current model for the cost benefit to be measurable.

Two models produce a manageable comparison surface: for any request, either the current model or the alternative handled it, and quality differences are attributable to a single substitution rather than a distribution across many models.

Selecting the alternative model.

Identify the dominant task categories in the production traffic. A customer service application is primarily Instruction Following and Knowledge. A coding assistant is primarily Coding and Agentic. A document analysis tool is primarily Long Context and Reasoning.

Select an alternative that performs well on those specific categories rather than one with the highest aggregate benchmark score. As covered in GMI Cloud's guide to choosing the right model for each prompt, task-specific benchmark performance predicts production quality more reliably than aggregate rankings.

Configure the Allowed Model Pool.

Set the pool to exactly these two models. GMI Router will only select from the allowed pool, including for fallback, so this constraint holds under all conditions including failure scenarios.

Set the initial mode to Quality.

Quality mode selects the highest-performing model for the detected task regardless of cost. This is the correct starting mode because it isolates the routing decision's quality impact from the cost optimization. If quality holds in Quality mode, the routing decision is working. Cost optimization comes later by shifting to Balanced or Cost mode.

Phase 3: Shadow Routing

Shadow routing sends production traffic to both the current fixed model and the routing layer, returns the fixed model's response to the user, and logs the routing layer's response for comparison. No user sees routed output during this phase.

Implementation.

For a defined fraction of production traffic (start at 10 percent, expand to 50 percent as confidence grows), send the request to both paths. Return the fixed model's response. Log both responses along with the routing metadata: which model the router selected, the detected task type, and the mode.

The cost of shadow routing is double inference cost on the shadowed fraction. At 10 percent of traffic, this adds 10 percent to the inference bill for the duration of the shadow period. Budget for this explicitly rather than being surprised by it.

What to measure during shadow routing:

Model distribution.

What fraction of requests route to each model in the pool? If 95 percent route to the current model, the routing layer is not finding opportunities to substitute, and the cost benefit will be small. If 95 percent route to the alternative, verify that the alternative is genuinely handling the traffic well before treating this as a positive signal.

Quality comparison.

Score both responses on the same rubric used for the baseline. The relevant metric is the fraction of requests where the routed response scores as well as or better than the fixed model response. Below 90 percent parity indicates a quality problem worth investigating before proceeding.

Latency comparison.

Compare time to first token and total generation time between paths. Routing adds the routing decision latency, and the selected model may be faster or slower than the fixed model. Confirm the total latency stays within the application's budget.

Cost comparison.

Compute the cost of the routed path against the fixed path on the shadowed traffic. This is the projected cost benefit, measured on real production traffic rather than estimated from benchmarks.

Cache hit rate.

Compare prompt cache hit rates between paths. If the routed path has significantly lower cache hits, investigate whether routing is distributing requests across endpoints in a way that fragments cache locality.

Duration.

Run shadow routing for at least one full traffic cycle, meaning a full week for applications with weekday and weekend variation, or a full month for applications with monthly patterns. Two weeks is a reasonable default that captures both weekday and weekend distributions.

Phase 4: Canary Rollout

When shadow routing data shows quality parity and the projected cost benefit is confirmed, begin routing real user traffic.

Start at 5 percent.

Route 5 percent of production traffic through the routing layer, returning the routed response to users. Keep 95 percent on the fixed model. This limits the blast radius if a quality problem appears that shadow evaluation did not catch.

Monitor the signals that shadow routing cannot measure.

Shadow routing compares outputs but does not capture user reaction. The canary phase adds:

User-visible quality signals: thumbs up and down rates, regeneration requests, session abandonment, support ticket volume tagged to the affected feature. Compare these between the routed cohort and the control cohort.

Downstream failure rates: for applications where the model output feeds into automated processing (JSON parsing, code execution, structured data extraction), track parse failure and execution failure rates. A model that produces slightly different formatting can break downstream automation in ways that quality scoring on the output text does not surface.

Per-model quality breakdown: split the quality metrics by which model handled the request. If the alternative model shows worse user signals than the current model, the routing decision may be sending it requests it handles poorly.

Expansion schedule.

5 percent for 3 days. If signals are clean, 25 percent for 3 days. Then 50 percent for a week. Then 100 percent. Hold at each stage long enough to accumulate a statistically meaningful sample of the user-visible signals, which for low-volume feedback signals like thumbs-down rate may require longer than the traffic volume alone suggests.

Rollback criteria, defined in advance.

Define the thresholds that trigger rollback before the canary starts: user-visible quality signal degradation beyond a defined percentage, downstream failure rate increase beyond a defined threshold, latency p99 exceeding the application's budget, or any critical incident attributable to the routed path.

Rollback is a configuration change: set Auto Mode off, or reduce the Allowed Model Pool to the single current model. Traffic returns to the fixed model immediately without a deployment.

Phase 5: Expand the Pool and Optimize the Mode

With routing established on a two-model pool at 100 percent traffic, the remaining optimization is expanding the pool and shifting the mode toward cost.

Expand the model pool incrementally.

Add one model at a time. Each addition changes the routing distribution, so the same measurement discipline applies: shadow the expanded pool against the current pool, compare quality and cost, then canary the expansion.

The value of each additional model diminishes. Adding a strong model for a task category that represents 30 percent of traffic produces meaningful benefit. Adding a fifth model that wins on a category representing 2 percent of traffic produces marginal benefit and additional output style variation for the application to handle.

Shift the mode toward cost.

Quality mode establishes that routing preserves quality. Once that is confirmed, Balanced mode captures cost savings by weighing cost alongside quality in the selection.

In GMI's internal evaluation across 12 benchmarks and 21 models, Quality mode delivered 84.9 percent quality at 0.72× the cost of always using GPT-5.5 (82.5 percent, 1.00× cost). Balanced mode delivered 81.3 percent quality at 0.22× cost, matching Opus 4.8's quality at 22 percent of its cost.

The mode shift should follow the same shadow-then-canary process as the initial migration. Balanced mode routes differently than Quality mode, which means the quality validation performed for Quality mode does not automatically transfer.

Consider per-workload mode configuration.

Not all traffic requires the same mode. Customer-facing interactive features may stay in Quality or Balanced mode. Internal tooling, batch processing, and non-critical background tasks may run in Cost mode. Configuring mode per workload rather than globally captures the cost benefit where quality tolerance allows without affecting the user-facing paths.

The Migration Timeline

For a production application with existing traffic:

Total elapsed time from start to 100 percent routed traffic: approximately 5 to 7 weeks. The majority of that time is measurement and confidence building rather than engineering work. The actual implementation is a configuration change; the timeline is dominated by accumulating enough production data to confirm quality parity.

Teams under time pressure can compress Phase 3 to one week and Phase 4 to two weeks, at the cost of less data supporting the quality parity conclusion. Compressing below that produces migrations where quality problems appear after full rollout rather than during canary.

Common Migration Failures

Skipping the baseline.

Without a pre-migration quality measurement on the production distribution, any quality complaint after migration is unattributable. Teams end up rolling back on anecdote rather than data, or persisting through real problems because they cannot confirm the problems are real.

Enabling the full model catalog immediately.

A wide pool maximizes theoretical cost benefit and maximizes the number of output style variations the application encounters. Debugging a quality issue across a 15-model pool is significantly harder than across a two-model pool.

Not auditing prompt portability.

Prompts loaded with workarounds for the current model's quirks perform unpredictably on other models. The migration produces quality variance that appears to be a routing problem but is actually a prompt problem.

Measuring quality on outputs only.

Downstream automation failures (JSON parse errors, code execution failures, format mismatches in templated pipelines) do not appear in output quality scoring. Track these separately during canary.

Treating the mode shift as free.

Moving from Quality mode to Balanced or Cost mode changes the routing distribution. The quality validation performed for one mode does not transfer to another. Each mode change requires its own shadow and canary cycle.

No defined rollback criteria.

Without pre-defined thresholds, rollback decisions become debates during an incident. Define the numbers before the canary starts.

GMI Router Controls That Support Migration

Three GMI Router features map directly to the migration phases.

Allowed Model Pool controls exactly which models routing can select. Starting with two models and expanding incrementally is a pool configuration change. The pool constraint applies to fallback selection as well as primary selection, so the migration's model scope holds even during failure conditions.

Auto Mode toggle enables or disables routing per workspace. During canary rollout, this is the mechanism for controlling which traffic routes and which stays pinned. During rollback, disabling Auto Mode returns all traffic to the fixed model without a deployment.

Three mode preferences (Cost, Balanced, Quality) allow the migration to begin in Quality mode, where the routing decision's quality impact is isolated from cost optimization, then shift toward cost as quality parity is confirmed.

Routing metadata in the API response identifies which model handled each request, the detected task type, the selected mode, and whether fallback was triggered. This metadata is what makes shadow routing measurement possible without additional instrumentation: the per-model quality breakdown, the model distribution analysis, and the cost attribution all derive from it.

Conclusion

Migrating from a fixed model to routing is a measurement exercise wrapped around a configuration change. The configuration change takes a day. The measurement that confirms quality parity on production traffic takes five to seven weeks, and skipping it is what produces the migrations that get rolled back.

The sequence that works: establish a quality baseline on production-representative traffic, audit prompts for model-specific dependencies, start with a two-model pool in Quality mode, shadow route for two weeks to compare quality and cost on real traffic, canary from 5 percent to 100 percent with pre-defined rollback thresholds, then expand the pool and shift toward Balanced or Cost mode with the same discipline applied to each change.

The payoff, based on GMI's internal evaluation across 12 benchmarks and 21 models: Quality mode routing produced higher quality than always using GPT-5.5 at 28 percent lower cost, and Balanced mode matched Opus 4.8's quality at 22 percent of its cost. Those numbers are the reason to migrate. The playbook above is how to get there without a rollback.

FAQs

How long does a migration from a fixed model to routing take? Approximately five to seven weeks from start to 100 percent routed traffic for a production application. The breakdown: 3 to 5 days establishing a quality baseline, 2 to 5 days auditing prompt portability, one day selecting the initial model pool, two weeks of shadow routing, and two to three weeks of incremental canary rollout. The majority of that time is measurement rather than engineering. The actual implementation is a configuration change; the timeline is dominated by accumulating enough production data to confirm quality parity before user traffic is affected.

What is shadow routing and why does it matter for migration? Shadow routing sends production traffic to both the current fixed model and the routing layer, returns the fixed model's response to the user, and logs the routing layer's response for comparison. No user sees routed output during this phase. It matters because it produces quality and cost comparison data on real production traffic without any user-facing risk. The cost is double inference on the shadowed fraction: at 10 percent of traffic, shadow routing adds 10 percent to the inference bill for the duration.

How many models should the initial routing pool contain? Two: the current production model plus one alternative selected for the task categories that dominate the application's traffic. A two-model pool produces a manageable comparison surface where quality differences are attributable to a single substitution, and limits the number of output style variations the application must handle. Expanding to a wide pool immediately maximizes theoretical cost benefit and maximizes debugging difficulty if quality problems appear.

What prompt changes are needed before enabling routing? Audit each production prompt for four categories of model-specific dependency: workaround instructions added to compensate for the current model's known weaknesses, few-shot examples calibrated to the current model's natural output style, provider-specific feature usage that does not transfer across models, and implicit capability assumptions about context window or tool calling format. Test simplified prompts against the current model first: if quality holds after removing a workaround, the workaround was unnecessary and its removal improves portability.

What triggers a rollback during canary rollout, and how fast is it? Rollback criteria should be defined before the canary starts: user-visible quality signal degradation beyond a defined percentage, downstream failure rate increase beyond a defined threshold, latency p99 exceeding the application's budget, or any critical incident attributable to the routed path. Rollback itself is a configuration change: disabling Auto Mode or reducing the Allowed Model Pool to the single current model returns traffic to the fixed model immediately, without a code change or deployment.