Routing Without a Judge Model: Why Sub-200ms Model Selection Needs a Different Architecture

Most LLM routing systems make their decision by asking another language model. A small classifier model reads the incoming prompt, decides what kind of task it is, and returns a routing recommendation. This works, and it is the architecture behind most routing implementations available today. It also adds 50 to 300 milliseconds to every request, introduces a second point of failure, and in some implementations modifies the prompt before it reaches the model that generates the answer. For interactive applications where the routing overhead is a visible fraction of total response time, and for agentic workloads where every step pays that overhead again, judge-based routing costs more than it saves.

• Judge-model routing adds 50 to 300 milliseconds per request. An LLM classifier reading a prompt and returning a task type is itself an inference call, subject to the same prefill latency, queuing, and provider variability as the request it is routing.
• GMI Router completes routing in under 200 milliseconds without a judge model. The routing decision runs on a model GMI trained specifically for the routing task, not a general-purpose LLM prompted to classify. The prompt is passed to the selected model verbatim, with no rewriting, no injected instructions, and no classification preamble.
• Prompt modification is the hidden cost of judge-based routing. Some routing implementations rewrite or annotate the prompt during classification. Any modification changes what the generating model sees, which invalidates cached prefixes, breaks prompt caching, and makes routing behavior harder to debug when output quality changes unexpectedly.
• KV-cache continuity is incompatible with prompt rewriting. Cache reuse requires the prefix to be byte-identical across requests. A router that modifies prompts produces a different prefix on every request, defeating prompt caching entirely and forfeiting the 60 to 90 percent input cost reduction that stable prefixes deliver.
• The measured result of judge-free routing: in GMI's internal evaluation across 12 benchmarks and 21 models, Quality mode delivered 84.9 percent quality at 0.72× the cost of always using GPT-5.5, which scored 82.5 percent at 1.00× cost. Balanced mode delivered equivalent quality to Claude Opus 4.8 at 0.22× the cost.
• Routing latency compounds in agentic workloads. A 30-step agent pipeline with 150ms of routing overhead per step accumulates 4.5 seconds of pure routing latency across the task. At sub-200ms with a specialized routing model, the same pipeline stays within the latency budget that judge-based routing exceeds.

How Judge-Model Routing Works and What It Costs

Judge-model routing uses a language model to make the routing decision. The architecture is straightforward: the incoming prompt is sent to a small, fast model with a system prompt instructing it to classify the task, estimate complexity, or recommend a model tier. The classifier returns a decision, and the router forwards the original request to the selected model.

The appeal is obvious. Language models are good at understanding what a prompt is asking for, which is exactly the judgment the routing decision requires. Implementation is straightforward because the classifier is just another API call with a carefully written system prompt.

The costs appear at production scale.

Cost 1: Latency, paid on every request.

The judge model call is a complete inference request. It has prefill latency proportional to the prompt length, queuing delay if the classifier endpoint is under load, network round-trip time to the classifier endpoint, and generation time for the classification output.

For a small classifier model on dedicated infrastructure with a short prompt, this lands at 50 to 100 milliseconds. For a classifier on shared serverless infrastructure, or with a long prompt that must be fully prefilled before classification, it reaches 200 to 300 milliseconds. This latency is paid before the actual generating model receives the request.

In a text chat application, 150 milliseconds of routing overhead is a small fraction of a 2-second total response. In a voice AI application with an 800-millisecond total budget, it consumes nearly 20 percent of the entire budget. In an agentic workload with 30 sequential steps, it accumulates to 4.5 seconds of pure routing time per task.

Cost 2: A second point of failure.

The judge model is an inference endpoint with its own availability characteristics. When the classifier is rate-limited, times out, or returns a malformed response, the routing system must decide what to do: retry the classification (adding more latency), fall back to a default model (losing the routing benefit), or fail the request entirely.

Every failure mode of the generating model now also applies to the classifier. The routing layer, which exists to improve reliability by enabling fallback, has itself become a reliability dependency.

Cost 3: Prompt modification and cache invalidation.

Some judge-based routing implementations pass the prompt to the classifier unchanged and forward the original prompt to the generating model. This is the correct implementation.

Others modify the prompt during routing: adding routing metadata, rewriting for clarity, injecting classification results as context, or restructuring the prompt to match the selected model's preferred format. Each modification means the generating model sees a different prompt than the application sent.

The immediate consequence is cache invalidation. Prompt caching requires the prefix to be byte-identical across requests to produce a cache hit. A router that appends routing metadata, rewrites the system prompt, or restructures the message ordering produces a unique prefix on every request. The cache hit rate drops to zero, and the 60 to 90 percent input cost reduction that prompt caching delivers is forfeited.

The secondary consequence is debuggability. When output quality changes and the prompt that reached the model differs from the prompt the application sent, root cause analysis requires reconstructing what the router did to the prompt before attributing the quality change to the model.

Cost 4: Classification quality is bounded by classifier capability.

A general-purpose model prompted to classify tasks performs the classification the way it performs any other task: reasonably well most of the time, with a failure rate that depends on prompt clarity and task ambiguity. It has not been trained specifically to predict which model will perform best on a given prompt. It is being asked to approximate that judgment from its general capabilities.

The Alternative: A Model Trained for Routing

GMI Router replaces the judge model with a model trained specifically for the routing task. This is not a general-purpose LLM given a classification prompt. It is a model whose training objective is predicting model performance on a given prompt across a benchmark-backed evaluation set.

Why a specialized routing model is faster.

A general-purpose LLM performing classification must process the prompt through a full transformer forward pass sized for general language understanding, then generate output tokens describing its classification. A specialized routing model performs a narrower task with an architecture sized for that task, producing a routing decision rather than a natural language classification that must then be parsed.

The practical result is routing completion in under 200 milliseconds, which for most production workloads is a small enough fraction of total latency that it does not measurably affect the user experience.

Why a specialized routing model is more accurate for routing.

The routing decision is not "what kind of task is this." It is "which model in the eligible pool will produce the best result for this prompt, given the current mode preference and cost constraints." A general classifier answers the first question; the routing system must then map task type to model through a separate scoring layer.

A model trained on the routing objective directly learns the mapping from prompt characteristics to model performance, using the benchmark data as the training signal. GMI Router's evaluation layer draws on 12 benchmarks covering the full task spectrum: Agentic Planning, GPQA Diamond, IFBench, LiveBench Coding, LiveBench Data Analysis, LiveBench Instruction, LiveBench Language, LiveBench Math, LiveBench Reasoning, LiveCodeBench, Terminal-Bench, and WildBench.

Why the prompt stays verbatim.

A specialized routing model reads the prompt to make its decision but does not modify it. The prompt that reaches the selected model is byte-identical to the prompt the application sent. This preserves three properties that prompt-modifying routers forfeit:

Prompt caching works. Stable prefixes remain byte-identical across requests, so cache hits occur as expected and the input cost reduction from caching is preserved.

Debugging is straightforward. When output quality changes, the prompt that reached the model is the prompt the application sent, so quality analysis does not require reconstructing router modifications.

Provider-side features behave predictably. Anthropic's cache_control markers, OpenAI's automatic prefix caching, and provider-specific prompt features all operate on the prompt as sent, without interference from routing-layer modifications.

Why KV-Cache Continuity Requires Judge-Free Routing

The interaction between routing and caching is where judge-model architectures compound their cost, particularly for multi-turn and agentic workloads.

Prompt caching depends on prefix stability. When turn 5 of a conversation shares its first 4,000 tokens with turn 4, the KV cache for those tokens can be reused rather than recomputed. This is the mechanism that keeps multi-turn conversations from becoming progressively more expensive as context accumulates.

A router that modifies the prompt breaks this. If the router appends routing metadata, the metadata differs per request and the prefix is no longer stable. If the router rewrites the system prompt to match the selected model's preferred format, and the selected model changes between turns, the system prompt changes with it. Either modification produces a cache miss where a cache hit should have occurred.

The cost compounds with conversation length. At turn 10 with 8,000 tokens of accumulated context, a cache miss means recomputing 8,000 tokens of prefill. At standard rates this is both a latency cost (400 to 800 milliseconds of prefill time) and a compute cost (full input token charges rather than the 10 to 50 percent cache-hit rate).

GMI Router is designed around this constraint. Routing decisions preserve KV-cache reuse across turns because the prompt is passed through unmodified and the routing layer does not introduce per-request variation into the prefix. For agentic workloads where each step builds on accumulated context, this is the difference between per-step costs that stay flat and per-step costs that grow linearly with session length.

Where Routing Latency Actually Matters

Routing overhead is not equally consequential across workload types. Understanding where it matters determines whether judge-free architecture is a nice-to-have or a requirement.

Interactive chat applications: moderate impact.

A text chat interface with a 1.5 to 3 second total response time absorbs 150 milliseconds of routing overhead without users noticing. Judge-based routing is acceptable here, though the latency is not free and the cache invalidation risk still applies if the router modifies prompts.

Voice AI: critical impact.

The voice AI latency budget is approximately 800 milliseconds end-to-end, split across speech-to-text, LLM inference, and text-to-speech. The LLM inference stage has a budget of 150 to 300 milliseconds for time to first token. Judge-model routing at 150 milliseconds consumes half of that budget before the generating model receives the request. Sub-200ms judge-free routing keeps the routing overhead small enough that the LLM stage stays within its allocation.

Agentic workloads: compounding impact.

Agent pipelines make many sequential model calls. A 30-step agent task pays the routing overhead 30 times. At 150 milliseconds per routing decision, the accumulated routing latency is 4.5 seconds per task, which is added to the actual inference and tool call time. At sub-200ms with judge-free routing, the same pipeline stays materially closer to its latency target.

The cache interaction compounds this further. Agentic workloads accumulate context across steps, which makes cache reuse increasingly valuable as the session progresses. A router that breaks cache continuity converts what should be flat per-step cost into cost that grows with step count.

Batch processing: minimal impact.

For async workloads with no latency sensitivity, routing overhead of any magnitude is acceptable. The cache invalidation concern still applies if the router modifies prompts, but the latency concern does not.

What the Measured Results Show

GMI's internal evaluation compared routing performance against fixed-model baselines across 12 benchmarks and 21 models.

Two findings are worth reading carefully.

Quality mode outperformed both fixed-model baselines on quality while costing less. This is not a cost-quality tradeoff. Routing in Quality mode produced higher benchmark quality than always using GPT-5.5 (84.9 percent versus 82.5 percent) at 28 percent lower cost. The mechanism is that no single model leads across all task categories, so routing each prompt to the category leader outperforms any fixed choice.

Balanced mode matched Opus 4.8's quality at 22 percent of its cost. For teams currently defaulting to a frontier model for all traffic, this represents an 84 percent cost reduction at equivalent measured quality.

These results depend on the routing decision being accurate. A routing system that misclassifies prompts sends requests to the wrong model and produces worse results than a fixed frontier model. The accuracy of the routing decision is the property that determines whether routing beats a fixed choice, which is why the routing model's training objective matters.

The Reliability Argument for Judge-Free Routing

Routing systems exist partly to improve reliability through automatic fallback. As covered in GMI Cloud's article on primary and fallback models, a well-configured routing layer selects a primary model and fallback candidates, then applies the fallback automatically when the primary fails, times out, or hits rate limits.

Judge-model routing undermines this by introducing a new dependency. If the classifier endpoint is unavailable, the routing decision cannot be made. The system must either fail the request, fall back to a default model without routing intelligence, or retry the classification and accept additional latency.

A specialized routing model running on the routing infrastructure rather than as an external API call removes this dependency. The routing decision does not depend on the availability of a separate inference endpoint that has its own rate limits, capacity constraints, and failure modes.

This matters most during exactly the conditions where routing is most valuable: platform-wide load peaks where some models are rate-limited and fallback routing is needed. A judge model on shared infrastructure during a load peak is itself likely to be slow or rate-limited, which means the routing layer degrades precisely when its fallback capability is most needed.

Implementation: What Judge-Free Routing Looks Like in Practice

For teams evaluating routing architectures, four properties distinguish judge-free from judge-based implementations.

Property 1: Routing latency measured independently.

Measure the time between request receipt and the routing decision, separate from the time spent in the generating model. This isolates the routing overhead. If the routing layer does not expose this timing, measure the difference between the total request latency with routing enabled and with a fixed model.

Property 2: Prompt verbatim guarantee.

Verify that the prompt reaching the generating model is byte-identical to the prompt sent by the application. This can be tested by sending a prompt with distinctive content and inspecting what the generating model received, or by monitoring cache hit rates: if a stable prefix produces zero cache hits under routing but cache hits without routing, the router is modifying the prompt.

Property 3: Cache hit rate under routing.

Compare the prompt cache hit rate with routing enabled against the rate with a fixed model. Equivalent rates indicate the router preserves cache continuity. A significant drop indicates prompt modification or endpoint distribution that breaks cache locality.

Property 4: Routing availability independent of model availability.

Test routing behavior when the primary model endpoint is unavailable. A judge-free architecture continues making routing decisions and applies fallback. A judge-based architecture on shared infrastructure may fail to route at all if the classifier shares infrastructure with the degraded model.

GMI Router: Architecture and Controls

GMI Router implements the judge-free architecture with the operational controls that production deployments require.

The four-stage routing process. Understand the prompt (task characteristics, complexity, domain), evaluate candidate models against the benchmark bank for the detected task profile, select the best-fit model within the mode preference and allowed pool constraints, and route the request with the prompt passed verbatim.

Three mode preferences. Cost mode maximizes savings within acceptable quality bounds. Balanced mode weighs quality and cost together. Quality mode selects the highest-performing model for the detected task regardless of cost. The measured results above show Quality and Balanced mode performance against fixed-model baselines.

Allowed Model Pool. Restrict routing to an explicit set of approved models. This is the governance control that ensures routing operates within organizationally approved boundaries, including for fallback selection. Enterprise teams with security-reviewed model lists configure the pool to match.

Auto Mode toggle. Enable or disable automatic routing per workspace or per request, allowing teams to route some traffic and pin other traffic to a specific model during migration or for workloads with specific model requirements.

No routing fees. During preview, routing is free. Teams pay only for the model that generates the response, which means the routing layer's cost benefit is not partially consumed by routing charges.

For teams evaluating whether routing fits their workload, the practical starting point is the analysis in GMI Cloud's guide to model routing, which covers the cost, quality, latency, and reliability dimensions that routing addresses.

Conclusion

Judge-model routing is the straightforward implementation of LLM routing and the reason most routing systems add 50 to 300 milliseconds of latency to every request. For text chat applications with multi-second response budgets, that overhead is tolerable. For voice AI with an 800-millisecond total budget, and for agentic workloads that pay the overhead on every one of dozens of sequential steps, it is not.

The architectural alternative is a model trained specifically for the routing task rather than a general-purpose LLM prompted to classify. This delivers sub-200ms routing decisions, removes the classifier as a separate reliability dependency, and preserves the prompt verbatim so that prompt caching and KV-cache continuity work as expected.

The measured outcome in GMI's evaluation across 12 benchmarks and 21 models: Quality mode produced higher quality than always using GPT-5.5, at 28 percent lower cost. Balanced mode matched Opus 4.8's quality at 22 percent of the cost. Those results depend on routing accuracy, which is why the routing model's training objective matters more than the convenience of prompting a general model to classify.

Start routing with GMI Router

FAQs

What is judge-model routing and why does it add latency? Judge-model routing uses a language model to classify the incoming prompt and recommend a model tier. The classifier call is a complete inference request with its own prefill latency, queuing delay, network round-trip, and generation time, typically adding 50 to 300 milliseconds before the generating model receives the request. A model trained specifically for routing performs a narrower task with an architecture sized for it, producing a routing decision rather than a natural language classification that must be parsed.

Why does prompt modification during routing matter? Prompt caching requires the prefix to be byte-identical across requests to produce a cache hit. A router that appends routing metadata, rewrites the system prompt, or restructures message ordering produces a unique prefix on every request, dropping the cache hit rate to zero and forfeiting the 60 to 90 percent input cost reduction that stable prefixes deliver. Prompt modification also complicates debugging, because the prompt reaching the model differs from the prompt the application sent.

How much does routing latency matter for different workload types? For text chat with a 1.5 to 3 second total response time, 150 milliseconds of routing overhead is tolerable. For voice AI with an 800-millisecond end-to-end budget and a 150 to 300 millisecond allocation for LLM time to first token, judge-based routing consumes half the LLM budget before generation begins. For agentic workloads, the overhead is paid on every step: a 30-step pipeline at 150 milliseconds per routing decision accumulates 4.5 seconds of pure routing latency per task.

Does routing actually produce better quality than always using a frontier model? In GMI's internal evaluation across 12 benchmarks and 21 models, Quality mode routing produced 84.9 percent quality at 0.72× cost, compared to 82.5 percent at 1.00× cost for always using GPT-5.5. The mechanism is that no single model leads across all task categories, so routing each prompt to the category leader outperforms any fixed choice. This depends on routing accuracy: a system that misclassifies prompts sends requests to the wrong model and underperforms a fixed frontier model.

How do you verify that a routing system preserves prompt caching? Compare the prompt cache hit rate with routing enabled against the rate with a fixed model on the same workload. Equivalent rates indicate the router preserves cache continuity. A significant drop indicates either prompt modification during routing or endpoint distribution that breaks cache locality. A second test: send a prompt with distinctive content and verify that what reached the generating model is byte-identical to what the application sent.

‍