Routing Metadata: What to Log When Models Are Selected Automatically

The standard LLM observability playbook assumes you know which model handled each request, because you specified it in the API call. Log the prompt, the completion, the token counts, the latency, and the cost, and you have a complete picture. Automatic model routing breaks that assumption in a specific way: the model becomes a per-request variable rather than a deployment constant, which means every metric you were already tracking now needs a model dimension to remain interpretable. A cost increase is meaningless without knowing whether the routing distribution shifted. A quality regression is undiagnosable without knowing which model produced the affected responses. A latency spike could be the routing layer, the selected model, or a fallback event, and standard observability cannot distinguish them.

• Routing turns four existing metrics into four segmented metrics. Cost, quality, latency, and error rate each need a per-model breakdown to remain actionable once the model varies per request. Aggregate numbers hide the routing distribution that produced them.
• GMI Router returns routing metadata in the API response, which means the model dimension is available without additional instrumentation: selected model, attempted primary model, fallback models, detected task type with confidence, selected mode, whether fallback triggered, and the fallback reason.
• Fallback events are invisible without explicit metadata. A request that fails on the primary model and succeeds on the fallback returns a normal response. Without the fallback_triggered flag, the incident that caused the fallback never appears in monitoring, and the pattern of primary model degradation goes undetected until it worsens.
• The routing decision itself is a thing to evaluate, not just a mechanism to use. Task type detection can be wrong. When it is, the request routes to a model selected for the wrong task category. Logging the detected task type with its confidence score makes this measurable.
• Cost attribution changes shape under routing. Effective cost per request is no longer a function of the endpoint you called; it is a function of the routing distribution. Two workloads with identical token volumes can differ substantially in cost because one routes more traffic to expensive models.
• OpenTelemetry's GenAI semantic conventions provide the foundation with gen_ai.request.model and token usage attributes. Routing metadata extends this with the fields that describe why that model was selected.

What Changes in Observability When the Model Is a Variable

Four metrics that were previously scalar become distributions once routing is enabled. Each requires a different analysis.

Cost: from a rate to a weighted average.

Without routing, cost per request is predictable: token count times the known rate for the model you called. With routing, cost per request depends on which model handled it, and the aggregate cost is a weighted average across the routing distribution.

A month-over-month cost increase now has three possible causes: token volume grew, the routing distribution shifted toward more expensive models, or the model pricing changed. Without per-model cost breakdown, these are indistinguishable.

The metric to track: cost per request broken down by selected model, plus the routing distribution (what percentage of requests went to each model). A distribution shift with flat token volume is the signal that something changed in the routing behavior, which may be a task type distribution change in the incoming traffic or a change in model availability affecting selection.

Quality: from one baseline to several.

Without routing, quality regressions have a small set of causes: a prompt change, a model version update, or a shift in the input distribution. With routing, a fourth cause appears: the routing distribution shifted and a model that handles a task category worse is now receiving more of it.

The metric to track: quality scores segmented by selected model and by detected task type. A quality drop concentrated in one model on one task type points to a routing accuracy problem. A quality drop spread evenly across models points to an input distribution change or a prompt issue.

Latency: from one profile to several, plus routing overhead.

Different models have different latency profiles. A routing distribution that shifts toward a slower model produces a latency increase that looks like infrastructure degradation.

Routing also adds its own latency. This should be measured separately from model inference latency, because the two have different causes and different fixes. A routing layer that adds 150 milliseconds per request has a different remedy than a model that is slow to first token.

The metrics to track: routing decision latency as a distinct measurement, and time to first token segmented by selected model.

Error rate: from one failure mode to two layers.

Without routing, an error is a model error. With routing, an error can occur at the routing layer (the routing decision failed) or at the inference layer (the selected model failed). These have different causes and different remediation.

The critical addition: fallback events. A request where the primary model failed and the fallback succeeded returns a normal response to the application. The error occurred, was handled, and is invisible without explicit metadata marking it.

The Routing Metadata Fields and What Each Enables

GMI Router returns these fields in the API response. Each supports a specific operational analysis.

selected_model: the model that generated the response.

This is the foundational field, and it is the one that makes every other metric segmentable. Without it, cost attribution, quality analysis by model, and latency profiling by model are all impossible.

Operational use: join this field to token counts for per-model cost attribution, and to quality scores for per-model quality analysis.

attempted_primary_model: the model routing selected first.

When fallback did not trigger, this equals selected_model. When fallback triggered, the difference between these two fields identifies which model failed.

Operational use: aggregate over time to build a reliability profile per model. A model whose attempted_primary_model count substantially exceeds its selected_model count is failing frequently.

fallback_model: the configured fallback candidates.

Returned whether or not fallback triggered, which provides transparency into what would have happened. This matters for governance: it confirms that the fallback candidates are within the allowed model pool.

primary_task_type and task_confidence: the detected task category and detection certainty.

The routing decision starts with classifying what the prompt is asking for. Logging the detected task type makes the classification auditable.

Operational use: two analyses. First, the task type distribution across your traffic, which tells you what your application actually does versus what you assumed it does. Second, quality segmented by task type, which identifies categories where routing is selecting poorly.

Low confidence scores are a specific signal worth alerting on. A cluster of requests with low task confidence indicates prompt patterns that the classifier finds ambiguous, and those requests are the most likely to be routed suboptimally.

selected_mode: which mode preference applied.

Cost, Balanced, or Quality. This matters when mode is configured per workspace or per request rather than globally, because it explains why two similar requests routed to different models.

fallback_triggered and fallback_reason: whether the fallback applied and why.

This is the field that converts an invisible failure into a monitored one. Fallback reasons map to distinct operational responses:

A high rate of rate-limit-triggered fallbacks (HTTP 429) indicates a quota problem: the primary model does not have enough capacity allocation for the traffic volume, and the fix is reserved capacity or a higher-tier quota.

A high rate of timeout-triggered fallbacks indicates that the primary model's latency is approaching the timeout threshold, and either the threshold needs adjustment or a faster model should be primary.

A high rate of 5xx-triggered fallbacks indicates provider instability, which may warrant replacing the primary model entirely.

The Four Analyses That Routing Metadata Enables

Analysis 1: Routing distribution over time.

Chart the percentage of requests routed to each model, by day or by hour. This is the single most informative routing metric because it changes when anything changes: traffic composition, model availability, mode configuration, or the allowed model pool.

What to look for: a sudden distribution shift with no configuration change. This usually means either the incoming task type distribution changed (your users are doing something different) or a model became unavailable and traffic redistributed.

Analysis 2: Cost per task type.

Join primary_task_type to token counts and per-model rates. This reveals which task categories are expensive, which is not always the categories teams expect.

The common finding: a task type representing 10 percent of requests consumes 40 percent of the cost, because it routes to an expensive model and produces long outputs. This is the highest-leverage optimization target, and it is invisible without task-type-segmented cost data.

Analysis 3: Quality delta between models on the same task type.

Segment quality scores by selected_model within each primary_task_type. This answers the question that determines whether routing is working: for each task category, is the model routing selects actually the best-performing one on your traffic?

When a lower-cost model shows equivalent quality to a higher-cost model on a given task type, shifting the mode toward Cost for that workload captures savings at no quality cost. When a model shows notably worse quality on a task type, that model should be excluded from the allowed pool for that workload.

Analysis 4: Fallback pattern analysis.

Aggregate fallback_triggered and fallback_reason by attempted_primary_model. This builds a reliability profile for each model in the pool.

Thresholds worth alerting on: fallback rate above 5 percent on any primary model warrants investigation. Above 20 percent, the primary model is effectively unavailable and should be replaced in the pool. Sustained 429-triggered fallback above 10 percent indicates a quota mismatch.

Instrumentation: Where the Metadata Goes

OpenTelemetry GenAI semantic conventions as the foundation.

The GenAI semantic conventions define standard span attributes for LLM calls: gen_ai.request.model, gen_ai.usage.input_tokens, and gen_ai.usage.output_tokens. Recording these as span attributes puts LLM telemetry into the same observability pipeline as the rest of the application.

Routing metadata extends this. The recommended pattern: record gen_ai.request.model as the model that actually ran (from selected_model), and add custom attributes for the routing-specific fields: the attempted primary, the task type, the confidence, the mode, and the fallback status.

Prompt and completion as span events, not attributes.

When prompt and completion content is large, recording it as span attributes bloats the trace. The convention is to record content as span events rather than attributes, which keeps the attribute set small and queryable while preserving the content for debugging.

PII scrubbing inside the instrumentation wrapper.

Prompt content frequently contains personal data. The correct pattern is a wrapper function that strips PII before writing span events, rather than relying on individual developers to remember sanitization at each call site. This makes the sanitization a property of the instrumentation layer rather than a discipline requirement.

Sampling strategy.

100 percent sampling in development and staging. In production, 10 to 30 percent head-based sampling at the trace level is typical for cost management, with tail-based sampling triggered on errors or high latency so that the traces that matter most are always captured.

A routing-specific addition: sample 100 percent of requests where fallback_triggered is true, regardless of the head-based sampling rate. Fallback events are rare and high-information, and losing 70 to 90 percent of them to sampling defeats the purpose of logging them.

Cost Attribution Under Routing

Cost attribution is where routing metadata has the most immediate business value.

The three-level attribution model.

Request level: cost per individual request, computed from the selected model's rate and the token counts. This is the atomic unit and the foundation for everything above it.

Workload level: cost aggregated by application, feature, or team. Requires tagging each request with a workload identifier that is preserved alongside the routing metadata.

Customer level: for products that pass AI costs through to customers or attribute them to business units, cost aggregated by end customer or tenant. Requires a customer identifier on each request.

What routing changes about budgeting.

Without routing, a budget forecast is token volume times a known rate. With routing, it is token volume times a weighted average rate that depends on the routing distribution, which is itself a function of the incoming task type mix.

This makes the forecast more accurate in one sense and less stable in another: routing produces lower average cost, but a shift in the task type distribution changes the average without any change in configuration.

The practical approach: forecast using the observed weighted average rate from the trailing period, and alert when the routing distribution deviates materially from the distribution the forecast assumed.

Budget alert thresholds.

Set alerts at 50, 80, and 100 percent of the period budget. The 50 percent alert early in a period is the signal that the run rate has changed, which with routing usually means the distribution shifted toward more expensive models.

Configure budgets at the team, workspace, or API key level rather than globally, so that a single workload's cost spike is attributable and containable rather than appearing as an undifferentiated organizational overrun.

Diagnosing Three Common Routing Incidents

Incident 1: Cost increased without a traffic increase.

Query: routing distribution for the current period compared to the prior period, and cost per request by model.

If the distribution shifted toward expensive models: check whether the task type distribution also shifted (your traffic changed) or whether the mode configuration changed. If neither, check whether a cheaper model became unavailable and traffic redistributed to more expensive alternatives.

If the distribution is unchanged: the cause is token volume per request, not routing. Check for prompt growth or output length increase.

Incident 2: Quality complaints concentrated in one feature.

Query: quality scores segmented by selected_model and primary_task_type, filtered to the affected feature's requests.

If one model shows notably worse quality on the relevant task type: the routing decision is selecting poorly for this category, and that model should be removed from the allowed pool for this workload or the mode adjusted.

If quality is uniformly worse across all models for this task type: the problem is not routing. Check the prompt, the input distribution, or whether this task type is at the boundary of what the model pool handles well.

Incident 3: Latency p99 increased.

Query: routing decision latency, and time to first token segmented by selected model, and fallback rate.

If routing decision latency increased: the routing layer is the cause.

If TTFT increased for one model: that model's provider is degraded.

If fallback rate increased: requests are failing on the primary and retrying on the fallback, which adds the failed attempt's latency to the total. This is the most common cause of a p99 increase that does not appear in any single model's latency profile, because the affected requests paid two inference attempts.

Governance and Audit Under Automatic Routing

Routing metadata serves a compliance function in addition to an operational one.

Proving which models processed which data.

For organizations with an approved model list, compliance may require demonstrating that only approved models processed the organization's data. The selected_model field in the request log is that evidence.

This requires that the log be retained for the applicable period and that it be queryable by time range and by data classification. For organizations under frameworks with audit log requirements, the routing metadata log is part of the audit trail.

Verifying that governance constraints held under failure.

The fallback_model field confirms that fallback candidates were within the allowed pool. This matters specifically because failure conditions are when governance is most likely to break: a routing system that selects the most available model during an incident, rather than the most compliant one, produces a governance violation precisely when the system is already degraded.

GMI Router selects both primary and fallback models from the allowed pool during recommendation rather than at failure time, and the metadata confirms which candidates were selected. As covered in GMI Cloud's article on primary and fallback models, pre-selecting fallbacks from the approved pool is what keeps governance intact under failure conditions.

Retention.

Routing metadata is small: a handful of fields per request. Retaining it longer than full prompt and completion content is both cheap and useful, because the routing analyses above benefit from long time series while content logs do not.

A practical split: retain routing metadata and token counts for the full audit retention period, and retain prompt and completion content for a shorter operational window with sampling.

The Minimum Viable Routing Observability Setup

For teams enabling routing on an existing production deployment, five instrumentation additions cover the essential analyses.

1. Log selected_model on every request and join it to existing token count and cost data. This alone enables cost attribution and per-model quality analysis.

2. Log fallback_triggered and fallback_reason, and alert on fallback rate above 5 percent for any model. This converts an invisible failure mode into a monitored one.

3. Log primary_task_type and build the task type distribution. This reveals what your application actually does and identifies the expensive categories.

4. Measure routing decision latency separately from model inference latency, so that a latency regression can be attributed to the correct layer.

5. Sample fallback events at 100 percent regardless of the general sampling rate, because they are rare and high-information.

These five additions take a few hours to implement against an API that already returns the metadata, and they cover the analyses that matter for the first months of a routing deployment. The more sophisticated analyses (quality delta by model within task type, cost per task type, routing accuracy evaluation) build on the same fields once the baseline is collecting.

For teams planning the transition to routing, GMI Cloud's production playbook for migrating from a fixed model covers the shadow routing phase where this metadata provides the comparison data that makes the migration decision quantitative.

Conclusion

Automatic routing makes the model a per-request variable, and every metric that was previously interpretable as a scalar becomes a distribution that requires a model dimension to read correctly. Cost is a weighted average across the routing distribution. Quality is a set of per-model baselines. Latency has a routing component and an inference component. Error rate has a routing layer and an inference layer, plus a category of handled failures that are invisible without explicit metadata.

The five fields that matter most: the model that ran, the model that was attempted first, whether fallback triggered and why, the detected task type, and the routing decision latency. Those five make the four core analyses possible: routing distribution over time, cost per task type, quality delta between models on the same task type, and fallback pattern analysis by model.

GMI Router returns this metadata in the API response, which means the instrumentation work is logging fields that are already present rather than building a separate measurement layer.

Try GMI Router

FAQs

What routing metadata fields are essential to log when using automatic model selection? Five fields cover the core analyses. The selected model, which makes every other metric segmentable by model and enables cost attribution. Whether fallback triggered and the fallback reason, which converts a handled failure from invisible to monitored. The detected task type, which reveals the actual composition of your traffic and identifies expensive categories. The attempted primary model, which combined with the selected model builds a reliability profile per model. Routing decision latency measured separately from inference latency, so a latency regression can be attributed to the correct layer.

Why are fallback events invisible without explicit routing metadata? A request that fails on the primary model and succeeds on the fallback returns a normal, successful response to the application. From the application's perspective, nothing went wrong: it sent a request and received a valid completion. The error occurred inside the routing layer, was handled, and left no trace in standard success and error rate metrics. Without a fallback_triggered flag in the response, the pattern of primary model degradation accumulates undetected until it worsens enough to cause visible failures. Sampling fallback events at 100 percent, regardless of the general sampling rate, is important because they are rare and high-information.

How does cost attribution change when routing selects models automatically? Without routing, cost per request is token count times a known rate. With routing, it is token count times a rate that varies per request depending on which model was selected, and the aggregate is a weighted average across the routing distribution. This means a cost increase has three possible causes that are indistinguishable without per-model breakdown: token volume grew, the routing distribution shifted toward more expensive models, or model pricing changed. The required instrumentation is joining the selected model field to token counts at the request level, then aggregating by workload and by customer for showback and budgeting.

How should routing metadata integrate with OpenTelemetry-based observability? The GenAI semantic conventions provide the foundation with gen_ai.request.model, gen_ai.usage.input_tokens, and gen_ai.usage.output_tokens as span attributes. Record the model that actually ran in gen_ai.request.model, and add custom span attributes for the routing-specific fields: attempted primary model, detected task type, task confidence, selected mode, and fallback status. Record prompt and completion content as span events rather than attributes when content is large, which keeps the attribute set small and queryable. PII scrubbing belongs inside the instrumentation wrapper rather than at individual call sites.

What does routing metadata provide for compliance and audit purposes? Two things. First, evidence of which models processed which data: for organizations with an approved model list, the selected model field in the retained request log demonstrates that only approved models handled the organization's data, which is the record an audit requires. Second, verification that governance held under failure: the fallback model field confirms that fallback candidates were within the allowed pool, which matters because failure conditions are when governance is most likely to break. A routing system that selects the most available model during an incident rather than the most compliant one produces a violation precisely when the system is already degraded. Routing metadata is small enough that retaining it for the full audit period is inexpensive, even when prompt and completion content is retained for a shorter operational window.