LLM Model Comparison: How to Run a Fair Head-to-Head Evaluation

Most model comparisons produce a number that cannot support the decision it is used to make. A team runs both candidates on 40 prompts, one scores 4.2 and the other 4.0, and the 4.2 model ships. Nobody computed whether a 0.2 gap on 40 samples is distinguishable from noise, whether both models ran under the same harness and sampling parameters, or whether the judge that produced those scores was systematically favouring longer responses. The comparison felt rigorous because it involved numbers. The statistical answer, had anyone computed it, would almost certainly have been that the two models are tied.

• Harness parity is the precondition for any valid comparison. Different evaluation harnesses shift coding benchmark scores by 10 to 26 points on the same model. A comparison where each model ran under its vendor’s preferred harness measures harness differences, not model differences.

• Pairwise comparison beats absolute scoring for subjective quality. Inter-rater reliability on absolute 1-to-5 helpfulness scales sits around 0.45 to 0.60 on public datasets. Judges are substantially more consistent answering “which of these two is better” than “rate this one from 1 to 5.”

• Naive pairwise can be worse than pointwise. Research on the comparative trap shows pairwise comparison amplifies biased preferences in LLM evaluators: on several models, pairwise with chain-of-thought scored materially lower than pointwise scoring on evaluator accuracy. The fix is generating independent per-response reasoning before the comparative decision.

• Overlapping confidence intervals mean no demonstrated gap. This is the single most important leaderboard-reading skill and the one most often skipped. Many hundreds to thousands of comparisons are typical for separating close models.

• GMI Cloud provides unified API access to 170-plus models, which removes the most common source of harness asymmetry: comparing models through different provider SDKs with different default parameters.

• Comparison must include cost and latency, not only quality. A model that wins on quality by 2 percentage points at three times the cost and twice the latency loses the production decision, and a quality-only comparison never surfaces that.

The Four Sources of Unfair Comparison

Before any statistics, four asymmetries can invalidate a head-to-head result. Each is straightforward to eliminate and each is routinely missed.

Harness asymmetry. The evaluation harness determines how the prompt is constructed, how tools are exposed, how many turns are permitted, how retries are handled, and how the final answer is extracted from the model’s output. These choices move scores substantially. On coding benchmarks specifically, harness differences alone produce score swings of 10 to 26 points on the same model.

Vendor-published benchmark tables frequently mix harnesses: one model evaluated under the vendor’s own agent framework, competitors evaluated under a third-party framework. The resulting table is not a model comparison.

The fix is running every candidate under the same harness. EleutherAI’s lm-evaluation-harness is the de facto standard for academic benchmarks and powers the Hugging Face Open LLM Leaderboard. For agentic and application-specific evaluation, a custom harness is usually necessary, and the requirement is that it is identical across candidates.

Prompt asymmetry. A prompt tuned against one model’s quirks disadvantages the other. Instructions added to suppress a known verbosity problem, few-shot examples calibrated to one model’s natural output style, and formatting directives that compensate for a specific weakness all transfer poorly.

The fix is evaluating on prompts that express what the task requires without encoding one model’s idiosyncrasies. Where a model genuinely needs a different prompt to perform well, run both prompt variants against both models and report all four cells, rather than pairing each model with its own optimised prompt and comparing the diagonal.

Sampling parameter asymmetry. Temperature, top-p, top-k, repetition penalty, and maximum output length all affect output quality. Comparing a model at temperature 0.2 against another at temperature 0.9 measures the temperature difference as much as the model difference.

The fix is identical sampling parameters across candidates, or, when a model genuinely requires different settings, a sweep across a small parameter grid for both models with the best configuration of each reported.

Operating point asymmetry. For any model or system with a tunable threshold, comparing at each system’s own preferred operating point is not a comparison. Security benchmark methodology in 2026 handles this explicitly: for every external comparison, the detector’s threshold is re-tuned to the competitor’s published false-positive rate, so head-to-head values are evaluated at matched operating points.

The same principle generalises. When comparing models on tasks with a precision-recall tradeoff, or comparing configurations with a latency-quality tradeoff, match the operating point on one axis and compare on the other.

Pointwise Scoring Versus Pairwise Comparison

Two evaluation structures produce different reliability properties, and the choice affects how many samples the comparison needs.

Pointwise, or rubric scoring, asks the judge to grade each response in isolation: how good is this response on a 1-to-5 scale for faithfulness, completeness, and tone. Every response is graded independently.

The weakness is that the judge must hold an absolute standard across runs, across models, and across months. Inter-rater reliability on absolute 1-to-5 helpfulness sits around 0.45 to 0.60 on most public datasets. Judge models inherit that human variance and add their own drift over time.

The strength is diagnostic value. Rubric scores tell you which dimension is weak, which is what you need when the decision is “how do I improve this” rather than “which of these two ships.”

Pairwise, or arena-style comparison, shows the judge two responses to the same input and asks which is better. The judge performs the comparison directly rather than approximating it through two absolute judgements.

The strength is reliability. Both humans and judge models are substantially more consistent answering “B is better than A” than assigning “A is a 7.2.” Aggregate win rate becomes a sharper ship signal than aggregate rubric score, especially for subjective qualities that a rubric blurs.

The practical recommendation from teams running both: rubrics give you the absolute number and the diagnostic axis, arena gives you the decision. Run both when the budget allows, and prioritise pairwise when the question is purely which model to ship.

The comparative trap. Pairwise is not automatically better, and this nuance is worth understanding before adopting it wholesale. Research on the comparative trap demonstrates that pairwise comparison amplifies biased preferences in LLM evaluators. On several models tested, pairwise with chain-of-thought produced materially lower evaluator accuracy than pointwise scoring: one weaker judge scored 52.35 on pointwise and 36.05 on pairwise with chain-of-thought.

The mechanism is that presenting two responses together invites the judge to rely on surface comparisons rather than substantive evaluation. The mitigation demonstrated in that research generates independent reasoning about each response before making the comparative decision, which restored and exceeded pointwise accuracy across the models tested.

The practical implication: a pairwise judge prompt should ask for per-response analysis first and the verdict second, not the verdict with a post-hoc justification.

Controlling Judge Bias

Four biases systematically corrupt LLM-as-judge comparisons. Each has a specific control.

Position bias. Judges favour one position, typically the first response presented. The control is presenting each pair twice with the order swapped and counting a result only when both orderings agree. Disagreement between orderings is recorded as a tie, which is the honest interpretation: the judge could not distinguish the responses independently of their position.

Length bias. Judges favour longer responses, which correlates with perceived thoroughness but not necessarily with quality. Two controls: instruct the judge explicitly that length is not a quality signal, and record response length alongside the verdict so that a systematic length-verdict correlation is detectable after the fact.

Self-preference bias. A judge model favours outputs from its own model family. The control is using a judge from a different family than either candidate, or running multiple judges from different families and taking a majority vote.

Style bias. Judges favour confident, well-formatted prose over hedged or plainly formatted responses of equivalent substance. This is the hardest to control because style and quality genuinely correlate. Partial mitigation: include in the rubric an explicit instruction that formatting and confidence are not quality signals, and sample verdicts for human review to check whether the judge is rewarding presentation over substance.

The robust protocol. Research practice for head-to-head evaluation without gold-standard references uses layered voting. Generate answers to the same query from both candidates. Randomise order. Present the pair to five independent judge models, each assessing the same pair three times. Apply majority voting within each judge to resolve that judge’s own variance, then a second majority vote across judges to produce the final verdict.

This is expensive at fifteen judge calls per comparison. A practical production adaptation runs three judges twice each with order randomised, which retains most of the variance reduction at 40 percent of the cost.

The Statistics Everyone Skips

This is the part that determines whether the comparison supports the decision, and it is routinely omitted.

Overlapping confidence intervals mean no demonstrated gap. If candidate A scores 84.2 percent with a confidence interval of 81.1 to 87.3, and candidate B scores 82.5 percent with an interval of 79.4 to 85.6, the models are statistically tied. The 1.7-point raw gap is not evidence of a difference. Reading a leaderboard without reading the intervals produces confident conclusions from indistinguishable numbers.

Bootstrap the intervals. Resample the evaluation set with replacement, recompute the score on each resample, and take the empirical distribution of results as the confidence interval. This works for accuracy, for win rate, and for Elo or Bradley-Terry ratings derived from pairwise comparisons.

Use paired significance tests for A-versus-B. Both models run on the same items, which means the comparison is paired rather than independent. Paired tests are substantially more powerful than unpaired ones because they remove item difficulty as a source of variance. For binary outcomes, McNemar’s test. For continuous scores, the paired t-test or Wilcoxon signed-rank. Permutation tests work for both and make fewer distributional assumptions.

The small-sample warning. Confidence intervals computed from the central limit theorem dramatically underestimate uncertainty on specialised benchmarks with fewer than a few hundred datapoints. This finding, published at ICML 2025, matters because most practical evaluation sets are 50 to 200 items, which is exactly the regime where CLT-based intervals mislead. Bootstrap and Bayesian alternatives are the recommended substitutes at these sizes.

Size the test set before running it. Power analysis answers how many items are needed to detect a difference of a given size at a given confidence level. Running it first prevents the common outcome of spending the evaluation budget and discovering the result is inconclusive.

For pairwise comparison specifically, many hundreds to thousands of comparisons are typical for separating close models. If the models are genuinely close, a 50-comparison evaluation will not distinguish them regardless of how carefully it is run.

Where the noise lives. Variance decomposition identifies whether evaluation noise comes from item selection, from sampling seeds, or from judge variance. This matters because the remedy differs: item variance is reduced by a larger or better-stratified test set, seed variance by averaging across multiple generations per item, and judge variance by multiple judges or repeated judging.

Getting more signal per item. Item response theory methods extract more information from each evaluation item by modelling item difficulty alongside model ability. The tinyBenchmarks work demonstrated that MMLU rankings are recoverable from roughly 100 curated items rather than the full 14,000, which is a 140-fold reduction in evaluation cost for the same ranking conclusion. More recent work on factorised active querying reports up to fivefold effective sample-size gains while preserving valid frequentist coverage.

Elo, Bradley-Terry, and Their Limits

Pairwise verdicts need to be converted into rankings, and the two standard methods have known limitations worth understanding before relying on them.

Elo was designed for chess, a domain with stable, well-defined outcomes and a single dimension of skill. LLM performance is multifaceted and context-dependent, which makes Elo a less natural fit. Published analysis has found Elo producing inconsistent and sometimes unreliable results when applied to language model rankings, particularly when the comparison set is small or the models are close.

Elo is also order-dependent: the rating depends on the sequence in which comparisons were processed, which means the same set of verdicts can produce different ratings depending on processing order.

Bradley-Terry fits a single latent strength parameter per model to the full set of pairwise outcomes simultaneously, which removes the order dependence. This is the method behind the LMSYS Chatbot Arena leaderboard, combined with bootstrap confidence intervals and model tier grouping rather than strict ranking.

The tier grouping practice is worth copying. Rather than presenting a strict rank order, group models whose confidence intervals overlap into tiers. This communicates the actual state of knowledge: these three models are indistinguishable at the current sample size, and this group is measurably better than that group.

Beyond Quality: The Comparison That Supports a Production Decision

A quality-only comparison answers the wrong question. The production decision is which model to deploy, which depends on quality, cost, and latency together.

The practical approach records four values per candidate per evaluation item: the quality score or pairwise verdict, the input and output token counts, the time to first token, and the total generation time. From these, the derived comparisons are cost per item at each model’s rate, latency at p50 and p95, and quality per unit cost.

The common outcome of adding these dimensions: the model that won on quality is not the model that ships. A candidate that wins 54 percent of pairwise comparisons at one third the cost and half the latency is the correct production choice over one that wins 46 percent at premium pricing, because the quality difference is within the confidence interval and the cost difference is not.

This is also the finding that motivates routing rather than selection. As covered in GMI Cloud’s guide to model routing, no single model leads across all task categories, which means the comparison result frequently differs by task type within the same application.

Segmenting by Task Type

An aggregate comparison across a mixed evaluation set hides the pattern that matters most.

Two models with identical aggregate win rates can have completely different profiles: one winning decisively on coding and losing on long-context retrieval, the other the reverse. The aggregate says they are tied. The segmented view says route coding to the first and retrieval to the second.

The practical requirement is tagging each evaluation item with its task type before running the comparison, then reporting win rate per task type alongside the aggregate. With 8 task categories and 200 evaluation items, each category has roughly 25 items, which is too few for a statistically confident per-category conclusion but sufficient to identify which categories warrant a focused follow-up comparison.

As covered in GMI Cloud’s guide to choosing the right model for each prompt, task-specific performance predicts production quality more reliably than aggregate rankings, which is the reason the segmented view is worth the additional tagging effort.

A Working Protocol

The following sequence produces a comparison that supports a production decision, at a cost most teams can absorb.

Step 1: Define the decision. Write down what the comparison will decide and what result would change the decision. A comparison run without a pre-specified decision rule tends to be interpreted in favour of whichever model the team already preferred.

Step 2: Build the evaluation set. 100 to 200 items sampled from production traffic or a realistic simulation of it, tagged by task type, with personal data redacted. Include the edge cases and the ambiguous inputs, not only the clean examples.

Step 3: Fix the harness. One harness, one prompt template, one set of sampling parameters, applied identically to both candidates. Record the configuration alongside the results so the comparison is reproducible.

Step 4: Generate. Run both candidates on every item. Record output, token counts, TTFT, total latency, and any errors. For items where output variance matters, generate three times per item per model and evaluate all generations.

Step 5: Judge. Pairwise with order randomisation, each pair evaluated in both orders, with per-response reasoning generated before the comparative verdict. Three judges from different model families if budget allows, one otherwise. Record disagreements as ties.

Step 6: Compute intervals. Bootstrap the win rate. Run a paired significance test. Report the interval, not only the point estimate.

Step 7: Segment. Win rate per task type alongside the aggregate. Identify categories where the models diverge.

Step 8: Add cost and latency. Cost per item at each model’s rate, p50 and p95 latency. Compare the full picture rather than the quality axis alone.

Step 9: Decide, or declare a tie. If the confidence intervals overlap and the cost and latency profiles are similar, the honest conclusion is that the models are equivalent for this workload and the decision should be made on other grounds: data residency, vendor relationship, or operational familiarity.

Running the Comparison on GMI Cloud

Two properties of the evaluation infrastructure affect comparison validity.

Unified API access removes provider asymmetry. Comparing models through different provider SDKs introduces differences in default parameters, retry behaviour, timeout handling, and response parsing that are invisible in the results but present in the numbers. GMI Cloud’s model library provides access to 170-plus models through a single OpenAI-compatible API, which means the same client code, the same parameter handling, and the same error semantics apply to every candidate.

Consistent hardware removes infrastructure asymmetry. Latency comparison is only meaningful when both models run on comparable hardware under comparable load. A candidate measured on a warm dedicated endpoint against one measured on a cold shared endpoint produces a latency comparison that reflects the infrastructure rather than the models.

For teams comparing self-hosted candidates, GMI Prime Inference provides reserved dedicated capacity with model weights pre-loaded, which means both candidates are measured warm on the same GPU class. For teams comparing managed endpoints, the relevant control is measuring both at the same time of day under the same platform load conditions.

Hourly billing makes thorough comparison affordable. A 200-item evaluation run three times per model across two candidates is 1,200 generations plus judge calls. GMI Cloud’s on-demand infrastructure bills hourly with no minimum commitment, which makes the full protocol above practical rather than something to cut short for budget reasons.

The broader methodology for fair infrastructure comparison is covered in GMI Cloud’s article on benchmarking AI inference providers fairly, which addresses the same class of asymmetry at the provider level rather than the model level.

Conclusion

A model comparison is valid when the two candidates were measured under identical conditions and the observed difference exceeds the measurement noise. Most published and internal comparisons satisfy neither condition: harnesses differ, prompts are tuned per model, sampling parameters vary, and no confidence interval is computed.

The corrections are mechanical. Fix the harness, the prompt, and the sampling parameters across candidates. Use pairwise comparison with order randomisation and per-response reasoning before the verdict. Bootstrap the confidence interval and run a paired significance test. Report overlapping intervals as a tie rather than reading a rank order into indistinguishable numbers. Segment by task type, because aggregate parity frequently hides per-category divergence that changes the deployment decision.

Add cost and latency to the comparison, because the production question is which model to deploy rather than which model scores higher, and those are different questions often with different answers.