How Do Different Large Language Models Perform in Benchmarks?

GMI Cloud Blog | AI Infrastructure Guide | gmicloud.ai

On current benchmarks, GPT-4o and Claude 3.5 Opus lead on general reasoning (MMLU 87-90%), DeepSeek-R1 and OpenAI o3 lead on mathematical reasoning (MATH 90%+), Code Llama and DeepSeek Coder lead among open-source code models (HumanEval 70-85%), and Llama 3 405B is the top open-source general-purpose model (MMLU ~86%).

But benchmark scores tell an incomplete story. This guide covers the major benchmarks, current standings, their limitations, and how to interpret the numbers for real-world decisions.

For GPU infrastructure to run your own benchmarks, providers like GMI Cloud offer H100/H200 instances alongside a 100+ model library.

The Major LLM Benchmarks

Six benchmarks define how the industry evaluates LLM performance. Each measures something different.

MMLU (Massive Multitask Language Understanding)

Tests broad knowledge across 57 academic subjects: biology, history, law, medicine, math, and more. Scores represent accuracy on multiple-choice questions. It's the most widely cited general-capability benchmark.

What it measures: Breadth of factual knowledge and reasoning across domains. What it misses: Creative writing quality, instruction following, and real-world task completion.

HumanEval

Tests code generation by asking the model to write Python functions that pass unit tests. Scores represent the percentage of problems solved correctly (pass@1).

What it measures: Functional code generation ability. What it misses: Code readability, architecture design, debugging skills, and performance in languages other than Python.

MATH

Tests mathematical problem-solving with competition-level math questions. Problems range from algebra to number theory to geometry.

What it measures: Mathematical reasoning and multi-step problem solving. What it misses: Applied math in real contexts, statistical reasoning, and mathematical modeling.

MT-Bench

Tests multi-turn conversation quality across 8 categories (writing, roleplay, reasoning, math, coding, extraction, STEM, humanities). Scores are rated by GPT-4 as a judge on a 1-10 scale.

What it measures: Conversational quality and instruction following across turns. What it misses: Uses an LLM as the judge, which introduces bias toward models that sound similar to the judge.

GPQA (Graduate-Level Google-Proof Q&A)

Tests expert-level reasoning with questions written by domain PhD holders. Questions are designed to be unsearchable, requiring genuine reasoning rather than retrieval.

What it measures: Deep expert-level reasoning. What it misses: Narrow question set. Performance may not generalize to other expert tasks.

Chatbot Arena ELO

Humans chat with two anonymous models side-by-side and vote for the better response. ELO ratings are calculated from thousands of human preference votes.

What it measures: Real human preference in open-ended conversation. What it misses: Voters may prefer confident-sounding responses over accurate ones. Voting population skews toward tech-savvy English speakers.

With these benchmarks defined, here's how the leading models compare across all six.

Current LLM Benchmark Standings

GPT-4o

• MMLU: ~88%
• HumanEval: ~90%
• MATH: ~76%
• MT-Bench: 9.0+
• GPQA: ~53%
• Arena ELO: Top 3

Claude 3.5 Opus

• MMLU: ~89%
• HumanEval: ~92%
• MATH: ~78%
• MT-Bench: 9.0+
• GPQA: ~60%
• Arena ELO: Top 3

Gemini 2.5 Pro

• MMLU: ~87%
• HumanEval: ~85%
• MATH: ~83%
• MT-Bench: 8.8+
• GPQA: ~59%
• Arena ELO: Top 5

o3 (OpenAI)

• MMLU: ~92%
• HumanEval: ~92%
• MATH: ~96%
• MT-Bench: N/A
• GPQA: ~80%+
• Arena ELO: N/A

DeepSeek-R1

• MMLU: ~90%
• HumanEval: ~87%
• MATH: ~97%
• MT-Bench: 8.5+
• GPQA: ~72%
• Arena ELO: Top 10

Llama 3 405B

• MMLU: ~86%
• HumanEval: ~81%
• MATH: ~73%
• MT-Bench: 8.5+
• GPQA: ~48%
• Arena ELO: Top 15

Llama 3 70B

• MMLU: ~82%
• HumanEval: ~73%
• MATH: ~68%
• MT-Bench: 8.2+
• GPQA: ~40%
• Arena ELO: Top 20

Scores are approximate and based on publicly reported results as of early 2025. Exact numbers vary by evaluation methodology, prompting strategy, and model version. Sources: model technical reports, lmsys.org/chatbot-arena, open benchmark leaderboards.

Key takeaways from the table: Reasoning models (o3, DeepSeek-R1) dominate MATH and GPQA but may not lead on conversational benchmarks. Claude and GPT-4o are closely matched on most general benchmarks. Llama 3 405B closes the gap with closed-source models but doesn't match them on the hardest reasoning tasks.

These numbers are useful but have important limitations.

Why Benchmarks Tell an Incomplete Story

Four limitations mean you should never choose a model based on benchmark scores alone.

Data Contamination

Models may have been exposed to benchmark questions during training. If the model has "seen" MMLU questions, its score reflects memorization rather than reasoning. This is difficult to detect and makes cross-model comparisons unreliable.

Benchmark Saturation

When multiple models score 85-90% on MMLU, the remaining differences may be within noise. A 2% score difference is unlikely to be meaningful in practice. The benchmark has stopped discriminating between top models.

Benchmarks Don't Measure Production Metrics

No benchmark measures inference latency, cost per request, reliability under load, or how the model handles your specific domain. A model that scores 2% higher on MMLU but costs 5x more per request isn't the better choice for most production applications.

Self-Reported Scores Are Selective

Model providers choose which benchmarks to highlight. They'll publish the scores where they lead and omit the ones where they lag. Always look for independent evaluations (Chatbot Arena, third-party benchmark runs) alongside provider-reported numbers.

Given these limitations, here's how to use benchmarks correctly.

How to Interpret Benchmarks

Use benchmarks for shortlisting, not final decisions. Narrow your candidates to 2-3 models based on benchmark performance, then evaluate on your actual task.

Match the benchmark to your use case. If you're building a coding tool, HumanEval matters more than MMLU. If you're building a chatbot, Arena ELO and MT-Bench matter more than MATH.

Prefer human evaluation over automated scores. Arena ELO reflects real user preference. Automated benchmarks reflect test performance. When they disagree, human preference is usually more predictive of real-world satisfaction.

Always validate on your own data. Run your top 2-3 candidates on a representative sample of your actual inputs. Your task-specific performance may differ significantly from benchmark rankings.

For running your own evaluation, here's what you need.

Running Your Own Benchmarks

To evaluate models on your specific workload, you need either API access (for closed-source models) or GPU infrastructure (for open-weight models you self-host).

For self-hosted benchmarking, H100 (80 GB) handles 70B models at FP8. H200 (141 GB) accommodates larger models or enables higher-concurrency testing. Per NVIDIA's H200 Product Brief (2024), the H200 delivers up to 1.9x inference speedup on Llama 2 70B vs. H100 (TensorRT-LLM, FP8, batch 64, 128/2048 tokens).

Cloud platforms like GMI Cloud offer GPU instances (H100 ~$2.10/GPU-hour, H200 ~$2.50/GPU-hour; check gmicloud.ai/pricing for current rates) for running your own evaluation suite, plus a model library for API-based testing across image, video, and audio models.

FAQ

Which single benchmark is most useful?

Chatbot Arena ELO, because it reflects real human preference in open-ended tasks. But no single benchmark is sufficient. Use at least 2-3 benchmarks relevant to your use case, plus your own task-specific evaluation.

Why do reasoning models (o3, DeepSeek-R1) score so much higher on MATH?

They use inference-time compute (chain-of-thought, tree search) to spend more computation per problem. This dramatically improves accuracy on mathematical reasoning but increases latency and cost per request.

Are benchmark scores comparable across different model sizes?

Yes, but context matters. Llama 3 70B scoring 82% on MMLU while GPT-4o scores 88% doesn't mean GPT-4o is categorically better. It means GPT-4o has a 6-point edge on broad knowledge, while costing significantly more per request. The 70B model may be the better production choice at sufficient quality.

How often do benchmark rankings change?

The top positions shift every 2-4 months as new models release. However, the same names (OpenAI, Anthropic, Google, Meta, DeepSeek) have dominated the top 10 for the past year. Dramatic ranking changes are becoming less common as the field matures.

Tab 43