Courses·Evaluation·7 min read

LLM Benchmarks

LLM benchmarks are standardized test suites designed to measure specific capabilities of language models, forming the primary (if imperfect) basis for comparing models across the industry.

One-Line Summary: LLM benchmarks are standardized test suites designed to measure specific capabilities of language models, forming the primary (if imperfect) basis for comparing models across the industry.

Prerequisites: Understanding of what LLMs are and how they generate text, basic familiarity with few-shot prompting, awareness that models can be evaluated on specific tasks.

What Are LLM Benchmarks?

Think of LLM benchmarks like standardized tests for humans -- the SAT, GRE, or bar exam. Each test measures a different slice of ability. No single test captures everything a person can do, but taken together they paint a useful (if incomplete) picture. LLM benchmarks work the same way: each one probes a specific capability, and researchers combine scores across many benchmarks to form an overall assessment.

flowchart TD
    R1["LLM benchmark taxonomy: knowledge (MMLU)"]
    C2["knowledge (MMLU)"]
    R1 --> C2
    C3["reasoning (GSM8K, MATH)"]
    R1 --> C3
    C4["coding (HumanEval)"]
    R1 --> C4
    C5["safety (TruthfulQA)"]
    R1 --> C5

Just as with human standardized tests, there are serious problems. Students can study to the test. Test designers can make questions too easy or too hard. Cultural biases creep in. And eventually, top performers all score near-perfectly, making the test useless for distinguishing among the best. Every one of these problems has an exact analogue in the LLM benchmark world.

How It Works

flowchart LR
    S1["Benchmark saturation chart"]
    S2["how models have approached ceiling on olde"]
    S1 --> S2

The Major Benchmarks

MMLU (Massive Multitask Language Understanding): A collection of 15,908 multiple-choice questions spanning 57 subjects from elementary mathematics to professional law and medicine. It measures breadth of knowledge. Models are given a question and four answer choices and must select the correct one. Originally published in 2021, top models now score above 90%, leading to the creation of MMLU-Pro with harder, 10-option questions.

HellaSwag (Harder Endings, Longer contexts, and Low-shot Activities for Situations With Adversarial Generations): Tests commonsense reasoning by asking models to complete a scenario from four options. The incorrect options were generated by an earlier language model and filtered to be adversarially misleading. It probes whether a model understands how everyday situations typically unfold. Top models now exceed 95%.

HumanEval: A code generation benchmark consisting of 164 Python programming problems. Each problem includes a function signature, docstring, and a set of unit tests. The model must generate a working function body. Evaluated using "pass@k" -- the probability that at least one of k generated samples passes all tests. Extended versions include HumanEval+ with stricter tests and MultiPL-E for non-Python languages.

GSM8K (Grade School Math 8K): Contains 8,500 grade-school-level math word problems requiring multi-step arithmetic reasoning. The problems are linguistically diverse but mathematically straightforward (addition, subtraction, multiplication, division). This benchmark specifically tests chain-of-thought reasoning ability. Top models now solve over 95% of problems, prompting the creation of harder alternatives.

TruthfulQA: Measures whether a model generates truthful answers to questions where humans commonly hold misconceptions. It contains 817 questions across 38 categories including health, law, finance, and politics. A model scores well not by being knowledgeable but by avoiding confidently stated falsehoods. This benchmark specifically targets the tendency of LLMs to reproduce popular misconceptions from training data.

MATH: A dataset of 12,500 competition-level mathematics problems from AMC, AIME, and similar competitions, spanning seven subjects (algebra, geometry, number theory, etc.) at five difficulty levels. Unlike GSM8K, these problems require genuine mathematical insight. Performance has improved dramatically with chain-of-thought prompting and tool use.

ARC (AI2 Reasoning Challenge): A set of 7,787 science exam questions from grade 3 through grade 9. The "Challenge" subset contains only questions that simple retrieval and co-occurrence methods fail on, ensuring that solving them requires actual reasoning rather than pattern matching.

MT-Bench: Evaluates conversational ability through 80 multi-turn questions across eight categories (writing, roleplay, reasoning, math, coding, extraction, STEM, humanities). Responses are scored by GPT-4 on a scale of 1-10. This is one of the first benchmarks designed specifically for chat-tuned models rather than base models.

Chatbot Arena (LMSYS): A live evaluation platform where users submit prompts and vote on which of two anonymous models gives a better response. Votes are aggregated into ELO ratings (the same system used in chess). This is widely considered the most ecologically valid benchmark because it uses real user queries and preferences rather than pre-designed test questions.

How Evaluation Works

Most benchmarks use one of these evaluation protocols:

Multiple choice: The model selects from predefined options. Scoring is exact match. Used by MMLU, HellaSwag, ARC.
Open generation with automated checking: The model generates free-form output that is checked against reference answers or unit tests. Used by HumanEval, GSM8K, MATH.
LLM-as-judge: A strong model (typically GPT-4) evaluates the quality of responses. Used by MT-Bench, AlpacaEval.
Human preference: Real users choose between outputs. Used by Chatbot Arena.

Few-shot prompting is standard: models are given several examples of correctly answered questions before being asked to answer new ones. The number of shots (0-shot, 5-shot, etc.) significantly affects scores and must be reported for results to be meaningful.

Why It Matters

Benchmarks serve as the common language of the LLM field. When a company announces a new model, benchmark scores are the primary evidence they present. When researchers propose a new training technique, benchmark improvements are the proof it works. Without standardized benchmarks, every claim about model quality would be anecdotal.

However, the benchmark ecosystem faces several serious challenges:

Benchmark saturation: When top models all score above 95% on a benchmark, it can no longer distinguish between them. MMLU, HellaSwag, and GSM8K are all approaching this ceiling. The field responds by creating harder versions (MMLU-Pro, MATH-500, GSM-Hard), but this is an ongoing arms race.

Leaderboard gaming: Organizations have strong incentives to optimize specifically for benchmark performance. This can range from legitimate practices (training on similar data distributions) to problematic ones (training directly on benchmark test sets). The line between "teaching to the test" and "contamination" is often blurry.

Narrow measurement: Each benchmark measures a thin slice of capability. A model that excels on MMLU might fail catastrophically at following nuanced instructions. A model that aces HumanEval might write insecure code. Real-world usefulness is far more complex than any benchmark suite captures.

Key Technical Details

Benchmark scores are sensitive to prompt format. The same model can score significantly differently depending on how questions are presented (e.g., "Answer: " vs "The answer is" as the completion prefix).
Few-shot count matters enormously. Zero-shot MMLU scores can be 10-15 points lower than 5-shot scores for the same model.
Evaluation harnesses like EleutherAI's lm-evaluation-harness and Stanford's HELM attempt to standardize evaluation protocols, but implementations still vary across organizations.
Contamination detection is non-trivial. Checking for exact string matches is insufficient because paraphrased or reformatted test questions can still leak information.
Aggregate scores (like the "Open LLM Leaderboard" average) obscure important differences in capability profiles between models.

Common Misconceptions

"Higher benchmark scores mean a better model." Better at what? A model scoring 2 points higher on MMLU but 10 points lower on HumanEval is not uniformly better. The right model depends entirely on the use case.
"Benchmarks measure intelligence." Benchmarks measure performance on specific, narrow tasks under specific evaluation protocols. They do not measure general intelligence, creativity, or real-world usefulness.
"Published benchmark scores are always comparable." Different evaluation harnesses, prompt templates, few-shot counts, and even random seeds can produce different scores for the same model on the same benchmark. Always check the methodology.
"New models beating old benchmarks means AI is getting smarter." It might also mean training data is getting more contaminated, benchmarks are getting stale, or evaluation protocols are being gamed.

Connections to Other Concepts

perplexity.md: An intrinsic metric that complements extrinsic benchmark evaluation. Low perplexity is necessary but not sufficient for good benchmark performance.
llm-as-judge.md: The evaluation method used by MT-Bench, AlpacaEval, and other modern benchmarks that go beyond multiple choice.
benchmark-contamination-detection.md: The central threat to benchmark validity, where test data leaks into training sets.
rlhf.md: The training technique most directly targeted at improving performance on human-preference benchmarks like Chatbot Arena.
chain-of-thought-prompting.md: The prompting technique that unlocked dramatic improvements on reasoning benchmarks like GSM8K and MATH.
scaling-laws.md: Predict benchmark performance as a function of model size and training compute.