Putting AI Benchmarks in Context: What They Show - and What They Don’t

Almost every major model release today comes with a benchmark table. Scores for coding, mathematical reasoning, scientific questions, or agentic tasks are meant to show where a new model stands relative to its predecessors and competitors. Such comparisons matter. Without standardized tests, technical progress would be much harder to measure in a consistent and traceable way.

Problems begin when a result from one specific test is turned into a general ranking. “Model A scores higher than Model B on benchmark X” quickly becomes “Model A is better.” A single benchmark usually does not support that conclusion.

The limits of even an established test became particularly clear in 2026 with SWE-bench Verified, a widely used benchmark for autonomous software engineering tasks. In February, OpenAI announced that it would no longer use the benchmark to evaluate its frontier models. When examining tasks that models frequently failed to solve, the company found both indications that test material had been present in training data and problems with the tasks themselves. These included incomplete requirements, tests that did not reliably recognize functionally correct solutions, and cases where a successful solution depended on implementation details that were not apparent from the task description. OpenAI documented the reasons for the decision in February 2026.

In July, OpenAI audited SWE-Bench Pro, which it had previously recommended as an alternative. Depending on the review method, the company classified 27.4% or 34.1% of the examined tasks as flawed. OpenAI subsequently withdrew its recommendation of SWE-Bench Pro as well.

This does not make SWE-bench retrospectively worthless. Rather, it illustrates something that is easy to overlook when focusing on benchmark scores: a benchmark is a measurement instrument with a defined task space, an evaluation procedure, and limits.

Put simply, an AI benchmark consists of tasks, a defined evaluation procedure, and a metric used to score the responses. It may measure, for example, the number of multiple-choice questions answered correctly, whether software bugs are successfully fixed, or performance in a simulated environment.

Abstract properties such as “intelligence,” “reasoning,” or “overall model quality,” by contrast, cannot be observed directly. To measure them, researchers first have to decide which concrete tasks should serve as proxies for the capability in question.

How difficult this step can be is illustrated by a 2025 NeurIPS study of 445 LLM benchmarks. Its focus is what researchers call construct validity: whether the tasks and metrics actually support reliable conclusions about the capability a benchmark claims to measure. The authors found recurring weaknesses in how such capabilities were defined, how they were operationalized in concrete tasks, and which conclusions were subsequently drawn from the results. The study therefore provides a methodological basis for asking what a benchmark score actually tells us.

A well-designed coding benchmark can certainly tell us something about coding capabilities. But what it tells us remains limited to the task space it covers. A model that answers academic multiple-choice questions very reliably is not automatically particularly good at extracting information from long company documents. And successfully fixing bugs in public GitHub repositories does not tell us how the same model will perform in a long-established internal codebase.

Before asking which model scores higher, there is another question to answer: what did this test actually measure?

Mathematical reasoning, coding, language understanding, tool use, and long-context processing place different demands on a model. They cannot meaningfully be tested with the same set of tasks. Current model reports accordingly contain a range of different evaluations.

Google DeepMind’s Gemini 3.1 Pro model card, for example, reports separate results for scientific knowledge, abstract reasoning, coding, multimodal tasks, agentic tool use, multilingual capabilities, and long-context tasks. A model may lead on one of these tests while trailing a competitor on another.

Any overall ranking inevitably obscures such differences. Which capabilities matter depends on what the model is ultimately expected to do.

Language itself can change the evaluation. Many established benchmarks were initially developed with a strong focus on English-language tasks. MMLU-ProX addresses precisely this issue by comparing challenging questions across 29 languages. In the study of 36 models, performance differs considerably between languages in some cases. MMLU-ProX therefore explicitly treats language as a separate evaluation dimension.

A high score on an English-language benchmark therefore does not establish how well the same model will handle German technical documents, company-specific abbreviations, or typical internal document structures.

A benchmark is supposed to make differences between models visible. It can do so as long as its tasks remain sufficiently difficult for the systems being tested and clearly distinguish weaker from stronger models.

As model performance improves, that gap can shrink. If more and more models solve a large share of the tasks correctly, only a small number of test cases remain on which differences become visible. The tasks themselves do not become invalid. But the benchmark becomes less useful for comparing top-performing models because their scores move increasingly close together. This effect is known as benchmark saturation.

A systematic study published at ICML 2026 analyzed 60 widely used LLM benchmarks and found signs of saturation in almost half of them. Older benchmarks were affected more strongly. The authors do not define saturation simply as reaching 100%, but in terms of a test’s declining ability to reliably differentiate between high-performing models. The study therefore treats benchmark saturation as a measurable loss of discriminative power.

Saturation needs to be distinguished from other problems. A benchmark can be methodologically sound and still become too easy for the current generation of models. Contamination or benchmark-specific optimization, by contrast, affect different parts of the evaluation process.

When scores approach 100%, it is tempting to say that a model has “solved” a benchmark. For a technical assessment, however, that wording is too imprecise.

A recent example is GPT-6 Astra on ARC-AGI-3. At the same “max” reasoning effort, Astra scores 62.7% using ARC Prize’s Standard harness and 98.6% using an OpenAI-provided Provider Adapter harness. Among other things, the Provider Adapter can preserve internal reasoning state between requests and use compaction for longer interactions. ARC Prize reports the two evaluation setups separately.

The 98.6% result tells us, first of all, that the system solved almost all of the benchmark’s tasks under that specific evaluation procedure. It does not tell us whether the underlying capability has been solved in a general sense. Nor can contamination or deliberate “gaming” of the benchmark be inferred from the score alone.

At the same time, the large gap between 62.7% and 98.6% illustrates how strongly evaluation conditions can affect the result. In more complex evaluations, factors such as tool access, agent loops, persistent state, reasoning budget, and the number of allowed attempts can all matter.

Current model cards therefore increasingly document such conditions explicitly. For Gemini 3.1 Pro, for example, the model card states whether tools were used, which harness was used, or whether only a single attempt was permitted. The same benchmark name therefore does not guarantee identical evaluation conditions.

Data contamination occurs when training and evaluation data overlap. A model may then already have encountered tasks, solutions, or very similar material during training that are later used to measure its capabilities.

With publicly available benchmarks, this is difficult to rule out completely. A 2025 EMNLP survey therefore describes contamination as a growing problem for static LLM evaluation and examines different approaches for making tests more robust against it.

SWE-bench shows how concrete this problem can become. Its tasks are based on public open-source repositories. In its analysis, OpenAI reported that the frontier models being tested were able, on individual tasks, to reproduce original human-written patches or specific wording. The company interpreted this as an indication that at least some of the material may have been encountered during training.

That is not the same as simple memorization. For interpreting the benchmark, the uncertainty itself is enough to matter: was the model truly solving an entirely unseen task, or did prior exposure to test material play a role?

Another problem arises when either the task or its automated scoring does not reliably determine whether a model has performed correctly.

For SWE-bench Verified, OpenAI examined 138 tasks on which one of its own models frequently failed even after many attempts. In this deliberately selected subset, the company found relevant problems in the tests or task descriptions in 59.4% of cases. Because the review deliberately focused on tasks with unusually high failure rates, that figure cannot be extrapolated to the benchmark as a whole. But it does show that a failure does not necessarily demonstrate a weakness in the model.

A 2025 EMNLP paper describes a related effect. Across several LLM benchmarks, semantically correct answers were sometimes scored as incorrect because they did not match the expected output format. Some of the apparent prompt sensitivity was therefore a product of the scoring process itself.

Interpreting a benchmark score consequently means assessing more than the model being tested. The quality of the tasks and the scoring procedure is part of the result as well.

If Model A scores 87% on a benchmark and Model B scores 85%, one thing is clear for that test: under the chosen conditions, Model A performed better. What has not yet been established is whether the two-percentage-point gap represents a stable difference in the capability being measured.

Every benchmark is based on a finite set of tasks. A different but comparable sample may challenge the models somewhat differently. Generative systems introduce further sources of variation: repeated runs can produce different answers, and semantically similar prompt variants can also change model behavior. An ACL 2026 paper examines this prompt sensitivity systematically. In addition, not every scoring procedure treats borderline cases in the same way.

When results are close together, the relevant question is whether the ranking remains stable under these variations. A consistent gap across many tasks and repeated evaluations can be meaningful even if it amounts to only a few percentage points. A single score, however, does not reveal that stability by itself.

The U.S. National Institute of Standards and Technology (NIST) similarly notes in its statistical treatment of AI benchmarks that mean scores without information about uncertainty, evaluation assumptions, and evaluation design describe only part of the result. The size of the gap alone therefore does not determine how robust a comparison is.

How strongly a ranking can depend on the specific application is illustrated by a recent study on relation extraction. The task is to identify relationships between entities in text—for example, which person is associated with which company or which organization is associated with a particular process.

Among other experiments, the authors fine-tuned a Qwen2.5 model with just 0.5 billion parameters specifically for relation extraction. Across the general-domain datasets included in the study, the model achieved a micro-F1 score of 0.83. Put simply, micro-F1 summarizes how accurately and comprehensively the target relations are identified; higher values are better. GPT-5.4 achieved 0.69 in a zero-shot setting, while Claude Sonnet 4.6 achieved 0.66. The results come from a preprint published in June 2026.

It is notable that a model with only 0.5 billion parameters ranks ahead of current frontier models on this clearly defined task.

The Qwen model had been fine-tuned for relation extraction, while GPT-5.4 and Claude Sonnet 4.6 were evaluated zero-shot. For the task examined, the 0.5B model was clearly ahead of both frontier models.

Many tasks within companies are considerably narrower than the capabilities that general-purpose model benchmarks are intended to cover: extracting specific information from documents, classifying text, producing defined output formats, or working within a limited specialist domain. The strongest model overall will not necessarily perform best on such tasks.

Public benchmarks are useful for an initial shortlist. They show which models perform well on standardized tasks and which candidates deserve closer consideration.

But the actual performance question for a specific application remains unanswered. A general reasoning or coding benchmark does not tell you how reliably a model will extract particular information from the German-language technical documents actually used in an organization, how it will handle unusual document structures, or how consistently it will follow a required output format.

For such questions, a manageable set of representative tasks is often more informative than another general leaderboard: the actual types of documents involved, typical user questions, difficult edge cases, and predefined criteria for what counts as a correct or usable answer. This does not necessarily require a large evaluation project. What matters more is that the test tasks closely resemble the intended use.

A model choice also depends on factors that a general capability benchmark does not measure. Response time, hardware requirements, costs, data flows, and integration requirements can all matter once the model is deployed. Some of these characteristics can in turn be benchmarked separately; MLPerf Endpoints, for example, measures throughput, Time to First Token, and concurrency.

The Measure guidance in the NIST AI Risk Management Framework Playbook follows the same basic principle: measurement methods and metrics should be aligned with the actual context of use, while characteristics not covered by an existing evaluation should be considered separately.

A leaderboard can therefore show which models merit closer consideration. Whether any of them is actually better suited to a specific application can only be assessed using tasks and conditions that resemble that application as closely as possible.