Executive Summary

The case for retirement compresses into four numbers. Scores saturated within a narrow band; more than half of the hard tasks turned out to have flawed grading criteria; a large share of those flaws were tests that forced a single narrow answer; and on some tasks the model showed signs of having seen the answer in advance.

SWE-bench Verified was the de facto standard exam for measuring a coding agent's skill. It takes issues drawn from real open-source repositories, has a model fix them in code, and grades the result by whether the repository's own tests pass. After OpenAI released it in 2024, it became a shared coordinate for comparing frontier models, and OpenAI itself used the benchmark in the internal metrics it relies on to track dangerous capabilities.

In February 2026, the company that built that benchmark declared this exam could no longer measure frontier capability. The surface signal was saturation. Over the past six months the top score rose only from 74.9% to 80.9%. Once the top of the leaderboard starts crowding into a narrow band, there is no way to tell whether the still-unsolved 20% reflects a real limit of the models or problems that were built wrong in the first place. It is the point where the scores keep rising but what those scores mean grows blurry.

▲ Pebblous original diagram — six-month change in SOTA score (74.9%→80.9%) and the remaining gap

Saturation alone does not retire a benchmark. When OpenAI opened up the remaining problems to look, two more fundamental defects surfaced: problems whose grading criteria were simply wrong, and signs that the answers had already leaked into the models' training data.

OpenAI selected 138 hard tasks that its latest model, o3, had consistently failed across 64 independent runs, and audited them. Several experienced software engineers cross-reviewed each task. The results shook confidence in the benchmark. In 59.4% of the 138 tasks, the test design or the problem statement itself was flawed. Even when a model produced functionally correct code, the grading test rejected it because it forced one specific implementation.

Of those, 35.5% were what OpenAI called overly narrow tests. The problem says "fix this bug," but the grading test demands "fix it in exactly this way." When the bug is resolved by a different route, the answer is actually correct, yet the test marks it wrong. On problems like these, a low score is not evidence that the model failed — it is evidence that the answer key was too narrow.

2.1Traces left in the chain of thought

The more decisive signal came from the contamination side. During the audit, GPT-5.2 solved 31 tasks that had been classified as "nearly impossible." In one Django-related task, the grading test required a specific parameter that never appears anywhere in the problem statement. Yet the model's chain of thought already contained release-note information describing that change to the codebase. It had not reasoned its way to the answer — it had seen the answer somewhere before.

▲ Pebblous original diagram — reconstructing the leak path OpenAI found in its audit

OpenAI compared this to handing students the problems and answers before the exam. Even if that student did not memorize the answer wholesale, the mere fact of having seen it once gives them an edge over the others. The score becomes a function of exposure rather than skill. The very ability the benchmark set out to measure was one the benchmark could no longer measure honestly.

Benchmark contamination is not unique to SWE-bench. In knowledge and math benchmarks such as MMLU and GSM8K, public questions and answers mixing into training data to inflate scores has already been measured quantitatively. That general background is covered in an earlier Pebblous piece on benchmark contamination. Here the focus is on the structural trap specific to coding benchmarks.

SWE-bench's source is public GitHub repositories. The problem, the real commit that serves as the answer, and the tests used for grading all already exist inside those repositories. And those same public repositories are also a training-data source for large models. To learn code, you have to scrape GitHub, and when you scrape GitHub, the benchmark's answers and tests come along with it. The data used for training and the data used for evaluation come from the same well.

▲ Pebblous original diagram — training data and evaluation data drawn from the same public repository

With multiple-choice questions, the answer is one of A/B/C/D, so exposure is at least somewhat gaugeable. A code repository is different. Issues, pull requests, commit messages, release notes, and code-review discussions are woven into one context, so information absent from the problem statement is scattered across other corners of the repository. Just as the release note leaked the answer in the earlier Django case, a coding benchmark can leak not just a single answer but the entire context surrounding it into the training data.

The techniques for detecting contamination after the fact are actually growing more refined. The Kernel Divergence Score (KDS), proposed at ICML 2025, is one example. It briefly fine-tunes a model on the benchmark data, then uses kernel similarity to measure how much the sample representations changed before and after. The core intuition: fine-tuning on data a model is seeing for the first time shifts its representations sharply, but data it has already seen in training barely moves them. The less a benchmark's representations shift, the stronger the signal that it has already leaked.

▲ Pebblous original diagram — how KDS separates contaminated from clean data by representation shift

In controlled contamination experiments, KDS correlated almost perfectly with the level of contamination and was more accurate than earlier methods that count character overlap. In other words, the question "is this benchmark contaminated?" is one we can answer better and better.

The trouble is what comes next. Detecting contamination does not give you a way to revive an already-contaminated benchmark. You cannot erase the memory of a model that has seen the answers, and you cannot put a problem that has once spread across the web back behind closed doors. The only options left to OpenAI were retirement and replacement. While it builds a new contamination-resistant evaluation system, it recommended using SWE-bench Pro — made up of 1,865 multilingual tasks — instead. Notably, the same model is reported to score much lower on Pro than on Verified, and that gap itself shows how much froth had built up in the Verified numbers.

Discussions of data quality have long stayed on training data. Are the labels accurate? Is there bias? Is the provenance clear? The retirement of SWE-bench demands that the same questions extend to evaluation data. However good the training data behind a model, if the exam measuring its skill is contaminated, the resulting numbers cannot be trusted.

Translated into the language of a Pebblous reader, this is a problem of evaluation-data freshness and provenance. When the evaluation set was created, whether its timing overlaps with the model's training cutoff, and whether it has ever been made public together determine the expiration date of that evaluation. Just as you attach source and license history to training data, you have to attach exposure history to evaluation data and track it.

For an organization trying to keep drawing up uncontaminated evaluation sets, three practices become the working standard. First, holdout isolation: physically isolate the evaluation set used for grading so the training pipeline can never touch it. Second, timing control: create or refresh evaluation items after the model's training cutoff so they cannot be memorized in advance. Third, exposure logging: record which evaluation set was published when and where, along with a version, so you can later trace back whether a model has seen the exam behind a given score.

Official announcement

• 1.OpenAI. (2026). "Why SWE-bench Verified no longer measures frontier coding capabilities." OpenAI. Audit of 138 tasks, 59.4% flawed, GPT-5.2 contamination signals; SWE-bench Pro recommended.

Industry sources

• 2.Kili Technology. (2026). "AI Benchmarks Guide — The Top Evaluations in 2026 (and Why They're Not Enough)." Kili Technology. SWE-bench Pro's 1,865 multilingual tasks; overview of benchmark saturation and the real-world gap.

Academic research

• 3.Choi, H. K., Khanov, M., Wei, H., & Li, Y. (2025). "How Contaminated Is Your Benchmark? Measuring Dataset Leakage with Kernel Divergence." ICML 2025. Measures contamination through embedding kernel shifts before and after fine-tuning.