OpenAI has retracted its recommendation of SWE-Bench Pro, a widely used benchmark designed to evaluate AI models’ coding skills, after discovering that nearly a third of the benchmark’s tasks are flawed. This move sends ripples across the AI developer community, where benchmark scores have become a key marketing tool to showcase model capabilities.
SWE-Bench Pro was designed to offer a more realistic gauge of ”agent-like” programming performance. Unlike simpler tests such as HumanEval, it pulls tasks from real-world coding repositories, requiring models to fix bugs so new tests pass without breaking existing functionality. Its practical approach has attracted attention from research labs and companies building AI coding assistants.
OpenAI’s internal review revealed significant issues. Out of 731 public tasks, automated checks flagged 27.4% as broken, while a detailed evaluation by five senior engineers raised that figure to 34.1%. The problems fell into four main categories: overly strict tests, vague problem descriptions, insufficient test coverage, and misleading wording that could cause models to fail despite providing working solutions-or pass with incomplete fixes.
These defects undermine SWE-Bench Pro’s core purpose: to reliably distinguish true model limitations from dataset noise. OpenAI points out that it previously identified similar flaws in SWE-Bench Verified, for which it had recommended users migrate to SWE-Bench Pro as a cleaner alternative. Now, that recommendation has been withdrawn. This is especially notable since recent AI model launches from OpenAI, Anthropic, and Google have often cited SWE-Bench results to demonstrate progress.
Rapid improvement in SWE-Bench Pro scores raises concerns
OpenAI’s skepticism extends beyond the percentage of faulty tasks to the benchmark’s reported rapid improvement. Top model scores soared from 23.3% to 80.3% in just eight months. While such gains might suggest real leaps in AI capability, they often signal benchmark issues like data leakage, ambiguous tasks, or shortcuts that let models ”game” the test.
To investigate, OpenAI set up a quality control pipeline where automated systems analyzed problem statements, model attempts, metadata, and test failures. AI ”investigator agents” based on Codex reviewed contentious cases in test environments and repositories. Human engineers then weighed in, identifying even more broken tasks and multiple issues within single problems.
This problem affects the broader AI industry because many popular AI code benchmarks rely on datasets mined from public GitHub repositories, pull requests, and issue trackers. These artifacts were created to support human developers, not as rigorous exams for AI. As a result, task descriptions, final patches, and unit tests often don’t align perfectly-what works for maintainers may not be a fair test for AI models.
Amid this, interest is growing in alternatives like LiveCodeBench, a benchmark designed to reduce risks of data leakage and overfitting on public tasks. Still, the core question remains: do these benchmarks truly reflect a model’s ability to write and debug code in authentic development workflows, or just its skill at passing certain tests? When companies base release decisions and safety assessments on these numbers, measurement errors become costly.
For OpenAI, this issue is especially sensitive because coding benchmarks influence deployment policies under its internal Preparedness Framework. If a benchmark inflates or underestimates a model’s abilities, it affects research priorities, model restrictions, and safety arguments-not just leaderboard rankings. OpenAI plans to develop new task sets designed by experienced developers specifically to evaluate AI coding models, moving away from post-hoc collections sourced from live software projects.
The immediate impact is clear: SWE-Bench Pro results require more cautious interpretation, and past records should be scrutinized closely. The next challenge for the AI coding landscape will be introducing fresh benchmarks. Their success depends on whether major AI labs accept them as reliable industry standards or view them as just another promotional tool.