SWE-bench Verified, Pro and What Replaced Them: By the Numbers

What SWE-bench Verified measured

The original SWE-bench, released in 2023, turned 2,294 real GitHub issues from 12 Python repositories into coding tasks.¹ A model gets the issue text and the repository as it stood before the fix, and must produce a code change. It passes a task only if two sets of hidden tests pass afterwards: tests that failed before the fix, and regression tests that check nothing else broke.²

OpenAI published SWE-bench Verified on August 13, 2024.¹¹ Expert software engineers reviewed 1,699 SWE-bench problems, three reviewers each, and kept a curated set of 500 that they judged fair to solve.² It became a standard metric reported in frontier model releases.²

A Verified score also depends on the harness, the agent scaffold that lets the model read files, run commands and submit a patch. Epoch AI's first review of the benchmark, in June 2025, found that results were "highly dependent on the scaffold used".⁴ For that reason, swebench.com's default "Bash Only" view runs every model in the same minimal agent, mini-SWE-agent, "so scores compare models rather than harnesses".¹ The table below uses only runs in one harness version, v2.0.0.

The saturation numbers

Thirteen entries have been run in mini-SWE-agent v2.0.0, one attempt per task, between February 17 and 26, 2026. The top four score between 75.6% and 76.8%, and the open-weight MiniMax M2.5 ties Gemini 3 Flash at 75.8%.¹ Across all 180 Verified entries and every harness, the newest is dated February 26, 2026.¹

Vendor-reported scores, in each lab's own setup, ran higher. When OpenAI stopped reporting the benchmark, it wrote that the state of the art had moved "from 74.9% to 80.9% in the last 6 months", citing an aggregator of reported results. Those are Vendor-stated figures and aren't comparable with the shared-harness runs below.²

What the audits found

Flawed tests. On February 23, 2026, OpenAI published an audit of 138 Verified tasks that its o3 model did not consistently solve over 64 runs. At least six experienced engineers reviewed each one. OpenAI found that 59.4% of the 138 "contained material issues in test design and/or problem description".²¹¹ It sorted them into three groups:

• 35.5% had "narrow" tests that demand one specific implementation, so functionally correct fixes fail.
• 18.8% had "wide" tests that check for functionality the task never asked for.
• 5.1% had other issues.²

Contamination. The tasks, their repositories and their reference fixes (the "gold patches") are all public. OpenAI ran an automated probe in which GPT-5 tried to draw task details out of other models. It reported that GPT-5.2-Chat, Claude Opus 4.5 and Gemini 3 Flash Preview could each reproduce a gold patch or verbatim task details for some tasks, so "all frontier models we tested" had seen at least some problems and solutions in training.² OpenAI said gains on Verified "increasingly reflect how much the model was exposed to the benchmark at training time". It stopped reporting the benchmark and recommended SWE-bench Pro's public split instead.²

An independent verdict. Epoch AI's benchmark review, dated September 3, 2026, rates SWE-bench Verified "Flawed". By Epoch's reading of OpenAI's audit, at least 16.4% of all 500 tasks are broken. On that basis, Epoch judged the benchmark unlikely to meet its bar of fewer than 20% of tasks having accuracy-affecting issues. The review also notes that 100% of tasks and 100% of solutions are public.⁴

Exploitable scoring. In April 2026, researchers at UC Berkeley's Center for Responsible, Decentralized Intelligence reported scoring 100% on Verified's 500 tasks without fixing any bug. Their agent added a short test-configuration file that made every test report a pass. They reported the same 100% on SWE-bench Pro's 731 public tasks and on Terminal-Bench's 89 tasks, because in each case the grader trusts output from an environment the agent controls.⁶

SWE-bench Pro, and its own audit

Scale AI built SWE-bench Pro from three subsets. Only the first is scored publicly:⁵

• A public set of 731 tasks from open-source repositories under strong copyleft licenses such as the GPL, chosen as a legal deterrent against training on the code.
• A private set of 276 tasks from 18 startups' proprietary codebases.
• A held-out set of 858 tasks.

At launch, Scale wrote that top models scored "around 23%" on the public set, against 70%+ on Verified.⁵ By October 13, 2026, the top public-set entry on Scale's leaderboard was Meta's Muse Spark 1.1 at 61.50% (± 3.10), added July 9, 2026. Next came GPT-5.4 at its xHigh setting with 59.10%, both run in the mini-swe-agent harness.⁵ OpenAI's own account puts frontier pass rates on the public split at 23.3%, rising to 80.3% within eight months. Those are Vendor-stated figures in an unnamed setup, so they don't line up against Scale's runs.³

On July 8, 2026, OpenAI audited Pro too.³¹¹

• An automated filter flagged 286 tasks for review.
• Agent-assisted review, with a researcher's final judgment, marked 200 tasks (27.4%) as broken.
• A parallel campaign in which five engineers reviewed each task marked 249 (34.1%).

The failures fell into four types: overly strict tests, underspecified prompts, low-coverage tests that let incomplete fixes pass, and misleading prompts.³ OpenAI estimated that about 30% of Pro's tasks are broken and wrote, "we retract our earlier recommendation to adopt SWE-Bench Pro." It named no replacement.³

What came next

Instead of a single successor, OpenAI called for new benchmarks "built by experienced software developers specifically to test model capabilities".³ In February it had pointed to privately authored tasks graded by trained reviewers, as in its GDPval evaluation, calling that approach resource-intensive.²

Terminal-Bench, hosted by Stanford, Harbor and the Laude Institute, chose continuous versioning: 2.0 in November 2025, 2.1 in May 2026, 3.0 on July 30, 2026, and 4.0 on August 28, 2026.¹⁰ Version 3.0 launched with 74 tasks across seven domains. Its tasks run in an agent container that is separate from the verifier container, which the maintainers say blocks many reward-hacking routes.⁹ Version 4.0 changed the task set:⁸

• It removed 8 tasks: two as saturated (every model class in the latest generation solved them 5 times out of 5), two for refusals, two because solutions were public and two for unresolved quality issues.
• It fixed 19 more.
• Version 4.1 is planned to add tamper-resistant verifiers.

Its top entry is Anthropic's Opus 5.5 in Claude Code at the max setting, scoring 64.8% ± 3.1% over 330 trials, with a release date of September 22, 2026.⁷ The SWE-bench team has since released new evaluations: CodeClash in November 2025, and ProgramBench in May 2026, which tests whether models can build meaningful software from scratch.¹

The snapshot below sets the three leaderboards side by side. Each score comes from a different task set, harness and grading rule, so the rows aren't comparable with each other.