“SWE-bench” is not one benchmark — it is four, with wildly different difficulty and, as of 2026, one of them (Verified) publicly disowned by the lab that funded its creation over contamination. If a model card just says “SWE-bench: 88%” without naming the variant, treat the number as marketing copy, not evidence. Here is what Original, Lite, Verified, and Pro actually measure, why OpenAI walked away from the version everyone still quotes, and where the scores stand this month across trackers that don’t agree with each other.
Four benchmarks sharing one name
SWE-bench started as a single dataset: Jimenez et al., 2023 pulled 2,294 real GitHub issue-and-pull-request pairs from 12 popular Python repositories and turned each into a task — read the issue, patch the codebase, pass the hidden tests. That original set is barely reported today. It’s expensive to run at full scale, and a meaningful fraction of its tasks turned out to be underspecified: ambiguous bug reports, flaky test runners, or fixes that required context no model (or human) could infer from the issue text alone.
Three narrower variants replaced it, each solving a different problem with the original:
- SWE-bench Lite — 300 tasks, curated by the original authors for cheaper, faster evaluation. Tasks referencing images, external hyperlinks, or other PRs/issues were stripped out to leave self-contained problems. The tradeoff: 300 tasks means less statistical power, so a two-or-three point gap between models is close to noise.
- SWE-bench Verified — 500 tasks, announced by OpenAI on August 13, 2024 in partnership with the Princeton SWE-bench team. OpenAI paid 93 professional software engineers to independently review 1,699 candidate tasks, three reviewers per task, filtering out anything with broken tests or an unsolvable spec. For over a year this was the closest thing the field had to a trusted, human-audited coding benchmark.
- SWE-bench Pro — introduced by Scale AI in 2025 and now OpenAI’s recommended replacement for Verified. It comprises 1,865 total human-verified and augmented tasks split into three subsets: a Public set of 731 tasks deliberately drawn from copyleft (GPL-licensed) repositories — a legal deterrent meant to keep the solutions out of future training corpora — a Commercial set of 276 tasks from proprietary startup codebases that can never be published, and a Held-out set of 858 tasks mirroring the public set with a disjoint repository list. Every task ships with human-reviewed fail2pass and pass2pass tests.
Why OpenAI stopped using its own benchmark
On February 23, 2026, OpenAI published a post explaining that it had stopped evaluating frontier models on SWE-bench Verified. The stated reason: contamination had made the benchmark unreliable as a capability signal. OpenAI’s own audit found that roughly 59% of the tasks its models failed contained fundamentally broken tests — meaning a chunk of the “unsolved” 12% wasn’t measuring model weakness at all. Separately, the company found evidence that frontier models across vendors, including GPT-5.2, Claude Opus 4.5, and Gemini 3 Flash, had likely been trained on data that included Verified’s solutions, since the underlying GitHub repositories keep getting re-crawled into pretraining corpora after the benchmark’s answers become public. A model that has seen the patch before isn’t being tested on software engineering; it’s being tested on retrieval.
This is the same failure mode documented across other frontier tools — a benchmark’s public success becomes the mechanism of its own decay. OpenAI’s recommended fix is to move evaluation to SWE-bench Pro, which uses restrictive licensing and held-out subsets specifically to blunt this effect.
The scoreboard, July 2026 — and where trackers disagree
Because Verified is now considered compromised, treat any single “Verified” number as a lower-confidence signal — and the trackers illustrate why. Within the same week in July 2026:
- BenchLM.ai (July 19, 2026) reports Claude Mythos 5 leading Verified at 95.5%, with Claude Fable 5 at 95% and Claude Opus 4.8 at 88.6%.
- LLM-Stats, updated the same month, puts Claude Fable 5 on top at 95.0% — but an earlier snapshot from the same aggregator had GPT-5.5 at 88.7% and Claude Opus 4.8 at 88.6% leading instead.
An eight-point spread on which model is “best” at the same benchmark, in the same month, is not measurement noise — it’s the contamination problem OpenAI flagged, playing out in public in real time.
Pro tells a similarly inconsistent story, and here the gap is even starker because it crosses first-party and third-party sources:
| Source | Leading model | Score | Date |
|---|---|---|---|
| Scale AI, official public leaderboard | GPT-5.4 (xHigh) | 59.1% | Jun 28, 2026 |
| LLM-Stats vendor aggregate (active models) | Claude Opus 4.8 | 69.2% | Jul 2026 |
| BenchLM.ai | Claude Mythos 5 | 80.3% | Jul 16, 2026 |
| SWE-bench Lite, swebench.com-adjacent trackers | Claude Opus 4.6 | 62.7% | Jul 20, 2026 |
A 21-point gap between Scale’s own leaderboard (59.1%) and a third-party aggregator (80.3%) on the same benchmark is too large to explain as rounding or a few weeks of model releases. It most likely reflects different subsets being scored (public vs. commercial vs. held-out), different agent scaffolds layered on top of the same base model, or self-reported vendor numbers making it into aggregator tables without being independently re-run. Scale runs and controls the Pro dataset, which makes its own public leaderboard the higher-confidence source of the two — treat third-party Pro numbers as directional at best until they specify which subset and scaffold produced them.
Scaffold matters more than most model-card comparisons let on: Princeton’s own mini-SWE-agent, a deliberately simple agent loop, has scored around 65% on Verified — competitive with results that other papers attribute to much heavier multi-rollout, review-augmented systems. The harness wrapped around a model routinely moves the score as much as the model swap does, which is exactly the kind of variable a rigorous LLM evaluation has to hold constant before comparing anything else.
What this means for reading a vendor’s claim
Three checks before citing a SWE-bench number: which variant (Original/Lite/Verified/Pro — they are not comparable to each other), which subset if it’s Pro (public/commercial/held-out score differently by design), and which scaffold ran the model (a strong agent loop can add ten-plus points over a bare model call). The pattern echoes what’s already visible in other saturating benchmarks — GPQA Diamond’s frontier cluster has compressed to the point where tracker disagreement is itself the finding, not a footnote. SWE-bench Verified got there faster, and for a more structural reason: its answers are sitting in public GitHub history, and pretraining doesn’t ask permission before ingesting them.
The contamination mechanism here is also a documented limitation of the models being scored, not just the benchmark — a model that’s “solving” tasks by pattern-matching memorized patches is a different failure mode than one reasoning through the fix from scratch, and it fails differently in production when the exact GitHub issue it memorized isn’t the one in front of it.
FAQ
Is SWE-bench Verified still meaningful in 2026?
As a rough capability signal for older models, somewhat. As a leaderboard for ranking current frontier models against each other, OpenAI’s own February 2026 audit says no — contamination and broken tests have eroded its reliability, which is why it recommends SWE-bench Pro instead.
Why is SWE-bench Pro so much harder than Verified?
Pro draws from a broader set of repositories and languages, enforces stricter fail2pass/pass2pass test review, and deliberately uses copyleft-licensed and held-out repositories to resist the exact contamination pattern that undermined Verified. Scores in the 55–70% range on Pro’s official public leaderboard correspond to 80%+ on Verified for the same models.
Why do SWE-bench trackers disagree so much on the “top” model?
Because they’re often not measuring the same thing: different subsets (public vs. commercial vs. held-out on Pro), different agent scaffolds on top of the same base model, and in some cases self-reported vendor numbers that haven’t been independently re-run. When a spread exceeds roughly five points, check which of these differs before trusting the ranking.
Last updated July 20, 2026. This page is refreshed as benchmarks and scores move.