SWE-bench Explained: Original vs Lite vs Verified vs Pro

“SWE-bench” is not one benchmark — it is four, with wildly different difficulty and, as of 2026, one of them (Verified) publicly disowned by the lab that funded its creation over contamination. If a model card just says “SWE-bench: 88%” without naming the variant, treat the number as marketing copy, not evidence. Here is what Original, Lite, Verified, and Pro actually measure, why OpenAI walked away from the version everyone still quotes, and where the scores stand this month across trackers that don’t agree with each other.

Four benchmarks sharing one name

SWE-bench started as a single dataset: Jimenez et al., 2023 pulled 2,294 real GitHub issue-and-pull-request pairs from 12 popular Python repositories and turned each into a task — read the issue, patch the codebase, pass the hidden tests. That original set is barely reported today. It’s expensive to run at full scale, and a meaningful fraction of its tasks turned out to be underspecified: ambiguous bug reports, flaky test runners, or fixes that required context no model (or human) could infer from the issue text alone.

Three narrower variants replaced it, each solving a different problem with the original:

• SWE-bench Lite — 300 tasks, curated by the original authors for cheaper, faster evaluation. Tasks referencing images, external hyperlinks, or other PRs/issues were stripped out to leave self-contained problems. The tradeoff: 300 tasks means less statistical power, so a two-or-three point gap between models is close to noise.
• SWE-bench Verified — 500 tasks, announced by OpenAI on August 13, 2024 in partnership with the Princeton SWE-bench team. OpenAI paid 93 professional software engineers to independently review 1,699 candidate tasks, three reviewers per task, filtering out anything with broken tests or an unsolvable spec. For over a year this was the closest thing the field had to a trusted, human-audited coding benchmark.
• SWE-bench Pro — introduced by Scale AI in 2025 and now OpenAI’s recommended replacement for Verified. It comprises 1,865 total human-verified and augmented tasks split into three subsets: a Public set of 731 tasks deliberately drawn from copyleft (GPL-licensed) repositories — a legal deterrent meant to keep the solutions out of future training corpora — a Commercial set of 276 tasks from proprietary startup codebases that can never be published, and a Held-out set of 858 tasks mirroring the public set with a disjoint repository list. Every task ships with human-reviewed fail2pass and pass2pass tests.

Why OpenAI stopped using its own benchmark

On February 23, 2026, OpenAI published a post explaining that it had stopped evaluating frontier models on SWE-bench Verified. The stated reason: contamination had made the benchmark unreliable as a capability signal. OpenAI’s own audit found that roughly 59% of the tasks its models failed contained fundamentally broken tests — meaning a chunk of the “unsolved” 12% wasn’t measuring model weakness at all. Separately, the company found evidence that frontier models across vendors, including GPT-5.2, Claude Opus 4.5, and Gemini 3 Flash, had likely been trained on data that included Verified’s solutions, since the underlying GitHub repositories keep getting re-crawled into pretraining corpora after the benchmark’s answers become public. A model that has seen the patch before isn’t being tested on software engineering; it’s being tested on retrieval.

This is the same failure mode documented across other frontier tools — a benchmark’s public success becomes the mechanism of its own decay. OpenAI’s February recommended fix was to move evaluation to SWE-bench Pro, which uses restrictive licensing and held-out subsets specifically to blunt this effect.

Update, September 2026: OpenAI has since retracted that recommendation, too. In a July 8, 2026 post, “Separating signal from noise in coding evaluations,” OpenAI reported that an internal audit of SWE-bench Pro’s 731-task public split found roughly 30% of the tasks are themselves broken — an automated pipeline flagged 27.4% as broken, and a human annotation campaign with five experienced engineers independently flagged 34.1%. The failure modes mirror what plagued Verified: overly strict hidden tests that require implementation details never specified in the prompt, under-specified problem statements, narrow test coverage that lets incomplete fixes pass, and tasks whose instructions actively point toward the wrong fix. OpenAI’s stated reason for building Pro — that GitHub issues and PRs were authored for human collaborators, not as clean, independent AI-eval tasks — turns out to undercut Pro the same way it undercut Verified. The company now says the field needs benchmarks purpose-built by experienced software engineers for AI evaluation, rather than harvested from real repositories after the fact. Treat any single SWE-bench Pro score, including the ones in the table below, as similarly contamination- and construction-flawed until a given tracker specifies how it filtered for broken tasks.

The scoreboard, July 2026 — and where trackers disagree

Because Verified is now considered compromised, treat any single “Verified” number as a lower-confidence signal — and the trackers illustrate why. Within the same week in July 2026:

• BenchLM.ai (July 19, 2026) reports Claude Mythos 5 leading Verified at 95.5%, with Claude Fable 5 at 95% and Claude Opus 4.8 at 88.6%.
• LLM-Stats, updated the same month, puts Claude Fable 5 on top at 95.0% — but an earlier snapshot from the same aggregator had GPT-5.5 at 88.7% and Claude Opus 4.8 at 88.6% leading instead.

An eight-point spread on which model is “best” at the same benchmark, in the same month, is not measurement noise — it’s the contamination problem OpenAI flagged, playing out in public in real time.

Pro tells a similarly inconsistent story, and here the gap is even starker because it crosses first-party and third-party sources:

Source	Leading model	Score	Date
Scale AI, official public leaderboard	Muse Spark 1.1 (Meta)	61.5%	Sep 2026
LLM-Stats vendor aggregate (active models)	Claude Opus 4.8	69.2%	Jul 2026
BenchLM.ai	Claude Opus 5.5	89.9%	Sep 25, 2026
SWE-bench Lite, swebench.com-adjacent trackers	Claude Opus 4.6	62.7%	Jul 20, 2026

The gap between Scale’s own leaderboard and third-party aggregators has actually widened since our last check: Scale’s official public leaderboard now shows Meta’s Muse Spark 1.1 leading at 61.5% (up from GPT-5.4’s 59.1%), while BenchLM’s September 2026 table shows Claude Opus 5.5 at 89.9% — a roughly 28-point spread on the same nominal benchmark. That gap most likely reflects different subsets being scored (public vs. commercial vs. held-out), different agent scaffolds layered on top of the same base model, or self-reported vendor numbers making it into aggregator tables without being independently re-run — on top of the roughly 30% broken-task rate OpenAI documented above, which applies to every row in this table regardless of source. Scale runs and controls the Pro dataset, which makes its own public leaderboard the higher-confidence source of the two — treat third-party Pro numbers as directional at best until they specify which subset and scaffold produced them, and treat both as inflated by an unknown share of unsolvable tasks.

Scaffold matters more than most model-card comparisons let on: Princeton’s own mini-SWE-agent, a deliberately simple agent loop, has scored around 65% on Verified — competitive with results that other papers attribute to much heavier multi-rollout, review-augmented systems. The harness wrapped around a model routinely moves the score as much as the model swap does, which is exactly the kind of variable a rigorous LLM evaluation has to hold constant before comparing anything else.

What this means for reading a vendor’s claim

Three checks before citing a SWE-bench number: which variant (Original/Lite/Verified/Pro — they are not comparable to each other), which subset if it’s Pro (public/commercial/held-out score differently by design), and which scaffold ran the model (a strong agent loop can add ten-plus points over a bare model call). The pattern echoes what’s already visible in other saturating benchmarks — GPQA Diamond’s frontier cluster has compressed to the point where tracker disagreement is itself the finding, not a footnote. SWE-bench Verified got there faster, and for a more structural reason: its answers are sitting in public GitHub history, and pretraining doesn’t ask permission before ingesting them.

Commercial coding tools inherit the same problem one layer up: a 2026 head-to-head puts GitHub Copilot at 56% and Cursor at 51.7% on SWE-bench, while other 2026 aggregations report Copilot’s SWE-bench Verified score at 72.5–72.8% for roughly the same period — a 17-point spread driven by which underlying model each tool routed to and which harness version was tested. See our GitHub Copilot vs Cursor limitations breakdown for the full scoreboard.

The contamination mechanism here is also a documented limitation of the models being scored, not just the benchmark — a model that’s “solving” tasks by pattern-matching memorized patches is a different failure mode than one reasoning through the fix from scratch, and it fails differently in production when the exact GitHub issue it memorized isn’t the one in front of it.

FAQ

Is SWE-bench Verified still meaningful in 2026?
As a rough capability signal for older models, somewhat. As a leaderboard for ranking current frontier models against each other, OpenAI’s own February 2026 audit says no — contamination and broken tests have eroded its reliability, which is why it recommends SWE-bench Pro instead.

Why is SWE-bench Pro so much harder than Verified?
Pro draws from a broader set of repositories and languages, enforces stricter fail2pass/pass2pass test review, and deliberately uses copyleft-licensed and held-out repositories to resist the exact contamination pattern that undermined Verified — a different anti-contamination strategy than LiveCodeBench’s rolling, dated problem releases, aimed at the same underlying risk. Scores in the 55–70% range on Pro’s official public leaderboard correspond to 80%+ on Verified for the same models.

Why do SWE-bench trackers disagree so much on the “top” model?
Because they’re often not measuring the same thing: different subsets (public vs. commercial vs. held-out on Pro), different agent scaffolds on top of the same base model, and in some cases self-reported vendor numbers that haven’t been independently re-run. When a spread exceeds roughly five points, check which of these differs before trusting the ranking.

Last updated September 27, 2026. This page is refreshed as benchmarks and scores move.