OSWorld’s original 369-task benchmark is now effectively saturated — top systems clear 85% against a 72.36% human baseline — so the field moved the goalposts to OSWorld 2.0, where the best reported system still completes only 20.6% of long-horizon workflows outright. That gap between a “solved” headline benchmark and a much harder successor is the real story of computer-use evaluation in 2026, and it’s worth understanding both halves before trusting any single leaderboard number.
What OSWorld Actually Tests
OSWorld, introduced by a team including Tianbao Xie and Danyang Zhang in an April 2024 paper presented at NeurIPS, is a real computer-use environment rather than a static question set. Agents operate live Ubuntu, Windows, or macOS virtual machines across 369 tasks spanning web apps, desktop software, file I/O, and multi-step workflows that touch several applications at once. Success is checked with execution-based validators that inspect the final machine state — did the file get renamed, did the setting actually change — not by comparing text output to a reference answer.
The original paper’s headline number is the one worth remembering: humans completed 72.36% of tasks, while the best model at the time managed only 12.24%, failing mostly on GUI grounding (clicking the right pixel) and operational knowledge (knowing where a setting lives). That 60-point gap is why OSWorld became the reference benchmark for computer-use agents, and it’s the same gap this guide to evaluating LLMs would flag as a case where a benchmark’s ceiling matters as much as any single score.
OSWorld-Verified: A Repair Job, Then a Ceiling Problem
The original task set had scoring bugs and ambiguous instructions, so the maintainers shipped OSWorld-Verified in July 2025 as a corrected release, per BenchLM’s benchmark page. It keeps the 369 tasks but allows eight Google Drive tasks to be excluded when they can’t be configured, producing an “official” 361-task run. Within two years of launch, frontier systems have blown past the original human baseline on this version: Anthropic’s Claude 5 system card reports 85.4% for a Claude Mythos preview build and 85.0% for both the shipped Claude Mythos 5 and Claude Fable 5, all using a revised evaluation harness with a 100-step budget, according to the Steel.dev OSWorld leaderboard. That’s the same saturation pattern this site has documented on other early frontier benchmarks — see the MMLU write-up on why the original test got retired once scores stopped separating models.
Three Trackers, Three Different Leaders
Ask “who’s winning OSWorld” on the same day and you’ll get three different answers, which is itself useful information about how self-reported agent benchmarks work. As of late August 2026:
- Steel.dev has Claude Mythos Preview on top at 85.4%, sourced to Anthropic’s Claude 5 system card, with Qwen3 VL 235B as the strongest open-weight entry at 66.7%.
- BenchLM.ai instead ranks Qwen3.8 Max first at 86.1%, ahead of Claude Mythos 5 and Claude Fable 5, both tied at 85%, across 30 tracked model rows.
- llm-stats.com shows a completely different leader — ByteDance’s Seed 2.1 Pro at 78.8%, with Seed 2.1 Turbo second at 76.4% — and doesn’t list Qwen3.8 Max or the Claude Mythos/Fable line at all in its 20-model table, dated August 31, 2026.
None of these trackers is “wrong” in a strict sense; they’re aggregating different, incomplete slices of self-reported submissions, and OSWorld has no central, continuously-updated official leaderboard forcing every vendor to submit through one pipeline. The practical lesson: treat any single “OSWorld leader” claim as a snapshot of what one tracker happened to ingest, not as a settled ranking, and check the setup notes (step budget, tool-call mode, thinking effort) before comparing two rows at all.
Why OSWorld 2.0 Exists — and Why Its Scores Are So Low
The saturation on OSWorld-Verified is precisely why the same research group shipped OSWorld 2.0. Its 108 tasks are long-horizon, end-to-end workflows — filing an expense report, completing a multi-step visa application, working through a hotel booking flow — that take human users a median of about 1.6 hours and roughly 318 tool calls per task, versus about 30 in the original, according to the OSWorld 2.0 leaderboard and the official project page. Scores are reported two ways: a partial score (fraction of an average 27.25 fine-grained checkpoints reached) and binary completion (every checkpoint passed). The gap between them is the headline: OpenAI’s self-reported GPT-5.6 Sol leads on partial score at 62.6%, but binary completion for the best system — Claude Opus 4.8 with batched tool calls — is just 20.6%. Every other tracked system, including GPT-5.5 and Claude’s Sonnet and Opus 4.6/4.7 variants, completes fewer than one in five of these workflows outright, and cost per run for the higher-effort configurations reportedly runs from roughly $1,550 to $3,870. That’s the answer to “why are computer-use scores so low”: on tasks that resemble actual multi-hour office work rather than single, well-scoped actions, current agents still fail constantly, they just fail later and more expensively.
What’s Actually Failing: Grounding, Hidden Operations, Cognitive Errors
A June 2026 failure-mode study, “Naive Visual Memory is Not Enough”, taxonomizes GUI agent errors into four categories mapped to the perception-reasoning-action pipeline: cognitive failure, visual state misunderstanding, hidden operation blindness, and grounding error. On OSWorld specifically, the study reports cognitive failure at 82.6% and hidden operation blindness — missing commands buried in menus, ribbons, or keyboard shortcuts — at 67.1%, reflecting how menu-heavy and long-horizon desktop tasks are. The same paper found that simply feeding agents more full-screenshot memory doesn’t fix this: it reduces some state-tracking errors but increases grounding and hidden-operation failures, which is why the authors proposed action-grounded memory (storing the specific image crop tied to a successful action) instead, reporting a 33.3% relative improvement in task success. This is the same class of problem covered in this site’s map of agent benchmarks and in the tool-calling-specific breakdown in tau-bench explained — computer-use failure and tool-calling failure are close cousins, both rooted in an agent misjudging what state it’s actually in.
OSWorld Scores at a Glance
| System / Tracker | Benchmark version | Score | Source |
|---|---|---|---|
| Claude Mythos Preview | OSWorld-Verified (Steel.dev) | 85.4% | Claude 5 system card, Jun 2026 |
| Qwen3.8 Max | OSWorld-Verified (BenchLM.ai) | 86.1% | Self-reported, Aug 2026 |
| Seed 2.1 Pro (ByteDance) | OSWorld (llm-stats.com) | 78.8% | Self-reported, tracked Aug 2026 |
| GPT-5.6 Sol | OSWorld 2.0, partial score | 62.6% | OpenAI, Jul 2026 |
| Claude Opus 4.8 (batched tools) | OSWorld 2.0, binary completion | 20.6% | Author-run, official harness, Jun 2026 |
| Human baseline | Original OSWorld (2024) | 72.36% | OSWorld paper, arXiv 2404.07972 |
FAQ
Is OSWorld the same as OSWorld-Verified?
No. OSWorld-Verified is a July 2025 corrected release of the same 369-task set, allowing up to eight Google Drive tasks to be excluded for a 361-task run. Scores from the two are usually close but not guaranteed identical, since Verified also standardizes some evaluation bugs from the original release.
Are OSWorld and OSWorld 2.0 scores comparable?
No, and every source that publishes both says so explicitly. OSWorld 2.0 uses 108 different tasks, roughly ten times the tool calls per task, a 500-step budget, and a partial-checkpoint scoring metric instead of simple pass/fail, so an 85% on OSWorld-Verified and a 62.6% partial score on OSWorld 2.0 are not measuring the same thing.
Why do different leaderboards disagree on who leads OSWorld?
Because there’s no single mandatory submission pipeline — trackers like Steel.dev, BenchLM.ai, and llm-stats.com each ingest a different, incomplete subset of self-reported vendor results, on different update schedules, which is why the same week produced three different “#1” systems (Claude Mythos Preview, Qwen3.8 Max, and Seed 2.1 Pro) across three trackers.
Last updated September 1, 2026. This page is refreshed as benchmarks and scores move.