Terminal-Bench is a 89-task benchmark that drops AI agents into real Docker terminal environments — compiling kernels, recovering corrupted databases, cross-compiling Doom — and the top independently-verified score as of August 2026 is 89.5%, up from a stated “less than 65%” for frontier models at the benchmark’s own launch just months earlier.
That jump is the story. Terminal-Bench, built by Stanford researchers and the Laude Institute, has become one of the most-cited agentic coding benchmarks of 2026 precisely because it was designed to be hard to game — and it’s being gamed anyway, or agents are genuinely getting better, or both. The trackers don’t fully agree on which. For the broader question of what “good at Terminal-Bench” should and shouldn’t tell you about a model, see our guide to evaluating LLMs.
What Terminal-Bench Actually Tests
Unlike function-level coding benchmarks, Terminal-Bench measures whether an agent can complete an entire real-world workflow inside a command-line environment with no GUI. Each of the 89 tasks in the current version ships with four parts: a natural-language instruction, a containerized Docker environment, a programmatic verification test suite, and a human-written oracle solution, per the benchmark’s own description on Artificial Analysis’s leaderboard page. Tasks span software engineering, system administration, data processing, model training, and security.
The example tasks published alongside the leaderboard are deliberately unglamorous and deliberately hard: standing up a Git server over SSH that deploys two branches to separate HTTPS endpoints via an Nginx post-receive hook within a 3-second window; recovering as many rows as possible from a SQLite database corrupted by binary truncation; and — the one that gets screenshotted — cross-compiling Doom’s source for a MIPS target so it runs inside a JavaScript VM. None of these have a single obviously-correct command; they require exploration, multi-step tool use, and error recovery, which is the point.
From 1.0 to 2.1, in Under 18 Months
Terminal-Bench 1.0 launched on May 19, 2025, with 80 tasks and a reference scaffold called Terminus, led at Stanford by Mike A. Merrill and at the Laude Institute by Alexander G. Shaw, according to the project’s own release history. A substantially revised Terminal-Bench 2.0 followed on November 7, 2025: 89 more rigorously verified tasks and a new evaluation harness. The accompanying paper (Merrill et al., arXiv:2601.11868) stated that “frontier models and agents score less than 65%” on the new set — a deliberate design goal, not a bug.
Terminal-Bench 2.1, the version most trackers report today, is not a new task set but a “verified refresh”: patched Dockerfiles (adding missing packages like procps so background-process tooling actually works) and corrected instruction-test mismatches across roughly a dozen of the 89 tasks, per Artificial Analysis’s methodology notes. That matters for anyone reading leaderboards: 2.1 scores are not directly comparable to 2.0 scores, because 2.1 closed off several ways agents were passing tasks for the wrong reasons, as CodingFleet’s leaderboard tracking also notes.
The Scores — and Why Two Trackers Don’t Match
This is where Terminal-Bench becomes a useful case study in benchmark methodology rather than just a leaderboard. Two credible trackers, looking at adjacent versions with different evaluation setups, report meaningfully different pictures:
| Source | Version | Methodology | #1 | #2 | #3 |
|---|---|---|---|---|---|
| Artificial Analysis (independent) | v2.1 | Terminus 2 agent harness, e2b sandbox, pass@1 averaged over 3 repeats per task | GPT-5.6 Sol (xhigh) — 89.5% | Claude Opus 5 (Adaptive Reasoning, Max Effort) — 89.1% | Grok 4.6 (high) — 88.4% |
| tbench.ai snapshot, mirrored by BenchLM.ai | v2.0 | Vendor-reported scores; BenchLM explicitly flags this table as display-only and excludes it from its own composite rankings | GPT-5.6 Sol — 91.9% | Claude Mythos 5 — 88.0% | GPT-5.6 Terra — 87.4% |
A few things stand out. First, the same model family — OpenAI’s GPT-5.6 Sol — tops both tables, but at 91.9% on the easier, vendor-reported v2.0 snapshot and 89.5% on the harder, independently-run v2.1 (with an “xhigh” reasoning-effort setting disclosed by AA and not by the vendor snapshot). Second, BenchLM itself declines to fold this benchmark into its overall model rankings and marks it “display only,” which is a meaningful editorial signal about how much trust to place in self-reported CLI-agent numbers generally. Third, the harness matters as much as the model: Artificial Analysis runs everything through the same Terminus 2 scaffold in an e2b sandbox specifically so that scaffold differences don’t get conflated with model differences — a control that vendor-reported numbers typically don’t disclose.
Why the Numbers Moved So Fast
Sub-65% to high-80s in a matter of months looks, at a glance, like the kind of trajectory that eventually saturates a benchmark — the same pattern already visible in HumanEval, where contamination and ceiling effects made the original benchmark stop being useful. Terminal-Bench’s designers built in some resistance to this: task verification is programmatic rather than judge-based, environments are fresh Docker containers rather than static prompts, and the 2.1 refresh actively closed loopholes rather than just adding harder tasks. Whether that’s enough to keep Terminal-Bench discriminating at the frontier for another year, or whether it follows the same saturation curve as its predecessors, is the open question — and it’s exactly the kind of question the gap between the two tables above is already hinting at.
How It Differs From SWE-bench, tau-bench, and OSWorld
Terminal-Bench occupies a specific niche in the broader landscape mapped out in our AI agent benchmarks guide. SWE-bench tests an agent’s ability to resolve a specific GitHub issue inside an existing, already-scaffolded repository — a narrower, more code-centric task. tau-bench tests multi-turn tool-calling against simulated APIs in domains like retail and airlines, with no real shell access at all. Terminal-Bench sits at the systems-level end: it wants agents to set up servers, recover from corrupted state, cross-compile for unusual targets, and generally do the kind of ambient sysadmin and build work that a competent engineer does without thinking about it — work that has no single API call to fall back on.
What To Watch
For anyone tracking this benchmark going forward, three things are worth watching rather than any single leaderboard snapshot: whether independent re-runs (Artificial Analysis, or similar third-party harnesses) continue to trail vendor-reported numbers by a consistent margin; whether a Terminal-Bench 3.0 or “Hard” variant follows the same pattern as the 1.0-to-2.0 jump once 2.1 saturates; and whether the reasoning-effort settings that models are run at (the “xhigh” and “Max Effort” tags scattered through these tables) get standardized, since right now they’re one of the biggest uncontrolled variables between rows.
FAQ
Is Terminal-Bench the same as SWE-bench?
No. SWE-bench scores an agent’s ability to produce a patch that resolves a specific GitHub issue in an existing repository. Terminal-Bench scores end-to-end command-line workflows — server setup, data recovery, cross-compilation — verified by a programmatic test suite rather than a single patch diff. See our SWE-bench breakdown for the contrast.
Why do Terminal-Bench scores vary so much between sources?
Because version (2.0 vs. 2.1), agent harness (e.g., Terminus 2 vs. an unspecified vendor scaffold), sandbox environment, reasoning-effort setting, and number of repeats per task (pass@1 vs. averaged pass@1) are all inconsistently disclosed across trackers. Artificial Analysis and BenchLM’s mirror of the tbench.ai snapshot, compared above, illustrate the gap directly.
Is Terminal-Bench at risk of saturating like HumanEval or MMLU did?
It’s trending that way in raw numbers — sub-65% to high-80s in under a year — but the benchmark’s programmatic verification and containerized environments make it structurally harder to game than static prompt benchmarks like MMLU. The 2.1 “verified refresh,” which patched roughly a dozen tasks specifically to close scoring loopholes, is itself evidence the maintainers are actively fighting saturation rather than ignoring it.
Last updated August 18, 2026. This page is refreshed as benchmarks and scores move.
Pingback: AI Agent Benchmarks: A Map of tau-bench, OSWorld, Terminal-bench and More - Tech Blog