Terminal-Bench is a 89-task benchmark that drops AI agents into real Docker terminal environments — compiling kernels, recovering corrupted databases, cross-compiling Doom — and the top independently-verified score on that 89-task set as of September 2026 is 91.4%, up from a stated “less than 65%” for frontier models at the benchmark’s own launch just months earlier. Terminal-Bench 4.0, a harder 66-task replacement now live, knocks the same class of models back to roughly 60%.
That jump is the story. Terminal-Bench, built by Stanford researchers and the Laude Institute, has become one of the most-cited agentic coding benchmarks of 2026 precisely because it was designed to be hard to game — and it’s being gamed anyway, or agents are genuinely getting better, or both. The trackers don’t fully agree on which. For the broader question of what “good at Terminal-Bench” should and shouldn’t tell you about a model, see our guide to evaluating LLMs.
What Terminal-Bench Actually Tests
Unlike function-level coding benchmarks, Terminal-Bench measures whether an agent can complete an entire real-world workflow inside a command-line environment with no GUI. Each of the 89 tasks in the current version ships with four parts: a natural-language instruction, a containerized Docker environment, a programmatic verification test suite, and a human-written oracle solution, per the benchmark’s own description on Artificial Analysis’s leaderboard page. Tasks span software engineering, system administration, data processing, model training, and security.
The example tasks published alongside the leaderboard are deliberately unglamorous and deliberately hard: standing up a Git server over SSH that deploys two branches to separate HTTPS endpoints via an Nginx post-receive hook within a 3-second window; recovering as many rows as possible from a SQLite database corrupted by binary truncation; and — the one that gets screenshotted — cross-compiling Doom’s source for a MIPS target so it runs inside a JavaScript VM. None of these have a single obviously-correct command; they require exploration, multi-step tool use, and error recovery, which is the point.
From 1.0 to 2.1, in Under 18 Months
Terminal-Bench 1.0 launched on May 19, 2025, with 80 tasks and a reference scaffold called Terminus, led at Stanford by Mike A. Merrill and at the Laude Institute by Alexander G. Shaw, according to the project’s own release history. A substantially revised Terminal-Bench 2.0 followed on November 7, 2025: 89 more rigorously verified tasks and a new evaluation harness. The accompanying paper (Merrill et al., arXiv:2601.11868) stated that “frontier models and agents score less than 65%” on the new set — a deliberate design goal, not a bug.
Terminal-Bench 2.1, the version most trackers report today, is not a new task set but a “verified refresh”: patched Dockerfiles (adding missing packages like procps so background-process tooling actually works) and corrected instruction-test mismatches across roughly a dozen of the 89 tasks, per Artificial Analysis’s methodology notes. That matters for anyone reading leaderboards: 2.1 scores are not directly comparable to 2.0 scores, because 2.1 closed off several ways agents were passing tasks for the wrong reasons, as CodingFleet’s leaderboard tracking also notes.
The Scores — and Why Two Trackers Don’t Match
This is where Terminal-Bench becomes a useful case study in benchmark methodology rather than just a leaderboard. Two credible trackers, looking at adjacent versions with different evaluation setups, report meaningfully different pictures:
| Source | Version | Methodology | #1 | #2 | #3 |
|---|---|---|---|---|---|
| Artificial Analysis (independent) | v2.1 | Terminus 2 agent harness, e2b sandbox, pass@1 averaged over 3 repeats per task | Claude Fable 5.1 (Adaptive Reasoning, Max Effort) — 91.4% | Claude Fable 5.1 (Adaptive Reasoning, Xhigh Effort) — 91.0% | Claude Fable 5.1 (Adaptive Reasoning, High Effort) — 89.9% |
| tbench.ai snapshot, mirrored by BenchLM.ai | v2.0 | Vendor-reported scores; BenchLM explicitly flags this table as display-only and excludes it from its own composite rankings | GPT-5.6 Sol — 91.9% | Claude Mythos 5 — 88.0% | GPT-5.6 Terra — 87.4% |
| llm-stats.com | v2.1 | Aggregated scores across 42 models; the tracker’s own verification panel lists 0 verified and 42 self-reported results | DeepSeek-V4.1-Flash — 90.6% | MiMo-V2.6-Pro — 89.9% | Gemini 3.8 Flash — 89.4% |
A few things stand out. First, the three tables don’t even agree on which lab leads: Artificial Analysis’s independently-run v2.1 puts three Claude Fable 5.1 effort settings in the top three (91.4% / 91.0% / 89.9%), the vendor-reported v2.0 snapshot puts OpenAI’s GPT-5.6 Sol on top at 91.9%, and llm-stats’s all-self-reported v2.1 table puts DeepSeek-V4.1-Flash on top at 90.6% while placing Claude Fable 5 fifteenth at 84.3%. A ~7-point swing for the same family on nominally the same benchmark version is a scaffold-and-sourcing artifact, not a capability finding. Second, BenchLM itself declines to fold this benchmark into its overall model rankings and marks it “display only,” which is a meaningful editorial signal about how much trust to place in self-reported CLI-agent numbers generally. Third, the harness matters as much as the model: Artificial Analysis runs everything through the same Terminus 2 scaffold in an e2b sandbox specifically so that scaffold differences don’t get conflated with model differences — a control that vendor-reported numbers typically don’t disclose.
Why the Numbers Moved So Fast
Sub-65% to high-80s in a matter of months looks, at a glance, like the kind of trajectory that eventually saturates a benchmark — the same pattern already visible in HumanEval, where contamination and ceiling effects made the original benchmark stop being useful. Terminal-Bench’s designers built in some resistance to this: task verification is programmatic rather than judge-based, environments are fresh Docker containers rather than static prompts, and the 2.1 refresh actively closed loopholes rather than just adding harder tasks. Whether that’s enough to keep Terminal-Bench discriminating at the frontier for another year, or whether it follows the same saturation curve as its predecessors, is the open question — and it’s exactly the kind of question the gap between the two tables above is already hinting at.
Terminal-Bench 4.0 Resets the Board
The saturation question the previous section leaves open has now been answered by the maintainers themselves. Terminal-Bench 4.0 is live: a harder 66-task set spanning software, machine learning, science, operations, security, hardware, and media, with recalibrated compute and time allowances, improved instructions, environments and verifiers, and eight tasks removed for being saturated, refusal-prone, publicly solved, or affected by unresolved quality issues, per Artificial Analysis’s methodology notes on its v4.0 leaderboard. It is not a refresh of 2.1 the way 2.1 was a refresh of 2.0 — it is a different, smaller, harder task set, and scores are not comparable across the two.
The reset is severe. Running all 66 tasks through the mini-swe-agent harness at pass@1 averaged over three repeats, Artificial Analysis has Claude Opus 5.5 (Adaptive Reasoning, Max Effort) at 59.6%, tied with the same model at Xhigh effort and with GPT-6 Astra (xhigh), also at 59.6%. Frontier agents that clear 90% on the 89-task 2.1 set land near 60% on 4.0 — which is roughly where 2.0 sat at launch, and is the clearest available evidence that the high-80s numbers on 2.1 reflected a task set the field had caught up to rather than a solved capability. Terminal-Bench 4.0 is also the version Artificial Analysis folds into its Intelligence Index; 2.1 is now tagged legacy on the same site.
The example tasks published with 4.0 signal the difficulty step directly: a zero-downtime MySQL-to-PostgreSQL cutover on a live API server, graded on replayed customer traffic with any 5xx, stale read or lost row failing the task at first occurrence; photonic waveguide routing against physical bend-radius and clearance constraints; and a parametric FreeCAD spring-clip script that must produce a real PartDesign feature tree rather than a baked shape.
How It Differs From SWE-bench, tau-bench, and OSWorld
Terminal-Bench occupies a specific niche in the broader landscape mapped out in our AI agent benchmarks guide. SWE-bench tests an agent’s ability to resolve a specific GitHub issue inside an existing, already-scaffolded repository — a narrower, more code-centric task. tau-bench tests multi-turn tool-calling against simulated APIs in domains like retail and airlines, with no real shell access at all. Terminal-Bench sits at the systems-level end: it wants agents to set up servers, recover from corrupted state, cross-compile for unusual targets, and generally do the kind of ambient sysadmin and build work that a competent engineer does without thinking about it — work that has no single API call to fall back on.
What To Watch
For anyone tracking this benchmark going forward, three things are worth watching rather than any single leaderboard snapshot: whether independent re-runs (Artificial Analysis, or similar third-party harnesses) continue to trail vendor-reported numbers by a consistent margin; whether 4.0 follows 2.1’s trajectory from roughly 60% to the high 80s, and how fast (a separate Terminal-Bench Hard leaderboard also now runs alongside it); and whether the reasoning-effort settings that models are run at (the “xhigh” and “Max Effort” tags scattered through these tables) get standardized, since right now they’re one of the biggest uncontrolled variables between rows.
FAQ
Is Terminal-Bench the same as SWE-bench?
No. SWE-bench scores an agent’s ability to produce a patch that resolves a specific GitHub issue in an existing repository. Terminal-Bench scores end-to-end command-line workflows — server setup, data recovery, cross-compilation — verified by a programmatic test suite rather than a single patch diff. See our SWE-bench breakdown for the contrast.
Why do Terminal-Bench scores vary so much between sources?
Because version (2.0 vs. 2.1), agent harness (e.g., Terminus 2 vs. an unspecified vendor scaffold), sandbox environment, reasoning-effort setting, and number of repeats per task (pass@1 vs. averaged pass@1) are all inconsistently disclosed across trackers. Artificial Analysis and BenchLM’s mirror of the tbench.ai snapshot, compared above, illustrate the gap directly.
Is Terminal-Bench at risk of saturating like HumanEval or MMLU did?
It’s trending that way in raw numbers — sub-65% to high-80s in under a year — but the benchmark’s programmatic verification and containerized environments make it structurally harder to game than static prompt benchmarks like MMLU. The 2.1 “verified refresh,” which patched roughly a dozen tasks specifically to close scoring loopholes, and the September 2026 release of the harder 66-task 4.0 set — on which frontier agents score around 60% — are both evidence the maintainers are actively fighting saturation rather than ignoring it. The pattern to expect is not a permanent ceiling but a recurring reset.
Last updated September 24, 2026. This page is refreshed as benchmarks and scores move.
Pingback: AI Agent Benchmarks: A Map of tau-bench, OSWorld, Terminal-bench and More - Tech Blog
Pingback: State of AI Benchmarks: Q3 2026 - Tech Blog