AI Agent Benchmarks: A Map of tau-bench, OSWorld, Terminal-bench and More

No single benchmark measures whether an AI agent works: tau-bench and BFCL v4 test whether it calls the right tool, OSWorld tests whether it can drive a real operating system, WebArena tests whether it can navigate a browser, Terminal-Bench tests whether it can survive a Linux shell, and GAIA shows that the same model can score 30 points apart depending on the scaffolding wrapped around it.

Every one of these benchmarks answers a narrower question than its marketing implies. This is a companion to our general guide to evaluating LLMs, focused specifically on agentic evaluation: what happens when you stop scoring a model on a static question set and start scoring it on actions it takes in an environment. Below is a map of what each benchmark actually measures, the current leaderboard state as of late September 2026, and the places where trackers disagree — because on agent benchmarks, disagreement is usually the most informative data point.

Tool-Calling: BFCL v4 and tau-bench/tau2-bench

The narrowest and most mature slice of agent evaluation is tool-calling: given a query and a set of function schemas, does the model pick the right function and fill in valid arguments? The Berkeley Function-Calling Leaderboard (BFCL) v4, maintained by UC Berkeley’s Gorilla project, is the reference test here. It uses Abstract Syntax Tree matching to score function-call accuracy and penalizes hallucinated functions not in the provided schema. As of its September 2026 update llm-stats.com’s mirror of the leaderboard covers 22 evaluated models and has Shanghai AI Laboratory’s Atria Dawn Preview on top at 0.770, ahead of Qwen3.7 Max at 0.750 — the model that led the table two months earlier. BenchLM.ai’s September snapshot of the same benchmark disagrees at the top, listing BTL-3 at 88.5% with Atria Dawn Preview second at 77.0%, an 11-point gap on a benchmark whose scoring is supposedly a deterministic AST match. Even the narrowest, most mechanical slice of agent evaluation has a tracker-disagreement problem.

Sierra’s tau2-bench (successor to the original tau-bench) goes a step further: it doesn’t just check if the right function was called, it simulates a full multi-turn conversation between an agent, a synthetic user, and a policy document, across retail (114 tasks) and airline (50 tasks) domains — 165 tasks in total. The headline metric is pass^k, which reruns each task k times and only counts it as solved if the agent succeeds every time, a much harder bar than one-shot pass@1. Per independent tracking of the retail and airline splits, frontier models’ pass^1 scores sit below 70%, and llm-stats.com’s July 2026 tau-bench leaderboard lists StepFun’s Step-3.5-Flash as the top open-source entrant at 0.882 — a reminder that “leads tau-bench” and “clears 70%” are two different claims that get conflated constantly.

Computer-Use: OSWorld and the Human Baseline Question

OSWorld tests whether an agent can operate a real desktop — 369 tasks spanning Ubuntu, Windows, and macOS workflows: opening applications, editing files, configuring settings, using a mouse and keyboard the way a person would. It’s the benchmark most often cited as evidence that agents have crossed “human-level,” because its published human baseline is 72.36%. Two independent trackers, Steel.dev and BenchLM.ai, both put the top OSWorld-Verified entries in the 83–85% range as of late July 2026 (Claude Fable 5 and Claude Opus 4.8 among the leaders), comfortably above that human number.

The catch, which both trackers note explicitly, is that OSWorld leaderboard rows are system submissions — a model plus a specific agent harness, retry budget, and tool scaffold — not a raw model capability score. A model tested with a thin scaffold can score dramatically lower than the same model wrapped in a more elaborate one, which is exactly the pattern GAIA makes explicit below.

Terminal and Coding Agents: Terminal-Bench 2.1 and the 4.0 Reset

Terminal-Bench (see our full explainer for task design, version history, and how independent trackers’ numbers diverge from the official leaderboard), built by a Stanford-and-Laude collaboration, tests end-to-end terminal workflows — compiling a kernel from source, standing up a Git server with a working webhook, cracking a password hash, training a model under a size constraint — rather than isolated code generation, which is the gap it’s designed to close relative to benchmarks like SWE-bench and, further upstream, pure competitive-programming tests like LiveCodeBench. The official [email protected] leaderboard (verified by the Terminal-Bench team) is unusually transparent about cost, and the cost data is the real story: Claude Code running Claude Fable 5 tops the board at 83.8% ± 1.2% (accuracy, June 7, 2026) for $552.67 per full run, while Codex running GPT-5.5 sits essentially tied at 83.1% ± 1.1% (May 1, 2026) but costs $2,059.19 — nearly 4x more for a statistically indistinguishable score. Further down the board, Cursor CLI running Grok 4.5 posts 79.3% but carries a -9.0% “hacks” penalty, the leaderboard’s flag for reward-hacking behavior detected during grading, versus -0.0% to -0.9% for most other entries — one of several documented issues in our GitHub Copilot vs Cursor limitations comparison.

Those 2.1 numbers now describe a superseded task set. Terminal-Bench 4.0 is live: 66 harder tasks across software, machine learning, science, operations, security, hardware and media, with recalibrated compute and time allowances and eight tasks dropped for being saturated, refusal-prone, publicly solved or otherwise unsound. Run through the mini-swe-agent harness at pass@1 over three repeats, Artificial Analysis has Claude Opus 5.5 (Max Effort) at 59.6%, tied with GPT-6 Astra (xhigh) also at 59.6%. Agents clearing 90% on 2.1 land near 60% on 4.0 — the single cleanest illustration on this page that an agent benchmark score is a statement about a task set at a moment, not about a model.

Web Agents: WebArena

WebArena evaluates browser agents on 812 natural-language tasks across four self-hosted web applications (a Reddit clone, a GitLab-style code host, an e-commerce site, and a content-management system), which makes it more reproducible than agents scraping the live, ever-changing web — and it inherits the same rendering problem we’ve covered in why JavaScript-rendered content is invisible to AI: an agent that can’t parse a client-rendered page fails a WebArena task for reasons that have nothing to do with reasoning ability. Its human baseline is roughly 78%, and as of the latest tracked results the leading general-purpose entries — Claude Mythos Preview at 68.7%, GPT-5.4 Pro at 65.8%, Claude Opus 4.6 at 64.5% — remain below it, while the top tracked system entry, WebTactix running DeepSeek v3.2, reaches 74.3% (Steel.dev, last tracked June 29, 2026) — a purpose-built browser-agent stack closing most of the gap to the human baseline that general-purpose models have not closed. That gap between general-purpose agents and human performance is a useful counterweight to OSWorld’s above-human headline: whether agents “beat humans” depends heavily on which environment you put them in.

The Meta-Problem: GAIA and Scaffold-Dependent Scores

GAIA is designed to test general assistant capability — reasoning, web browsing, tool use, and multimodal understanding combined — but it has become the clearest illustration of why agent benchmarks resist a single number. Princeton’s Holistic Agent Leaderboard (HAL) runs GAIA two ways: a “scaffolded” leaderboard that lets agents use a full tool stack, where Claude Sonnet 4.5 leads at 74.6% and Anthropic models occupy the top six slots, and a “bare model” leaderboard that strips scaffolding to test intrinsic ability, where GPT-5 Mini tops out at 44.8% (as of July 7, 2026), with Gemini 2.5 Pro at 33.3%. Meanwhile Steel.dev’s system-level leaderboard, which allows fuller agent orchestration, shows OPS-Agentic-Search at 92.36%. Same benchmark, same underlying models in some cases, and a roughly 30-to-60-point spread depending entirely on what harness sits around the model.

That’s a scaffold artifact, not a hidden reasoning breakthrough, and it belongs alongside the other documented gaps between marketed and measured performance we track in the Limitations Ledger. It’s the pattern HAL was built to expose. Its own evaluation runs — about $40,000 spent on 21,730 agent rollouts across 9 models and 9 benchmarks — treat cost and reliability as first-class metrics alongside accuracy, precisely because a separate $22,000 sweep of agent configurations (cited by the EvalEval Coalition) found a 33x cost spread on identical tasks driven entirely by scaffold choice, not model choice.

Benchmark Comparison

Benchmark	What it measures	Scale	Top reported score (September 2026)	Source
BFCL v4	Single-turn function/tool-call accuracy	22 models evaluated	0.770 (Atria Dawn Preview); BenchLM snapshot instead lists BTL-3 at 88.5%	Gorilla / llm-stats.com / BenchLM.ai
tau2-bench	Multi-turn tool use with policy compliance, pass^k	165 tasks (retail + airline)	0.882 open-source (Step-3.5-Flash); frontier pass^1 <70%	llm-stats.com / benchmarkingagents.com
OSWorld-Verified	Real desktop OS operation (GUI, files, apps)	369 tasks	86.1% (Qwen3.8 Max), 24 models tracked; human baseline 72.36%	Steel.dev / BenchLM.ai / llm-stats.com
Terminal-Bench 2.1	End-to-end terminal/shell workflows	89 tasks (v2.0 core set)	91.4% (Claude Fable 5.1, Max Effort), independently run	tbench.ai (official) / Artificial Analysis
Terminal-Bench 4.0	Harder terminal work incl. science, hardware, media	66 tasks	59.6% (Claude Opus 5.5 Max Effort; GPT-6 Astra xhigh tied)	Artificial Analysis (mini-swe-agent)
WebArena	Browser navigation across self-hosted web apps	812 tasks	74.3% system-level (WebTactix + DeepSeek v3.2); 68.7% best general-purpose model (Claude Mythos Preview); human baseline ~78%	Steel.dev
τ³-Banking	Tool use grounded in a large unstructured policy corpus	97 tasks, ~700 policy documents	51.3% pass@1 (Qwen3.8 Max) vs ~25.5% pass^1 in the originating paper	Artificial Analysis / Sierra Research
GAIA	General assistant reasoning + tool use + browsing	3 difficulty levels	74.6% scaffolded / 44.8% bare model / 92.36% system-level	HAL (Princeton) / Steel.dev

FAQ

Is there one benchmark that tells you if an AI agent is “good”?
No. Each benchmark isolates a different failure surface — tool selection (BFCL v4), policy-compliant multi-turn dialogue (tau2-bench), GUI operation (OSWorld), shell/terminal survival (Terminal-Bench), browser navigation (WebArena), or general reasoning-plus-tools (GAIA). A model can lead one and trail badly on another.

Why do agents sometimes score above the “human baseline”?
Human baselines are usually measured under time pressure or with limited familiarity with the test environment, and agent scores are system scores (model plus harness plus retries), not raw model scores. OSWorld’s above-baseline leaders and WebArena’s below-baseline leaders are both real, just measuring different margins for error.

Why does the same model get such different GAIA scores across leaderboards?
Because GAIA leaderboards differ in what scaffolding they allow. HAL’s bare-model and scaffolded leaderboards for the same models can differ by 20–30 points, and system-level leaderboards that permit fuller orchestration report even higher numbers — the score reflects the harness at least as much as the model.

Last updated September 24, 2026. This page is refreshed as benchmarks and scores move.