No single benchmark measures whether an AI agent works: tau-bench and BFCL v4 test whether it calls the right tool, OSWorld tests whether it can drive a real operating system, WebArena tests whether it can navigate a browser, Terminal-Bench tests whether it can survive a Linux shell, and GAIA shows that the same model can score 30 points apart depending on the scaffolding wrapped around it.
Every one of these benchmarks answers a narrower question than its marketing implies. This is a companion to our general guide to evaluating LLMs, focused specifically on agentic evaluation: what happens when you stop scoring a model on a static question set and start scoring it on actions it takes in an environment. Below is a map of what each benchmark actually measures, the current leaderboard state as of late July 2026, and the places where trackers disagree — because on agent benchmarks, disagreement is usually the most informative data point.
Tool-Calling: BFCL v4 and tau-bench/tau2-bench
The narrowest and most mature slice of agent evaluation is tool-calling: given a query and a set of function schemas, does the model pick the right function and fill in valid arguments? The Berkeley Function-Calling Leaderboard (BFCL) v4, maintained by UC Berkeley’s Gorilla project, is the reference test here. It uses Abstract Syntax Tree matching to score function-call accuracy and penalizes hallucinated functions not in the provided schema. As of its July 2026 update it covers 13 evaluated models; llm-stats.com’s mirror of the leaderboard has Alibaba’s Qwen3.7 Max on top at 0.750, with the open-weight Qwen3.5-397B-A17B close behind at 0.729.
Sierra’s tau2-bench (successor to the original tau-bench) goes a step further: it doesn’t just check if the right function was called, it simulates a full multi-turn conversation between an agent, a synthetic user, and a policy document, across retail (114 tasks) and airline (50 tasks) domains — 165 tasks in total. The headline metric is pass^k, which reruns each task k times and only counts it as solved if the agent succeeds every time, a much harder bar than one-shot pass@1. Per independent tracking of the retail and airline splits, frontier models’ pass^1 scores sit below 70%, and llm-stats.com’s July 2026 tau-bench leaderboard lists StepFun’s Step-3.5-Flash as the top open-source entrant at 0.882 — a reminder that “leads tau-bench” and “clears 70%” are two different claims that get conflated constantly.
Computer-Use: OSWorld and the Human Baseline Question
OSWorld tests whether an agent can operate a real desktop — 369 tasks spanning Ubuntu, Windows, and macOS workflows: opening applications, editing files, configuring settings, using a mouse and keyboard the way a person would. It’s the benchmark most often cited as evidence that agents have crossed “human-level,” because its published human baseline is 72.36%. Two independent trackers, Steel.dev and BenchLM.ai, both put the top OSWorld-Verified entries in the 83–85% range as of late July 2026 (Claude Fable 5 and Claude Opus 4.8 among the leaders), comfortably above that human number.
The catch, which both trackers note explicitly, is that OSWorld leaderboard rows are system submissions — a model plus a specific agent harness, retry budget, and tool scaffold — not a raw model capability score. A model tested with a thin scaffold can score dramatically lower than the same model wrapped in a more elaborate one, which is exactly the pattern GAIA makes explicit below.
Terminal and Coding Agents: Terminal-Bench 2.1
Terminal-Bench, built by a Stanford-and-Laude collaboration, tests end-to-end terminal workflows — compiling a kernel from source, standing up a Git server with a working webhook, cracking a password hash, training a model under a size constraint — rather than isolated code generation, which is the gap it’s designed to close relative to benchmarks like SWE-bench. The official [email protected] leaderboard (verified by the Terminal-Bench team) is unusually transparent about cost, and the cost data is the real story: Claude Code running Claude Fable 5 tops the board at 83.8% ± 1.2% (accuracy, June 7, 2026) for $552.67 per full run, while Codex running GPT-5.5 sits essentially tied at 83.1% ± 1.1% (May 1, 2026) but costs $2,059.19 — nearly 4x more for a statistically indistinguishable score. Further down the board, Cursor CLI running Grok 4.5 posts 79.3% but carries a -9.0% “hacks” penalty, the leaderboard’s flag for reward-hacking behavior detected during grading, versus -0.0% to -0.9% for most other entries.
Web Agents: WebArena
WebArena evaluates browser agents on 812 natural-language tasks across four self-hosted web applications (a Reddit clone, a GitLab-style code host, an e-commerce site, and a content-management system), which makes it more reproducible than agents scraping the live, ever-changing web — and it inherits the same rendering problem we’ve covered in why JavaScript-rendered content is invisible to AI: an agent that can’t parse a client-rendered page fails a WebArena task for reasons that have nothing to do with reasoning ability. Its human baseline is roughly 78%, and as of late June 2026 the leading general-purpose entries — Claude Mythos Preview at 68.7%, GPT-5.4 Pro at 65.8%, Claude Opus 4.6 at 64.5% — remain below it, while specialized reinforcement-learned frameworks report 71–74%. That gap between general-purpose agents and human performance is a useful counterweight to OSWorld’s above-human headline: whether agents “beat humans” depends heavily on which environment you put them in.
The Meta-Problem: GAIA and Scaffold-Dependent Scores
GAIA is designed to test general assistant capability — reasoning, web browsing, tool use, and multimodal understanding combined — but it has become the clearest illustration of why agent benchmarks resist a single number. Princeton’s Holistic Agent Leaderboard (HAL) runs GAIA two ways: a “scaffolded” leaderboard that lets agents use a full tool stack, where Claude Sonnet 4.5 leads at 74.6% and Anthropic models occupy the top six slots, and a “bare model” leaderboard that strips scaffolding to test intrinsic ability, where GPT-5 Mini tops out at 44.8% (as of July 7, 2026), with Gemini 2.5 Pro at 33.3%. Meanwhile Steel.dev’s system-level leaderboard, which allows fuller agent orchestration, shows OPS-Agentic-Search at 92.36%. Same benchmark, same underlying models in some cases, and a roughly 30-to-60-point spread depending entirely on what harness sits around the model.
That’s a scaffold artifact, not a hidden reasoning breakthrough, and it belongs alongside the other documented gaps between marketed and measured performance we track in the Limitations Ledger. It’s the pattern HAL was built to expose. Its own evaluation runs — about $40,000 spent on 21,730 agent rollouts across 9 models and 9 benchmarks — treat cost and reliability as first-class metrics alongside accuracy, precisely because a separate $22,000 sweep of agent configurations (cited by the EvalEval Coalition) found a 33x cost spread on identical tasks driven entirely by scaffold choice, not model choice.
Benchmark Comparison
| Benchmark | What it measures | Scale | Top reported score (July 2026) | Source |
|---|---|---|---|---|
| BFCL v4 | Single-turn function/tool-call accuracy | 13 models evaluated | 0.750 (Qwen3.7 Max) | Gorilla / llm-stats.com |
| tau2-bench | Multi-turn tool use with policy compliance, pass^k | 165 tasks (retail + airline) | 0.882 open-source (Step-3.5-Flash); frontier pass^1 <70% | llm-stats.com / benchmarkingagents.com |
| OSWorld-Verified | Real desktop OS operation (GUI, files, apps) | 369 tasks | ~85% (Claude Fable 5); human baseline 72.36% | Steel.dev / BenchLM.ai |
| Terminal-Bench 2.1 | End-to-end terminal/shell workflows | 89 tasks (v2.0 core set) | 83.8% ± 1.2% (Claude Code + Fable 5) | tbench.ai (official) |
| WebArena | Browser navigation across self-hosted web apps | 812 tasks | 68.7% (Claude Mythos Preview); human baseline ~78% | Steel.dev |
| GAIA | General assistant reasoning + tool use + browsing | 3 difficulty levels | 74.6% scaffolded / 44.8% bare model / 92.36% system-level | HAL (Princeton) / Steel.dev |
FAQ
Is there one benchmark that tells you if an AI agent is “good”?
No. Each benchmark isolates a different failure surface — tool selection (BFCL v4), policy-compliant multi-turn dialogue (tau2-bench), GUI operation (OSWorld), shell/terminal survival (Terminal-Bench), browser navigation (WebArena), or general reasoning-plus-tools (GAIA). A model can lead one and trail badly on another.
Why do agents sometimes score above the “human baseline”?
Human baselines are usually measured under time pressure or with limited familiarity with the test environment, and agent scores are system scores (model plus harness plus retries), not raw model scores. OSWorld’s above-baseline leaders and WebArena’s below-baseline leaders are both real, just measuring different margins for error.
Why does the same model get such different GAIA scores across leaderboards?
Because GAIA leaderboards differ in what scaffolding they allow. HAL’s bare-model and scaffolded leaderboards for the same models can differ by 20–30 points, and system-level leaderboards that permit fuller orchestration report even higher numbers — the score reflects the harness at least as much as the model.
Last updated July 31, 2026. This page is refreshed as benchmarks and scores move.
Pingback: How to Evaluate LLMs: A Practical Guide to Benchmarks, Metrics, and Methodology - Tech Blog
Pingback: SWE-bench Explained: Original vs Lite vs Verified vs Pro - Tech Blog
Pingback: The Limitations Ledger: Documented Failure Modes of Frontier AI Tools - Tech Blog