τ-bench (tau-bench) measures whether an AI agent can complete a multi-turn customer-service task by calling real tools while following written policy — and its headline scores from different trackers routinely disagree by 30-plus points because they’re testing different domains, scaffolds, and benchmark versions, not the same thing.
What tau-bench Actually Tests
τ-bench was introduced by Sierra’s research team (Shunyu Yao, Noah Shinn, Pedram Razavi, Karthik Narasimhan) in a June 2024 paper, “τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains” (arXiv:2406.12045). Unlike static QA benchmarks such as MMLU or GPQA, τ-bench doesn’t hand the model a question and grade the answer. It drops an LLM agent into a simulated customer-service conversation — airline changes, retail returns, telecom troubleshooting — where a second LLM plays the customer, and the agent must use a fixed set of tools (lookup order, issue refund, change seat) while obeying a written policy document it’s given at the start.
Success is graded on the end state of a database, not the transcript: did the refund actually get issued for the correct amount, did the flight actually get rebooked. That structural choice is why τ-bench reads as an “agentic” benchmark rather than a knowledge benchmark, and why it sits alongside tool-calling suites like BFCL and multi-environment benchmarks like AgentBench rather than next to MMLU-Pro.
The Metric Nobody Reads Correctly: pass^k
τ-bench’s most-cited contribution isn’t the tasks — it’s the metric. Most agent benchmarks report pass@k: did the model succeed on at least one of k attempts. τ-bench instead reports pass^k: did the model succeed on every one of k independent attempts at the identical task. The original paper argues pass@k rewards lucky sampling, while pass^k measures something closer to what a business actually needs from a deployed agent — reliability under repetition, not a single good roll. Because pass^k is strictly harder to satisfy as k rises, published pass^1/pass^2/pass^3/pass^4 numbers for the same model always decline as k increases, and how steeply they decline is itself a signal: a model that drops sharply from pass^1 to pass^4 is inconsistent even when its single-shot number looks strong.
From tau-bench to τ²-bench to τ³-bench
The original 2024 release covered two domains, airline and retail, with the agent as the only tool-user and the simulated customer as a passive information source. In 2025, Sierra and Princeton researchers (Victor Barres, Honghua Dong, Soham Ray, Xujie Si, Karthik Narasimhan) published τ²-bench, “Evaluating Conversational Agents in a Dual-Control Environment” (arXiv:2506.07982), adding a telecom domain where the simulated user also holds tools — toggling airplane mode, granting an app permission — so the agent has to coordinate actions rather than just execute them. That paper’s own numbers illustrate why domain choice moves scores dramatically: GPT-4.1’s pass^1 was 74% on retail and 56% on airline, but fell to 34% on the dual-control telecom domain (Barres et al., 2025). Guiding a user through troubleshooting is measurably harder for a model than executing tool calls unilaterally.
The benchmark kept evolving into 2026. The maintained repository (sierra-research/tau2-bench) shipped τ³-bench in March 2026, adding a knowledge-retrieval domain (`banking_knowledge`) documented in the τ-Knowledge paper (arXiv:2603.04370, Shi et al., 2026), a full-duplex voice evaluation mode documented in the τ-Voice paper (arXiv:2603.13686, Ray et al., 2026), and more than 75 task-quality fixes based on the SABER error-analysis paper (Cuadron et al., ICLR 2026 workshop). Each version change is a reason older leaderboard numbers stop being comparable to new ones — the repo’s own versioning notes flag this explicitly, and a July 2026 v1.0.1 grading fix specifically changed scores on the banking_knowledge domain.
Where Two Trackers Land on the Same Benchmark
Here’s the τ²-bench Telecom snapshot from two trackers that both cite the official benchmark but run their own harnesses. BenchLM.ai’s ledger (data verified August 12, 2026) mirrors Artificial Analysis’s independently run telecom implementation for most rows, with named-provider rows for others; Artificial Analysis runs its own evaluation directly and publishes token, cost, and latency data alongside score.
| Model | τ²-bench Telecom score | Source |
|---|---|---|
| GLM-5.2 | 99.1% | BenchLM.ai / Artificial Analysis, Aug 12, 2026 |
| GPT-5.4 | 98.9% | BenchLM.ai, Aug 12, 2026 |
| Claude Fable 5 | 98.5% | BenchLM.ai / Artificial Analysis, Aug 12, 2026 |
| Claude Opus 4.8 | 94.4% | BenchLM.ai / Artificial Analysis, Aug 12, 2026 |
| Gemini 3 Pro | 87.1% | BenchLM.ai / Artificial Analysis, Aug 12, 2026 |
| GPT-5.2 | 84.8% | BenchLM.ai / Artificial Analysis, Aug 12, 2026 |
| Claude Sonnet 4.6 | 79.5% | BenchLM.ai / Artificial Analysis, Aug 12, 2026 |
| Gemini 2.5 Pro | 54.1% | BenchLM.ai / Artificial Analysis, Aug 12, 2026 |
| GPT-4.1 | 47.1% | BenchLM.ai / Artificial Analysis, Aug 12, 2026 |
That’s one harness’s telecom-only numbers, and on their own they look like a saturating benchmark — a dozen-plus models cluster above 90%. But the moment you bring in a second evaluation with a different scaffold, the picture changes. The Holistic Agent Leaderboard (HAL) project (arXiv:2510.11977) ran its own τ-bench Airline evaluations and reported Claude Opus 4 (high reasoning, May 2025 release) at 44.0%, GPT-5 (medium reasoning, August 2025 release) at 30.0%, and Gemini 2.0 Flash (February 2025 release) at 22.0% — all far below the telecom-domain numbers above, for comparably capable models. A third source, the AISBench/EvalScope implementation using a standardized “openai-v1-chat” scaffold, reported airline at 38.0%, retail at 21.05%, and telecom at 33.33% pass^1 (EvalScope docs) — again in a completely different band than either BenchLM/Artificial Analysis or HAL.
Why the Trackers Disagree
None of these three sources are wrong; they’re measuring different things and calling it the same benchmark name. The domain matters — telecom’s dual-control tasks and airline’s single-control tasks are not interchangeable difficulty. The scaffold matters — τ-bench’s own leaderboard submission guide requires disclosing the agent’s harness, the user-simulator model, the prompt, and the trial count, precisely because changing any of those shifts scores independent of underlying model capability. The benchmark version matters — τ³-bench’s task-quality fixes changed which tasks are even gradeable correctly. And trial count matters for pass^k specifically: a pass^1 number from one trial is a different statistical claim than a pass^4 number averaged over four. Sierra’s own submission requirements ask for at least four trials per domain for a reason — fewer trials means more noise in exactly the kind of reliability signal pass^k is built to catch.
The practical implication for anyone using τ-bench scores to pick a model: match domain, scaffold, and version before treating two numbers as comparable, exactly as this site’s practical guide to evaluating LLMs recommends doing for any benchmark. A 99% telecom score and a 44% airline score can both be true statements about capable models — they’re not a contradiction, they’re a demonstration that “tau-bench score” alone is an underspecified claim. The same caution that applies to reading SWE-bench’s Lite/Verified/Pro variants as different benchmarks despite the shared name applies here.
FAQ
Is τ-bench the same as τ²-bench?
No. τ-bench (2024) covers airline and retail with the agent as the only tool-user. τ²-bench (2025) added a dual-control telecom domain where the simulated user also acts through tools, which is why scores on the two aren’t directly comparable (Barres et al., 2025, arXiv:2506.07982).
What does pass^4 mean on a τ-bench leaderboard?
It’s the share of tasks a model solved on all four independent attempts, not just one. It’s a stricter, reliability-focused measure than the more common pass@k, and it will always be lower than or equal to a model’s pass^1 score on the same tasks.
Why do τ-bench scores I find online vary so much for the same model?
Because “τ-bench score” without a domain, benchmark version, agent scaffold, and trial count attached is an incomplete citation. The BenchLM/Artificial Analysis telecom snapshot, the Holistic Agent Leaderboard’s airline runs, and the EvalScope standardized-scaffold runs are all legitimate but non-comparable measurements of different setups.
Last updated August 13, 2026. This page is refreshed as benchmarks and scores move.