tau-bench Explained: How Tool-Calling Is Really Measured
τ-bench grades AI agents on real tool calls, not answers. Here’s how pass^k works, how τ² and τ³-bench evolved, and why trackers disagree by 30+ points.
τ-bench grades AI agents on real tool calls, not answers. Here’s how pass^k works, how τ² and τ³-bench evolved, and why trackers disagree by 30+ points.
AI engines don’t read your brand name — they resolve it to a graph node. Here’s how entity resolution works, and where it breaks.
Copilot and Cursor both shipped RCE patches and usage-based billing in 2026. Here’s every documented limitation, with sources.
tau-bench, BFCL v4, OSWorld, Terminal-Bench, WebArena, and GAIA each test a different slice of agent behavior — and their leaderboards often disagree.
GPTBot, ClaudeBot, and PerplexityBot don’t execute JavaScript. Measured crawler data shows what that means for AI visibility, and how to fix it.
Perplexity has the lowest citation error rate of 8 AI search tools in the one rigorous study available — 37%. Here’s what that number does and doesn’t mean.
HumanEval scores now cluster at 88-99% across frontier models, but three trackers disagree by 20 points. Here’s the contamination evidence and what replaced it.
Bing confirms its LLMs read schema. Google calls most AI-citation claims about it “wishful thinking.” Here’s what’s actually documented vs. marketing hype.
Gemini 3 Pro’s 10M-token window sounds huge, but MRCR v2 shows retrieval accuracy falling to ~25% at 1M tokens — trailing Claude and GPT-5.4.
MMLU is saturated and frontier labs have stopped reporting it. MMLU-Pro replaced it — but trackers disagree on who leads, and label errors linger.