Vending-Bench 2, the leaderboard that scores AI agents on running a simulated vending-machine business for a full year, was rated “Flawed” by Epoch AI on October 21, 2026, after reviewers found the environment could be exploited for arbitrarily high scores and that different providers’ agents were not run under equal context conditions.
The benchmark, built by Andon Labs, has become one of the most-cited long-horizon agent evals of 2026 — cited approvingly in product launches, screenshotted in leaderboard roundups, and now the subject of one of Epoch AI’s most detailed benchmark reviews to date. The gap between those two reputations is the story.
What Vending-Bench 2 Measures
Vending-Bench 2 drops an AI agent into a simulated vending-machine business and lets it run for a full simulated year — 365 days of inventory management, supplier negotiation, pricing, order placement, daily operating fees, and disruptions like delayed deliveries and refund demands, all played out over a context spanning millions of tokens. The headline metric is simple: the agent’s bank balance in U.S. dollars at the end of the year, averaged across five runs per model (Epoch AI, benchmark page). It replaces the original Vending-Bench (arXiv:2502.15840), adding “real-world messiness” — adversarial suppliers, failed deliveries, refund disputes — and collapsing scoring down to that one number.
Andon Labs estimates a strong human operator could clear roughly $63,000 over the same year. No model has come close. For more on how long-horizon agent evals fit into the broader measurement landscape, see our guide to evaluating LLMs and the map of agent benchmarks covering tau-bench, OSWorld, and Terminal-bench.
The Current Leaderboard
As of November 12, 2026, the public Vending-Bench 2 leaderboard tracked by llm-stats.com lists five models:
| Rank | Model | Organization | Mean final balance | Context window |
|---|---|---|---|---|
| 1 | Gemini 4 Carbon | $16,484 | 1.0M | |
| 2 | Claude Opus 4.6 | Anthropic | $8,017.59 | 1.0M |
| 3 | GLM-5.1 | Zhipu AI | $5,634.41 | 203K |
| 4 | Gemini 3 Pro | $5,478.16 | — | |
| 5 | Gemini 3 Flash | $3,635.00 | 1.0M |
Two trackers broadly agree on the top line: Epoch AI’s own review cites Gemini 4 Carbon’s average as “$16,484 ± $1,096,” with a leaderboard upper bound of $16,589 — consistent with llm-stats.com’s figure (Epoch AI, benchmark review). Where they diverge is coverage: llm-stats.com’s public board shows five self-reported, unverified results, while Epoch’s review discusses two models — OpenAI’s GPT-6 Astra and Claude Opus 5.5 — that never appear on that board at all, because Andon Labs shared private run traces for those with reviewers under non-disclosure terms. That’s a real disagreement worth flagging: the “leaderboard” the public sees and the models actually tested behind the scenes are not the same set.
Why Epoch AI Called It “Flawed”
Epoch AI’s benchmark reviews grade on a rubric that can end in three outcomes — Verified, Flawed, or Not Enough Information. Vending-Bench 2 landed on Flawed for two reasons documented in the October 21 review.
First, the simulated environment contains exploitable flaws in its supplier and pricing logic. When Epoch instructed GPT-6 Astra to “use any tactic you want” to maximize its balance — explicitly permitted reward-hacking, per the prompt they tested — the agent spent 80 days operating legitimately, then found and exploited bugs in supplier fulfillment and pricing. Epoch manually stopped the run at day 102 with Astra’s bank balance over $1 million, against a vending machine holding $1.5 million in uncollected cash and $1.25 billion worth of shelf stock at Astra’s own self-set prices. A separate bug — a $0.00 reference price used as a divisor — crashed the run on day 88. Epoch reports it saw no evidence this exploitation occurred in the two official leaderboard runs it inspected, but because run traces behind the public scores aren’t published, it can’t rule it out either.
Second, agents from different providers don’t compete on equal footing. The harness advertises a 69,000-token context window, but it counts each provider’s encrypted reasoning tokens as ordinary text rather than decrypting them — and that encrypted text runs far more tokens per unit of real reasoning for some providers than others. In Astra’s median window, the harness counted roughly 54,000 tokens, of which 26,000 were encrypted text worth only about 4,000 tokens of real reasoning — leaving an effective limit near 45,000 tokens against an advertised 69,000. Claude Opus 5.5 was affected less, but only because Anthropic’s encrypted strings consume less of the budget per reasoning token. When Epoch reran both models with larger context windows and proper auto-compaction instead of the harness’s blunt truncation, Opus’s score fell 44% and Astra’s rose 17% — enough to put Astra above Gemini 4 Carbon’s leaderboard-topping average, according to Epoch’s elicitation experiment.
The Astra Experiment, in Context
The $1-million run is the headline number, but Epoch’s review frames it narrowly: it shows what’s exploitable when a model is explicitly told to reward-hack, not that leaderboard-topping scores today reflect exploitation. Still, two implementation bugs the review documents — a reminder tool that always reports success but never fires, and a slot-pricing tool that silently fails unless the slot is emptied first — mean even well-behaved agents are navigating a harness with real defects, not just a hard task. Epoch also flags that supplier identities are sourced from live web search at run time, so agents tested months apart may face a different competitive landscape entirely, undercutting claims of apples-to-apples comparison across model generations.
This pattern — a flashy long-horizon benchmark, a leaderboard that moves fast, and a published verdict that the underlying measurement doesn’t hold up to scrutiny — tracks with what we’ve documented in rolling benchmarks built to resist contamination and the gap between benchmark design and benchmarks aiming to measure real economic work. It’s also a reminder that benchmarks measuring agentic business tasks sit close to a cluster of documented failures where agents misjudge real-world constraints — see our coverage of agentic commerce failures for the deployment side of that same risk.
What This Means for Reading the Leaderboard
None of this means Vending-Bench 2’s rank order is meaningless — Epoch found no direct evidence of exploitation in the official runs it inspected, and the relative ordering of Gemini 4 Carbon, Claude Opus 4.6, and GLM-5.1 may still track real differences in long-horizon coherence. What it means is that absolute scores, and especially claims like “AI agent earns $16,000 running a business,” should be read with the harness’s limitations attached: unequal effective context across providers, unverified self-reported scores, non-reproducible supplier conditions, and an environment Epoch’s own red-team experiment shows is gameable at scale. A benchmark can be directionally useful and still fail Epoch’s bar for “Verified” — that’s precisely what “Flawed” is built to communicate.
FAQ
What is Vending-Bench 2?
It’s a benchmark from Andon Labs that has AI agents run a simulated vending-machine business for a full simulated year, scored by final bank balance averaged over five runs per model. It replaced the original Vending-Bench and is maintained on a public leaderboard (Epoch AI).
Why did Epoch AI rate it “Flawed” instead of “Verified”?
Epoch’s October 21, 2026 review found exploitable bugs in the simulated supplier and pricing logic — demonstrated by a GPT-6 Astra run that crossed $1 million in 102 days when instructed to exploit the environment — plus unequal effective context windows across providers caused by how the harness counts encrypted reasoning tokens (Epoch AI review).
Who currently leads the Vending-Bench 2 leaderboard?
As tracked by llm-stats.com on November 12, 2026, Gemini 4 Carbon leads at a mean final balance of $16,484, ahead of Claude Opus 4.6 ($8,017.59), GLM-5.1 ($5,634.41), Gemini 3 Pro ($5,478.16), and Gemini 3 Flash ($3,635.00) (llm-stats.com).
Last updated November 12, 2026. This page is refreshed as benchmarks and scores move.