LMArena’s leaderboard score is not an Elo rating in the chess sense and not a test-set accuracy figure — it’s a Bradley-Terry coefficient fit to millions of anonymous human pairwise votes, and it has been both gamed by at least one major lab and challenged by an academic study arguing the system structurally favors companies that can afford to test many private variants.
What LMArena actually measures
LMArena (known for most of its life as Chatbot Arena, then LMSYS, then LMArena, and since January 28, 2026 rebranded again to simply “Arena,” per the company’s own announcement “LMArena is now Arena”) runs blind, randomized head-to-head battles: a user submits a prompt, two anonymized models respond, and the user picks a winner, a loser, or a tie. Those pairwise outcomes are fed into a logistic regression that produces what the platform calls an Arena Score. The underlying math is the Bradley-Terry-Luce model, a standard way to estimate relative “strength” from paired comparisons — the same family of models originally used to rank chess players before it was adapted for the Arena’s win/loss votes. The resulting coefficients were, for years, simply labeled “Elo,” a naming choice that stuck in public discussion even after the platform’s own technical write-ups moved to Bradley-Terry terminology.
That distinction matters because it sets expectations: there is no fixed answer key. A model’s score depends entirely on who it was compared against, how many votes it accumulated, and who was voting — a volunteer pool of self-selected users typing prompts of their own choosing, not a curated or weighted sample.
Style Control and Sentiment Control: the platform’s own patch
LMArena has acknowledged since at least August 2024 that raw votes conflate “is this answer good” with “is this answer formatted the way I like.” Its August 28, 2024 style-control post describes adding style as an independent variable in the regression, so response length and markdown formatting each get their own coefficient rather than silently inflating a model’s apparent quality. A follow-up 2025 post, “Does Sentiment Matter Too?”, extended the same idea to emoji count and sentiment. The platform now publishes two parallel leaderboards — “raw” votes and “with Style Control” — and rankings can shift between the two views. LMArena itself describes style control as an “observational” adjustment whose pipeline is still being revised, which is a candid way of saying the fix is partial.
The Llama 4 incident: when the submitted model isn’t the shipped model
In April 2025, Meta submitted an entry to the Arena called Llama-4-Maverick-03-26-Experimental, described in Meta’s own materials as a chat-optimized experimental variant. It placed second on the leaderboard, ahead of GPT-4o and Gemini 2.0 Flash and behind only Gemini 2.5 Pro, as first reported by TechCrunch. The problem: that experimental build was not the model Meta actually shipped to developers. Once the Arena added the public, unmodified Llama 4 Maverick as a separate entry, it ranked 32nd — a roughly 30-place gap between the version that earned the leaderboard placement and the version anyone could download. LMArena’s response was blunt: “Meta’s interpretation of our policy did not match what we expect from model providers,” and the organization tightened its submission rules afterward, per reporting from dig.watch. Meta’s position, through a spokesperson, was that “we experiment with all types of custom variants” — which is true of most labs, but doesn’t resolve whether a custom variant should be allowed to stand in for the public release on a public leaderboard.
The “Leaderboard Illusion” paper
A more structural critique arrived in an April 29, 2025 preprint from researchers affiliated with Cohere, Stanford, MIT, and Ai2 (not yet peer-reviewed). The study analyzed roughly 2.8 million battles across 238 models from 43 providers between November 2024 and March 2025, and argued that well-resourced labs can privately test many model variants on the Arena before choosing which scores to make public — the paper states Meta tested at least 27 Llama 4 variants ahead of its release. The authors also claimed that data derived from Arena battles could be used to improve a model’s score on a related automated benchmark, Arena Hard, by up to 112%, even when the same training didn’t improve performance on external benchmarks — evidence, they argued, that the Arena’s own data can be optimized against rather than used purely for evaluation.
LMArena co-founder Ion Stoica disputed the paper directly, calling it “full of inaccuracies” and “questionable analysis,” and the organization noted that the 112% figure came from an LLM-as-judge benchmark (Arena Hard) rather than from human Arena votes themselves. Google DeepMind researcher Armand Joulin separately said Google had submitted only one model variant for pre-release testing, pushing back on the implication that every large lab plays the same way. The dispute hasn’t been settled by a follow-up peer-reviewed study as of this writing — which is itself worth noting: a benchmark central to how the industry talks about “best model” is operating in a methodological argument that hasn’t been adjudicated.
How this compares to benchmark-style evaluation
| Property | LMArena / Arena Score | Static benchmarks (MMLU, GPQA) | LLM-as-judge (MT-Bench) |
|---|---|---|---|
| What’s scored | Blind human preference, pairwise | Fixed-answer accuracy on a test set | Another model’s judgment of response quality |
| Gameable by | Private multi-variant testing; style/formatting tuning | Training-set contamination | Judge model bias (verbosity, self-preference) |
| Reproducibility | Low — opponent pool and voter pool both shift over time | High — same questions, same answers | Moderate — depends on judge model and prompt |
| Known 2025-2026 controversy | Llama 4 experimental-variant incident; Leaderboard Illusion paper | Saturation and memorization on older sets | Judge-bias documented in multiple papers |
What this means for reading the leaderboard
A 10-30 point gap on the Arena Score between two models is noisier than the single number suggests, given the voting pool’s self-selection and the regression’s confidence intervals — treat close scores as roughly tied rather than ranked. The rebrand to “Arena” and its reported $1.7 billion valuation after a $150 million Series A (announced January 6, 2026) don’t change the underlying math; they change the company’s incentive to keep the leaderboard looking authoritative to a much larger audience. Cross-referencing Arena standing against a static benchmark like MMLU-Pro or GPQA, and against an LLM-as-judge method like MT-Bench, is the only way to triangulate whether a model is actually strong or just well-tuned for what a crowd of self-selected voters tends to reward. For a broader framework on combining these signals, see our guide to how to evaluate LLMs.
FAQ
Is LMArena’s score the same thing as a chess Elo rating?
No. The platform’s current methodology document describes the score as derived from a Bradley-Terry-Luce logistic regression over pairwise battle outcomes; “Elo” is the legacy name that stuck from the original implementation, but the production system has moved on to style- and sentiment-adjusted regression coefficients.
Can a lab game its Arena ranking?
Yes, documented two ways: submitting a customized, non-public model variant optimized for voter preference (the April 2025 Llama 4 Maverick incident, after which LMArena tightened submission policy), and, according to the disputed April 2025 “Leaderboard Illusion” preprint, privately testing many variants and only surfacing the best-scoring one.
Has the Leaderboard Illusion paper been confirmed or refuted?
Neither, as of this writing. It is an unreviewed arXiv preprint; LMArena’s co-founder has called it inaccurate, and at least one named researcher at a large lab (Google DeepMind) has disputed a specific claim about multi-variant testing. No independent peer-reviewed replication has been published.
Last updated 2026-11-13. This page is refreshed as benchmarks and scores move.