MT-Bench proved that GPT-4 could judge chatbot quality with over 80% agreement with human raters — the same level humans get agreeing with each other — but that same 2023 paper also documented the position, verbosity, and self-enhancement biases that 2026 research now shows are harder to remove than the original authors implied.
What MT-Bench actually measures
MT-Bench is a fixed set of 80 multi-turn questions spanning eight categories — writing, roleplay, reasoning, math, coding, extraction, STEM, and humanities/social science. A model answers each question across two conversational turns, and a “judge” (originally GPT-4) scores the response on a 1–10 scale, or picks a winner in a head-to-head pairwise comparison. It shipped alongside Chatbot Arena in the paper “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena” (Zheng et al., submitted June 2023, NeurIPS 2023 Datasets and Benchmarks Track, revised through v4 in December 2023). The paper’s central claim: strong LLM judges agree with human preference over 80% of the time, matching the rate at which humans agree with other humans, which is why LLM-as-a-judge became the default scalable substitute for expensive human evaluation.
That framing is the reason MT-Bench matters beyond its own leaderboard. It’s the paper that legitimized using one model to grade another’s homework — a practice now embedded in RLHF pipelines, model-release evaluations, and most third-party benchmark suites that need to score open-ended text rather than multiple-choice answers.
The biases the original paper already flagged
Zheng et al. didn’t just claim high agreement — they also named the failure modes, and named them clearly enough that they’re still the vocabulary used in 2026 papers: position bias (favoring whichever answer sits in the first slot), verbosity bias (favoring longer answers regardless of quality), and self-enhancement bias (a model preferring outputs from its own family). The paper measured position bias at roughly 10 to 15 points of win-rate swing depending on slot order, and used a “repetitive list” attack — rephrasing an answer’s content into unnecessarily long prose with no new information — to show GPT-4 could be pushed toward the padded version.
| Bias type | What it does | Reported magnitude | Common mitigation |
|---|---|---|---|
| Position bias | Favors the response in a fixed slot (A or B) regardless of content | ~10–15 point win-rate swing (Zheng et al., 2023); “always-A” degenerate behavior in 8 of 9 tested judges in a 2026 rubric-based study (arXiv:2602.02219) | Swap answer order and average scores, or record disagreement as a tie |
| Verbosity bias | Favors longer, padded answers over shorter accurate ones | Demonstrated via “repetitive list” attack in the original paper; a June 2026 large-scale study across 21 judges found verbosity bias comparatively small next to other error sources (arXiv:2606.19544) | Explicit length-normalization instructions in the judge prompt |
| Self-enhancement bias | A judge model rates outputs from its own family more favorably | Self-preference measured at roughly 10–25% in recent audits, though one 2026 analysis (Roytburg et al.) attributes close to 90% of the apparent effect to evaluator uncertainty rather than genuine favoritism | Cross-family judge ensembles; blind the judge to model identity |
Why MT-Bench stopped discriminating frontier models
Eighty questions was enough separation in 2023, when the field was comparing Vicuna and LLaMA variants. It is not enough now. LMSYS’s own follow-up work, “From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder”, reports that Arena-Hard-Auto — its 500-prompt successor curated from live Chatbot Arena traffic — achieves roughly 3x the separability of MT-Bench between models and about 98.6% correlation with human preference rankings. That gap is the practical reason most frontier labs no longer headline MT-Bench scores in model release reports. Checking the current independent tracker at llm-stats.com/benchmarks/mt-bench confirms the pattern from the other direction: as of its last update (July 2026) the leaderboard lists only 12 evaluated models, topped by Hermes 3 70B at roughly 8.99 out of 10 — a thin, mid-tier field, not the frontier-model horse race you’d see on SWE-bench or GPQA trackers today. MT-Bench is still cited for instruction-following sanity checks, but as a tool for telling today’s top models apart, it’s functionally retired.
What 2026 research adds: it’s worse than “known biases”
The bigger problem isn’t the three named biases — it’s that the standard way of validating a judge doesn’t hold up. “Reliability without Validity” (June 2026) ran the largest systematic evaluation of LLM-as-a-judge to date: 21 judges from nine providers, tested on MT-Bench, JudgeBench, and RewardBench under three separate protocols, totaling roughly 541,000 individual judgments across 118 runs. Its headline finding is a methodological one: raw exact-match agreement — the metric MT-Bench popularized — doesn’t correct for chance and systematically overstates how discriminating a judge really is. Once the authors applied Cohen’s kappa instead, agreement on MT-Bench specifically dropped by 33 to 41 percentage points from the exact-match number. The same study found judge rankings shift by up to 14 positions depending on which benchmark is used to evaluate them, and identified a “consistency-bias paradox” in two production-deployed judges — models that were internally consistent but consistently wrong in the same biased direction, which is arguably worse than being randomly noisy.
Position bias research has also gotten more specific. A 2026 study on rubric-based pairwise grading, “Am I More Pointwise or Pairwise?”, found 8 of 9 tested judge configurations defaulted to picking whichever answer sat in slot A — a degenerate pattern, not a mild statistical tilt.
Mitigations that are actually load-bearing
Two techniques show up repeatedly across the 2026 literature as the practical defaults, not because they eliminate bias but because they measurably reduce it:
Position swapping. Evaluate every pair in both orders (A-then-B and B-then-A). In the score-based variant, average the two scores; in the comparison-based variant, treat a flipped verdict as a tie rather than trusting either single run.
Cross-family judge ensembles. Because self-enhancement bias runs along model-family lines, using judges from different labs and aggregating by majority or weighted vote cancels out family-specific priors. Practitioner guidance circulating in mid-2026 (a vendor blog, not a peer-reviewed source, so treat it as practice rather than proof) points to three-judge panels — for example Claude Sonnet 4.5, GPT-5.1, and Gemini 2.5 Pro — as a working default, at roughly 3x the inference cost of a single judge.
Neither fixes the deeper problem “Reliability without Validity” surfaces: even a debiased judge’s agreement rate depends heavily on which chance-correction metric you report. A benchmark’s methodology section listing “80%+ agreement” without specifying exact-match versus kappa is a number worth re-deriving before you cite it.
FAQ
Is MT-Bench still worth running on a new model?
As a cheap sanity check on instruction-following and multi-turn coherence, yes. As a way to rank frontier models against each other, no — its separability is roughly a third of Arena-Hard-Auto’s, per LMSYS’s own benchmarking, and independent trackers now show a thin, mid-tier field rather than current frontier models.
Does self-enhancement bias mean GPT-4 always prefers GPT-4 outputs?
Not always, and not by as much as early reporting suggested. Measured self-preference sits around 10–25% in recent audits, but at least one 2026 re-analysis attributes most of that effect to the judge’s own uncertainty rather than genuine favoritism — a distinction that matters for how much you should discount a same-family judge’s score.
What’s the single biggest methodological problem with LLM-as-a-judge right now?
Chance-correction, not the named biases. The June 2026 large-scale evaluation found that switching from raw exact-match agreement to Cohen’s kappa dropped MT-Bench agreement figures by 33 to 41 percentage points — meaning a headline “80% agreement” claim can mean very different things depending on which metric produced it.
For the broader framework these benchmarks fit into, see our guide to evaluating LLMs, and for how judge unreliability connects to a model making things up outright, see how hallucination is measured. MT-Bench’s saturation story parallels two other benchmarks we’ve covered: HumanEval’s contamination problem and SWE-bench’s tiered difficulty fix. If you’re evaluating tool-calling rather than open-ended chat, tau-bench measures that differently than any judge-scored benchmark can.
Last updated August 28, 2026. This page is refreshed as benchmarks and scores move.