MLE-bench: Can AI Agents Actually Do Machine Learning Engineering?

The best AI agents now clear OpenAI’s MLE-bench at roughly 64% — up from 17% in 2024 — but that gain depends heavily on which leaderboard and which scaffold you trust.

What MLE-bench Actually Tests

MLE-bench, introduced by OpenAI in “MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering” (arXiv:2410.07095, October 2024), drops an agent into 75 real Kaggle competitions and scores it against the actual medal thresholds other human competitors earned: bronze, silver, gold. Unlike HumanEval or SWE-bench, which grade isolated code snippets or single-repo patches, MLE-bench grades the whole pipeline — data cleaning, feature engineering, model selection, and submission formatting — against a human leaderboard that already exists. The original paper reported that OpenAI’s own o1-preview, scaffolded with the open-source AIDE agent framework, reached bronze-medal level in just 16.9% of competitions.

The Official Leaderboard vs. the Vendor Numbers

The maintained leaderboard on the MLE-bench GitHub README and mirrored at mlebench.com shows Famou-Agent 2.0 running Gemini-3-Pro-Preview at the top with 64.44% (±1.18), narrowly ahead of AIBuildAI running Claude-Opus-4.6 at 63.11%. That’s the progression researchers point to: 17.1% in 2024 (o1-preview + AIDE), 59.6% in 2025 (Famou-Agent 2.0 + Gemini-2.5-Pro), 64.4% in early 2026 (Famou-Agent 2.0 + Gemini-3-Pro-Preview), per the running tally aggregated by aiadoption.org.

Separately, an aggregator site, sugra.systems, lists model-card-reported figures that don’t match the official board at all: Claude Sonnet 5 (released 2026-06-30) at 66.9% on the full benchmark, and Claude Opus 4.6 at 75.7% on the “Lite” subset. These numbers come from vendor model cards, not the standardized leaderboard harness, so scaffold, run length, and task subset all differ — they are not directly comparable to the GitHub-maintained scores, even though they’re frequently quoted as if they were. A third source, benchmarklist.com, ranks Gemini 3.6 Flash first at 63.9% — but that ranking is scoped to open-weight models only, which is a different competition entirely.

Where the Gains Concentrate

For the current leading agent, performance isn’t uniform across difficulty: it succeeds on 80.3% of low-difficulty competitions, 64% of medium-difficulty ones, and 42.2% of high-difficulty ones. That gradient matters more than the single headline score — it shows most of the reported progress comes from clearing easier competitions more reliably, not from suddenly matching top human teams on the hardest ones.

Source Top score Model / agent Scope
GitHub README (official) 64.44% Famou-Agent 2.0 + Gemini-3-Pro-Preview Full leaderboard, early 2026
GitHub README (official) 63.11% AIBuildAI + Claude-Opus-4.6 Full leaderboard, early 2026
sugra.systems (vendor-reported) 66.9% Claude Sonnet 5 Full benchmark, model-card figure
sugra.systems (vendor-reported) 75.7% Claude Opus 4.6 Lite subset, model-card figure
benchmarklist.com 63.9% Gemini 3.6 Flash Open-weight models only

Why the Numbers Disagree

This is a case where disagreement is itself the finding, not noise to average away. The official leaderboard uses a fixed scaffold and run budget so agents are compared on equal footing; vendor model cards report whatever internal harness and prompt scaffold produced the best number, then publish it without the same reproducibility guarantees. A reader comparing “Claude Opus 4.6: 75.7%” against “Famou-Agent 2.0 + Gemini-3-Pro-Preview: 64.44%” is not comparing the same thing — one is a self-reported Lite-subset number, the other a standardized full-benchmark leaderboard entry. Anyone citing MLE-bench scores should specify which board, which subset, and whether the number is independently reproducible before drawing conclusions about which model is “better” at ML engineering.

This same gap between benchmark branding and methodology comparability shows up across the field — see our guide to evaluating LLMs for how to read any benchmark claim with the right skepticism.

What the Benchmark Doesn’t Capture

MLE-bench’s medal-threshold grading has a structural blind spot: it scores against the final leaderboard standings of competitions that already closed, which means every human benchmark score it compares against was produced with unlimited iteration time, team collaboration, and no compute budget constraint reported to the grader. Agents in the benchmark run under a fixed time and token budget (commonly 24-hour or similarly capped runs in the published leaderboard entries), so a score like 64.44% says an agent can approach competitive data-science performance under a hard deadline — not that it matches an unconstrained human team’s process. The original paper’s authors were explicit that MLE-bench is meant to track engineering competence on self-contained tasks, not general research judgment, novel experiment design, or the kind of multi-week iteration real ML teams do in production.

There’s also a scope question worth flagging for anyone using MLE-bench to argue AI agents are close to replacing ML engineers: all 75 competitions are historical Kaggle contests with known datasets, meaning some level of prior exposure during pretraining can’t be fully ruled out for recent frontier models, even though the benchmark’s authors took steps to check for contamination at release in 2024. As with HumanEval’s documented contamination problems, a benchmark’s trustworthiness tends to decay the longer it sits on the public internet with solutions attached.

Practical Implications for Teams Evaluating These Claims

If a vendor cites an MLE-bench score in a model announcement, the useful follow-up question isn’t “is that good?” — it’s “which harness, which subset, and is it on the maintained leaderboard?” A 75.7% figure on the Lite subset and a 64.44% figure on the full official leaderboard are both true statements that can’t be ranked against each other without knowing the scope. Teams building internal evaluation pipelines around ML-engineering agents should treat MLE-bench the way they’d treat SWE-bench’s Lite/Verified/Pro split: as a family of related-but-distinct benchmarks rather than one number, and should re-run the standardized harness themselves on representative tasks before trusting a marketing claim.

How MLE-bench Fits Among Agent Benchmarks

MLE-bench sits in a growing family of benchmarks built specifically to test agentic, multi-step task completion rather than single-turn question answering — the same category as tau-bench for tool-calling and Terminal-bench for agentic coding in a shell environment. Our map of agent benchmarks covers how these evaluations differ in what “success” means: MLE-bench’s medal thresholds are an unusually concrete, externally-validated success criterion compared to many agent benchmarks that rely on LLM-as-judge scoring or binary task completion, which is part of why it’s become a reference point despite the leaderboard-versus-vendor-number confusion documented above.

FAQ

Is MLE-bench the same as SWE-bench? No. SWE-bench grades single-repository bug-fix patches against real GitHub issues; MLE-bench grades end-to-end Kaggle competition submissions against human medal thresholds — a broader, messier task that includes data handling and iteration, not just a patch diff.

Why do vendor-reported MLE-bench scores look higher than the official leaderboard? They typically come from internal model-card evaluations using a different scaffold, prompt strategy, or task subset (often the easier “Lite” set) than the standardized GitHub leaderboard harness, so the numbers aren’t apples-to-apples.

Has any agent matched human gold-medal performance across the board? No current agent clears anywhere near 100% of high-difficulty competitions at gold-medal level; the leading agent’s 42.2% success rate on high-difficulty tasks shows a meaningful gap remains at the hardest end of the benchmark.

Last updated 2026-11-13. This page is refreshed as benchmarks and scores move.

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top