A "#1 on the leaderboard" claim is true about one specific test, run one specific way, on one specific day. It says much less about whether that model is the best one for your work. Before you switch tools or pay for a bigger plan because of a ranking, find out which board it came from, who ran it, and what it measures.
This matters more in 2026 because leaderboards became a business. Arena, the public ranking site formerly known as LMArena, raised $200 million at a $3.1 billion valuation this year, nearly double the $1.7 billion it was valued at earlier in 2026 (Phemex). Arena says its annualized revenue has passed $100 million (Arena). Every model launch now leads with a rank, and the rank sells.
None of this means the boards are fake. Most are built by serious people and they measure something real. The trouble is that each one measures a different something, and a launch post will quote whichever one flatters the model. This guide explains how the main boards work, where each one bends, and how to read a claim in about two minutes.
Four kinds of board, four different questions
Most rankings you'll meet fall into one of four groups. The groups matter more than any single board, because a number only means something once you know which group it came from.
Human-preference arenas ask people which answer they like better. Independent suites run a fixed set of tests on every model under the same conditions. Task benchmarks check one skill, such as fixing real software bugs, with an automatic grader. Vendor tables are the scores a lab prints in its own launch post or model card, usually with settings the lab picked.
The groups don't compete so much as answer different questions. Preference tells you which model people enjoy talking to. A suite tells you how models compare when somebody neutral holds the stopwatch. A task benchmark tells you whether a model can do one narrow job. A vendor table tells you what the lab wants you to notice.
| Board type | Example | What it measures | Main weak spot |
|---|---|---|---|
| Human preference | Arena text and WebDev | Which answer people vote for | Rewards style and length, can be gamed |
| Independent suite | Artificial Analysis, Epoch AI hub | Scores on a fixed test set | Only as good as the tests it picked |
| Task benchmark | SWE-bench, Humanity's Last Exam | One skill with automatic grading | Contamination, flawed tasks, saturation |
| Vendor table | Launch posts, model cards | Whatever the lab chose to report | Settings and prompts you can't check |
How an arena vote becomes a rating
Arena began as Chatbot Arena, a research project at UC Berkeley. The idea is simple. You type a prompt, two anonymous models answer, and you pick the better reply. The researchers' original paper describes a platform that had collected more than 240,000 votes in its first months and found that crowd votes agreed well with expert judgments (arXiv).
Each vote is one tiny record: model A beat model B on this prompt. Millions of those records go into a statistical model that assigns every model a single score. Arena's own write-ups describe the method as a Bradley-Terry regression, a close cousin of the Elo system used in chess (LMSYS). A model's score goes up when it beats strong opponents and down when it loses to weak ones, and the text leaderboard shows an interval next to each score (a "±" figure), not just the point estimate.
That interval is the part launch posts skip. If two models sit a few points apart and their intervals overlap, the honest reading is a tie. A "#1" that rests on a gap of two or three points is usually a statistical coin flip.
Arena has also grown well past chat. Its leaderboard now has separate boards for text, WebDev, vision, search, image generation and editing, and video, and a newer agent board that reports cost per task next to the win rate (Arena). The cost column is worth reading. A model that ranks one place higher at several times the price is a different recommendation.
What an arena can't tell you
Preference voting has three well-documented problems. The first is style. People like long, well-formatted answers, so a model can climb by writing more and adding headers. The Arena team's own analysis found answer length was the dominant style effect, and that correcting for length and markdown moved models around. In that 2024 analysis, GPT-4o-mini fell from sixth to eleventh place while Claude 3.5 Sonnet rose from sixth to fourth (LMSYS). Arena now offers style-controlled views, and it's worth using them.
The second problem is access. A 2025 paper called "The Leaderboard Illusion", with authors including Sayash Kapoor and Noah Smith, argued that big labs got structural advantages on Arena. It reported that Meta tested 27 private model variants before the Llama 4 release and published only the best score, and it estimated that Google and OpenAI received about 19.2% and 20.4% of all Arena data, while 83 open-weight models together received about 29.7% (arXiv).
The paper also reported that training on Arena-style data can lift performance on the Arena distribution by up to 112%, which is an incentive to tune for the board and not for general ability. These are the authors' estimates and claims, and they reflect a snapshot from when the paper was written. They are still the clearest outside description of how a preference board can be gamed without anyone breaking a rule.
The third problem is that preference isn't correctness. A confident, friendly answer can win a vote and still contain a wrong fact. For chat feel and writing, preference is a fair guide. For code that must run or a figure that must be right, it's a weak one.
Arena's current policy page addresses some of this. It says pre-release models are tested under anonymous names until scores stabilize (a minimum of 1,000 votes), that those scores are shared privately with the provider, and that a model's score is marked "preliminary" after public release until fresh votes arrive. It also says every battle includes at least one publicly available model and that at least 20% of battles involve only public models (Arena). The practical rule: a "preliminary" tag means the number isn't settled yet.
Independent suites: Artificial Analysis and Epoch AI
If preference boards ask what people like, independent suites ask what a model can do when everyone gets the same test. Artificial Analysis is the best-known one. Its Intelligence Index, now at version 4.3.2, combines 10 evaluations in four groups: agents (30% of the weight), general knowledge and reasoning (30%), coding (20%) and scientific reasoning (20%). Every model is run under the same conditions with pass@1 scoring, meaning one attempt per question, and the site says it keeps its own copies of the datasets. For subjective grading it uses a panel of three frontier models from different labs. The index is mostly English and text, and the site benchmarks multilingual and multimodal ability separately (Artificial Analysis).
The weights are an editorial choice. A different weighting, with less on agents and more on coding, would reorder some models. That's not a flaw, but it means the "overall" number is one opinion about what matters. If you care mostly about coding, read the coding results, not the headline.
Epoch AI runs a benchmarking hub that mixes tests it administers itself, such as GPQA Diamond, FrontierMath and SWE-bench Verified, with results from outside sources. Its most useful finding for readers is about vendor numbers. Claude 3.5 Sonnet was reported at 65% on GPQA Diamond. Epoch measured 55%, plus or minus 3, across 16 runs, and noted that labs often don't publish the prompt or temperature they used (Epoch AI). A 10-point gap from setup alone is a good reason to prefer reruns by a third party. The hub is funded by the UK AI Security Institute and its data is free to reuse.
Independent doesn't mean unbiased. A third party still picks the tests, and any index that includes LLM judges inherits the judges' habits. Arena's own study of AI judges found that models favor their own answers. Across 1,460 Text Arena battles judged by 12 models, the judges picked their own responses about 70% more often than humans did (Arena).
Benchmarks wear out
Task benchmarks have the opposite problem. They're objective, but they age fast. A benchmark launches hard, labs optimize for it, scores climb, the test leaks into training data, and soon every top model sits near the ceiling and the ranking stops meaning anything.
Humanity's Last Exam is a good example of the response. It has 2,500 questions across more than a hundred subjects, written by nearly 1,000 experts at over 500 institutions, and it keeps a private test set to catch overfitting. The authors built it because older tests like MMLU had been pushed past 90% by frontier models (arXiv). It was published in Nature in 2026, and even its creators expected scores to climb quickly.
SWE-bench shows the full lifecycle. The original benchmark had 2,294 real GitHub issues from 12 Python projects, and the best model of the time, Claude 2, solved 1.96% of them (per the original paper). A cleaned 500-task subset called Verified became the standard number in coding launches. Then the audit arrived.
OpenAI audited 138 Verified problems that its o3 model did not reliably solve, each reviewed by at least six engineers. It found that 59.4% had material problems in their tests or descriptions. In a separate contamination test, GPT-5.2, Claude Opus 4.5 and Gemini 3 Flash Preview could each reproduce details of original fixes, a sign the tasks had leaked into training data. OpenAI says it has stopped reporting Verified scores and recommends SWE-bench Pro instead (OpenAI). AI Weekly's summary adds that top models sat around 80% on Verified and roughly 35 points lower on Pro, and that a later outside audit questioned Pro's graders too (AI Weekly).
The lesson isn't that coding benchmarks are useless. It's that a score from a retired or leaky test is an old number. When a launch post quotes SWE-bench, check the variant, and check what agent scaffolding wrapped the model. Arena's "HarnessTax" study of 21 model and scaffolding pairs found that the choice of scaffolding had little effect on task success but changed cost by up to 5x for similar results, on a small sample of 30 tasks per benchmark (Arena).
A checklist for any #1 claim
Here's the two-minute version. Work through it before you believe a ranking, and especially before you spend money because of one.
- Find the exact board and version. "Tops the benchmarks" isn't a claim you can check. "First on Arena text with style control" is.
- Find who ran it. A lab's own run on its own prompts is a marketing number. A rerun by a third party carries more weight.
- Read the interval and the vote count. If the lead is smaller than the interval, it's a tie.
- Check whether the model is public. A score for something you can't try today can't help you choose.
- Ask whether the test matches your task. Chat preference won't predict how a model handles your spreadsheets or your codebase.
- Look for a second, different board that agrees. One board can be wrong or gamed. Two unrelated ones agreeing is much harder to fake.
- Put price and speed next to the score. The cheaper model that sits two places lower is often the better buy.
You can score a claim yourself with the checklist below. It's weighted toward the things that are hardest to fake, such as third-party runs and agreement across boards.
Match the board to your question
Because the boards answer different questions, the useful move is to start from your question and pick the board second. If you're choosing an assistant for writing and conversation, a preference arena is a decent first filter. If you want a repeatable comparison of raw ability, use an independent suite. If you're picking a coding tool, look at task benchmarks built on real repositories and then test on your own code.
The tree gives a starting point, not a verdict. Whatever board you land on, a short test on your own work beats any public rank. Take 10 real tasks you do every week, run them through two or three finalists, and compare the outputs side by side. It takes an afternoon and it measures the one thing no leaderboard can, which is your work.
To shop by category, the ToolPotion AI Models & LLMs directory and the AI models board list models side by side, and the guide to matching AI model tiers to tasks covers when a flagship is worth paying for. For a broader roundup, see Top 15 AI Models & LLMs in 2026, or ChatGPT alternatives in 2026 if one assistant is your starting point.
Where the numbers are least reliable
A few situations call for extra suspicion. New models in their first days have fewer votes and wider intervals, and Arena marks those scores preliminary for a reason. Open-weight models, the kind you can download from Hugging Face, are the other blind spot: the Leaderboard Illusion authors estimated they get a smaller share of Arena data, so their scores can be noisier. Hosted models can also change behind the same name, so a score from launch week may not describe the model you use months later.
Specialized work is the biggest gap. Legal drafting, medical coding, a niche language and a company's internal codebase are all poorly covered by public boards. Real-SWE Benchmark is one answer: it evaluates frontier models on private enterprise codebases, which are less likely to have leaked into training data than public repositories are. Treat a board like that as a useful second opinion, not a replacement for your own test.
And if you're a casual user, you may not need any of this. For everyday writing and questions, the top handful of models from OpenAI, Anthropic and Google are close enough that the choice will come down to price, interface and habit. Spending an hour on rankings to pick between near-ties is the wrong trade.
Frequently asked questions
Is Arena the same as LMArena?
Yes. LMArena grew out of the Chatbot Arena research project at UC Berkeley and now goes by Arena, at arena.ai. The older name still turns up in articles and in older papers.
Which leaderboard is the most trustworthy?
None is trustworthy in every case, and the question depends on what you want to know. For repeatability, a third-party suite that runs every model itself is the strongest. For what people prefer, Arena has the most data. The most reliable evidence is two unrelated boards agreeing, and then a short test on your own tasks.
What is benchmark contamination?
It's when a model has seen a test's questions or answers during training, so a high score reflects memory instead of skill. OpenAI's audit of SWE-bench Verified found frontier models could reproduce original fixes for some tasks. Private test sets, like the one Humanity's Last Exam keeps, are one defense.
Do vendors cheat on leaderboards?
The evidence supports a narrower claim. Labs choose which scores to publish, which settings to use and, according to the Leaderboard Illusion paper, which private variants to reveal. That's selective reporting inside the rules, not necessarily rule-breaking. It's the reason independent reruns matter.
Should I switch models every time a new one hits #1?
No. A rank gap often sits inside the margin of error, and switching has real costs in prompts, workflows and testing. Switch when a model beats your current one on your own tasks, or when it matches it at a clearly lower price.
