SOTA or Luck? The Winner’s Curse in LLM Leaderboards
Xu et al.
Ranking noisy benchmark scores overstates the leaders. An \(m\)-out-of-\(N\) bootstrap reduces the ranking bias from order \(N^{-1/2}\) to \(o(N^{-1/2})\) in \(N\) tasks.
Source published · In the Daily Brief of 19 November 2026
Filed under Learning theory, Probability
Leaderboards rank models by average scores over \(N\) benchmark tasks, and selecting the top of a noisy ranking rewards luck as well as ability. The authors call the resulting distortion rank-induced bias and correct it with an \(m\)-out-of-\(N\) bootstrap, assuming independent, identically distributed bounded tasks but allowing arbitrary correlation between models on the same task.
When the population scores fall into tiers separated by positive gaps, they prove the bootstrap cuts the bias from \(\Theta(N^{-1/2})\) to \(o(N^{-1/2})\). Across 14 benchmarks the correction to the leader ranges from about 0.09 to 3.86 percentage points; on SWE-bench Verified, 13 of 173 models land in the leading tier.