Learning theory Papers

SOTA or Luck? The Winner’s Curse in LLM Leaderboards

Xu et al.

Ranking noisy benchmark scores overstates the leaders. An \(m\)-out-of-\(N\) bootstrap reduces the ranking bias from order \(N^{-1/2}\) to \(o(N^{-1/2})\) in \(N\) tasks.

Filed under Learning theory, Probability

Leaderboards rank models by average scores over \(N\) benchmark tasks, and selecting the top of a noisy ranking rewards luck as well as ability. The authors call the resulting distortion rank-induced bias and correct it with an \(m\)-out-of-\(N\) bootstrap, assuming independent, identically distributed bounded tasks but allowing arbitrary correlation between models on the same task.

When the population scores fall into tiers separated by positive gaps, they prove the bootstrap cuts the bias from \(\Theta(N^{-1/2})\) to \(o(N^{-1/2})\). Across 14 benchmarks the correction to the leader ranges from about 0.09 to 3.86 percentage points; on SWE-bench Verified, 13 of 173 models land in the leading tier.