Quantitative finance Papers that became canon

An AI-Powered Multiverse Analysis of Empirical Finance Research: Evidence from Reused Natural Experiments

Gao and Xiao

Coding agents rebuilt the data of 324 finance papers that reuse 13 natural experiments and re-estimated their claims over 62,409 defensible specifications. Only 42% of main estimates were reproduced, and on average a quarter of specifications kept the published sign with \(p < 0.10\).

Filed under Quantitative finance, Agents

Reused natural experiments invite multiple testing: many papers draw on the same shock to study different outcomes. The agents reproduced each paper’s main estimate under its own specification and then varied the defensible choices, which gives a measure of how much a published claim depends on analysis decisions.

Regression-discontinuity claims held up better than difference-in-differences claims. Within an experiment, first-order effects, larger published \(t\)-statistics, available data or code, top-journal publication and fewer earlier papers on the same experiment all went with greater robustness. Claims on heavily reused experiments were not less robust on average. Read alongside a pre-registered audit of 91 LLM trading papers, which found that half of those with a trading test deduct no costs and that almost none of the rest test after the model’s training cutoff, the week’s message for empirical finance was the same: automated checking is now cheap enough to be routine.