Comparing tail forecasts: the number of exceedances is the sample size

Scores for tail forecasts are known in advance except on days when the outcome falls beyond a forecast, so Coelho argues that the number of those exceedances is the real sample size. For intervals at miscoverage level \(\alpha\), the observations required grow like \(1/\alpha\) while the exceedances required do not. A confidence sequence valid at all times attains this rate with no distributional assumption, whereas Hoeffding-type procedures need \(1/\alpha\) times more.

When exceedances are scarce, the Diebold-Mariano test declares the less conservative forecaster better with at least, asymptotically, the probability of seeing no exceedance, whichever is actually better. Small-sample corrections do not fix this, while a bootstrap-t version largely does.

In 250-day windows of 97.5% forecasts for seven stock indices, 18.4% of significant results favouring the less conservative model contradicted the ranking on the remaining days, against 6.9% for the more conservative one. The finding matters for anyone validating value-at-risk or expected shortfall models on short samples.