Are LLM Research Agents Reliable Judges of Predictability? Evidence from Planted-Signal Worlds

Researchers ask language model agents whether a forecasting model beats a static benchmark, a verdict that real data cannot check. This paper scores pre-specified runs of three frontier models and one open-weight model in worlds where the truth is known. The agents detect weak signals more often than a pre-specified out-of-sample test. Held to an agent’s false-positive rate, a conventional in-sample test does as well or better.

The agents also state miscalibrated probabilities and change their verdicts after a sentence about prior research, with magnitudes that depend on the model. The author concludes that agent verdicts are informative but need a neutral brief, a decision rule fixed outside the agent and a sealed holdout.