Does Reasoning Make Language Models Agree?
Gorecki and Hardt
Across nine models and three classification tasks, enabling reasoning raises agreement between models, while individuals who elicit longer reasoning traces receive less consistent predictions.
Source published · In the Daily Brief of 3 November 2026
Filed under Reasoning and inference, Large language models
Reasoning might raise agreement if models anchor on the same features, or lower it if stochastic traces add branch points. Mila Gorecki and Moritz Hardt separate reasoning as an intervention from reasoning as an observed behaviour, comparing controlled changes in effort with agreement across observed trace lengths, each measured against the agreement expected from accuracy.
Both patterns persist after accounting for accuracy. Asking models to reason more promotes agreement; seeing them reason longer signals disagreement.