Thinking Harder Doesn’t Lead to Better Choices: Reasoning Effort Increases the Confidence but Not the Correctness of LLM Shopping Agents
Krishnamoorthy and Han
Across ten frontier models and three product-choice benchmarks, raising the thinking effort did not improve decisions in a statistically meaningful way but did raise the models’ stated confidence.
Source published · In the Daily Brief of 19 November 2026
Filed under Reasoning and inference, Agents
The authors test language models acting as shopping agents on benchmarks built from WebShop, Amazon’s SQID and Wayfair’s WANDS, asking whether more inference-time thinking yields better purchases and whether reported confidence predicts errors.
More thinking effort left decision quality statistically unchanged while raising stated confidence significantly, and most models were poorly calibrated at every effort level. Errors were strongly shared: where several models failed on a task, many picked the same wrong product, which limits what majority voting across models can recover.