Reasoning and inference Papers

Thinking Harder Doesn’t Lead to Better Choices: Reasoning Effort Increases the Confidence but Not the Correctness of LLM Shopping Agents

Krishnamoorthy and Han

Across ten frontier models and three product-choice benchmarks, raising the thinking effort did not improve decisions in a statistically meaningful way but did raise the models’ stated confidence.

Filed under Reasoning and inference, Agents

The authors test language models acting as shopping agents on benchmarks built from WebShop, Amazon’s SQID and Wayfair’s WANDS, asking whether more inference-time thinking yields better purchases and whether reported confidence predicts errors.

More thinking effort left decision quality statistically unchanged while raising stated confidence significantly, and most models were poorly calibrated at every effort level. Errors were strongly shared: where several models failed on a task, many picked the same wrong product, which limits what majority voting across models can recover.