Max Planck study finds reasoning makes models more likely to err togetherMachine translation

New Max Planck paper finds that turning on reasoning makes different LLMs agree more, even on mistakes, while long traces mark where they disagree.

When several LLMs judge the same person, a wrong answer from one model could be fixed by switching to another. That only works if the models do not all make the same mistakes.

The authors tested 9 Qwen and GPT models on 3 prediction tasks built from US census surveys, with reasoning on and off. They measured agreement beyond what each model's accuracy alone would predict.

With reasoning on, models agreed more, and the share of people all 9 models got wrong rose by 2.0 to 2.4 points. For people who drew the longest traces, adjusted agreement fell to about 0.31 to 0.39.

For decisions about people, check whether anyone gets a correct answer from your model set, since more agreement can also mean more shared errors.