Original post · AI translation
标题：“推理会让语言模型达成一致吗？”
马克斯·普朗克研究所的一篇新论文发现，启用推理会使不同的大语言模型更加一致，即使是在犯错时也是如此，而较长的推理轨迹则标示出它们存在分歧的地方。
当几个大语言模型对同一个人作出判断时，一个模型给出的错误答案可能通过换用另一个模型得到纠正。但这只有在这些模型并非都犯同样错误时才行得通。
作者在基于美国人口普查调查构建的 3 项预测任务上，测试了 9 个 Qwen 和 GPT 模型，分别开启和关闭推理。他们测量了超出仅凭各模型准确率所能预测的一致程度。
开启推理后，模型之间的一致程度提高，9 个模型全部判断错误的人所占比例上升了 2.0 至 2.4 个百分点。对于引发最长推理轨迹的人，调整后的一致程度降至约 0.31 至 0.39。
在对人作出决策时，要检查你使用的模型组中是否有任何模型给出了正确答案，因为一致程度提高也可能意味着共同犯错增多。
Original
New Max Planck paper finds that turning on reasoning makes different LLMs agree more, even on mistakes, while long traces mark where they disagree. When several LLMs judge the same person, a wrong answer from one model could be fixed by switching to another. That only works if the models do not all make the same mistakes. The authors tested 9 Qwen and GPT models on 3 prediction tasks built from US census surveys, with reasoning on and off. They measured agreement beyond what each model's accuracy alone would predict. With reasoning on, models agreed more, and the share of people all 9 models got wrong rose by 2.0 to 2.4 points. For people who drew the longest traces, adjusted agreement fell to about 0.31 to 0.39. For decisions about people, check whether anyone gets a correct answer from your model set, since more agreement can also mean more shared errors.