aifollow.news
Back Rohan Paul
Rohan Paul· @rohanpaul_ai · X· · Original publication time AI score60

New Stanford paper finds all 8 tested LLMs favor their own medical answersMachine translation

AI introduction

Researchers had eight models from OpenAI, Anthropic, Google and Qwen answer 620 real clinician questions, then rank the anonymous answers. Each model ranked its own answer 1.22 positions higher on average than the other judges did. The paper recommends using judges from several model families and reporting results for each judge.

Original post · AI translation

@rohanpaul_ai
Translation · refer to the original

斯坦福一篇新论文发现,接受测试的全部8个大语言模型都给自己的医疗回答排出高于其他模型所给的名次,即使这些回答是匿名的。

医疗AI基准测试常用大语言模型给其他大语言模型的回答评分。如果评审模型偏爱自己的输出,基准测试结果就会向该模型倾斜。

研究人员让来自OpenAI、Anthropic、Google和Qwen的8个模型回答620个真实的临床医生问题,然后对全部8份匿名回答排序。他们比较了每个评审模型给自己答案的排名,以及其他7个评审模型给同一答案的排名。

每个评审模型都偏爱自己的答案,平均高出1.22个名次;GPT-5.6 Sol的差距接近3个名次。即使没有评分细则,这种偏差依然存在;公布模型名称也仅使偏差缩小0.094个名次。

如果使用大语言模型评审,应采用来自多个模型系列的评审小组,并分别报告每个评审模型的结果,而不是只报告1个汇总分数。

Source: Rohan Paul · x.com

Research
Found an error? Send a correction