研究发现医疗大模型评测存在自我偏好Chinese only
一篇斯坦福论文测试了来自 OpenAI、Anthropic、Google 和 Qwen 的 8 个模型:它们回答 620 道真实临床医生问题,再对匿名答案排序。每个模型都更偏好自己的答案,平均高出 1.22 个名次;即使公开模型名称,偏差也只减少 0.094 个名次。
1 developments2 reports1 sources
Event developments 1
New Stanford paper finds all 8 tested LLMs favor their own medical answers
Researchers had eight models from OpenAI, Anthropic, Google and Qwen answer 620 real clinician questions, then rank the anonymous answers. Each model ranked its own answer 1.22 positions higher on average than the other judges did. The paper recommends using judges from several model families and reporting results for each judge.
View source reports · 1 sources · 2 reports
- Rohan Paul (@rohanpaul_ai) · New Stanford paper finds all 8 tested LLMs favor their own medical answers
- Rohan Paul (@rohanpaul_ai) · Study finds self-preference in medical LLM evaluation
All reports 2
Study finds self-preference in medical LLM evaluation
A Stanford paper tested eight models from OpenAI, Anthropic, Google and Qwen on 620 real clinician questions, then had them rank the anonymized answers. Every model ranked its own answer higher than the other judges did, by 1.22 positions on average; revealing model names reduced the bias by only 0.094 positions.
1 more source reported this
- Rohan Paul (@rohanpaul_ai)
New Stanford paper finds all 8 tested LLMs favor their own medical answers
Researchers had eight models from OpenAI, Anthropic, Google and Qwen answer 620 real clinician questions, then rank the anonymous answers. Each model ranked its own answer 1.22 positions higher on average than the other judges did. The paper recommends using judges from several model families and reporting results for each judge.
1 more source reported this
- Rohan Paul (@rohanpaul_ai)