aifollow.news
返回 Rohan Paul
Rohan Paul· @rohanpaul_ai · X· · 原发布时间 AI 评分60

斯坦福新论文发现:8个受测大模型评判医疗回答时均偏爱自己的答案

AI 导读

研究让来自 OpenAI、Anthropic、Google 和 Qwen 的8个模型回答620个真实临床医生问题,再为匿名答案排序。每个模型给自己答案的排名,平均比其他评审模型高1.22个位次;论文建议使用不同模型家族组成评审组,并分别报告各评审结果。

原帖引用 · 原文

@rohanpaul_ai

New Stanford paper finds that all 8 LLMs tested rank their own medical answers higher than other models rank them, even when answers are anonymous.

Medical AI benchmarks often use LLMs to grade other LLMs' answers. If a judge favors its own outputs, the benchmark tilts toward that model.

They had 8 models from OpenAI, Anthropic, Google and Qwen answer 620 real clinician questions, then rank all 8 anonymous answers. They compared where each judge placed its own answer with where the other 7 judges placed that same answer.

Every judge favored its own answer, by 1.22 rank positions on average and nearly 3 for GPT-5.6 Sol. The bias held without a rubric, and revealing model names cut it by only 0.094 positions.

If you use LLM judges, use a panel from several model families and report results per judge instead of 1 pooled score.

来源:Rohan Paul · x.com

论文
发现内容有误?提交纠错