aifollow.news
返回 Rohan Paul
Rohan Paul· @rohanpaul_ai · X· · 原发布时间 AI 评分60

研究发现医疗大模型评测存在自我偏好

AI 导读

一篇斯坦福论文测试了来自 OpenAI、Anthropic、Google 和 Qwen 的 8 个模型:它们回答 620 道真实临床医生问题,再对匿名答案排序。每个模型都更偏好自己的答案,平均高出 1.22 个名次;即使公开模型名称,偏差也只减少 0.094 个名次。

原帖引用 · 原文

@rohanpaul_ai

– https://t.co/rcuhEy3o0D

Title: "The Judge Is Not Impartial: Self-Preference in Medical LLM Evaluation"

回复Rohan Paul@rohanpaul_ai
New Stanford paper finds that all 8 LLMs tested rank their own medical answers higher than other models rank them, even when answers are anonymous. Medical AI benchmarks often use LLMs to grade other LLMs' answers. If a judge favors its own outputs, the benchmark tilts toward that model. They had 8 models from OpenAI, Anthropic, Google and Qwen answer 620 real clinician questions, then rank all 8 anonymous answers. They compared where each judge placed its own answer with where the other 7 judges placed that same answer. Every judge favored its own answer, by 1.22 rank positions on average and nearly 3 for GPT-5.6 Sol. The bias held without a rubric, and revealing model names cut it by only 0.094 positions. If you use LLM judges, use a panel from several model families and report results per judge instead of 1 pooled score.
在 X 查看这条帖子 ↗

来源:Rohan Paul · x.com

论文
发现内容有误?提交纠错