aifollow.news
返回 Rohan Paul
Rohan Paul· @rohanpaul_ai · X· · 原发布时间 AI 评分60

密歇根大学研究称,编程智能体分析无效应数据时频繁报告假发现

AI 导读

一则介绍密歇根大学论文的帖子称,研究人员让4个前沿智能体在没有真实效应的模拟数据中寻找发现;它们在54%至98%的运行中报告了低于0.05的p值,结果均为假阳性。研究通过在子样本上重新运行智能体来校正选择偏差,模拟数据中的假阳性率降至5.1%,在3篇已发表论文的数据上为5.9%。

原帖引用 · 原文

该语言的正文暂不可用,当前显示已有版本。

@rohanpaul_ai

– https://t.co/BtQivEw79K

Title: "Inference after coding agents"

回复Rohan Paul@rohanpaul_ai
New University of Michigan paper finds that AI agents asked to analyze data where nothing was going on still reported a discovery in up to 98% of cases. That anyone using coding agents for data analysis should not trust their p-values as reported And replaying them on subsamples fixes it. Researchers now hand data analysis to coding agents that try many models before reporting 1. Classic p-values assume the test was chosen before seeing the data. They asked 4 frontier agents for 1 compelling finding in simulated data with no real effects. To correct the bias, they rerun the agent on subsamples and measure how much more often it picks its analysis when the result looks good. The agents reported a p-value under 0.05 in 54% to 98% of runs, all false. The correction cut false positives to 5.1% in simulations and 5.9% on data from 3 published papers. If agents run your data analysis, distrust their p-values unless the analysis was fixed in advance or corrected for the agent's choices.
在 X 查看这条帖子 ↗

来源:Rohan Paul · x.com

论文
发现内容有误?提交纠错