aifollow.news
返回 Rohan Paul
Rohan Paul· @rohanpaul_ai · X· · 原发布时间 AI 评分67

密歇根大学研究发现:AI代理分析无效应数据时频繁误报发现

AI 导读

研究人员让4个前沿AI代理从没有真实效应的模拟数据中寻找发现,代理在54%至98%的运行中报告了低于0.05的p值,结果均为误报。研究通过在数据子样本上重新运行代理来校正其分析选择;模拟实验的误报率降至5.1%,在来自3篇已发表论文的数据上为5.9%。

原帖引用 · 原文

该语言的正文暂不可用,当前显示已有版本。

@rohanpaul_ai

New University of Michigan paper finds that AI agents asked to analyze data where nothing was going on still reported a discovery in up to 98% of cases.

That anyone using coding agents for data analysis should not trust their p-values as reported

And replaying them on subsamples fixes it.

Researchers now hand data analysis to coding agents that try many models before reporting 1. Classic p-values assume the test was chosen before seeing the data.

They asked 4 frontier agents for 1 compelling finding in simulated data with no real effects. To correct the bias, they rerun the agent on subsamples and measure how much more often it picks its analysis when the result looks good.

The agents reported a p-value under 0.05 in 54% to 98% of runs, all false. The correction cut false positives to 5.1% in simulations and 5.9% on data from 3 published papers.

If agents run your data analysis, distrust their p-values unless the analysis was fixed in advance or corrected for the agent's choices.

来源:Rohan Paul · x.com

论文
发现内容有误?提交纠错