密歇根大学研究发现:AI代理分析无效应数据时频繁误报发现Chinese only
研究人员让4个前沿AI代理从没有真实效应的模拟数据中寻找发现,代理在54%至98%的运行中报告了低于0.05的p值,结果均为误报。研究通过在数据子样本上重新运行代理来校正其分析选择;模拟实验的误报率降至5.1%,在来自3篇已发表论文的数据上为5.9%。
1 developments2 reports1 sources
Event developments 1
University of Michigan study finds AI agents frequently report false discoveries in data with no real effects
Researchers asked four frontier AI agents to find a compelling result in simulated data with no real effects. The agents reported a p-value below 0.05 in 54% to 98% of runs, all false positives. A correction that reruns the agents on subsamples reduced the false-positive rate to 5.1% in simulations and 5.9% on data from three published papers.
View source reports · 1 sources · 2 reports
- Rohan Paul (@rohanpaul_ai) · University of Michigan study finds AI agents frequently report false discoveries in data with no real effects
- Rohan Paul (@rohanpaul_ai) · University of Michigan study reports frequent false findings when coding agents analyze data with no real effects
All reports 2
University of Michigan study reports frequent false findings when coding agents analyze data with no real effects
A post describing a University of Michigan paper says researchers asked four frontier agents to find a compelling result in simulated data with no real effects. The agents reported p-values below 0.05 in 54% to 98% of runs, all false positives. A correction that reruns agents on subsamples reduced the false-positive rate to 5.1% in simulations and 5.9% on data from three published papers.
1 more source reported this
- Rohan Paul (@rohanpaul_ai)
University of Michigan study finds AI agents frequently report false discoveries in data with no real effects
Researchers asked four frontier AI agents to find a compelling result in simulated data with no real effects. The agents reported a p-value below 0.05 in 54% to 98% of runs, all false positives. A correction that reruns the agents on subsamples reduced the false-positive rate to 5.1% in simulations and 5.9% on data from three published papers.
1 more source reported this
- Rohan Paul (@rohanpaul_ai)