aifollow.news
Back Rohan Paul
Rohan Paul· @rohanpaul_ai · X· · Original publication time AI score60

University of Michigan study reports frequent false findings when coding agents analyze data with no real effectsMachine translation

AI introduction

A post describing a University of Michigan paper says researchers asked four frontier agents to find a compelling result in simulated data with no real effects. The agents reported p-values below 0.05 in 54% to 98% of runs, all false positives. A correction that reruns agents on subsamples reduced the false-positive rate to 5.1% in simulations and 5.9% on data from three published papers.

Original post · AI translation

@rohanpaul_ai
Translation · refer to the original

– https://t.co/BtQivEw79K

标题:“编程智能体之后的推断”

ReplyRohan Paul@rohanpaul_ai
Translation
密歇根大学的一篇新论文发现,即使让 AI 智能体分析实际上没有任何效应的数据,它们仍会在多达 98% 的情况下报告有所发现。 任何使用编程智能体进行数据分析的人,都不应轻信它们报告的 p 值。 而在子样本上重新运行它们可以解决这个问题。 研究人员如今把数据分析交给编程智能体;这些智能体会尝试许多模型,然后报告其中 1 个。传统的 p 值假定检验是在查看数据之前选定的。 他们要求 4 个前沿智能体从没有真实效应的模拟数据中找出 1 项有说服力的发现。为了纠正这种偏差,他们在子样本上重新运行智能体,并测量当结果看起来不错时,智能体选择其所用分析方法的频率会提高多少。 这些智能体在 54% 至 98% 的运行中报告了低于 0.05 的 p 值,但这些结果全是误报。这种校正使模拟中的假阳性率降至 5.1%,在来自 3 篇已发表论文的数据上降至 5.9%。 如果智能体负责你的数据分析,除非分析方法已事先确定,或已针对智能体的选择进行校正,否则不要轻信它们给出的 p 值。
Original

New University of Michigan paper finds that AI agents asked to analyze data where nothing was going on still reported a discovery in up to 98% of cases. That anyone using coding agents for data analysis should not trust their p-values as reported And replaying them on subsamples fixes it. Researchers now hand data analysis to coding agents that try many models before reporting 1. Classic p-values assume the test was chosen before seeing the data. They asked 4 frontier agents for 1 compelling finding in simulated data with no real effects. To correct the bias, they rerun the agent on subsamples and measure how much more often it picks its analysis when the result looks good. The agents reported a p-value under 0.05 in 54% to 98% of runs, all false. The correction cut false positives to 5.1% in simulations and 5.9% on data from 3 published papers. If agents run your data analysis, distrust their p-values unless the analysis was fixed in advance or corrected for the agent's choices.

View this post on X ↗

Source: Rohan Paul · x.com

Research
Found an error? Send a correction