原帖引用 · AI 翻译
标题:“编程智能体之后的推断”
密歇根大学的一篇新论文发现,即使让 AI 智能体分析实际上没有任何效应的数据,它们仍会在多达 98% 的情况下报告有所发现。
任何使用编程智能体进行数据分析的人,都不应轻信它们报告的 p 值。
而在子样本上重新运行它们可以解决这个问题。
研究人员如今把数据分析交给编程智能体;这些智能体会尝试许多模型,然后报告其中 1 个。传统的 p 值假定检验是在查看数据之前选定的。
他们要求 4 个前沿智能体从没有真实效应的模拟数据中找出 1 项有说服力的发现。为了纠正这种偏差,他们在子样本上重新运行智能体,并测量当结果看起来不错时,智能体选择其所用分析方法的频率会提高多少。
这些智能体在 54% 至 98% 的运行中报告了低于 0.05 的 p 值,但这些结果全是误报。这种校正使模拟中的假阳性率降至 5.1%,在来自 3 篇已发表论文的数据上降至 5.9%。
如果智能体负责你的数据分析,除非分析方法已事先确定,或已针对智能体的选择进行校正,否则不要轻信它们给出的 p 值。
原文
New University of Michigan paper finds that AI agents asked to analyze data where nothing was going on still reported a discovery in up to 98% of cases. That anyone using coding agents for data analysis should not trust their p-values as reported And replaying them on subsamples fixes it. Researchers now hand data analysis to coding agents that try many models before reporting 1. Classic p-values assume the test was chosen before seeing the data. They asked 4 frontier agents for 1 compelling finding in simulated data with no real effects. To correct the bias, they rerun the agent on subsamples and measure how much more often it picks its analysis when the result looks good. The agents reported a p-value under 0.05 in 54% to 98% of runs, all false. The correction cut false positives to 5.1% in simulations and 5.9% on data from 3 published papers. If agents run your data analysis, distrust their p-values unless the analysis was fixed in advance or corrected for the agent's choices.