aifollow.news
Back Rohan Paul
Rohan Paul· @rohanpaul_ai · X· · Original publication time AI score60

University of Michigan study reports frequent false findings when coding agents analyze data with no real effectsMachine translation

AI introduction

A post describing a University of Michigan paper says researchers asked four frontier agents to find a compelling result in simulated data with no real effects. The agents reported p-values below 0.05 in 54% to 98% of runs, all false positives. A correction that reruns agents on subsamples reduced the false-positive rate to 5.1% in simulations and 5.9% on data from three published papers.

Original post · Original

The article text is unavailable in this language; an existing version is shown.

@rohanpaul_ai

– https://t.co/BtQivEw79K

Title: "Inference after coding agents"

ReplyRohan Paul@rohanpaul_ai
New University of Michigan paper finds that AI agents asked to analyze data where nothing was going on still reported a discovery in up to 98% of cases. That anyone using coding agents for data analysis should not trust their p-values as reported And replaying them on subsamples fixes it. Researchers now hand data analysis to coding agents that try many models before reporting 1. Classic p-values assume the test was chosen before seeing the data. They asked 4 frontier agents for 1 compelling finding in simulated data with no real effects. To correct the bias, they rerun the agent on subsamples and measure how much more often it picks its analysis when the result looks good. The agents reported a p-value under 0.05 in 54% to 98% of runs, all false. The correction cut false positives to 5.1% in simulations and 5.9% on data from 3 published papers. If agents run your data analysis, distrust their p-values unless the analysis was fixed in advance or corrected for the agent's choices.
View this post on X ↗

Source: Rohan Paul · x.com

Research
Found an error? Send a correction