aifollow.news
返回 Rohan Paul
Rohan Paul· @rohanpaul_ai · X· · 原发布时间 AI 评分54

研究发现:弱语言模型评审可能拖累 AutoML 搜索

AI 导读

一项关于语言模型引导 AutoML 搜索的研究,在固定的 96 种流水线配置中比较了有无评审模型的结果。Llama-3.1-8B 担任评审时,在 8 个数据集中的 7 个上落后于无评审基线,AUC 最多下降 0.245;理想评审带来的增益最多约为 0.005 AUC。

原帖引用 · 原文

该语言的正文暂不可用,当前显示已有版本。

@rohanpaul_ai

– https://t.co/42gliJsErv

Title: "The Critic Can Hurt: On the Safety of Critics in LLM-Guided AutoML Search"

回复Rohan Paul@rohanpaul_ai
New Amazon + University of Massachusetts paper finds that adding an LLM critic to a search loop has small upside, about 0.005 AUC at best, and much larger downside. Weak LLM critic can push AutoML search below plain random search Agent systems often add an LLM critic that reads past results and tells the search what to try next. Teams check whether it helps, rarely whether it hurts. They ran the same tabular model search with and without a critic, over a fixed pool of 96 pipeline setups. Random critic mistakes cost almost nothing since the loop keeps its best result, but advice that steadily points the wrong way wastes tries. A Llama-3.1-8B critic scored below the no-critic baseline on 7 of 8 datasets, losing up to 0.245 AUC. An ideal critic gained at most about 0.005 AUC, and the harm mostly faded with 250 setups and 20 rounds. Before trusting an LLM critic, even a large one, run the same budget without it and drop it if it loses.
在 X 查看这条帖子 ↗

来源:Rohan Paul · x.com

论文
发现内容有误?提交纠错