aifollow.news
返回 Rohan Paul
Rohan Paul· @rohanpaul_ai · X· · 原发布时间 AI 评分54

Amazon 与 University of Massachusetts 研究发现,弱 LLM 评审可能拖累模型搜索

AI 导读

研究在固定的 96 种表格模型配置中,对比加入与不加入 LLM 评审的搜索。Llama-3.1-8B 评审在 8 个数据集中的 7 个上低于无评审基线,最多损失 0.245 AUC;理想评审的收益最多约为 0.005 AUC。在 250 种配置、20 轮搜索下,负面影响大多消退。

原帖引用 · 原文

该语言的正文暂不可用,当前显示已有版本。

@rohanpaul_ai

New Amazon + University of Massachusetts paper finds that adding an LLM critic to a search loop has small upside, about 0.005 AUC at best, and much larger downside.

Weak LLM critic can push AutoML search below plain random search

Agent systems often add an LLM critic that reads past results and tells the search what to try next. Teams check whether it helps, rarely whether it hurts.

They ran the same tabular model search with and without a critic, over a fixed pool of 96 pipeline setups. Random critic mistakes cost almost nothing since the loop keeps its best result, but advice that steadily points the wrong way wastes tries.

A Llama-3.1-8B critic scored below the no-critic baseline on 7 of 8 datasets, losing up to 0.245 AUC. An ideal critic gained at most about 0.005 AUC, and the harm mostly faded with 250 setups and 20 rounds.

Before trusting an LLM critic, even a large one, run the same budget without it and drop it if it loses.

来源:Rohan Paul · x.com

论文
发现内容有误?提交纠错