aifollow.news
Back Rohan Paul
Rohan Paul· @rohanpaul_ai · X· · Original publication time AI score54

Study finds a weak LLM critic can hurt AutoML searchMachine translation

AI introduction

A study of LLM-guided AutoML search compared runs with and without a critic over a fixed pool of 96 pipeline setups. With Llama-3.1-8B as the critic, results fell below the no-critic baseline on 7 of 8 datasets, with losses of up to 0.245 AUC; an ideal critic gained at most about 0.005 AUC.

Original post · Original

The article text is unavailable in this language; an existing version is shown.

@rohanpaul_ai

– https://t.co/42gliJsErv

Title: "The Critic Can Hurt: On the Safety of Critics in LLM-Guided AutoML Search"

ReplyRohan Paul@rohanpaul_ai
New Amazon + University of Massachusetts paper finds that adding an LLM critic to a search loop has small upside, about 0.005 AUC at best, and much larger downside. Weak LLM critic can push AutoML search below plain random search Agent systems often add an LLM critic that reads past results and tells the search what to try next. Teams check whether it helps, rarely whether it hurts. They ran the same tabular model search with and without a critic, over a fixed pool of 96 pipeline setups. Random critic mistakes cost almost nothing since the loop keeps its best result, but advice that steadily points the wrong way wastes tries. A Llama-3.1-8B critic scored below the no-critic baseline on 7 of 8 datasets, losing up to 0.245 AUC. An ideal critic gained at most about 0.005 AUC, and the harm mostly faded with 250 setups and 20 rounds. Before trusting an LLM critic, even a large one, run the same budget without it and drop it if it loses.
View this post on X ↗

Source: Rohan Paul · x.com

Research
Found an error? Send a correction