原帖引用 · 原文
该语言的正文暂不可用,当前显示已有版本。
Rohan Paul@rohanpaul_ai
@SynScience @aayambansal @ishaangangwanii Measuring whether an agent can predict what happens when an experiment scales up is how senior researchers get judged too
Introducing FrontierML, a benchmark for evaluating AI agents on autonomous empirical ML R&D.
Most existing benchmarks for ML "research" give the agent a validation set or verifier that closely tracks its final score. In practice, human researchers rarely have that. Much of empirical research is deciding how to measure progress on a problem and how far to trust those measurements.
In FrontierML's 25 open-ended problems, agents have to operationalize progress through their own experiments, constructing local proxies and calibrating how far to trust them, while working toward an objective they never directly observe. Their submitted model or method is then scored on a held-out evaluation.
在 X 查看这条帖子 ↗
· UTC+8