aifollow.news
返回 Rohan Paul
Rohan Paul· @rohanpaul_ai · X· · 原发布时间 AI 评分54

Synthetic Sciences 推出机器学习研究基准 FrontierML

AI 导读

FrontierML 用25道机器学习研究题测试 AI 智能体能否自行设计评估方法、开展实验。每项任务给智能体12小时和 GPU 资源,但不公开最终评分目标;在已测试的5个前沿模型中,没有一个在任何题目上达到其设定的较高成绩门槛。

原帖引用 · 原文

该语言的正文暂不可用,当前显示已有版本。

@rohanpaul_ai

the paper for FrontierML

https://t.co/opuO8j305L https://t.co/WqIsBlwWcl

回复Rohan Paul@rohanpaul_ai
full technical report of FrontierML https://t.co/V8T2ZBIbEu
在 X 查看这条帖子 ↗
相关上下文Rohan Paul@rohanpaul_ai
Synthetic Sciences, a Y Combinator startup, has launched FrontierML, a benchmark so hard that none of the 5 frontier AI models it tested hit the strong-result bar on any of its 25 ML research problems. They hide each objective, so agents must build their own tests and decide how much to trust them. This is a test of whether AI agents can do machine learning research on their own without ever seeing the score they’ll be graded on. Each agent gets 12 hours and real GPUs, and has to work out for itself how to measure progress. FrontierML runs each task as a loop with the objective kept out of reach: > The agent gets an open-ended ML problem and 12 hours in Lab0 > It designs a local proxy for progress > It runs experiments on H100s and calibrates how far that proxy can be trusted > It submits a model or method > FrontierML scores it on a held-out evaluation Synthetic Sciences argues step 3 is where much of empirical research happens. Here, a lot of the score rides on it.
在 X 查看这条帖子 ↗

来源:Rohan Paul · x.com

论文
发现内容有误?提交纠错