aifollow.news
Back Rohan Paul
Rohan Paul· @rohanpaul_ai · X· · Original publication time AI score54

Synthetic Sciences launches FrontierML benchmark for machine learning researchMachine translation

AI introduction

FrontierML uses 25 machine learning research problems to test whether AI agents can devise their own evaluations and run experiments. Agents get 12 hours and GPU access per task, but the final scoring objective is hidden; none of the five frontier models tested met the benchmark's strong-result bar on any problem.

Original post · Original

The article text is unavailable in this language; an existing version is shown.

@rohanpaul_ai

the paper for FrontierML

https://t.co/opuO8j305L https://t.co/WqIsBlwWcl

ReplyRohan Paul@rohanpaul_ai
full technical report of FrontierML https://t.co/V8T2ZBIbEu
View this post on X ↗
Related contextRohan Paul@rohanpaul_ai
Synthetic Sciences, a Y Combinator startup, has launched FrontierML, a benchmark so hard that none of the 5 frontier AI models it tested hit the strong-result bar on any of its 25 ML research problems. They hide each objective, so agents must build their own tests and decide how much to trust them. This is a test of whether AI agents can do machine learning research on their own without ever seeing the score they’ll be graded on. Each agent gets 12 hours and real GPUs, and has to work out for itself how to measure progress. FrontierML runs each task as a loop with the objective kept out of reach: > The agent gets an open-ended ML problem and 12 hours in Lab0 > It designs a local proxy for progress > It runs experiments on H100s and calibrates how far that proxy can be trusted > It submits a model or method > FrontierML scores it on a held-out evaluation Synthetic Sciences argues step 3 is where much of empirical research happens. Here, a lot of the score rides on it.
View this post on X ↗

Source: Rohan Paul · x.com

Research
Found an error? Send a correction