Article · AI translation
Stanford, Harvard, Cornell... more than 10 top teams have gathered around one table.
By Luyu, reporting from Aofei Temple
QbitAI | Official WeChat account QbitAI
By Luyu, reporting from Aofei Temple
QbitAI | Official WeChat account QbitAI
Wait, have benchmarks really evolved this far now???
This time, instead of answering test questions, AI is given a set of experimental conditions and asked to conduct its own research and predict the results of real wet lab experiments in advance...
For example, if the same reaction is run under several conditions, which one yields more of the target product? Among a batch of antibody variants, which has higher expression?
Keep the answers hidden until the AI submits its predictions, then compare them with the measured results.
And you know what? This is now genuinely worth testing.
Over the past 2 years, AI for Science has become increasingly involved: AI can already search the literature and use software on its own, and can spend hours continuously working through an analytical task.
Naturally, people are moving to the next question: What scientific research abilities do we actually want AI to have?
Predicting wet lab results is one ability with considerable practical value.
It requires AI to understand the experimental objective and identify relevant mechanisms in existing knowledge so it can better help researchers choose directions worth exploring further.
The new benchmark led by UniPat AI, Science Prediction Universe (SPU), recognizes this point.
It has 116 tasks and 4207 wet lab experiments spanning chemistry, materials science, biology, and medicine. Its benchmark advisers and wet lab collaborators also come from universities including Stanford, Cornell, and Harvard, as well as leading research institutions and biotechnology companies.
Each model gets an 8-hour work session and an H20 GPU, then decides for itself how to use search and computing resources to produce experimental predictions.
The results surprised everyone.
Even models as strong as Opus 5.5 and GPT-6 Astra are still, by the numbers, a long way from truly being able to make these predictions...
SPU has certainly given every scientific research model a tough challenge.
Research abilities that once seemed intangible have now become numbers that can be evaluated consistently and studied over time.
That provides a more concrete basis for discussing how AI can develop scientific judgment step by step.
Wet lab prediction: an ability worth testing on its own
Returning to wet lab prediction, this is certainly not a new question.
There have been many moments in the history of science when someone made a judgment before an experiment and then let the process decide: if it was right, it stood; if it was wrong, it was overturned.
The reasoning behind such judgments constitutes a scientist's taste.
With good taste and accurate intuition, scientists can reach major results faster while conserving valuable experimental resources.
After all, even a single wet lab experiment takes work: chemists must prepare reagents, set reaction conditions, and then isolate the product; biologists must also go through lengthy protein expression... The time and tools are costly in themselves, and disappointing results can mean repeating the process for several rounds.
That is why predicting before starting an experiment is one of the most valuable steps in the research process.
The difference is that scientists used to make these predictions on their own. In the age of AI, models also need this ability, along with a test that can score it.
Existing benchmarks do not fully cover it. For example, BrainBench asks models to judge experimental results directly; ScienceAgentBench focuses on analyzing existing data; and long-horizon task benchmarks such as Terminal-Bench 4.0 and Long-Horizon-TB mainly use preset tests to check task outputs.
SPU differs from each of these.
It puts AI's independent research process and its prediction of wet lab results in the same test, assessing prediction quality as well as the ability to use scientific tools and carry out long-horizon tasks.
SPU also turns this kind of judgment into clearly defined tasks.
The 100 public tasks come from 85 recently published papers. Domain experts extracted candidate options, materials, conditions, and measurement methods from the studies, retaining the full quantitative results as the basis for scoring.
Another 16 private tasks come from 4 sets of unpublished collaborative wet lab experiments involving VHH nanobodies, reaction conditions, electrolytes, and Magrolimab variants.
In total, there are 32 subfields and 4207 experiments. Published research supplies tasks across disciplines, with measured results hidden from the models, while unpublished experiments add a test independent of answers in published papers.
To some extent, this helps prevent models from having seen the questions in advance.
The task selection process is carefully considered, too.
Experts prioritize studies with complete measurements and clearly defined conditions. Simple problems that can be ranked by following a single trend are excluded outright, as are problems whose values are so close that they offer little useful evidence.
The tasks that remain are harder and better at distinguishing performance.
For each prediction run, a model gets 8 hours, 8 CPU cores, 16 GiB of memory, and an H20 GPU. It may also search online and install scientific software.
During that time, the model gathers evidence and chooses computational methods on its own, then submits a predicted ranking of each comparison group from lowest to highest.
To prevent contamination from memorized answers, scoring takes place entirely in a separate, offline container. AI is prohibited from searching for the original study results, and tool outputs are automatically scanned for identifiers such as the source paper's DOI and title.
Finally, the system compares the predicted and measured rankings using Spearman's correlation coefficient.
The closer the coefficient is to 1, the more closely the ranking matches the experimental results.
SPU also attends to details that many benchmarks overlook.
For example, when considering a final score, SPU asks whether it could simply be due to luck.
It therefore generates an additional 4000 random rankings for each task, uses the 95th percentile as the threshold for random performance, and checks whether the model's submission clearly exceeds that benchmark. Each task has its own corresponding reference point.
The overall design of SPU can fairly be called scientifically rigorous.
Models must decide for themselves, step by step, what is worth calculating and then submit their final predictions.
This design helps measure AI's current ability to predict wet lab results and gives research teams a common metric to guide continued improvements in experimental work.
With that said, let's look at the first report card for leading models. Several details deserve a closer look.
A new test leaves leading models struggling across the board
SPU's first systematic evaluation covered 12 leading models in 13 configurations. Here, a configuration means a combination of model, reasoning effort, and model framework.
First, the overall results.
In a word: unimpressive.
Across all scored tasks, Claude Opus 5.5 had the highest average score, at 0.486. Yet only half of its tasks exceeded the random-performance threshold.
The picture was much the same across the 13 configurations: the share of tasks exceeding the random-performance threshold was only 34%–52%.
In other words, every model tested remains a long way from reliably predicting wet lab results.
Still, there were differences between models. Their average performance fell into 2 tiers:
a leading group (6 models) and a trailing group (7 models).
On the same tasks, the leading group scored about 0.15 points higher on average than the other 7.
Another detail is worth noting. About 9% of the leading group's task scores were below 0; for the other 7 configurations, the share reached as high as 22%.
Here, 0 is the average score for valid random rankings, so a negative score means the predicted and actual rankings are negatively correlated.
Where exactly did the gap come from? The research team dug deeper.
They first combined the predictions of the other models on each task into a reference ranking. Then they paired up experiments, compared the pairs with measured results, and checked whether each model had ranked them correctly on its own.
For experiment pairs that the reference ranking got right, the leading group ranked 86% correctly, versus 78% for the other 7 configurations. For pairs that the reference ranking got wrong, both groups independently ranked about 30% correctly.
So part of the leading group's advantage lay precisely in cases where the models tended to predict correctly together: the leading group was consistently more reliable there.
The run logs offer more specific clues about why models depart from the correct judgment.
Consider the solvent task: the same polymerization reaction was run with 12 different solvents, and the task was to rank them by the mass of polymer ultimately isolated.
Opus 5.5 scored 0.874, while Qwen3.8 scored -0.126. Pairing the 12 candidate solvent conditions yielded 66 pairs; the former ranked 56 correctly and the latter 30.
The analysis found that the 2 models focused on different factors from the start:
Qwen3.8 ranked the solvents mainly by how much ethylene they could dissolve, while Opus 5.5 paid more attention to the miscibility of the starting materials and the catalyst's compatibility with the solvent.
Although Qwen3.8 also identified this issue during its calculations, that discovery ultimately did not directly change the basis it used for its final ranking.
The most likely point for a prediction to go wrong, then, is the choice of criteria for judgment. Models also need to learn to adjust their rankings in response to new clues so those discoveries actually affect the answer.
A closer look shows that these differences also vary by discipline and task.
Among scored public tasks, Opus 5.5 had the highest average score in chemistry, Fable 5.1 in materials science, GPT-6 Astra Medium in biology, and Opus 5 in medicine.
P.S. Fable 5.1 and Opus 5.5 sometimes refused to answer.
So if you plan to involve AI in a particular kind of research, it may be worth finding a relevant task and testing the model with SPU before running formal experiments, to establish that it really is suitable.
The private-task results make this point more concrete.
Of the 13 configurations, 12 had average scores significantly above random rankings, with no clear difference detected between configurations. By contrast, variation across tasks was especially striking.
For the same batch of electrolytes, the ranking correlation coefficients across 13 configurations clustered between 0.89 and 0.95 when predicting ionic conductivity. When the target changed to the electrochemical window, the 12 scored configurations received scores from -0.55 to 0.07, none above the random-performance threshold.
The models stayed the same, but changing the measurement target clearly changed the distribution of scores.
So the next time we evaluate a model, saying it is good at chemistry is not specific enough; we also need to ask which measurement target and experimental conditions it faces.
Here is another rather counterintuitive finding:
Generally speaking, giving a model more time and greater reasoning effort should keep improving its performance, right?
That is what people generally think, and what they do.
But SPU's results delivered a sharp reality check—
Under the same 8-hour compute allowance, the research team raised GPT-6 Astra's reasoning effort from medium to maximum. Its median runtime increased to 4.8 times the previous level, and its median API cost rose to 5.3 times the previous level.
But its prediction score... barely moved.
Huh? The research team went back and examined the model's run logs.
On the 79 completed tasks, GPT-6 Astra at maximum reasoning effort had typically already produced an answer by the time it reached the median point of 26% of its run.
The model kept working afterward, calculating and revising, but those changes were minor and did not produce a visible difference in the results.
In other words, more reasoning effort does not necessarily lead to better results.
This finding also offers a lesson for allocating model resources: adding compute without a clear purpose can consume time and effort without improving performance.
Instead, researchers can examine the run logs more closely, first determine which judgment any new evidence bears on, and then increase computing effort selectively.
A similar pattern appears in tool use.
GPT-6 Astra Max used a median of 4 categories of methods per run, compared with 1 for Medium, but no significant difference in scores was detected between the 2 configurations.
The reason was much the same: useful tools were not applied where they mattered most.
For example, although the model called on Monte Carlo sampling much of the time, it only added an error range to the final ranking and did not directly affect the result itself.
So using tools well matters more than using lots of them.
GPU usage also offers an interesting perspective.
Even with ample GPU computing power available, GPU computations were successfully run only 255 times, fewer than one-fifth of the attempts.
GPU usage varied across models and disciplines, so the number of GPU calls alone says little about a model's scientific research capabilities.
To determine whether computing power helped, we also need to examine what it was used to calculate, whether the results helped with prediction, and how those results affected the final ranking.
The key is to direct computing power toward physical quantities relevant to the ranking so that the model can make more accurate predictions.
Considering costs as well, the average API cost per run across configurations ranged from $0.22 to $65.91, a 300-fold difference.
This means SPU provides both predictive performance and usage costs, making it a useful reference for teams preparing to deploy scientific research models.
From overall scores to trajectories on the same questions, SPU lays out model differences in every detail of scientific judgment, including computing power and costs.
How much do the values calculated by AI actually help with predictions? When a prediction is wrong, which judgments should be revisited? These questions can be analyzed further using experimental results and run records.
The next round of improvements thus has clear directions to try:
According to SPU, models can identify incorrect assumptions sooner and make more informed decisions about whether to retain or revise a ranking. After making changes, they can be tested again on the same tasks to see whether their predictions improve.
Bringing “testing understanding” into the model's exam
SPU, one could say, closely aligns with the development of AI4S.
AI4S evaluations are gradually moving deeper into real experimental work, and benchmarks focused on carrying out experiments have also emerged over the course of this year.
Beyond execution, prediction—an unavoidable step in scientific research and one of the hardest to quantify—finally has a way to be measured.
The benefits show up in 2 ways:
Researchers can use this to analyze which measurement targets are easier to predict, which tasks differ the most, and how an incorrect judgment arises. This can save costly, scarce experimental resources and shorten decision times.
For model development, it also provides a more concrete basis for improvement.
As the range of experiment types grows, the evaluation can answer more specific questions: for which materials, experimental conditions, and measurement methods are AI-generated rankings more reliable and more useful as a reference?
Scientists can thus continue building progress on the same kinds of problems.
In fact, SPU is only the tip of the iceberg among UniPat AI's series of evaluations based on real-world work:
ScienceVision: examines how models gather evidence from 3D scientific objects and answer questions;
PaperBenchX: turns published research into executable, verifiable paper-reproduction tasks;
SaaS-Bench and EvoCode-Bench: respectively examine work in real software workflows and development amid continuously changing requirements.
…
These efforts span scientific research, office work, and software development. They share a concern with how AI completes specific tasks in actual work and how its results can be verified.
This also indirectly reflects UniPat AI's distinctive understanding of the industry and forward-looking judgment about AI's move into scientific research practice and even the real world.
UniPat AI also demonstrates a new model of AI collaboration:
Domain experts take part in selecting and reviewing tasks, partner laboratories provide unpublished tasks, and the project team organizes experiments from different disciplines into a unified evaluation.
Take SPU: led by UniPat AI, it has advisory members affiliated with Stanford, Cornell, Harvard, UIC, Peking University, HKUST, and FutureHouse. Wet-lab collaborators from 优睿赛思, 爱博泰克, 深度原理, and 恩和生物 participated in the related experimental work.
This brings interdisciplinary knowledge and experimental resources into a single evaluation process. SPU reflects both the selection of problems and the collaboration needed to turn scientific research questions into executable tasks.
The 100 public tasks associated with SPU are currently planned for release as complete Harbor task packages; 16 private tasks will remain private for controlled evaluation. The dataset and code have already been released as open source.
Admittedly, today's AI is still a novice when faced with this SPU framework.
It can use tools at remarkable speed, install 3 software packages in 8 hours, complete DFT calculations, and then run Monte Carlo simulations. But when it comes to the most crucial judgment, it has yet to learn to ask itself: Is the quantity I calculated the one the experiment actually needs to measure?
But as tasks like these continue to accumulate, the kinds of scientific research work AI can take on will also become clearer.
SPU has measured that distance. From here on, every exam will record AI moving closer to the real laboratory bench.
Project homepage: https://spu.unipat.ai Paper link:
https://arxiv.org/abs/2611.15049 GitHub open-source link:
https://github.com/UniPat-AI/SPU