aifollow.news
Back IT之家 科技新闻
IT之家 科技新闻· · Original publication time AI score54

OpenAI releases MedChemBench to assess AI reasoning in medicinal chemistryMachine translation

OpenAI 发布 MedChemBench,评估 AI 模型的药物化学推理能力

AI-assisted summary

MedChemBench contains 99 questions drawn from papers and patents, testing models on molecular structures, experimental evidence, compound selection and property prediction. OpenAI's results show stronger performance on molecular structures but clear weaknesses in compound selection and numerical property prediction. The company says scores on historical questions cannot establish whether models can discover new drugs.

Article · AI translation

IT Home reported on November 21 that OpenAI released the MedChemBench benchmark on November 20 local time to assess AI models’ reasoning abilities in medicinal chemistry research, including identifying molecular structures, interpreting experimental results, comparing compounds, and predicting their properties. OpenAI said the models performed strongly in reconstructing molecular structures and explaining experimental evidence, but still had considerable room for improvement in compound selection and numerical property prediction.

IT Home learned from the official introduction that the benchmark contains 99 questions drawn from 117 papers and patents, involving 761 distinct molecular structures. Most questions provide models with the context available at a particular point in a study, including the biological target, the researchers’ goals, compounds previously studied, and some experimental results.

The official evaluation is divided into four parts. IT Home provides the details below:

Molecular structure: assembling molecules from structural diagrams and chemical groups provided in the questions, assessing whether atom connectivity and stereochemistry are correct;

Scientific explanation: providing explanations supported by the given evidence, such as why one compound was chosen over others;

Ranking and selection: whether the ranking of compounds and the final choice meet the question’s requirements;

Property prediction: whether predictions match the numerical values and categories in the original materials, covering measured and calculated molecular properties.

According to officially released information, at the xhigh reasoning level, GPT-6.2 Astra achieved a usable score of 84.5%, GPT-6 Astra scored 83.3%, GPT-6.1 Sol scored 82.0%, and GPT-5.6 Sol scored 58.2%. Officials said that raising the reasoning level can improve scores, but the gaps between models remain clear.

In the individual categories, Astra at the xhigh level earned a usable score of 95.4% for molecular structure, but only about half the available points in each of compound selection and numerical property prediction.

Officials gave an example of a question that asked models to assemble a molecule from a given central structure and two side groups while preserving an 18-atom ring.

At the xhigh level shared by 11 models, all 10 responses from GPT-6 Astra were correct; all 10 responses from GPT-5.6 Sol and all 10 from GPT-5.6 Luna were wrong. The former assembled a ring with only 15 atoms, while the latter split the molecule into two disconnected fragments.

Another question was drawn from research on PRMT5 inhibitors. PRMT5 is a protein of interest in cancer drug development. Models had to predict how strongly three related compounds would block PRMT5 inside cells, how easily they would cross the cell barrier, and how quickly liver enzymes would break them down in laboratory tests.

The subsequent experiments withheld from the models in the question showed a trade-off: compound 23 could block PRMT5 inside cells at a lower concentration, while compound 24 crossed the cell barrier more easily and was broken down more slowly in the liver enzyme test.

Officials said these results cannot establish whether models can discover new drugs or improve laboratory decisions. The possibility that they had previously encountered published materials cannot be ruled out, and each response was a single attempt without tools, feedback, or new experiments. Although the benchmark is based on historical evidence rather than real-world drug discovery, its results help identify where models have advanced and where challenges remain. The team believes MedChemBench will support continued progress and looks forward to sharing updates as models and their areas of application advance.

Found an error? Send a correction