Original post · AI translation
标题：“追溯大语言模型的科学发现能力与其训练数据的关系”
这里有一些奇怪的发现。
约翰斯·霍普金斯大学的一篇新论文发现，当大语言模型训练数据中更多事实只出现过1次时，模型会做出更多科学发现。
在2个模型系列中，随着训练数据中仅出现过1次的事实占比从0%升至100%，发现率稳步上升。
发现和幻觉都意味着生成模型在训练中从未见过的内容。早期理论将幻觉与“单次出现的事实”（即恰好只出现过1次的事实）联系起来，并认为这种未见过的内容大多是错误的。
作者扩展了这一理论，表明单次出现的事实也决定了模型能有多少发现。他们在材料和 DNA 数据集上对 gpt-oss-20b 和 gemma-3-12b-it 进行了微调，将单次出现的事实占比从0%调整到100%。
以准确还原训练数据中留出的材料或 DNA 序列作为发现的评分标准，在所有设置下，发现率都随着单次出现的事实占比稳步上升。训练3轮或5轮会使原本只出现1次的内容被重复看到，也会使这一趋势消失。
构建科学模型时，应保留罕见的单次出现样本并避免重复数据，同时预期更多发现会伴随更多幻觉。
Original
Strange findings here. New Johns Hopkins paper finds that LLMs make more scientific discoveries when more of their training facts appear only once. discovery rates rose steadily as the share of once-seen facts in training went from 0% to 100%, for 2 model families. Discovery and hallucination both mean producing something the model never saw in training. Earlier theory tied hallucination to "monofacts", facts seen exactly once, and treated that unseen space as mostly false. The authors extend that theory to show monofacts also set how much a model can discover. They fine-tuned gpt-oss-20b and gemma-3-12b-it on materials and DNA datasets, varying the monofact share from 0% to 100%. Discovery, scored as exactly recovering materials or DNA sequences held out from training, rose steadily with the monofact share in every setting. Training for 3 or 5 epochs, which turns single sightings into repeats, made the trend disappear. When building science models, keep rare one-off examples and avoid repeating data, and expect more hallucinations to come with more discoveries.