Original post · Original
Strange findings here.
New Johns Hopkins paper finds that LLMs make more scientific discoveries when more of their training facts appear only once.
discovery rates rose steadily as the share of once-seen facts in training went from 0% to 100%, for 2 model families.
Discovery and hallucination both mean producing something the model never saw in training. Earlier theory tied hallucination to "monofacts", facts seen exactly once, and treated that unseen space as mostly false.
The authors extend that theory to show monofacts also set how much a model can discover. They fine-tuned gpt-oss-20b and gemma-3-12b-it on materials and DNA datasets, varying the monofact share from 0% to 100%.
Discovery, scored as exactly recovering materials or DNA sequences held out from training, rose steadily with the monofact share in every setting. Training for 3 or 5 epochs, which turns single sightings into repeats, made the trend disappear.
When building science models, keep rare one-off examples and avoid repeating data, and expect more hallucinations to come with more discoveries.