Robel Yemiru retweetledi
New AI x Science research:
Frontier models have gaps in their scientific knowledge. Our new eval tests how models recall significant published findings without tools, in Alzheimer’s disease.
GPT-5.5 performed best among released models, but the benchmark remains far from saturated.
Questions are based on prominent findings, often the central claims from highly cited papers, that were published before the models’ cutoff dates.
Does it impact AI x Science discovery agents? In the topics that the LLMs have gaps in “parametric” knowledge, agents using the model overlook potential discoveries -- even with tools and internet search.
The conclusion: training models more intensively on the latest scientific findings is a promising way to make scientific agents more useful!
“More information can be pulled into context. More information should be pulled into context. But there will always be a marginal query that the agent does not run, and the shape of this frontier is determined by internal knowledge.”
Link to research blog below:
English
