Bendall Lab

This paper from Harvard and MIT quietly answers the most important AI question nobody benchmarks properly: Can LLMs actually discover science, or are they just good at talking about it? The paper is called “Evaluating Large Language Models in Scientific Discovery”, and instead of asking models trivia questions, it tests something much harder: Can models form hypotheses, design experiments, interpret results, and update beliefs like real scientists? Here’s what the authors did differently 👇 • They evaluate LLMs across the full discovery loop hypothesis → experiment → observation → revision • Tasks span biology, chemistry, and physics, not toy puzzles • Models must work with incomplete data, noisy results, and false leads • Success is measured by scientific progress, not fluency or confidence What they found is sobering. LLMs are decent at suggesting hypotheses, but brittle at everything that follows. ✓ They overfit to surface patterns ✓ They struggle to abandon bad hypotheses even when evidence contradicts them ✓ They confuse correlation for causation ✓ They hallucinate explanations when experiments fail ✓ They optimize for plausibility, not truth Most striking result: `High benchmark scores do not correlate with scientific discovery ability.` Some top models that dominate standard reasoning tests completely fail when forced to run iterative experiments and update theories. Why this matters: Real science is not one-shot reasoning. It’s feedback, failure, revision, and restraint. LLMs today: • Talk like scientists • Write like scientists • But don’t think like scientists yet The paper’s core takeaway: Scientific intelligence is not language intelligence. It requires memory, hypothesis tracking, causal reasoning, and the ability to say “I was wrong.” Until models can reliably do that, claims about “AI scientists” are mostly premature. This paper doesn’t hype AI. It defines the gap we still need to close. And that’s exactly why it’s important.

Celebrating Christina Curtis and her well-deserved Paul Marks Prize! Some tumors are born bad. Christina uses math to find them early, predict their paths, and outsmart them. A win for cancer biology and for MSK to cheer.

I am deeply honored to be named by the Stanford Medicine Alumni Association this year for the “Arthur Kornberg and Paul Berg Lifetime Achievement Award in Biomedical Sciences.” As a Stanford graduate student in the 1980s, I took a remarkable class in plasmid and phage replication from Dr. Kornberg. I marveled at his incisive manner of teaching complex concepts in science. I learned fundamentals of retroviral gene transfer from postdoctoral fellows and technical staff in Dr. Berg’s laboratory. I was in constant awe of Dr. Berg’s statesmanship on the world stage as a spokesman for science. Arthur Kornberg, M.D. and Paul Berg, Ph.D. were luminaries long before I was a graduate student. Professor Kornberg won the Nobel in Physiology or Medicine for isolating DNA polymerase and demonstrating how it could replicate DNA in a test tube. Professor Berg won the Nobel in Chemistry by showing the first inter-species DNA hybrid of bacterial and eukaryotic viral DNA and replicating it in bacteria. Professor Berg was instrumental in pointing out the dilemmas around recombinant DNA at the foundational Asilomar Conference in 1975, a critical meeting focusing on the ethical uses of rec-DNA technology. Dr. Kornberg’s and Dr. Berg’s efforts deeply touched biomedical sciences and helped patients worldwide. They symbolized science's dual role in fundamental discovery and service to humanity.