Lorenzo

If you had to live in one of the regions, which would you choose?

MIT and Harvard argue LLMs are nowhere near doing real scientific discovery. They published a paper called “Evaluating Large Language Models in Scientific Discovery.” Every week, tech labs claim an LLM has made a breakthrough in biology, physics, or chemistry. But this proves they are faking it. For years, AI benchmarks have tested models using static, multiple-choice science trivia. Models ace these tests, leading everyone to believe AI is right on the verge of autonomous scientific discovery. Researchers built a new evaluation framework called SDE to test what happens when you take LLMs out of the multiple-choice quiz and put them into real, open-ended research projects. They tested frontier models across biology, chemistry, materials science, and physics. The results are sobering. When forced to handle the actual loop of discovery—proposing a testable hypothesis, designing simulations, running experiments, and interpreting ambiguous results iteratively, current LLMs fall apart. There is a massive, glaring performance gap between passing standard science benchmarks and doing real science. Why do they fail? Because real science requires iterative reasoning, handling imperfect evidence, and adapting to unexpected observations. LLMs are built to predict the next token based on existing internet data. They can regurgitate a textbook explanation of photosynthesis or quantum mechanics instantly. But when placed inside an uncharted loop where the textbook doesn't have the answer yet, they hit a wall. Worse still, the researchers discovered diminishing returns. Simply scaling up model sizes and adding raw compute isn't fixing the gap. Top-tier models from different providers share the exact same blind spots. We are miles away from general scientific superintelligence. The tech industry is selling a narrative that AI is about to automate labs, run clinical trials, and invent materials on autopilot. But right now, AI isn't doing science. It's just remembering it.

There are essentially two main options to remedy this: 1. Find ways to perform active inference, so that the model adapts its learned program in contact with a new data distribution at test time. Would likely lead to some meaningful progress, but it isn't the ultimate solution, more of an incremental improvement. 2. Change the training mechanism to something more robust than SGD, such as the MDL principle. This would pretty much require moving away from deep learning (curve fitting) altogether and embracing discrete program search instead (which I have advocated for many years as a way to tackle reasoning problems...)

To address the limits of deep learning and avoid stalling, the field of AI started by applying patch (1), which started being demoed 9 months later in December 2024 and has now become completely ubiquitous. However, long term, it is simply inevitable that AI will move to patch (2).

Worth noting that to this day, base LLMs (no test time compute) *still* perform poorly on the ARC 1 benchmark from 2019 (on unseen tasks) despite continued massive scaling (~100,000x since 2019). We can say that without the switch to the test-time compute paradigm, AI would still not be capable of the kind of advanced reasoning that sota systems display today. Scaling the single-pass, static, next-token prediction paradigm of the GPT-2 through GPT-4 era was running into a capability asymptote. To bypass this plateau, test-time adaptation was a necessary evolutionary step.