Mark Histed

I've been talking to AI models a lot, and I don't think they reason at a PhD level at all. They seem to be good at math style problems, where you tell them A, B and C are true, and then ask them to figure out D. They're extremely bad at anything involving what I would call mature scholarship. Basically where A, B, and C are partially confirmed to various extents in the literature, and there are multiple conflicting, competing perspectives on what might be true. When it comes to this, they reason like naive undergrads. They try to force everything into one box called "the truth". If a framework is a standard part of their training data, like Bayesianism, they do seem to be able to write about things from that perspective. But if they need to construct perspectives on the fly, and keep track of competing frameworks, based on a novel research direction, they easily get lost about who is saying what and why. This is basic scholarship. The ability to apprehend the state of the literature on a given topic. It is literally the minimum of what you need to do to be a PhD level scholar. And AI models are terrible at it.

A new paper published in Nature Astronomy says if LLM can easily replicate what counts as your scientific contribution, then the deeper problem is not the model, but the fact that the work was too routine, formulaic, or low-value to begin with. --- nature .com/articles/s41550-026-02837-2

The findings suggest that Psychology research, despite it's bad reputation, is overall more robust than Economics, which is often put forward as the gold standard of social science research.

Tyler Cowen in his new book on why economists write about topics outside their area of expertise: “The dirty little secret is that what distinguishes economics as a field, right now, is a mix of higher standards, harder work, better math, and higher IQs.” This is clearly true.