Bioinfhotep

This is more evidence that current frontier models remain completely reliant on content-level memorization, as opposed to higher-level generalizable knowledge (such as metalearning knowledge, problem-solving strategies...)

Terence Tao responding to a question on what advice he would give someone considering a career in math in 2026: 'Yeah, so we live in a time of change. It is, as I said, we live in a particularly unpredictable era. And I think things that we've taken for granted for centuries may not hold anymore. So, yeah, the way we... do everything, not just mathematics, will change. In many ways, I would prefer the much more boring, quiet era where things are much the same as they were 10 years ago, 20 years ago. But I think one just has to embrace that there's going to be a lot of change and that, you know, the things that you study, some of them may become obsolete or revolutionized, but some things will be retained. There'll be a lot of opportunities for things that you wouldn't be able to do before. So, I mean, in math, you previously had to basically go through years and years of education to be a math PhD before you could contribute to the frontier of math research. But now it's quite possible at the high school level or whatever, that you could get involved in a math project and actually make a real contribution because of all these AI tools and lean and everything else. So there'll be a lot of non-traditional opportunities to learn. So you need a very adaptable mindset. There'll be one for pursuing things just for curiosity, for playing around. And I mean, you still need to get your credentials. I mean, I think for a while it would still be important to sort of still go through traditional education and learn math and science and so forth the old-fashioned way for a while. Yeah, but you should also be open to very, very different ways of doing science, some of which don't exist yet. Yeah, so it's a scary time, but also very exciting.'

🚨 Shocking: Frontier LLMs score 85-95% on standard coding benchmarks. We gave them equivalent problems in languages they couldn't have memorized. They collapsed to 0-11%. Presenting EsoLang-Bench. Accepted to the Logical Reasoning and ICBINB workshops at ICLR 2026 🧵

🚨 Shocking: Frontier LLMs score 85-95% on standard coding benchmarks. We gave them equivalent problems in languages they couldn't have memorized. They collapsed to 0-11%. Presenting EsoLang-Bench. Accepted to the Logical Reasoning and ICBINB workshops at ICLR 2026 🧵

7/ After the paper was finalized, we ran agentic systems that mimic how humans would learn to solve problems in esoteric languages. We supplied our agents with a custom harness + tools on the same benchmark. They absolutely crushed the benchmark. Stay tuned 👀

Someone needs to get Hermes agent a genetic sequencing skill