Mayur Naik

It is easy to overlook but hard to overstate how big a role "test-time inference" has played in modern LLM gains in reasoning. I myself wasn't sure, so I put GPT-5 and Gemini-2.5 to a classic programming puzzle called variable shadowing (en.wikipedia.org/wiki/Variable_…): Both models in "Thinking" mode solved it correctly but took around 2 minutes. The thinking was elaborate (and correct): "I'm currently focused on rewriting the user's C-like code, aiming to replace all instances of 'y' with 'x'. However, I've hit a snag. The crucial part is to avoid this substitution in specific scenarios. " More surprisingly, both models in "Instant" mode got it consistently wrong, giving a glimpse into their pre-training levels of reasoning which seem to have hardly improved over the years if at all. Is there a middle ground which reliably answers this simple question correctly without thinking for too long? The best solution today remains prompt engineering. Appending "Think step by step before answering" to the prompt causes both models to engage in a brief monologue which turns out to be crucial to subsequently generating the correct program. The recent "Auto" mode of LLMs is intended to automatically strike this balance and forgo the need for prompt engineering. But it currently errs on the side of over-thinking, or worse, under-thinking. LLMs will undoubtedly get better at calibrating their "Auto"mode but I am equally excited about emerging architectures like diffusion models, which are not constrained by left-to-right generation and can iteratively refine their answer the way humans typically do.

@PennEngineers doctoral student @jiani_huang_ai (@cis_penn) presents ESCA at @NeurIPSConf 2025, a system that helps embodied AI agents better understand their surroundings by creating context-aware descriptions of a scene. Research advised by Professor Mayur Naik (@AI4Code).