Keyon Vafa

Can an AI model predict perfectly and still have a terrible world model? What would that even mean? Our new ICML paper formalizes these questions One result tells the story: A transformer trained on 10M solar systems nails planetary orbits. But it botches gravitational laws 🧵

Really interesting work exploring whether/why next-token predictors would encode "useless" features

@adilsoubki @MilesCranmer @rachelczhang The work done by these people really inspired me to work on building SymTorch: @keyonV @NeelNanda5 @ch402 @eigensteve @PeterWBattaglia

Say you have trained your deep learning model. It works. But do you know what it has actually learned? 🚀 We’ve built SymTorch: a library that translates deep learning models into human-readable equations. I've attached here a quick video demonstrating how SymTorch works.

Super interesting paper from @AnthropicAI Fellows Program on model breakdown as task complexity increases. The longer the model has to reason, the more unpredictable it becomes: not consistently wrong, not completely random, just pursuing strange goals that are neither systematically aligned nor misaligned. Reminds me of @keyonV and co. human generalization function paper. This research suggests that human beliefs about model performance will be increasingly miscalibrated at longer reasoning length.

New post with @AndreyFradkin on the obstacles for AI agentic commerce. Imagine an AI that could optimize your credit card points across airlines and hotels to book a trip to Tokyo. The technology exists, at least in theory. So why can't you use an agent to book a flight, or shop for groceries? There are two main obstacles: 1) The interface of the internet was optimized for human interactions. What's needed is a machine-readable internet twin. Problem: the platforms best positioned to develop this twin often have an incentive not to. 2) The regulatory framework for agentic commerce is a mess. In many cases, it's legally ambiguous whether you can "bring your own" agent to shop for you (and agent providers get sued); in other cases, a platform can simply revoke permission and deny service. Firms have incentive to develop their own agents that you interact with, rather than allow you to BYO an agent aligned with your interests. To foster competition and improve consumer welfare, a new regulatory framework is needed. We outline what one might look like. aleximas.substack.com/p/why-cant-you…

@BenSManning @alexolegimas @arpitrage Thanks Ben! I agree with your questions. Something I've been thinking a bit about is the behavioral work showing where heuristics work for people/where they don't. For LLMs, beyond the yes/no world model question, can we characterize where the lack of world models work vs don't?

@alexolegimas @arpitrage Also, if something like this measure of completeness along many dimensions exists, PLEASE TELL ME!

This is an awesome post @alexolegimas and @arpitrage - I've been thinking a lot about this stuff too. Screenshot is from a revision section I’ve been writing with @johnjhorton and @apo_filippas. The core point is pretty obvious, almost to the point of being boring: you don’t need a perfectly “true” model of the world to be useful. Newtonian mechanics is wrong, quantum is more right, Newton is still incredibly predictive over a huge domain. But what feels newly salient with LLMs is that we don’t actually have good ways to measure the differences that matter. How incomplete is a model’s world model in the dimensions that show up under intervention? Where does predictive competence stop being reliable when you push on the system? Work like @keyonV NYC navigation + planets papers makes this very concrete: strong surface-level prediction can coexist with brittle or incoherent internal structure that only reveals itself under the “wrong” perturbations (I'm obsessed with these papers - they're so good)

@universeinanegg Related to NYC traversal, you may find @keyonV’s paper relevant: arxiv.org/abs/2406.03689

New paper: We train Activation Oracles: LLMs that decode their own neural activations and answer questions about them in natural language. We find surprising generalization. For instance, our AOs uncover misaligned goals in fine-tuned models, without training to do so.

Many think LLM-simulated participants can transform behavioral science. But social scientists lack accessible discussion of what it means to validate these models. Under what conditions can we trust LLMs to learn about human parameters? Our paper maps the validation landscape. 1/

I wrote a review paper about statistical methods in generative AI; specifically, about using statistical tools along with genAI models for making AI more reliable, for evaluation, etc. See here: arxiv.org/abs/2509.07054! I have identified four main areas where statistical thinking can be helpful. These are just a subset of what is out there; other topics have been well-covered in other reviews. 1. Designing "statistical wrappers" around a model, for instance, changing behavior of a trained model (e.g., abstaining), where a score, e.g., an "unsafety score" is too high. The key connection to statistics is to use the quantiles of the loss (on a calibration set) to set the critical threshold, thus enabling conformal-type high probability guarantees. 2. Closely related, methods for uncertainty quantification, which enable the model to express uncertainty in an answer. A crucial component here is "calibration", whereby the uncertainty is required to reflect reality. 3. Statistical methods for AI evaluation: Specifically, tools for statistical inference (e.g., confidence intervals) on model performance. Exciting recent work proposes careful statistical models for leveraging a very small high-quality dataset, possibly combined with much larger low-quality datasets, for accurate evaluation. 4. Experiment design and interventions. Careful AI experiments to understand and steer models may require interventions such as modifying experimental settings in a controlled manner. This brings up connections to classical experimental design in statistics. This connection has largely remained implicit so far, and my review aims to make it more explicit; hoping that experimental design principles will become useful here. This review references the work of many, including @HamedSHassani @obastani @tatsu_hashimoto @yuekai_sun @CsabaSzepesvari @ml_angelopoulos @stats_stephen @yaniv_romano @yaringal @KilianQW @_onionesque +their teams, and some work that I was also involved in. Hopefully, my review will be helpful to orient yourself in this exciting area. Nonetheless, since the area is rapidly expanding, it is possible that I missed important references. Please feel free to let me know of anything that I should add/change!

@danielrock @alexolegimas @pontus_rendahl It's a good idea, not sure if I've seen it. It's similar to some of the world model papers that simulate data from a specific DGP (e.g. Othello here arxiv.org/abs/2210.13382 or chess here arxiv.org/abs/2102.13249) and test on distribution shifts

@alexolegimas @pontus_rendahl This is another fun world model question. Once again, hey @keyonV

Why not just simulate, say, and NK model and use the data to form a neural net (or whatever). If it uncovers the deep parameters, the discussion is settled no?

Can vision transformers learn without images?🤔👀 Our latest work shows that pretraining ViTs on procedural symbolic data (eg sequences of balanced parentheses) makes subsequent standard training (eg on ImageNet) more data efficient! How is this possible?! ⬇️🧵

Our final invited speaker Keyon Vafa @keyonV shares insights on Evaluating Implicit World Models! Right after this we'll have six contributed talks and our best paper awards 🏆

I'll be speaking at the Language, Agent, and World Models (LAW) Workshop this afternoon at 2:15 in Upper Level Ballroom 20D!

The San Diego party ain’t over yet🌴 LAW 2025 is happening today at @NeurIPSConf 🥳 in📍Ballroom 20D This is the first year of LAW workshop and we are honored to have a world-class lineup of speakers to reshape the directions on the unification of language, agent and world models: @sherryyangML @chelseabfinn @fchollet @philipjohnball Stephen Spencer @ericxing @danijarh Ying Nian Wu @keyonV 🚀Huge thanks to our sponsors: @LambdaAPI, nexovaai.io Also, thanks to our amazing co-organizers: @ziqiao_ma @realJessyLin @melaniesclar @jianwen_xie @KelseyRAllen @alsuhr @jacobandreas @tianminshu @ZhitingHu sites.google.com/view/law-2025 neurips.cc/virtual/2025/l… #NeurIPS #NeurIPS2025 #LLMs #AgenticAI #WorldModels #AI #ML #RL #Reasoning #EmbodiedAI #VLMs

🔥 We are pleased to announce the talk title for our #LAW2025 #NeurIPS workshop Join us on 📅Sunday, Dec 7th in📍Upper Level Ballroom 20D Full schedule here: sites.google.com/view/law-2025/… We look forward to seeing you all this Sunday! #NeurIPS2025 #AI #ML #WorldModel #Agents #LLMs

@danielrock @alexolegimas @alz_zyd_ @littmath @monomorphix13 You might also find interesting this paper showing that RL doesn't seem to improve general reasoning capabilities but rather extracts the right capabilities from the base model arxiv.org/abs/2504.13837

@alexolegimas @alz_zyd_ @littmath @monomorphix13 @keyonV

Read Wolfram's excellent "What is ChatGPT Doing..." (h/t @danielrock). He writes that we learned a lot about how language works from fact that GPT3, with only 175 billion weights, is able to emulate it so well. This implies it's computationally a lot simpler than we may have thought. But what about math? At time this was written (2023), GPT was still very bad at math. The models became very (very) good at math when the first reasoning model came out (o1), which relied a lot more on reinforcement learning rather than just brute force pretraining. Wonder what this says about math? Conceptually, language is a lot "fuzzier" than math: multiple words can sound "right" in the same spot in a sentence. This is what makes the probabilistic LLM architecture work. Math is less fuzzy. This is perhaps why the more "rule based" RL step was crucial. But this also implies formal math is less computationally complex than we thought. Thoughts? @littmath @alz_zyd_

Paper: arxiv.org/abs/2503.17482

In San Diego for NeurIPS! Today: Presenting a poster for our paper on steerability from 4:30-7:30 in Exhibit Hall CDE (#3715). Paper w/ Sarah Bentley, Jon Kleinberg, and @m_sendhil. Sunday: Giving a talk at the Language, Agent, and World Model workshop on testing world models.