Changdae Oh @ ICLR'26

Your LLM agent just mass-deleted a production database because it was confident it understood the task. It didn't. Avoiding these irreversible mistakes requires uncertainty quantification, a pressing open problem in the era of LLM agents. Check out our #ACL2026 paper: "Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunities" 🔍 Why this matters: LLM agents now book flights, modify databases, and execute code autonomously. Yet most UQ research still measures a single-turn QA setup. In contrast, agents follow multi-turn trajectories in which they interact with users, call tools, and receive environmental feedback. The gap between how we study UQ and how agents actually operate is enormous. ⚙️ A unified formulation: We present the first unified formulation of Agent UQ. It models the full trajectory (actions, observations, states) and decomposes uncertainty per turn via the chain rule. Under this formulation, single-step LLM UQ and multi-step reasoning UQ fall out as special cases. 🚧 Challenges: We identify four core challenges: from selecting the right UQ estimator when existing methods all break down in agentic settings to handling heterogeneous uncertainty sources (user, tools, environment) to the near-total lack of fine-grained agent benchmarks (we survey 44 and find that turn-level evaluation is extremely rare). 🌍 Implications and open problems: Agent UQ is the missing safety layer for healthcare agents triaging patients, SWE agents pushing code to prod, and agents controlling cyber-physical systems. We also surface open problems around solution multiplicity, multi-agent UQ, and self-evolving systems. We release code and data to help the community build on this. 📄 Paper: arxiv.org/abs/2602.05073 🌐 Project: agentuq.github.io 💻 Code: github.com/deeplearning-w… Huge shoutout to @changdaeoh, who spearheaded this effort. When we started the work, agent UQ was a loosely defined space with scattered ideas; Changdae brought the clarity, structure, and rigor that the field needed to move forward. Also thanks to all the collaborators: @seongheon_96 , To Eun Kim, @JiatongLi0418, @Wendi_Li_ , @Samuel861025 @xuefeng_du, Hamed Hassani, Paul Bogdan, Dawn Song

We've been in GRPO-tweaking mode for months (entropy bonuses, clipping hacks, length penalties). But what if the entire objective is wrong? Today, we're releasing LAD (Learning Advantage Distributions), the most elegant rethink of RL for LLM reasoning I've seen this year. #ACL2026 Here's the idea, how it works, and why we think it changes things. 🧵 The problem we kept hitting GRPO, DAPO, RLOO, and many other variants do the same thing at their core: maximize expected reward. And when you do that, your policy can collapse onto a single dominant reasoning path. Entrop regularization can act as a bolt onto the framework, but it doesn't fundamentally fix it from the ground up. The key insight 💡Stop maximizing. Start matching. We reframe the policy update as a distribution matching problem. Instead of pushing toward the single best response, we make the policy's output distribution match the full advantage-weighted target distribution by minimizing an f-divergence between the two (see our theory in Section 3.1). When you match the full advantage distribution, you naturally preserve probability mass across multiple valid reasoning paths. High-advantage responses get upweighted, yes, but the objective also suppresses overconfident probability growth on any single mode. Collapse prevention isn't an afterthought. What validated the theory We tested six divergence families. The result that convinced us we were on the right track: - Strict divergences (Total Variation, Hellinger, Jensen-Shannon) that enforce exact distributional matching consistently outperform weaker ones (such as KL). - The more faithfully you learn the full advantage distribution, the better the reasoning. This is exactly what the framework predicts. The results - In a controlled bandit setting. LAD recovers multiple-mode advantage distributions (see plot below). GRPO fundamentally cannot. This is the clearest demonstration that the paradigm difference is real, not just theoretical - In math and code reasoning tasks across multiple LLM backbones. LAD consistently outperforms GRPO on both accuracy AND generative diversity across benchmarks. Why this matters beyond benchmarks Pass@k scaling: If your model knows 5 valid reasoning paths instead of 1, sampling at inference becomes massively more effective. Simplicity: Instead of stacking "GRPO + entropy hack," you get one principled objective. Diversity preservation comes by design. Paper: arxiv.org/abs/2602.20132 Code is available; link in the paper. Huge credit to my amazing student @Wendi_Li_, who drove this work, thinks boldly, and made things happen.

I wrote a blogpost about writing machine learning research papers (e.g., NeurIPS, ICML, ICLR, etc.). The core idea is that most papers follow one of a predetermined set of templates. The post talks about each template, describes their rules, and offers examples...

Creating user simulators is a key to evaluating and training models for user-facing agentic applications. But are stronger LLMs better user simulators? TL;DR: not really. We ran the largest sim2real study for AI agents to date: 31 LLM simulators vs. 451 real humans across 165 tasks. Here's what we found (co-lead with @sunweiwei12).

🧵 Excited to share our new paper at #ICLR2026: LUMINA: Detecting Hallucinations in RAG System with Context–Knowledge Signals With @tanwimallick and @SharonYixuanLi Paper: openreview.net/pdf?id=oJgNNBN… Code: github.com/deeplearning-w… 1/N

Excited to announce our workshop on flow-based generative models at CMU: Frontiers of Flows for Generative AI March 26-27, Pittsburgh PA cmu-l3.github.io/flows2026/ We have an amazing lineup of featured talks, panel discussions, and lightning talks. Registration is now open!

When evaluating LVLMs, should we really be asking: “Did the model get the right answer?” or rather “Did the model truly integrate the visual input?” LVLMs can rely on shortcuts learned from the underlying language model, aka language prior. In our #ICLR2026 paper, we attempt to understand this phenomenon at a deeper, representation-level. 📄 “Understanding Language Prior of LVLMs by Contrasting Chain-of-Embedding”. arxiv.org/abs/2509.23050 ------- 1/ Problem: LVLMs often ignore visual evidence While LVLMs perform well on many benchmarks, they sometimes rely on language patterns rather than actual images. A simple example: show a model a green banana, and it may confidently describe it as “ripe and yellow” ---because that’s the most common linguistic pattern it has learned. 🍌 This raises a central question: Where inside the model does visual information begin to influence its reasoning? 2/ Motivation: Output-level probes fall short Most analyses inspect outputs, e.g., by removing the image or comparing predictions. But these methods cannot reveal when the model starts integrating vision and how strongly visual signals affect internal states. To address this, we need a representation-driven perspective. 🔍 3/ Approach: Contrasting Chain-of-Embedding (CoE) We trace hidden representations across the model’s depth for the same prompt: •once with the image •once without the image By comparing these trajectories layer by layer, we identify the exact point where visual input begins shaping the model’s internal computation. This leads to the discovery of the Visual Integration Point (VIP) ✨--- the layer at which the model “starts seeing.” We then define Total Visual Integration (TVI), a metric that quantifies how much visual influence accumulates after the VIP. 4/ Findings across 10 LVLMs and 6 benchmarks Across 60 evaluation settings, we observe: • VIP consistently appears across diverse architectures • Pre-VIP → representations behave like a language-only model • Post-VIP → visual signals increasingly reshape the embedding pathway • TVI correlates strongly with actual visual reasoning performance • TVI outperforms attention- and output-based proxies at identifying language prior TVI thus offers a more principled indicator of whether a model actually uses the image. 5/ Impact: A new lens on multimodal behavior Our framework has a few practical benefits. It enables (1) diagnosing over-reliance on language prior, (2) comparing LVLM architectures more rigorously, (3) informing better training and alignment strategies, and (4) improving robustness and grounding in real-world tasks. Shout out to my students for this insightful work: Lin Long, @Changdae_Oh, @seongheon_96 🌻 Please check out our paper for more details!

We’ve made huge strides in model & agent capability. Now it’s time to scale up measurement. We’re excited to support open benchmarks that capture every aspect of the brave new agentic world: complexity, long horizon, autonomy, and rich outputs. Work with us to make it happen!

Check out our #ICLR2026 oral paper (top ~1-1.5%). It's a slow-cooked research that probes a fundamental question many of you have wondered about: How do transformers actually learn semantic associations between tokens (e.g., “bird” and “flew”) during training? Semantic associations are foundational because they enable models to go beyond memorization and instead generalize and generate coherent text. tl;dr: This paper provides a formal theory for the emergence of semantic associations in attention-based language models, connecting training dynamics with linguistic insight and mechanistic interpretability. 📄 Read here: arxiv.org/abs/2601.19208 Congratulations to my students and co-authors: @shawnim00 @Changdae_Oh @Abell_Zhen_Fang

Really appreciate all the contributions and guidance from the awesome team😊🙏 students: @seongheon_96 @TEKnologyy @JiatongLi0418 @windy_lwd @Samuel861025 senior: @xuefeng_du @HamedSHassani Paul Bogdan @dawnsongtweets advisor: @SharonYixuanLi

We conclude with discussions on some promising applications and open problems that are going to be attractive future work directions.

[Fresh paper buzzing 🐝] Anyone else a bit scared by the extreme autonomy we’re seeing in OpenClaw (ClawdBot formerly)? We hope to move toward reliable autonomy: a new foundation & perspective on uncertainty quantification for LLM agents has just dropped arxiv.org/pdf/2602.05073

Excited to share our recent work selected as an ICLR Oral! 
We work towards answering how models learn to associate tokens and build semantic concepts. We find that early-stage features in attention-based models can be written as compositions of three basis features.

Large Language Models: A Mathematical Formulation arxiv.org/abs/2601.22170

🚀 Call for Papers! We invite submissions to #ICLR2026 workshop on Agentic AI in the Wild: From Hallucinations to Reliable Autonomy. This workshop focuses on reliability, uncertainty, and hallucination in agentic AI systems. 📅 Deadline: Jan 30, 2026 (AoE) 🔗 …ination-reliable-agentic-ai.github.io Co-organized with @Grigoris_c, Etsuko Ishii, @xuefeng_du, Katia Sycara.