Kangrui Wang

📢 Friendly reminder: the Foundation Models Meet Embodied Agents (FMEA) Workshop @ CVPR 2026 is hosting 4 embodied AI challenges, and submissions are OPEN! 💰 Each challenge offers $500 / $300 / $200 cash prizes for the top 3 teams. Whether you're working on VLMs, LLM planning, or VLA policies, there's a challenge waiting for you! 🧠 ENACT: Evaluating Embodied Cognition Do VLMs understand how the world evolves from egocentric interaction? Forward & inverse world modeling in a POMDP framework. Challenge page: enact-embodied-cognition.github.io/challenge/ Submission portal: eval.ai/web/challenges… Test set submission deadline: May 25, 2026 🏠 EmbodiedBench: Vision-Driven Embodied Agents Benchmark MLLMs on household task planning (EB-ALFRED) & spatial navigation (EB-Navigation). Open-source models <10B params. Challenge page: embodiedbench.github.io/challenge.html Submission portal: eval.ai/web/challenges… Test set submission deadline: May 25, 2026 🧩 Embodied Agent Interface: From Thought to Action Evaluate LLMs on Goal Interpretation, Subgoal Decomposition, Action Sequencing & Transition Modeling across BEHAVIOR & VirtualHome. Challenge page: eai-challenge-cvpr2026.github.io Submission portal: eval.ai/web/challenges… Test set submission deadline: May 25, 2026 🤖 RoboMME: Memory for Robotic Generalist Policies 16 manipulation tasks testing temporal, spatial, object & procedural memory. Can your model remember and act? Challenge page: robomme.github.io Submission portal: docs.google.com/forms/d/e/1FAI… Test set submission deadline: May 23, 2026 All rules, evaluation details, and starter kits can be found on each challenge's page. If you have any questions or need help getting started, join the Slack channel linked on each challenge page. That's our official communication channel where organizers are actively responding. There's still plenty of time to jump in. We'd love to see what the community builds. Looking forward to your submissions, and see you at CVPR! 🚀

@zhouzhibo2008 @wzenus 🤪

@James_KKW @wzenus KR哥哥好强

I’ve been applying RV filtering to the agentic tasks I’m working on and it boosts performance almost everywhere 🚀🚀 Surprisingly effective and highly recommend people give it a try.💪💪

Check out our latest SAIL blog post on VAGEN, a reinforcement learning framework that trains VLM agents to build internal world models through explicit visual state reasoning! ai.stanford.edu/blog/vagen/

📍 Can LLMs discover, abstract, and reuse higher-level tool skills across tasks? Existing tool-use benchmarks test solving tasks with fixed tools. But real workflows contain recurring structures where efficiency comes from reusable tool compositions, not isolated calls. We introduce SkillCraft: 126 tasks across 6 domains designed to test whether LLM agents can acquire compositional skills, not just call atomic tools. We also propose Skill Mode, a lightweight protocol with four MCP primitives that let agents compose, verify, cache, and reuse tool chains at test time. Our Key findings across evaluating 8 SOTA models: ⚡Skill Mode enables agents to self-discover and reuse skills, leading to higher success and efficiency than agents without it. The gains are larger for stronger models. 🧠 Stronger models (e.g., Claude) discover more generalizable skills, which transfer across tasks and even across models. 🔍 Deeper composition ≠ better — shallow, well-tested skills generalize best. 🔗 Paper: arxiv.org/abs/2603.00718 💻 Code: github.com/shiqichen17/Sk… 🏠 Page: skillcraft-website.github.io/page (1/7)

Welcome submissions to our FMEA workshop @ CVPR 2026 on foundation models for embodied agents 🚀

📍Theory of Space (accepted at #ICLR2026) Theory of Mind → hidden mental states Theory of Space → hidden spatial beliefs from passive observers “What do I know?” to active explorers “What don’t I know, and how do I reduce that uncertainty?” Theory of Space is to evaluate if foundation models can actively construct, revise, and exploit internal spatial beliefs. We quantify Active-Passive Gap. Not just measure task accuracy, but how much uncertainty is reduced per step, and how many steps are needed in total for agents to build stable spatial beliefs. Exploration should prioritize information gain and reduce uncertainty per step. Instead, we observe LLMs/VLMs explore redundantly with stalled belief updates. Key findings: 1. Active agents perform worse than rule based programs 2. Cognitive Map Failures & Belief Drift (beliefs about previously observed objects degrades over time; new updates corrupt earlier correct perceptions) 3. Poor Vision Identification & Belief Inertia in Belief Revision Website: theory-of-space.github.io Code: github.com/mll-lab-nu/The… Data: huggingface.co/datasets/MLL-L… Theory of Space is a joint effort of @NorthwesternEng, @StanfordAILab, @uwcse, @Cornell_CS. Led by the amazing @WilliamZhangNU, jointly done with @zihanhuang66, @YueYuew8314, @JieyuZhang20, @XLe41402, @wzihanw, @qineng_wang, @keshigeyan, @RuohanZhang76, @YejinChoinka, @RanjayKrishna, @jiajunwu_cs, @drfeifei

Today is the day! Cohosted with BEHAVIOR challenge, come and check the winning team solutions!

While discussing spatial intelligence of "VLMs", wanted to share an interesting finding we have in ICML25 paper: We actually opens the black box of why VLMs fail at even the simplest spatial question "where is A to B" - 90% of tokens are visual, yet they get only ~10% of the attention!! also, there's classic "attention sink" but for vision tokens - Just giving vision more attention? → Doesn't help - Sharpening attention to excite the correct part works: seeing more → seeing more on "right" part - Adaptive sharpening (sharpen when model is confident, smooth when unsure): controlled_A on What's Up benchmark: 48.2 → 98.2% This simple intervention is not a robust method, but more a close look to the attention behavior of VLMs in Spatial Reasoning from a mechanism interpretability lens (1/n): 1. What causes these failures? 2. How do these failures manifest through internal patterns? 3. Can we mitigate these errors by leveraging the identified signals Work done with @shiqi_chen17 @tongyao_zhu @ruochenz1018 @jinghan23 @jcniebles @megamor2 @junxian_he @jiajunwu_cs Siyang Gao.

Spatial intelligence has long been one of the biggest bottleneck for VLMs. Two years ago in Sept 2023, when I just started my postdoc, I still remember vividly how we are excited about GPT-4V and how our “What GPT-4V still can’t do” slides were completely dominated by geometric and spatial failures. Then I have learned so much about 3D and physical understanding from @jiajunwu_cs @drfeifei ever since. Now in 2025, spatial intelligence has become one of the center topics for VLM and multimodal community! MLL lab collected an Github Repo of the latest advances in Spatial Intelligence for VLMs. This is a pure community effort. If you are working on spatial intelligence in VLMs, please feel free to contribute! 🙌 Let us build this resource together! #SpatialIntelligence #VLM #MultimodalAI #AI

Fantastic work on self-play agents and world models, excited to be part of it!

Happy to contribute to the work😎😎

Special thanks to @qineng_wang for leading this awesome project!

How do AI models 'think' about space? Our latest research dives into the mental models that VLMs use to tackle spatial problems 🤖🧩

Can VLMs build Spatial Mental Models like humans? Reasoning from limited views? Reasoning from partial observations? Reasoning about unseen objects behind furniture / beyond current view? Check out MindCube! 🌐mll-lab-nu.github.io/mind-cube/ 📰arxiv.org/pdf/2506.21458 🤗huggingface.co/datasets/MLL-L… 👩‍💻github.com/mll-lab-nu/Min…

Finally available on arxiv! arxiv.org/abs/2506.18945

Special thanks to @Sanjana for the great leadership!

Excited to be part of this Best Paper Award-winning project! See our demos of natural language-controlled Franka robot stacking cube pyramids!

We are very excited announcing our MLL lab! We are looking for collaborators on RAGEN, VAGEN, Chain-of-experts, T*, LongVideoHaystack, foundation models for embodied agents, etc mll-lab-nu.github.io

Why does your RL training always collapse? In our new paper of RAGEN, we explore what breaks when you train LLM *Agents* with multi-turn reinforcement learning—and possibly how to fix it. 📄 github.com/RAGEN-AI/RAGEN… 🌐 ragen-ai.github.io 1/🧵👇