Junwon Seo

Video world model imaginations🌎💭can miss critical but plausible outcomes of robot actions. Introducing 𝙎𝙩𝙧𝙚𝙨𝙨𝘿𝙧𝙚𝙖𝙢: inference-time steering for video WMs, imagining plausible✅, high-impact⚠️ futures for 𝙧𝙤𝙗𝙪𝙨𝙩🛡️ policy evaluation and improvement. (1/15)

Diffusion / flow-based robot policies unlock two axes of test-time scaling: sequential (denoising steps) and parallel (samples). Both improve performance but cost latency on a robot, and knowing which to scale a priori is often unclear. ELASTIC learns it via RL!

I’m presenting Uncertainty-Aware Policy Steering at #RSS2026 in Sydney 🇦🇺 this week! ⏰ Imitation Learning 2 session on Wednesday, July 15 at 3:30pm Feel free to reach out if you’d like to chat about this work or anything else - super happy to connect!

1/ Inference-time policy steering lets robots imagine possible futures and pick the best one—but what if the best-looking future still feels wrong? In ViTaL, we argue that for contact-rich tasks, steering robots by using visual futures 👁️ 🌍 is not enough. Robots should also verify what they expect to feel 🤚🌍 as a result of their actions.

How do we make robot policies robust to rare but high-impact failures? Video #World #Models (WMs) are rapidly becoming a powerful tool for robotics, enabling policy evaluation and improvement by "imagining" future outcomes. But there's a catch: these imagined futures are typically nominal samples, making it easy to overlook the rare yet safety-critical events that matter most. In our new paper, StressDream: Steering Video World Models for Robust Policy Evaluation and Improvement, we explore a simple but powerful idea: 💡 Instead of passively sampling futures, actively steer world model imaginations toward high-impact yet still plausible scenarios. StressDream optimizes the initial diffusion noise at inference time, allowing us to generate targeted stress-test scenarios without retraining the world model. This enables: - More robust policy evaluation by exposing failure modes that random sampling often misses. - Improved policy optimization by training against challenging but realistic imagined futures. As generative world models become a foundation for #Physical #AI, the ability to systematically probe their "long tail" of plausible futures will be increasingly important for building reliable and trustworthy autonomous systems. 📌 𝖯𝗋𝗈𝗃𝖾𝖼𝗍 𝖯𝖺𝗀𝖾: junwon.me/StressDream/ 📄 𝖯𝖺𝗉𝖾𝗋: arxiv.org/abs/2606.00267 Work led by Junwon Seo, with a great set of collaborators: Sushant Veer, Thomas Ran Tian, Wenhao Ding, Apoorva Sharma, Karen Leung, Edward Schmerling, Andrea Bajcsy. @NVIDIADRIVE @NVIDIAAI #Robotics #WorldModels #PhysicalAISafety #AISafety #AutonomousSystems #RobotLearnin

I’m organizing the “𝗥𝗲𝘁𝗵𝗶𝗻𝗸𝗶𝗻𝗴 𝗨𝗻𝗰𝗲𝗿𝘁𝗮𝗶𝗻𝘁𝘆 𝗳𝗼𝗿 𝗠𝗼𝗱𝗲𝗿𝗻 𝗥𝗼𝗯𝗼𝘁𝗶𝗰𝘀 𝗣𝗮𝗿𝗮𝗱𝗶𝗴𝗺𝘀” workshop 🤖⁉️ at IROS 2026. Consider submitting a paper or attending 👇: 🌐 Website: lnkd.in/gmRpWBnY 📅 Workshop Date: September 27, 2026

*Very* impressive work from Junwon--he is able to generate synthetic but plausible rare events from a pre-trained video world model (WM): spilling beans during pouring, near-misses during driving, etc. One step closer to doing robust policy evaluation and improvement in WMs!

@vai_viswanathan Thanks for the wonderful summary!

Are robot policies susceptible to imposter syndrome? StressDream steers video world models to imagine the worst plausible outcome of an action. - Detecting task failures in imagined rollouts jumps from 54% to 94% recall, and using that to improve a robot policy lifts its real success rate from 39% to 71% . - optimizes the diffusion model’s initial noise toward a text-specified event (“coffee beans spill”), using a VLM for the semantic gradient and a “typical set” constraint that keeps the noise from drifting out of distribution into implausible video. - Why it matters: this approach surfaces real long-tail risks without hallucinating, letting policies be stress-tested against worst-case futures before they touch the real world.

This work was 𝒉𝒂𝒑𝒑𝒊𝒍𝒚 done in collaboration with the Autonomous Vehicle Research Group at NVIDIA, with @Veer_Sushant, @tianran_, @wenhaoding95, @apoorva__sharma, Karen Leung, Edward Schmerling, and @drmapavone, under the guidance of @andrea_bajcsy. (15/15)

There are many more video results on our interactive website and in the paper. You can also try StressDream with the code: 🌐Project Page: junwon.me/StressDream 📄Paper: arxiv.org/abs/2606.00267 💻Code: github.com/CMU-IntentLab/… (14/15)

Video world model imaginations🌎💭can miss critical but plausible outcomes of robot actions. Introducing 𝙎𝙩𝙧𝙚𝙨𝙨𝘿𝙧𝙚𝙖𝙢: inference-time steering for video WMs, imagining plausible✅, high-impact⚠️ futures for 𝙧𝙤𝙗𝙪𝙨𝙩🛡️ policy evaluation and improvement. (1/15)