Runpei Dong

Want to ask your humanoid to get novel objects you want from a novel table? We introduce HERO, first achieving open-vocabulary visual loco-manipulation using human language queries! Now we can ask the humanoid to see and grasp the target object (eg, a carrot instead of a tissue box) with whole-body coordination. How do we do it? Key designs and findings in this work: We propose a residual-aware end-effector tracking policy that tracks the target accurately in a closed loop. We find that the robot's forward kinematics are quite inaccurate, and thus we propose residual neural forward models that correct the end-effector FK and base leg odometry. We design a modular system powered by visual foundation models that achieves generalizable grasping capabilities with 83% success rate on novel daily objects and daily scenes. Check out our new project: Learning Humanoid End-Effector Control for Open-Vocabulary Visual Loco-Manipulation Project page: hero-humanoid.github.io Paper: arxiv.org/pdf/2602.16705

GR00T-VisualSim2Real is now open source! VIRAL and DoorMan are now available with training code, simulation assets, and the full recipe for bringing visual sim-to-real loco-manipulation skills to your own humanoids. Repo: github.com/NVlabs/GR00T-V…

I'm super excited and proud about this result. A fantastic team at Sony AI, led by Peter Duerr, put together just the right mix of science and engineering to create the first robot to beat a professional athlete in a real-world competitive sport!

A touch-aware humanoid manipulation policy that cleans the lab for you🧹🧪 Introducing Humanoid Touch Dream: a real-world system for dexterous, contact-rich humanoid loco-manipulation. Our key idea is simple: the policy predicts future hand forces and tactile latents alongside actions, within a single-stage training framework. humanoid-touch-dream.github.io 1/7

📢MoRight: Motion Control Done Right "What if your video model actually understood cause and effect?" Existing motion-controlled video models entangle camera and object motion, and treat everything as kinematic displacement. MoRight changes both. 🔥 Motion Causality — MoRight decomposes motion into actions & consequences. Give an action → MoRight predicts consequences (aka motion simulation) . Give a desired outcome → MoRight recovers the driving action (aka motion planning). Not merely displacing pixels. 🎬 Disentangled Control — MoRight separates camera and object motion, allowing users to independently control each of them. No entanglement. Project Page: research.nvidia.com/labs/sil/proje… Paper: arxiv.org/abs/2604.07348

I wrote a blog "Three Levels of TTT" — Test-Time Training, Meta Training, World Models, 3D & Self-Supervised Learning: evelinehong.github.io/ttt_three_leve… The three levels are: 🧠 Episode — hippocampus encodes fast, neocortex consolidates slow. No labels needed. 🌱 Individual Lifetime — there is no train/test split. Every minute is testing as well as training. 🌍 Natural Selection & Evolution — continuous adaptation integrates into the species' prior. Each level is the meta-training of the level below. Each level is the test-time training of the level above. Priors flow down to the lower level; consolidated adaptations flow up to the next higher level. 🧠 The self-supervised signal needs no labels — it comes from the structure of experience itself: what did I expect vs. what happened? What follows what? What appears together? 🌱 This consolidates across a lifetime — every minute is testing as well as training. Given the priors of the human species, an infant develops 3D perception, object permanence, intuitive physics — not from instruction, but from reaching, crawling, acting. The world teaches the rest through self-supervised learning. 🌍 But what gives us those priors? Two front-facing eyes, exactly the right distance apart for depth to emerge. Pain and proprioception as free error signals. A face-detection circuit running at birth. Billions of years of test-time feedback from individual lives, accumulated and frozen into hardware. Evolution doesn't optimize behavior — it optimizes the prior you start from.

Excited to introduce OmniClone, a robust teleoperation system for humanoid mobile manipulation. While systems like TWIST2 and SONIC paved the way, we put efforts into solving the critical stability and scaling gaps. 1/ 📊 Moving past "vibe-based" testing. We’ve built a comprehensive diagnostic benchmark to systematically evaluate whole-body teleoperation. No more trial-and-error—get the actionable insights needed for true policy optimization. 2/ 👤 Universal Human-to-Robot Mapping. Teleop often breaks when switching operators. OmniClone mitigates biases from hardware fluctuations and, crucially, diverse human body shapes, ensuring high-stability control regardless of the person in the suit. 3/ 🚀 System Optimizations for Whole-body Manipulation Policy. By optimizing for affordability and reproducibility, OmniClone provides the high-fidelity pipeline necessary to collect data and train humanoid whole-body policies at scale. fully The model checkpoints and deploy code are now fully released—welcome to play with it! 📦 📄 Paper: arxiv.org/abs/2603.14327 🌐 Project: omniclone.github.io 💻 Code: github.com/yixxuan-li/Omn…

Turning a behavior "Prior" to a high-performing "Pro" in hours⚡️with DICE-RL (Distribution Contractive RL Finetuning)💡 Check out @s_zhanyi 's 🧵 for the secret source 😉

🎾Introducing LATENT: Learning Athletic Humanoid Tennis Skills from Imperfect Human Motion Data Dynamic movements, agile whole-body coordination, and rapid reactions. A step toward athletic humanoid sports skills. Project: zzk273.github.io/LATENT/ Code: github.com/GalaxyGeneralR…

Humanoid visual loco-manipulation is extremely challenging, especially for tasks like apple pealing. This is a very impressive result, congrats!

For a long time, I was skeptical about action-conditioned video prediction for robotics. Many models look impressive, but once you ask them to handle long-horizon manipulation with real physical interaction, things quickly fall apart (e.g., Genie is amazing but mostly focused on navigation). This project changed my mind. I'm beyond excited to share Interactive World Simulator, a project we have been working on for the past ~1.5 years 🤖 One of the first world models that produces convincing results for long-horizon robotic manipulation involving complex physical interactions, across a diverse range of objects (rigid objects, deformables, ropes, object piles). It directly unlocks scalable data generation for robotic policy training and policy evaluation. Try it yourself (no installation needed): yixuanwang.me/interactive_wo… Play directly with the simulator in your browser. Key Takeaways: 1️⃣ 15 Hz long-horizon action-conditioned video prediction for 10+ minutes on a single RTX 4090 GPU 2️⃣ Visual and dynamic fidelity: people often ask how much sim data equals one real data point. In our experiments, it turns out to be close to one-to-one using the Interactive World Simulator 3️⃣ Stress testing matters: we emphasize interactive stress testing to understand robustness and stability and to build trust in the simulator 4️⃣ The model is trained with only ~6 hours of real-world random interaction data on a single GPU. Imagine what happens if we scale this 1000× or even 1M× Huge credit to @YXWangBot, who led this effort with countless hours of work on data collection, training recipes, and system design. I'm incredibly proud of the work he did here! Enjoy the demos and videos. We also fully open-sourced the codebase for anyone interested in applying this to their own tasks. #Robotics #RobotLearning #WorldModels #EmbodiedAI

Origami Robotics is building high-DOF robotic hands with in-joint motors and a co-designed data-collection glove to eliminate the embodiment gap by collecting high-quality, real-world data at scale. Congrats on the launch, @DanielXieee and @QuanliangX! ycombinator.com/launches/Pcl-o…

Excited to share our new work ULTRA! We explore how to enable autonomous humanoid loco-manipulation by specifying what should happen to the object, rather than commanding robot motions. ULTRA is a unified multimodal controller, supporting mocap reference tracking, object-goal control, and egocentric visual inputs.

@builds_robots Thanks!

@RunpeiDong congrats!

Want to ask your humanoid to get novel objects you want from a novel table? We introduce HERO, first achieving open-vocabulary visual loco-manipulation using human language queries! Now we can ask the humanoid to see and grasp the target object (eg, a carrot instead of a tissue box) with whole-body coordination. How do we do it? Key designs and findings in this work: We propose a residual-aware end-effector tracking policy that tracks the target accurately in a closed loop. We find that the robot's forward kinematics are quite inaccurate, and thus we propose residual neural forward models that correct the end-effector FK and base leg odometry. We design a modular system powered by visual foundation models that achieves generalizable grasping capabilities with 83% success rate on novel daily objects and daily scenes. Check out our new project: Learning Humanoid End-Effector Control for Open-Vocabulary Visual Loco-Manipulation Project page: hero-humanoid.github.io Paper: arxiv.org/pdf/2602.16705

Robust humanoid perceptive locomotion is still underexplored. Especially when different cameras see different terrains, paths get narrow, and payloads disturb balance... Introduce RPL, tackling this with one unified policy: • Challenging terrains (slopes, stairs and stepping stones); • Multiple directions; • Payloads; Trained in sim. Validated long-horizon in the real world. Watch the robot walk it all🦿 Details below👇

Thrilled to share our work AlphaOne🔥 at @emnlpmeeting 2025, @jyzhang1208 and I will be presenting this work online, and please feel free to join and talk to us!!! 📆Date: 8:00-9:00, Nov 7, Friday (Beijing Standard Time, UTC+8) 📺Session: Gather Session 4

ResMimic: learns a whole-body loco-manipulation policy on top of general motion tracking a policy Key ideas: (i) pre-train general motion tracking (ii) post-train task-specific residual policy with: (a) object tracking reward (b) contact reward (c) virtual object force curriculum

Humanoid motion tracking performance is greatly determined by retargeting quality! Introducing 𝗢𝗺𝗻𝗶𝗥𝗲𝘁𝗮𝗿𝗴𝗲𝘁🎯, generating high-quality interaction-preserving data from human motions for learning complex humanoid skills with 𝗺𝗶𝗻𝗶𝗺𝗮𝗹 RL: - 5 rewards, - 4 DR terms, - Proprio. ONLY, - NO history/curriculum. Ready for agile, human-like 🤖? (Best with 🎧) 🔗 omniretarget.github.io 🎥 1/9

Visual manipulation is really challenging for humanoids, and it is impressive to see such interesting results with a depth policy!

Check out Any2Track! Really impressive work on whole-body tracking!