Yunhai Han

Can a robot acquire real-world dexterous manipulation skills from just human videos? Meet Video2Sim2Real: full-stack autonomous dexterous skill acquisition from a single RGB-D human manipulation video — without robot data or expert intervention. Project: video2sim2real.github.io Paper: arxiv.org/abs/2606.08828

1/ 👀Vision tells robots where to go. 👋Touch tells robots about the interactions. 2/ Visual policies from teleoperated robot demos and human videos are scaling fast — but they don't have paired tactile data, so they still fail at the last millimeter of contact-rich manipulation. ❓So here is the question: How can we adapt tactile feedback into pretrained visual policies? Humans solve this naturally. 1️⃣ Learn from demonstrations through vision: we understand the task structure and the motion priors. 2️⃣ Then practice with touch: we interact with the world, feel what happens, and refine the motion. 🚀 We introduce OmniTacTune: Policy-Agnostic Real-World RL for Tactile Residual Adaptation of Visual Policies ⬇️ ✦ Adapting tactile feedback into existing visual policies ✦ No offline tactile demonstrations ✦ Real-world RL in 40–80 minutes, 5–40% → 85–100% ✦ Works across human flow policies, ACT, DP, π0.5 ✦ Works across different tactile representations 🌐 Website: colinyu1.github.io/omnitactune-si… 📖 Paper: arxiv.org/abs/2607.03723 📷 Video: youtube.com/watch?v=anRD7l… In this demo, we show four challenging contact-rich manipulation tasks and a 40-mins, one-take recording of an online RL training demo. 1/n

Real robot data is expensive. Real robot evaluations are slow. Excited to share SimFoundry - a system that turns real scenes into sim-ready worlds for training and benchmarking robots at scale - ✅Automated Scene Reconstruction with asset generation ✅Handles clutter, articulated objects, multiple robot embodiments ✅High Correlation Real-to-Sim Evals ✅Zero-shot Sim-to-Real ✅Generates diverse digital cousins Less manual environment authoring, more scalable feedback for robot learning. 🌐research.nvidia.com/labs/gear/simf… 🧵1/9

Check out our paper on learning forceful manipulation by separating task planning from interaction control!

@SanSanychUA Yes all codes will be released soon!

@HanYunhai The keyframe-based refinement idea maps elegantly to industrial digital twin workflows. In scan-to-CAD, we face the same problem — you cannot densely reconstruct every bolt and flange. Identifying contact/interaction/detachment moments in the physical asset lifecycle and optimizing geometry only at those critical keyframes is exactly how production-grade digital twins avoid the point-cloud-to-nowhere problem. Are you planning to release the simulator as open-source?

@HondaInvestor Yes and no. Yes - learning from human video is getting hot! No - there is significant difference in the method and tasks, meaning that there is still no unified solution yet. But I believe with more ideas exchanged we can make the learning framework more and more powerful.

@HanYunhai Is everyone working on the same thing? x.com/notmahi/status…

Can a robot acquire real-world dexterous manipulation skills from just human videos? Meet Video2Sim2Real: full-stack autonomous dexterous skill acquisition from a single RGB-D human manipulation video — without robot data or expert intervention. Project: video2sim2real.github.io Paper: arxiv.org/abs/2606.08828

Beyond EgoEngine and Video2Sim2Real: Turning Human Videos into Robot Experience: @liuyangcen112358/beyond-egoengine-and-video2sim2real-turning-human-videos-into-robot-experience-af0810f3b24c" target="_blank" rel="nofollow noopener">medium.com/@liuyangcen112…

Takeaways: - Retargeting a robot trajectory from a human video alone is insufficient; effective execution requires object-centric refinement that leverages hand-object interaction cues extracted from the video. - Such refinement does not need to be applied continuously over the entire trajectory; instead, it can focus on a small set of key manipulation frames that capture the most critical interaction effects. - Since this refinement is performed in a digital-twin simulator, reliable real-world deployment also requires effective sim-to-real transfer. To this end, we introduce a decoupled transfer strategy in which global IL adapts to geometric variations, while residual RL handles contact and physics discrepancies. Big thanks to the team! Yunhai Han, Jianuo Qiu @JianuoQiu, Linhao Bai @lbai46, Ziyu Xiao, Zihang Zeng, Yangcen Liu @Randle_Liu, Zhaodong Yang, Shalin Jain, Wenrui Ma, Jiaqi Fu @JiaqiFu17693, Yuqian Zheng, Manisha Natarajan, Muhammad Zubair Irshad @mzubairirshad, Kenneth Shaw @kenny__shaw, Matthew Gombolay @MatthewGombolay, Zsolt Kira @zsoltkira, and Harish Ravichandar @h_ravichandar.

We evaluate on 7 everyday dexterous tasks, including fruit placement, steak seasoning, toy rearrangement, tissue handover, book passing, and tray retrieval. In real-world trials with object-pose variations, Video2Sim2Real achieves a 95.7% success rate, significantly outperforming both RL-only and IL-only sim-to-real methods. More task videos and results are available on our project website.

@tomssilver Hi Tom, I really appreciate that you like our work! We’re also working on extensions to incorporate visual inputs and other capabilities, which we hope to publish soon.

This week's #PaperILike is "On the Utility of Koopman Operator Theory in Learning Dexterous Manipulation Skills" (Han et al., CoRL 2023). This and others have convinced me that I need to learn Koopman! Another perspective on abstraction learning. PDF: arxiv.org/abs/2303.13446

Project website: sites.google.com/view/immimic Arxiv Link: arxiv.org/abs/2509.10952 Big thanks to: @wshin1726 @HanYunhai @DanielZhenyang @h_ravichandar @danfei_xu

What if one unified method helps robots learn from human videos across many tasks, many robots? Meet ImMimic: Cross-Domain Imitation from Human Videos via Mapping and Interpolation (CoRL 2025 Oral Presentation🏆) @ICatGT Check it here sites.google.com/view/immimic!

Can robots learn from human videos for different embodiments? ImMimic: leveraging DTW mapping and MixUp interpolation to co-train from retargeted human hand poses and robot demonstrations. RSS Dexterous Manipulation Workshop: dex-manipulation.github.io/rss2025/.

#CoRL2024 accepted!🌈 Our work KOROL developed a linear dynamics model using object features that capture key information for robotic manipulation, outperforming models that rely on GT object states. Code: github.com/hychen-naza/KO…

Introducing MimicTouch, our new paper accepted by #CoRL2024 (also the Best Paper Award at the #NIPS2024 TouchProcessing Workshop). MimicTouch learns tactile-only policies (no visual feedback) for contact-rich manipulation directly from human hand demonstrations. (1/6)

@bauzavillalonga Just applied!

I'm hosting a student researcher at Google DeepMind in 2024! If you or some you know is interested in robotic manipulation, multi-modal learning, and want to work at London GDM then apply by Dec 15 (note it is tight!). Link: shorturl.at/fivV6 and lmk if you applied!