Ryan Hoque

192 posts

Ryan Hoque

Ryan Hoque

@ryan_hoque

Robotics Research @Meta. Ex-Apple, PhD @berkeley_ai

San Francisco, CA Beigetreten Eylül 2021
385 Folgt2K Follower
Angehefteter Tweet
Ryan Hoque
Ryan Hoque@ryan_hoque·
Imitation learning has a data scarcity problem. Introducing EgoDex from Apple, the largest and most diverse dataset of dexterous human manipulation to date — 829 hours of egocentric video + paired 3D hand poses across 194 tasks. Now on arxiv: arxiv.org/abs/2505.11709 (1/4)
English
15
91
605
113.5K
Ryan Hoque
Ryan Hoque@ryan_hoque·
Thrilled to see egodex continue to unlock research breakthroughs! Check this one out from colleagues in FAIR
Amir Bar@_amirbar

A year ago, I took a big bet and shifted my research to world models. We started with navigation, but the vision was broader: simulate any interaction with the environment, including fine grained manipulation. Today we introduce DexWM, a world model for dexterous manipulation. Trained on 900+ hours of human and robot video, DexWM lets us imagine, plan, and execute dexterous actions on a real robot.

English
0
0
5
880
Ryan Hoque retweetet
Amir Bar
Amir Bar@_amirbar·
very interesting question and so much to unpack. it started in the micro-kitchen. @DavidJFan suggested we talk to @JimmyTYYang1 to brainstorm on how to deploy a WM on a Franka arm. We knew we have a model architecture ("CDiT") which is likely to work. But we missed the right human video training data and a roboticist with capacity to lead. Then: 1) EgoDex, a new large-scale dataset by Apple dropped 2) @raktimgg joined our team and brought the expertise we didn't have. he immediately saw the potential.
Karim C@BrandGrowthOS

@_amirbar Year-long research bets are where breakthroughs hide, but also where most teams run out of runway. What was the signal that told you manipulation was ready to move from theory to implementation?

English
4
3
47
34.3K
Ryan Hoque
Ryan Hoque@ryan_hoque·
Big update: I've left Apple. It’s been a blast pushing the frontier of learning dexterity from human video. I've now joined Meta’s new Robotics Studio as a Research Scientist, where I'll be building some exciting new products with super talented people. Stay tuned!
English
28
6
610
65.7K
Ryan Hoque
Ryan Hoque@ryan_hoque·
Some nice work on EgoDex visualization by @pablovelagomez1
Pablo Vela@pablovelagomez1

Continued working on the ego-dex dataset, I ported the entire test set to @rerundotio and created a @Gradio app to view it! Links below VVV This allows for a straightforward way to explore each episode of the (test) dataset and better understand how the hand-tracking and slam systems performed. I had to sadly reencode the videos to AV1, which took up a ton of time (nearly 2 hours of wall time for just the test dataset) Next up is taking this representative dataset and making it amenable to training. I'll start with something easy, such as pose estimation, as it's what I'm most familiar with, but the goal is to allow RRD <-> Webdataset standard.

English
1
0
5
1.7K
Ritwik Gupta 🇺🇦
Ritwik Gupta 🇺🇦@Ritwik_G·
I'm excited to share that I’ll be joining @UofMaryland as an Assistant Professor in Computer Science, where I’ll be launching the Resilient AI and Grounded Sensing Lab. The RAGS Lab will build AI that works in chaotic environments. If you would like to partner, please DM me!
English
32
20
333
31.6K
Xiaolong Wang
Xiaolong Wang@xiaolonw·
Got my tenure! Very grateful to my students and collaborators.
Xiaolong Wang tweet media
English
95
2
980
33.9K
Michael Black
Michael Black@Michael_J_Black·
@ryan_hoque Looks great. You mention that it’s now public but I don’t find the link anywhere.
English
2
0
5
2.1K
Ryan Hoque
Ryan Hoque@ryan_hoque·
Imitation learning has a data scarcity problem. Introducing EgoDex from Apple, the largest and most diverse dataset of dexterous human manipulation to date — 829 hours of egocentric video + paired 3D hand poses across 194 tasks. Now on arxiv: arxiv.org/abs/2505.11709 (1/4)
English
15
91
605
113.5K
Ryan Hoque
Ryan Hoque@ryan_hoque·
@pablovelagomez1 See Appendix A.4. Make sure to replace [filename] appropriately (try test)
English
1
0
3
326
Pablo Vela
Pablo Vela@pablovelagomez1·
@ryan_hoque Awesome! I can’t find the link for the dataset, where can I find it?
English
1
0
0
329
Ryan Hoque retweetet
Peide Huang
Peide Huang@peide_huang·
🚨Introducing EgoDex, the largest ego-centric video dataset to-date that focuses on human dexterous manipulation, with structured annotations including 3D upper-body and hand tracking🤲, camera pose📷, and language annotation💬. Kudos to the team and looking forward to what the community can cook from it. Checkout our preprint on arXiv, and data is available for downloading NOW. I am at Atlanta attending ICRA. DMs are open and happy to chat in person. 📄Preprint: arxiv.org/abs/2505.11709 #ICRA #robotics #imitationlearning #dexterousmanipulation
Ryan Hoque@ryan_hoque

Imitation learning has a data scarcity problem. Introducing EgoDex from Apple, the largest and most diverse dataset of dexterous human manipulation to date — 829 hours of egocentric video + paired 3D hand poses across 194 tasks. Now on arxiv: arxiv.org/abs/2505.11709 (1/4)

English
2
7
62
18.5K
Ryan Hoque
Ryan Hoque@ryan_hoque·
The full dataset is now publicly available to the community, access details are in the paper. Sample code for data loading is coming soon. Enjoy!
English
2
2
14
1.9K
Ryan Hoque
Ryan Hoque@ryan_hoque·
We also propose new benchmarks and train imitation learning policies for dexterous trajectory prediction. Below are 30 Hz wrist and fingertip trajectories on the test set, where blue = ground truth, red = model predictions, and points get lighter up to 2 seconds in the future.
Ryan Hoque tweet media
English
1
2
10
2.3K
Allen Ren
Allen Ren@allenzren·
It has been a wonderful journey with Ani over the years, and I couldn’t have imagined a more brilliant and supportive advisor and friend. Ani shed light on my research when I knew little in robot learning and reminded me to always think beyond the envelope. All the best Ani!
Anirudha Majumdar@Majumdar_Ani

Congratulations Dr. Allen Ren @allenzren! What an incredible honor it's been to have you in our lab over the past 5.5 years, and to learn from you. Very excited to follow your next steps in bringing general-purpose AI into the physical world!

English
17
3
149
11.7K
Ryan Hoque retweetet
Peide Huang
Peide Huang@peide_huang·
🚀 New Research on Human-Robot Interaction! 🤖 How can humanoid robots communicate beyond words? Our framework, EMOTION, leverages Large Language Models (LLMs) to dynamically generate expressive gestures, enhancing non-verbal communication in robots. 🤯 Our experiments show that EMOTION can generate various expressive gestures from only TWO examples and match human-generated gestures in understandability & naturalness! 🔍 What’s inside? ✅ LLM-powered motion generation ✅ Human feedback to refine gestures (EMOTION++) ✅ 10 expressive gestures generated and evaluated (thumbs-up, stop, jazz-hands & more!) 📜 Read the full paper: arxiv.org/abs/2410.23234 🎬 Watch the video: machinelearning.apple.com/research/emoti… Let’s bring robots closer to human-like interactions! What gestures would you like to see next? 👇 Huge kudos to the amazing team at Apple that made this work @Yuhan_Hu_, Nataliya Nechyporenko, @talking_kim, @waltertalbott, @jian_zhang_. #Robotics #HRI #LLMs #HumanRobotInteraction #GestureGeneration #SocialRobots
English
15
82
392
26.6K
Ryan Hoque
Ryan Hoque@ryan_hoque·
@eshear Goodhart's Law, one of my favorite
English
0
0
0
117
Ryan Hoque
Ryan Hoque@ryan_hoque·
@oier_mees Thanks @oier_mees ! Latency is an issue, but there are ways to improve. For example, a faster IK solver (Genesis? ;)) could help as we are running IK for each new hand pose
English
1
0
1
326
Ryan Hoque retweetet
Peide Huang
Peide Huang@peide_huang·
🚨Ever worried that your collected data cannot be used for training robot policies? You may need a Vision Pro. 🔥Check out this new AR-enabled, in-the-wild data collection method from our team here at Apple! Kudos to @ryan_hoque and everyone in the team!🎊
Ryan Hoque@ryan_hoque

🚨 New research from my team at Apple - real-time augmented reality robot feedback with just your hands + Vision Pro! Paper: arxiv.org/abs/2412.10631 Short thread below -

English
0
3
13
2.2K

Entdecken

@DavidJFan @JimmyTYYang1 @raktimgg @pablovelagomez1 @Ritwik_G @UofMaryland @xiaolonw @Michael_J_Black