Yevgen Chebotar

Proud to introduce EgoScale: We pretrained a GR00T VLA model on 20K+ hours of egocentric human video and discovered that robot dexterity can be scaled, not with more robots, but with more human data. A thread on 🧵what we learned. 👇

SONIC is now open-source! Generalist whole-body teleoperation for EVERYONE! Our team has long been building comprehensive pipelines for whole-body control, kinematic planner, and teleoperation, and they will all be shared. This will be a continuous update; inference code + model already there, training code and gr00t integration coming soon! Code: github.com/NVlabs/GR00T-W… Docs: nvlabs.github.io/GR00T-WholeBod… Site: nvlabs.github.io/GEAR-SONIC/

We are moving towards world-model based backbones for robotic policies, pre-trained on web-scale videos and outputting robotic actions directly within the same diffusion model! There are new levels of transfer to unseen tasks and motions being unlocked, something that has been missing in VLM-based pre-training ever since the early days of RT-2 VLA models, which although generalizing well to new objects and semantics always struggled with new “verbs” and “motions”. We also observe signs of efficient cross-embodiment transfer to new robots with a little amount of data. There are a lot of optimizations that allow us to run a 14B world action model in real time! My favorite trick is DreamZero-Flash, which while still denoising videos and actions jointly, introduces separate noise schedules, such that actions can be denoised much faster than videos to enable higher frequency control! Check out the website, paper and especially the eval gallery! dreamzero0.github.io

Excited to join the NVIDIA GEAR team to help build the next generation of open robotic foundation models!

We've made great progress on Vision-Language-Action Models for humanoids in our new Helix model! Check out the technical report for more details: figure.ai/news/helix

The path to VLAs lies through VLMs. A very nice intro for everyone interested in working with Vision-Language Models: arxiv.org/abs/2405.17247

Congrats everyone, 170+ authors and contributors, great to see the robotic field coming together!

Some personal updates! Excited to join the team @Figure_robot to help building AI for the robot age! 🤖

Turns out classification loss works surprisingly well for value-based RL, also some nice gains when used with Q-Transfomer!

RT-H learns a hierarchy all the way from high-level tasks through low-level “language motions” to robot actions! ✅ Improved performance and generalization through better data sharing ✅ Automated grounded “bottom-up” labeling ✅ Ability to intervene and correct with language

Had a great time today with @YevgenChebotar and @QuanVng visiting @USCViterbi to give a talk on “Robot Learning in the Era of Foundation Models”. Slides out soon, packed with works from *just the past 5 months* 🤯 Thanks to @daniel_t_seita for hosting!

Presenting RT-2 poster at CoRL! robotics-transformer2.github.io

How many @CSatUSC grads does it take to create a breakthrough robot? Three, apparently! Congrats @YevgenChebotar @hausman_k and @ryancjulian who worked on Google DeepMind's revolutionary RT-2 AI model. Find out more ⤵️ viterbischool.usc.edu/news/2023/10/t… @gauravsukhatme @USCViterbi

@_ntr_p_ There was a technical problem with the old website address, the updated website is at qtransformer.github.io

@YevgenChebotar Would it be possible get the website up again? It is currently unavailable.

Offline RL strikes back! In our new Q-Transformer paper, we introduce a scalable framework for offline reinforcement learning using Transformers and autoregressive Q-Learning to learn from mixed-quality datasets! Website and paper: q-transformer.github.io 🧵

Exciting times for Robot Learning! 60 datasets from 22 different robots and 21 institutions combined in a single Open-X Embodiment data repository, resulting in over 1 million episodes and improved RT-X models! Amazing and a very important collaboration across the world! 🤖🌐

Joint work with @QuanVng, @AlexIrpan, @hausman_k, @xf1280, @Yao__Lu, @aviral_kumar2, @TianheYu, @AlexHerzog001, @KarlPertsch, @keerthanpg, @julianibarz, @ofirnachum, @Kanishka_Rao, @chelseabfinn, @svlevine

Our real robot policies significantly improve upon RT-1 and other baselines when trained on limited amount of human demonstrations by leveraging autonomously collected negatives and dynamic programming properties of Q-learning.

Excited to present RT-2, a large unified Vision-Language-Action model! By converting robot actions to strings, we can directly train large visual-language models to output actions while retaining their web-scale knowledge and generalization capabilities! robotics-transformer2.github.io