Austin Patel

LLMs learn new tasks in-context. It’s time robots do the same 🤖 Introducing Behavior Prompting: human shows one demo, and the robot adapts immediately. Turns out robot demos are great in-context prompts! And the magic: no paired human2robot data needed behavior-prompting.github.io (1/8)

Teleop systems are usually designed for a single embodiment, but do they have to be? Introducing ModPack 🎒: a modular teleoperation interface for bimanual mobile robots. A wearable backpack provides shared infrastructure, robot-specific leader arms adapt to different embodiments, and plug-and-play modules add capabilities like haptic feedback, active perception, and mobile manipulation. 🧵(1/n)

Long horizon bimanual mobile manipulation requires reasoning in many coordinate frames: base, L/R hands, etc. In which frame would policy work best? It really depends, so don’t pick. Mixture of Frames Policy: denoise in multiple frame in parallel. 🌐mofpo.github.io (1/9)

Last week, I defended my PhD dissertation, "Robots From Anyone: Shaping Behaviors and Embodiments with Handheld Grippers". You can find the talk here! youtu.be/Zlmdzv4rYzM

Behavior prompting is a new way to control your robot which we broadly hope: 1) provides an interface for humans to specify preferences to the robot via demonstration and 2) simplifies adaptation to new tasks/environments by providing a single demo in the target environment. Shoutout to my amazing co-authors: @ben_pekarek, Joel Enrique Castro Hernandez and @SongShuran Check out Behavior Prompting Policy: 📄 Paper: arxiv.org/abs/2606.30457 🌐 Website: behavior-prompting.github.io 💻 BPP Code: github.com/real-stanford/… 💻 iPhUMI Code: github.com/real-stanford/… (8/8)

We are also excited to release iPhUMI ("eye-foo-me”)! It solves the localization challenges with the GoPro UMI, enabling rapid data collection across diverse environments and tasks. During deployment, iPhUMI lets you command your robot via demonstration. github.com/real-stanford/… (7/8)

LLMs learn new tasks in-context. It’s time robots do the same 🤖 Introducing Behavior Prompting: human shows one demo, and the robot adapts immediately. Turns out robot demos are great in-context prompts! And the magic: no paired human2robot data needed behavior-prompting.github.io (1/8)

Can we enable robots to develop a sense of touch without forgetting what they learned from large-scale vision-only pretraining? Introducing MultiSensory World Model (MuSe) 🌍: A new approach for finetuning visuomotor policies on minimal data from new sensor modalities, such as force/torque (F/T) With Muse, touch learned later improves skills learned earlier — a small amount of F/T data on new tasks improves zero-shot on diverse pretraining tasks that were never supervised with F/T We believe MuSe provides a practical pathway towards training multisensory foundation models that leverage both abundant vision data, and smaller multisensory datasets 🧵👇

Your bimanual manipulators might need a Robot Neck 🤖🦒 Introducing Vision in Action: Learning Active Perception from Human Demonstrations ViA learns task-specific, active perceptual strategies—such as searching, tracking, and focusing—directly from human demos, enabling robust visuomotor policies under visual occlusions. 🧵👇

Thrilled to share our CoRL 2024 paper on learning from demonstrations for long-horizon manipulation! Check out real-world demos here: blade-bot.github.io. Weiyu will be presenting BLADE today! Very grateful to Weiyu for being an amazing mentor and co-lead on this journey!

We recently launched umi-data.github.io as a community-driven effort to pool UMI-related data together. 🦾 If you are using a UMI-like system, please consider adding your data here. 🤩🤝 No dataset is too small; small data WILL add up!📈

Without any finetuning, GET-Zero can zero-shot control a wide variety of hand designs, even if we remove joints or add link length extensions (shown in orange). None of the hand designs below were seen during training. Check out our project page: get-zero-paper.github.io

How can our model adapt to different embodiments? Graph Embodiment Transformer (GET) is an embodiment-aware transformer encoder that combines joint hardware properties with an attention-based embodiment graph encoding to flexibly represent a wide range of hand designs.

What if you could control new hand designs without a new policy? Introducing GET-Zero, an embodiment-aware policy that can zero-shot control a wide range of hand designs with a single set of network weights. get-zero-paper.github.io