Andy Zeng

The dark matter of robotics is “physical commonsense” Those tiny corrections, subtle recoveries that your hands do (and rarely notice). It’s everywhere, and yet—hard to pin down Second nature to us, but hard for machines They’re starting to emerge in our foundation models👇

@chichengcc Thank you Cheng! 🫶

Great write up on data for manipulation! Always a fan of Andy’s papers 🫶

Took time to make sure we did this right with results we could trust: blind A/B evals 📊 on success rates. Still feels crazy to me that data from random unrelated activities around the world yields strong pretraining transfer. Yet here we are. It's quite the beast. Stay tuned.

Glad you caught this 🙂 Never thought we’d go this deep into networking. Move enough robot data globally 🌎 and your NAT gateways might get silently banned mid-Atlantic... 🤦 The world is not ready yet for massive-scale robots—but its coming, and we’ll lay the groundwork for it.

ok actually i think this is probably the most underappreciated part. these guys are serious about scaling. it’s not just talk.

At Generalist, robotics is no longer limited by data. Breaking through the data wall has enabled us to become a true foundation model company, building and scaling models from the ground up for embodied intelligence. In today’s blog post we’re excited to share details about GEN-0, our first generation of embodied foundation models: • GEN-0 is a class of custom, natively cross-embodied, model architectures built from the ground up for the decadent dexterous behaviors Generalist has become known for. • We have scaled GEN-0 to 10B+ fully active parameters and continue to push the boundaries of scale for training and inference. • GEN-0 foundation models are thus far pretrained on 270,000+ hours of real world diverse manipulation data. We collect 10,000 hours of data per week and are accelerating. • We are now seeing that massive-scale pretraining leads to beautifully sample-efficient finetuning on downstream tasks, delivering on the fundamental promise of embodied foundation models. • The scale of our data trove has enabled us to conduct detailed science on pretraining. Scaling Laws are alive and well in robotics. All of the above and much more on our blog post: generalistai.com/blog/nov-04-20…

General dexterity involves "physical commonsense" —learning long tail of cases like: 🤏nudging objects to make space for fingers to grasp 🫴placing down slipping objects to get better grip etc... This👇shares more on how we think robots🦾can get there with models & lots of data

"Train a model to predict how many timesteps left until task success" - a simple, yet powerful way to get rewards from episodic BC data Lots of nuggets in the paper (including steps-to-go fn is distributionally multimodal) Kamyar's post 👇 on how it drives RL self-improvement

@danfei_xu Thank you for the kind words, Danfei!

One shot imitation learning! Brings back good memories from eons ago (aka 2016). This is probably my favorite demo of the year (so far). The smoothness and agility of these systems speak to the quality of the full stack system. Sometimes I tell my students that a good number of problems could be fixed with better robot controller and overall system integration (time sync etc.) rather than fancy learning algorithms and bigger models ...

@tomprimozic I suspect a combination of both! as long as it's within the context buffer

@andyzeng_ does it *remember* the original brick structure ("what to build") or does it attend to it continuously (peripheral vision)?

This is one-shot assembly: you show examples of what to build, and the robot just does it. (see original post: generalistai.com/blog) To share more on how this works, the robot is controlled in real time by a neural network that takes in video pixels and outputs 100Hz actions. The video below is part of the raw input passed directly into the model. I also like this view (at 1x speed) because it shows more of the (I think very cool) subtle moments of dexterity near the fingertips 👌 One-shot assembly seemed like a dream even just a year ago — it's not easy. It requires both the high-level reasoning of "what to build" (recognizing the geometry of the structures presented by the human), and the low-level visuomotor control of "how to build it" (purposefully re-orienting individual pieces and nudging them together in place). While possible to manually engineer a complex system for this (e.g. w/ hierarchical control, or explicit state representations), we were curious if our own Foundation model could do it all end-to-end with just some post-training data. Surprisingly, it just worked. Nothing about the recipe is substantially different than any other demo we’ve run in the past, and we’re excited about its implications on model capabilities: • On contextual reasoning, these models can (i) attend to task-related pixels in the peripheral view of the video inputs, and (ii) retain this knowledge in-context while ignoring irrelevant background. This is useful for generalizing to a wide range of real workflows: e.g. paying attention to what’s coming down the conveyor line, or glancing at the instructions displayed on a nearby monitor. • On dexterity, these models can produce contact-rich "commonsense" behaviors that can be difficult to pre-program or write language instructions for e.g. rolling a brick slightly to align its studs against the bottom of another, re-grasping to get a better grip or to move out of the way before a forceful press, or gently pushing the corners of a brick against the mat to rotate it in hand and stand it up vertically (i.e. extrinsic dexterity). These aspects work together to form a capability that resembles fast adaptation — a hallmark of intelligence, relevant for real use cases. This has also expanded my own perspective on what's possible with robot learning, using a recipe that's repeatable for many more skills. This milestone stands on top of the solid technical foundations we’ve built here at Generalist: hardcore controls & hardware, all in-house built models, and a data engine that "just works." We're a small group of hyper-focused engineers, and hands-down the highest talent-density team I’ve ever worked with. We're accelerating and scaling aggressively towards unlocking next-generation robot intelligence. Building Legos is just one example, and it's clear to me that we're headed towards a future where robots can do just about anything we want them to. Its coming, and we're going to make it happen.

it just worked 🤷‍♂️

Imagine having a "copy-paste" button 📋 for the physical world – a pretty good litmus test for embodied AGI

🎉Advanced Robotics Best Survey Paper Award has been awarded to our survey paper "Real-World Robot Applications of Foundation Models: A Review"! We are truly grateful to everyone who contributed! Thank you @__tmats__, Andrew, @jiaxianguo07, @chris_j_paxton, and @andyzeng_ !

@JonathanTo64772 @generalistai Thank you 🙏🙏 We ran the policies with “--butter” flag for the videos 😄

@adivganapathi @peteflorence Thank you for the kind words @adivganapathi !

So excited to keep tabs on this. I had the opportunity to work with @andyzeng_ and @peteflorence at Google Brain where they convinced me that end to end learning would eventually enable robots that can generalize. A lot of my research was built upon their foundational work, and I can easily say that they are two of the smartest people I’ve ever worked with. If anyone is solving robotics in this decade, my bet is on these guys!

Excited to be out of stealth! Since graduating from MIT, I’ve been building something incredibly FUN. Every day at work is full of laughter and “wows.” Super proud of the fantastic team making it all happen. Let’s go 🚀

There’s something satisfying to see the robot slotting in the box flaps so nicely in the end ... 😌