Russ Tedrake

@chetan_ Just to be clear, we can and are doing significantly more than pick and place with the scenes/assets created here. That's always been an emphasis for me.

@RussTedrake i think physics and diverse models are often kind of coupled problems for sim. tasks that require significantly more contact than pick-and-place are still hard to develop good sims for

I've been saying for years that the biggest challenge for simulation in robotics is not actually the physics engine (although you do have to get that right). The real challenge is capturing the *diversity* of the real world. There was no doubt that generative AI had the potential to change that, but it's still amazing to see it take shape. Watching Nick's incredibly fast progress has convinced me that content generation might not actually be a bottleneck anymore. This is a beautiful combination of hardened tools for e.g. low-level mesh processing with the latest tools for generative asset creation, wrapped in a powerful agentic workflow. Please do give it a try and share your feedback.

@JeremySMorgan3 We're working on and eagerly watching all of the progress in real2sim. I do think capturing particular scenes (e.g. the robot failed at a task in the field, and you want to reconstruct that in sim) is going to be very high value when we really get it right.

@RussTedrake This looks like a great work. What do you see as the next bottleneck? Diverse and large scale data collection in such a simulator?

It's a good question. We have articulated objects.. typically the only actuated object in the scene is the robot, which we insert manually (and you want to connect that to your control stack, etc, too). We've found that the basic friction parameters are pretty good, but I agree that this could be a nice way to further increase the fidelity.

@RussTedrake Yeah! Sim cannot unleash its power until building a sim becomes engineering free. Thanks for sharing. Do you know if there is future plan to support contact property (e.g friction) and actuated objects?

RL training in sim has been incredibly powerful for locomotion, because it turned out that training on a handful of terrains was sufficient to generalize to the real world. We haven't seen the same for manipulation. Open-world manipulation has, instead, been leaning on real world data collection (and imitation learning). The main reason for this, I would argue, is because we haven't captured the diversity of the manipulation problem in our simulators yet. It's still possible that we could find a minimal set of scenarios that capture the full complexity of the dexterity of manipulation, but manipulation also requires world knowledge (e.g. pick up the mugs from the handles, etc) so you have to get that from somewhere too -- strong multimodal base models and/or imitation. Assets + physics engines are not enough.

i have a question on your point that capturing the diversity of the real world is one of the harder problems here, or at least one of the problems that is important for robot generalization. my question is this: what is diversity useful for? is it for being able to learn a policy that has "seen" most of the distribution the set of all possible situations it will be in, or is it more for having a policy that can learn online and generalize in context (as people say with language models)? is this a fair question?

My personal journey on this has been in Drake (drake.mit.edu). First step: The assets have to be good. Garbage in, garbage out. (People under estimate how low quality most assets are... even the models you get straight from robot manufacturers.) Second step: Get the physics engine right. We've been pounding on getting better contact models (e.g. Drake's hydroelastic contact model) and stronger and stronger numerical methods for contact simulation in Drake. Alejandro's papers are a great summary of that work: scholar.google.com/citations?view…

@RussTedrake Russ - do you have recommended ways to get the dynamics right?

Introducing Large Video Planner (LVP-14B) — a robot foundation model that actually generalizes. LVP is built on video gen, not VLA. As my final work at @MIT, LVP has all its eval tasks proposed by third parties as a maximum stress test, but it excels!🤗 boyuan.space/large-video-pl…

Boston Dynamics and Toyota Research Institute are advancing humanoid robotics with Large Behavior Models (LBMs) for Atlas.

@Anweshac1211 yes. Of course the distribution and quality of the data matters.

@RussTedrake in short is it safe to say as data scales up the robot's performance increases?

TRI's latest Large Behavior Model (LBM) paper landed on arxiv last night! Check out our project website: toyotaresearchinstitute.github.io/lbm1/ One of our main goals for this paper was to put out a very careful and thorough study on the topic to help people understand the state of the technology, and to share a lot of details for how we're achieving it. youtube.com/watch?v=BEXFnr…

This was a massive effort by the entire team, with a number of individuals really pouring their hearts into this paper. The paper is packed full of (too many?) details. Your comments and feedback would be very welcome.

Probably my favorite plot from the paper, which sums it all up, is this one. The plot compares performance using different amounts of pretraining data used before training a new task: 0% (aka single task), 25, 50, or 100% of TRI’s data, then 100% of TRI’s data + all of the open-source robot data (the red line) that we’ve curated. It’s just awesome that the distributions over task completion are so tight and that trends as we increase data are so consistent. The results show clearly that with pretraining, we can train a novel skill with substantially less data or use the same amount of data and get much better task performance. And the benefits appear to continue with more data.

@danfei_xu Thanks @danfei_xu. Much appreciated.

Russ's recent talk at Stanford has to be my favorite in the past couple of years. I have asked everyone in my lab to watch it. youtube.com/watch?v=TN1M6v… IMO our community has accrued a huge amount of "research debt" (analogous to "technical debt") through flashy demos and less-than-ideal evaluation protocols. But it turns out verifying a basic hypothesis of "does pretraining on many tasks help you learn a new task" takes tremendous effort and discipline. Kudos to Russ & co at TRI LBM for paying off a big chunk of the debt.

Very proud of Nicholas, who recently shared scalable-real2sim.github.io (for physics-quality assets from a small amount of interaction with a robot) and is now following up with his work on scene-level generation.

In my mind, it's a bit like a biology paper that is focused on a particular animal model. I hope we'll learn more quickly from each other if we can make precise, substantiated claims about particular setups, so that as a field we can assemble those claims into a coherent picture.

This work really sharpened my thinking about sim+real cotraining.

Side note: I'm proud of the title of this paper, which we intentionally made pretty narrow/specific. I think that some of the most important work that we have to do as a field right now is careful empirical work to interrogate the properties of these models that we're creating.

One of the most interesting take-aways for me is that "high-performing policies need to know whether they are executing in sim or in real." A number of implications flow from that, including that sim+real cotraining can decrease performance if the visual gap is too small.

New Paper: "Scalable Real2Sim: Physics-Aware Asset Generation via Robotic Pick-and-Place Setups"! 🤖 We introduce a fully automated pipeline that generates simulation-ready assets for real-world objects—no manual intervention needed! 🌐 Website: scalable-real2sim.github.io (1/5)⬇️