Nicholas Pfaff

Meet SceneSmith: An agentic system that generates entire simulation-ready environments from a single text prompt. VLM agents collaborate to build scenes with dozens of objects per room, articulated furniture, and full physics properties. We believe environment generation is no longer the bottleneck for scalable robot training and evaluation in simulation. Website: scenesmith.github.io 👇🧵(1/8)

🧵1/ 🤔New paper: Do LLMs Benefit from Their Own Words? In multi-turn chats, models are typically given their own past responses as context. But do their own words always help… or can they sometimes be a distraction?

I haven't ever timed this 😅 However, we implemented a bunch of performance improvements targeting throughput over latency. Hence, it wouldn't be much faster than when generating ~25 scenes or so in parallel. The biggest bottleneck is API response times. Hence, we have an option to opt into OpenAI's priority tier that speeds this up by 50% but is twice as expensive. Switching to Gemini Flash (or other speed-optimized models should also make a big difference here).

@NicholasEPfaff Very cool! How long does it take to generate each simulation-ready scene if you generate one scene per run?

Meet SceneSmith: An agentic system that generates entire simulation-ready environments from a single text prompt. VLM agents collaborate to build scenes with dozens of objects per room, articulated furniture, and full physics properties. We believe environment generation is no longer the bottleneck for scalable robot training and evaluation in simulation. Website: scenesmith.github.io 👇🧵(1/8)

@vatsalbajaj We have not tried RL-based training yet. We do have teleop demos on the website that could be used for supervised learning. RL would be exciting to try!

@NicholasEPfaff this is incredible! Out of curiosity, have you used Scene Smith to generate RL environments in which to train robots? D’you have any demos of how one could train/ post train for robotics using SceneSmith?

@YufeiWang25 Very cool! Thanks for sharing. Using image priors is a promising way to improve spatial reasoning.

@NicholasEPfaff Really cool work! We had a very similar work Architect which also focuses on indoor scene generation (also specifically for small & manipulatable objects placement): wangyian-me.github.io/Architect/. Different method applied to similar problems, excited to see this new progress!

Agreed here. VLMs seem to struggle with spatial imagination. Setting a table with place settings that face in different directions (and not just toward the current image render) is a revealing case of this. Image generative models are much better at this. Maybe the next version will use agentic video models?

@NicholasEPfaff Very cool... Although did anyone else notice that the knife was backwards on the table? 😁 There will always be a fundamental lack of actual understanding in ML/AI.

@allen_ai This is awesome work! Curious—any plans to integrate SceneSmith-like agentic scene generation into MolmoSpaces? It feels like a natural combo: MolmoSpaces benchmark + SceneSmith prompt-to-sim scenes = infinite evaluation distribution. scenesmith.github.io

Introducing MolmoSpaces, a large-scale, fully open platform + benchmark for embodied AI research. 🤖 230k+ indoor scenes, 130k+ object models, & 42M annotated robotic grasps—all in one ecosystem.

Agentic Generation of Simulation-Ready Indoor Scenes and Robot Test Environments. 📍 Paper AND Code: Instead of hand-building scenes in simulation, you write one prompt. SceneSmith builds the world for you. > Room layout. > Furniture. > Wall and ceiling objects. > Small movable items. Each stage is handled by a team of VLM agents: one proposes, one critiques, one coordinates. The result is not just pretty scenes, but physics-ready environments. Every object: •Metric scale •Collision geometry •Estimated mass, inertia, friction •<2% object collisions •96% stable under gravity And it exports directly to MJX, USD, SDFormat. If you train or evaluate robot policies, environment creation is usually the bottleneck. SceneSmith turns it into an on-demand layer. You can generate dozens of diverse scenes per task and automatically evaluate policies across them, with 99.7% agreement to human labels. That means: •More robust policies •Faster benchmarking •No hand-written success predicates 205 participants preferred SceneSmith scenes 92% of the time for realism and 91% for prompt faithfulness. Environment generation is no longer the slow part of robot research. If you work on sim2real, policy scaling, or automated evaluation, this is worth bookmarking and sharing with your team. 📍GitHub: scenesmith.github.io Paper: arxiv.org/abs/2602.09153 Code: github.com/nepfaff/scenes… —- Weekly robotics and AI insights. Subscribe free: 22astronauts.com

We don't have an explicit argument for that. However, the input just gets sent to a VLM agent, which can natively take both text and image inputs. Hence, supporting this seems like a minor code change. You could already use a VLM to describe a set of images in great detail in text and use that. We have been doing that, and it works quite well.

@NicholasEPfaff @cohnthomas43 @ZakharovSergeyN @RickCory21 @RussTedrake Can it take photos as inputs too? It would be awesome. Very fascinating work!

@sippeyxp @RussTedrake All objects already have estimated friction, and we do support articulated objects.

@RussTedrake Yeah! Sim cannot unleash its power until building a sim becomes engineering free. Thanks for sharing. Do you know if there is future plan to support contact property (e.g friction) and actuated objects?

I've been saying for years that the biggest challenge for simulation in robotics is not actually the physics engine (although you do have to get that right). The real challenge is capturing the *diversity* of the real world. There was no doubt that generative AI had the potential to change that, but it's still amazing to see it take shape. Watching Nick's incredibly fast progress has convinced me that content generation might not actually be a bottleneck anymore. This is a beautiful combination of hardened tools for e.g. low-level mesh processing with the latest tools for generative asset creation, wrapped in a powerful agentic workflow. Please do give it a try and share your feedback.

@maxzpchen Agreed. I think we are at the point where we have to push the robot side to see how far we can get with these environments and whether they start breaking down anywhere.

One thing I’m especially curious about is how far this kind of unbounded, VLM-driven object vocabulary and physics estimation can go before you start needing task-specific, human-curated distributions again—for example, for household manipulation vs. warehouse vs. surgery—and whether we’ll see “benchmark overfitting” in simulation the same way we did in vision and NLP

@nicks_robots @cohnthomas43 @ZakharovSergeyN @RickCory21 @RussTedrake Glad you like it! Agreed. That would be really cool to see! We uploaded >1200 scenes to Hugging Face, so you could try it with those. huggingface.co/datasets/nepfa…

@NicholasEPfaff @cohnthomas43 @ZakharovSergeyN @RickCory21 @RussTedrake Hey Nic! Great stuff, love to see automatic 3D scene construction! This would be sick to pair with my haptic gloves :D

@ludwig_fr Love that!

@NicholasEPfaff Add some steam VR teleop I would give you some free teleop data^^

@hferrolho Glad you liked that one! A pain to simulate with those large forces 😂

@NicholasEPfaff Earthquake test was my favourite part in this thread! 😆 Cool stuff!

Correct. We worked on real2sim (replicating an actual environment in simulation) in the past: scalable-real2sim.github.io The goal of this approach is to match the distribution of real-world and simulated environments for increased scale, but it might not be great at replicating one particular environment from images (though we have used it for that as well).

@NicholasEPfaff @GChongkai I also saw a post from somebody to create a gaussian splat of your personal space which can be used for simulation. This would be better suited if you want to train in a specific environment vs what you have here if you want to build something general purpose, right ?

@cyrux004 @GChongkai Recent works have shown that this is very promising. For example: viral-humanoid.github.io sim-and-real-cotraining.github.io co-training.github.io I do think that you will want to co-train on both sim and real data jointly to get the biggest benefit from added simulation data.

@NicholasEPfaff @GChongkai how much does training in simulation translate in real world performance. ?

@JeremySMorgan3 Agreed. Maybe even some VLM-aided TAMP or similar to get diverse data from such a model-based planner. @cohnthomas43 set up a simple version of a model-based planner for these scenes, but we have only used it for evaluation so far.

@NicholasEPfaff This looks great! Nice videos also. Would be cool to hook up a motion planner in here with a task sampler to generate diverse demonstrations for VLA training

@ZhaoMandi Thank you! It ended up working way better than what I had hoped when we last talked about this

@NicholasEPfaff this looks incredible, congrats Nicholas!!

I think this will be very useful for all of training, evaluation, and development. Large-scale training data generation takes more effort to set up, but has a lot of potential. We had lots of fun testing one of the applications for automatic evaluation. We open-sourced all of our code and even released all of our already generated scenes on Hugging Face!

I was prototyping a system like this after stumbling on the google deepmind object dataset . Is this more useful as an API service for evaluation/ training or both? Will big companies just implement it internally or would be open to use a service?

You should be able to send commands from your real-world teleop system to the simulator instead of to the robot directly. We did open-source a lightweight mobile iiwa teleop example with a space mouse a while back: github.com/nepfaff/scene_…. However, there are much better recent approaches that use a VR headset for this teleop. Also, check out this company: evolverobotics.com

@NicholasEPfaff Great work! May I know how to teleoperate a mobile robot in simulation (with only one person)? Is there any mature pipeline for this?