Chris has provided the clearest breakdown of robot world models that I've seen. The simplest way to understand them: an LLM asks "what word comes next?" A world model asks "what happens next in the physical world if I do this?" 3 types are competing right now:
1. Action-conditioned (V-JEPA 2, Dreamer v4): predict what happens given a robot's action. The purest approach, but predictions collapse within seconds.
2. Video world models (NVIDIA DreamGen, 1x): generate a video of the task first, then reverse-engineer the motor commands from the frames. No action labels needed to train.
3. Joint world-action models (DreamZero, Fast WAM): predict both video and action simultaneously. Currently winning on benchmarks.
The best models don't even render video at test time. They just need to have learned what the world looks like in order to act in it.
This matters because today's AI is essentially like a very well-read intern. Great at desk work, useless at unloading a truck. You can simulate billions of chess games, but you can't bake a billion cakes. World models trained on internet-scale video are how robotics closes that gap.
Transformers made AI book-smart. World models give it physical intuition.
Great read👇🏾
Chris Paxton@chris_j_paxton
English
