Marc Bejarano retweetledi
I'll try to explain how this fits in with other training approaches for self-driving, and why I think this milestone is so important. Both for us, and robotics in general.
Training an end-to-end agent with RL in a fully learned simulator (aka world model) is the holy grail of robotics. It's a very generic strategy, it's expected to scale to all of robotics with very few caveats. Nobody has shipped a robotics product like this to users, but I believe we are currently the closest.
An end-to-end agent is just one that takes in all available inputs (video, IMU, ...) and directly outputs the actions to take. This isn't controversial anymore, but how to train those actions is the hard part.
The first instinct is to just collect data of human experts, and have your agent learn to predict the expert actions for the corresponding inputs, aka imitation learning. For driving we define those actions as acceleration and steering curvature. This is a good start, but an agent trained like this will completely fail in the real world. Why this happens is the subject of much debate, but my summary is that an agent needs to be exposed to its own mistakes during training to be able to recover from them. For our driving models trained this way this manifests as drifting out of lane with the agent making no attempt to recover.
One solution to this problem is to fine-tune on a curated dataset of recoveries. One example of this is letting humans label the "ideal" place to be on an image of a road (or top-down view), and having an MPC system generate trajectories to get there smoothly. Another is to just let your broken agent drive in the real world, and let a human supervisor take over when it makes mistakes and correct them. You can then add those corrections to the dataset, retrain, and ship the updated model out. If you do this iteration enough times you get a good agent.
These strategies have been how several self-driving companies have gotten great capability. But they are expensive, because they require humans in the loop or even realworld mistakes. Training in simulation allows you to do this training without needing real-world disengagements or human labeling. This is the strategy we've focused on for quite a while. You need a learned simulator that can match the diversity and fidelity of reality. This means video-game type simulations with assets are inadequate. Many companies have now trained world models as simulators for driving that do this quite well. To my knowledge, no self-driving system shipped today other than ours has trained their agents on-policy in such a simulator to achieve their capability. I would love to hear more if I'm wrong about this, I'm not always up to date with what other companies are doing.
Ideally we would train our agents on-policy in such a simulator with RL on a good reward function. For example a good reward function would be a GAN-style approach where a discriminator says if the agent's driving is similar to that of a known good human driver. State-of-the-art RL doesn't seem good enough for this yet, we have not succeeded at using RL in this way. Instead we train on-policy in the learned simulator, but still provide ground-truth actions. How these actions are generated is not trivial to describe and explained in detail in our 2025 CVPR paper.
We hope to move to reward-based learning soon. Learning based on rewards should allow us to train policies that are smarter, particularly at low-level control, which is a big limitation of our current approach. Reward-based learning will also scale better to generic robotic tasks other than driving.
blog.comma.ai/011release/
English
