Roger Girgis

In-flight weight updates have gone from a “weird trick” to a must to train LLMs with RL in the last few weeks. If you want to understand the on-policy and throughput benefits here’s the CoLM talk @DBahdanau and I gave: youtu.be/Z1uEuRKACRs

Scenario Dreamer has been accepted at #CVPR2025! Website: …ceton-computational-imaging.github.io/scenario-dream… We train a vectorized latent diffusion model to synthesize high-fidelity driving simulation environments (agents+map). Scenario Dreamer enables fully data-driven closed-loop generative simulation!

Resharing as CtRL-Sim was accepted at #CoRL2024, which we will present this week in Munich! Also, all the code is now available, so you can start training your own controllable and reactive driving agents: github.com/montrealroboti…

Check out CtRL-Sim later this week at #CoRL2024 in Munich! How can we generate interesting edge cases to test autonomous vehicles in simulation? We propose CtRL-Sim for closed-loop behavior simulation that enables fine-grained control over agent behaviors. CtRL-Sim leverages return-conditioned offline reinforcement learning with exponential tilting for multi-agent closed-loop, reactive, and controllable behavior simulation. CtRL-Sim can modify existing traffic scenarios to generate a wide range of agent behaviors by “tilting” each agent towards good (positive tilting) or bad (negative tilting) driving behaviors. Project Page: torc.ai/knowledge-cent… Work led by Luke Rowe and Roger Girgis with amazing collaborators Anthony Gosselin, Bruno Carrez, Florian Golemo, Liam Paull, and Chris Pal.

Introducing ReSearch: An iterative self-reflection algorithm that enhances LLM's self-restraint abilities: • Encouraging abstention when uncertain • Producing accurate, informative content when confident Result: Significant accuracy boost for Llama2 7B Chat and Mistral 7B! 🚀

We’re thrilled to introduce Ctrl-V! Our new video diffusion model uses 2D and 3D bounding boxes to predict & control object motions in videos. Learn more, read the full report & access our code on our website: oooolga.github.io/ctrl-v.github.…

Check out our awesome new work, CtRL-Sim, a framework for reactive and Controllable behaviour simulation using Factorized Return-conditioned supervised learning.

One of the hardest challenges in developing AI for autonomous vehicles is evaluating the performance of our driving models. Why? (A short 🧵on our latest research on multi-agent RL).

Your RL agent is not behaving as expected? Try Direct Behavior Specification via Constrained RL! In our #ICML2022 paper we propose to use a special family of constraints to specify behavior instead of forcing everything into a single reward. Blog is out: tinyurl.com/3z6zy6n2

Our paper (accepted at @CoLLAs_Conf) is out! arxiv.org/abs/2208.02377 We show that Meta-Learning generalization to novel OOD task distributions can be inferred from the neural activation dynamics from a few unlabeled examples, and we propose Activation-Based Early-Stopping (ABE).

We have open sourced github.com/nvr-avg/trajda…! It's a new, unified interface to many trajectory forecasting datasets, greatly simplifying the process of training and evaluating a forecasting model on multiple motion datasets! @iamborisi @NVIDIADRIVE

According to the @USDOT, someone runs a red light every ~20 min. Proud of the work by our team to make roads safer, like in this situation where the @Waymo driver in fully-auto mode (no human in the driver’s seat) safely reacts to a car blowing through a red light in Phoenix.

Joint work with @chrisjpal, @felipealcm, @shoddy_robots, @SamiraEKahou, @jimdsouza, @_FelixHeide_, and Martin Weiss at @Mila_Quebec and @Algolux @etsmtl @polymtl. 🚗+🤖s -->😄🙏 🧵4/4

…And the best thing? With our seed parameter trick, you can train it on a GTX 1080Ti in 3h. No more TPUs and days worth of training. Low computation => easier on the 🌎. 🧵3/4

Transformers are the go-to model for language, but they can also be used for motion prediction in cars 🚗 & robots 🤖. Our #ICLR2022 spotlight paper presents AutoBots, a fast & effective multi-agent trajectory forecasting Transformer 🧵1/4 arXiv: arxiv.org/abs/2104.00563