Anirudh Vemula

Now that I have started using twitter somewhat regularly, let me take a minute to advertise the RL theory lecture notes I have been developing with Sasha Rakhlin: arxiv.org/abs/2312.16730

🗣️ “Next-token predictors can’t plan!” ⚔️ ​​“False! Every distribution is expressible as product of next-token probabilities!” 🗣️ In work w/ @GregorBachmann1 , we carefully flesh out this emerging, fragmented debate & articulate a key new failure. 🔴 arxiv.org/abs/2403.06963

@Wenxuan_Zhou @yufei_ye Congrats Wenxuan!!

Life updates: Successfully finished my Ph.D. thesis defense! It’s been an incredible journey of exploring the possibilities of robots and RL. I’m actively seeking full-time scientist/engineer positions in AI/Robotics. Looking forward to new adventures! ⛵️

Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback paper page: huggingface.co/papers/2307.15… Reinforcement learning from human feedback (RLHF) is a technique for training AI systems to align with human goals. RLHF has emerged as the central method used to finetune state-of-the-art large language models (LLMs). Despite this popularity, there has been relatively little public work systematizing its flaws. In this paper, we (1) survey open problems and fundamental limitations of RLHF and related methods; (2) overview techniques to understand, improve, and complement RLHF in practice; and (3) propose auditing and disclosure standards to improve societal oversight of RLHF systems. Our work emphasizes the limitations of RLHF and highlights the importance of a multi-faceted approach to the development of safer AI systems.

@aravindr93 @philippswu @arjunmajum @kevinleestone @yixin_lin_ @IMordatch @pabbeel @ncklashansen @haosu_twitr @HarryXu12 @xiaolonw Also at ICML this week. We should catch up :)

✈️ Just landed in Hawaii 🌴 to present two cool projects at #ICML2023 🚀 Masked Trajectory Models (w/ @philippswu, @arjunmajum, @kevinleestone, @yixin_lin_ , @IMordatch, @pabbeel) 📚 LfS Revisited (w/ @ncklashansen, @haosu_twitr, @HarryXu12, @xiaolonw et al.) Details in 🧵👇

@chinganc_rl Also at ICML this week. We should catch up :)

Will be at #ICML2023 this week and present 4 cool papers on offline RL, lifelong RL and RL with exogenous processes. Looking forward to meeting new and old friends. Ping me if you wanna meet up. 🌺

On my way to Honolulu to present this work! Hit me up if you want to hike and check out cool beaches B) arxiv.org/abs/2303.00694

@debidatta Why did you tweet this??

I'm rarely as excited about a paper as our #ICML2023 paper: we develop an algorithm for doing inverse reinforcement w/o an expensive RL inner loop, providing an *exponential* speedup. Works *extremely* well in practice. Joint work w/ @sanjibac, @zstevenwu, and Drew Bagnell. [1/n]

I am delighted to say that I will be joining the Colorado School of Mines @CSatMines 💻🤖 as an Assistant Professor 👨‍🏫 this January! #academia #AcademicTwitter

Why is being laziness a fundamental virtue in both model based RL and IRL? Excited to share our new ICML'23 papers arxiv.org/abs/2303.00694 and arxiv.org/abs/2303.14623 that gets at the heart of this question. Check out my talk at my CMU to learn more! youtube.com/live/ndtnvmwHK…

@rkbanoth Nuvvu unnavu kada Mari atluntadi

Rey Enti maa Khammam ki antha scene unda? :o

@bremen79 Yes please

That's it! There is much more on this topic, so let me know if this is interesting to you and if you think I should write a blog post on it. 6/6

A mini-thread about optimization algorithms and "implicit preconditioners". If you optimize a function where the Hessian is ill-conditioned, gradient descent will be very slow (left fig). However, if you precondition it, it will go straight to the minimum (right fig). 1/6

@nanjiang_cs @g_k_swamy @yus167 Thanks for pointing out that connection @g_k_swamy Loved it! And thanks @nanjiang_cs def looking forward to catching up again after a long time at ICML

@g_k_swamy @vvanirudh @yus167 This one and your IRL paper are on my ICML "shopping list". My first in-person big conf ever since pandemic and so much looking forward to connecting with old & n w friends who like theory-inspired thinking in RL :)

Really impressive work that gets at the heart of what makes MBRL hard and how to fix it. Nice job @vvanirudh + @yus167!

Joint work with my awesome collaborators @yus167, @sanjibac, Aarti Singh, and Drew Bagnell

If this has been a long thread, this can be the only tweet to pay attention to the example figure to understand awesomeness of PDAM. MBPO: O(2^H) computation per iteration, and converges to bad model LAMPS-MM: O(H) computation per iteration and converges to good model

Our paper on a new (lazy) approach to model-based RL that is both computationally efficient and avoids the objective mismatch problem has been accepted for ICML! Excited to present it at Honolulu this summer! arxiv.org/abs/2303.00694