Bo Dai

RL is so back, as David Silver predicted. m.youtube.com/watch?v=pkpJMN…

@Haolun_Wu0203 Hi Haolun, we have considered even more aggressive setting: finetune the model with only access to the sample of LLM arxiv.org/pdf/2402.08219. The setting with last layer logits we considered as gray box.

🚀 New Research Alert: Logits are All We Need to Adapt Closed Models 🔒Many commercial Large Language Models (LLMs), e.g., GPT-4, are closed-source, limiting developers to steer content generation. 🤔Can we adapt closed-source LLMs when fine-tuning or accessing their internal weights is not possible? Check out our work by @gaurushh, @Haolun_Wu0203, Subhojyoti, @sanmikoyejo from Stanford @stai_research. 1/n

RL is sparkling again.

@shaneguML 1, Gamify the problem with clear reward; 2, RL

ChatGPT/RLHF was a distraction (though necessary). o3/gemini-2-flash-thinking/RL is the real game. Glad the community finally went over the local optimum. There's no looking back; it's all RL until AGI (for some domains). (Slide: Dave Silver)

@denny_zhou The power of RL with clear target

any benchmark—including ARC-AGI—can be rapidly solved, as long as the task provides a clear evaluation metric that can be used as a reward signal during fine-tuning.

@shaneguML Value function in RL is also approximating gradients

I chose RL because MuProp couldn't make Arvind/Ilya/Quoc neural programmer work better. So instead of approximating gradients, I decided to go RL. It was inspired by e2e training of tool-augmented language agent. Ofc another reason: Sergey Levine, and I'll talk another time.

Second thing Ilya told me was Tim Lillicrap, a Canadian neuroscientist turned DeepMind researcher, his random synaptic feedback. He was imagining end-to-end toolformer where you differentiate through both weights and discrete Google search in 2015. This was why I wrote MuProp (mu used to be Greek mu for mean-field network, but inspired by MuZero, I later tell people it's actually mu meaning nothingness in Japanese) - I wrote MuProp with him dustintran.com/blog/muprop-un… (concurrent work from John Schulman. Acceptance with zero experiments arxiv.org/abs/1506.05254 - Arvind's work. We were interns together who stayed up late. His work is the first non toy work where I applied newly developed MuProp (and failed) arxiv.org/abs/1511.04834 - Ilya also had another intern (19 year old Stanford undergrad) working on differentiating through Google search - Tim's work nature.com/articles/ncomm…. And Tim is my manager right now at DeepMind. My condition for coming back to Google was to report to Tim Lillicrap and that was the second best decision I made in 2023.

@yuchen_zhuang @haotiansun014 @yue___yu @chaozhangcs I will be there for helping the presentation. :)

(🧵6/N) Many thanks to our awesome collaborators: @haotiansun014, @yue___yu, Rushi Qiang, Qifan Wang, @chaozhangcs, @daibond_alpha Unfortunately, I will miss this year's #NeurIPS2024. But I am open to chat about LLMs for agentic reasoning and planning virtually. DM or email me📨!

Excited to present HYDRA 🐉 at #NeurIPS2024! 🚀 Our novel model-factorization framework combines personal behavior patterns 👤 with global knowledge 🌐 for truly personalized LLM generation. Achieves 9%+ gains over SOTA across 5 tasks 🏆 using personalized RAG. Learn more: arxiv.org/pdf/2406.02888

@chanpyb @iclr_conf The lesson is selecting the primary area carefully.

@daibond_alpha @iclr_conf Some interesting observations here: 1. It seems like the latter has "shorter reviews" and imo generally of lower quality than those of the former 2. The expectations seem to be different, maybe due to different primary areas? 3. ...

We scale up the diffusion transformer for SoTA performance with fast inference!

I did not even have 10 submissions…. There are two different “Bo Dai”.

@damekdavis @CsabaSzepesvari I see. The additional K steps of vanilla gradient descent updates will be helpful in this case. Interesting!

@daibond_alpha @CsabaSzepesvari So the method in our work is dealing with the (2a) case in theorem 1. Think of y^2 + x^4. Here \xi = 3/4, so linear convergence does not hold GNGD according to the theorem.

New work exponentially accelerates gradient descent w/ adaptive steps, which alternate * small constant steps and * exponentially large Polyak steps Theory works for losses w/ quartic growth (and beyond). Exampls incl. overparam matrix sensing and toy neural nets paper below

@damekdavis @CsabaSzepesvari Looking forward to your feedback!

@daibond_alpha @CsabaSzepesvari Neat, I'll take a look!

@dpkingma @AnthropicAI Congrats to you and @AnthropicAI !

Personal news: I'm joining @AnthropicAI! 😄 Anthropic's approach to AI development resonates significantly with my own beliefs; looking forward to contributing to Anthropic's mission of developing powerful AI systems responsibly. Can't wait to work with their talented team, including a number of great ex-colleagues from OpenAI and Google, and tackle the challenges ahead!

@Lumi871 @GTCSE Welcome, Lu!

Please consider joining us to explore the frontier on generative foundation model for decision making, planning, and reasoning.

My Twitter Interaction Circle ➡️ infinityweet.me/interaction-ci…

Our black-box adaptation for LLMs has been accepted to #ICML2024. We provide offline and online learning strategy for a value function to pivot the LLMs decoding procedure, only with the access to the output sentences of LLMs.