Abhinav Moudgil

@Antrunt @BorisAKnyazev @ebelilov Thanks! It's the same as in standard optimizers: performance degrades smoothly as you deviate more from optimal LR. 1e-4 or 1/20x tuned AdamW LR are good starting points for Celo2. More on LR sensitivity in appendix:

@amoudgl @BorisAKnyazev @ebelilov Nice work! How important is the LR compared to schedule free optimizers like Prodigy or the default ones like AdamW?

Introducing Celo2: Towards Learned Optimization Free Lunch We show that learned optimizers can generalize to practical tasks like GPT-3 1.3B pretraining and several out-of-distribution vision/RL tasks from limited meta-training (~4.5 GPU hours)! 🧵

Finally, I'd like to acknowledge the Google TPU research cloud program that made this research possible and sincerely thank @mrtnm @kvfrans @_chris_lu_ for their open-source contributions by releasing clean jax codebases with TPU/FSDP support.

Work done with @BorisAKnyazev and @ebelilov. If you are interested in this line of research or related topics, our lab is hiring: x.com/ebelilov/statu…