Ryan Bahlous-Boldi
86 posts
@RyanBoldi
PhD student @MIT_CSAIL | Continual RL, Open-Endedness, Evolution
ChatGPT several times where's best to go for spring break? It recommends Barcelona almost every time. This isn't a fluke. RL training rewards one best answer, so the model learns to commit to one mode and repeat it. Meet Multi-Answer RL: a simple RL method that trains LMs to reason through and output a distribution of answers in a single generation. [1/N]
Can language models learn useful priors without ever seeing language? We pre-pre-train transformers on neural cellular automata — fully synthetic, zero language. This improves language modeling by up to 6%, speeds up convergence by 40%, and strengthens downstream reasoning. Surprisingly, it even beats pre-pre-training on natural text! Blog: hanseungwook.github.io/blog/nca-pre-p… (1/n)
Check out our new Digital Red Queen work! Core War is a programming game where assembly programs fight against each other for control of a Turing-complete virtual machine. We ask what happens when an LLM drives an evolutionary arms race in this domain. We find that as you run our DRQ algorithm for longer, the resulting programs become more generally robust, while also showing evidence of convergence across independent runs - a sign of convergent evolution!
For agents to improve over time, they can’t afford to forget what they’ve already mastered. We found that supervised fine-tuning forgets more than RL when training on a new task! Want to find out why? 👇
How can we collect the best stepping stones for open-ended discovery? I am excited to share Learned Quality-Diversity — a family of algorithms meta-optimized to be efficient stepping stone collectors! 📄: arxiv.org/abs/2502.02190 🌟: github.com/maxencefaldor/… work done in collaboration with @RobertTLange from @SakanaAILabs and @CullyAntoine.
