Ryan Sullivan

Releasing PufferLib 4.0: Train agents in seconds

@abhishekunique7 This is really cool! Do you have any thoughts on how this approach relates to Go-Explore? arxiv.org/abs/2004.12919 It seems like the motivation is similar but you have a new way of generating interesting states to start from.

Excited to share the project that has surprised me the most in the last year! Large-scale RL in simulation, no demos and no reward engineering can solve dynamic, dexterous and contact rich tasks. The learned behaviors are reactive, forceful and use the environment for recovery in ways that are extremely challenging to bake in or teleoperate! You can play with the policies yourself to see: weirdlabuw.github.io/omnireset/ And, the learned behavior transfers to real world robots from RGB camera inputs! So what’s the trick - using simulator resets carefully! Let’s unpack (1/10)

The AI Scientist: Towards Fully Automated AI Research, Now Published in Nature!!✨ Today in Nature we share a comprehensive technical summary of our work on The AI Scientist, including new scaling law results showing how it improves with more compute and more intelligent foundation models. The AI Scientist autonomously creates its own research ideas, codes up and conducts experiments to test those ideas, creates figures to visualize the results, writes an entire scientific manuscript summarizing what it has discovered, and conducts its own “peer” review of the resulting paper. One of its papers–entirely AI generated–passed peer review at a top-tier AI conference workshop, a historic milestone marking the dawn of a new era of AI-accelerated scientific discovery. 🔬🧪✨🧬💡🔭 Paper nature.com/articles/s4158… Blog sakana.ai/ai-scientist-n… Work done in collaboration with a great team from Sakana, Oxford, and my lab at UBC. Thanks and congratulations everyone! @_chris_lu_ @cong_ml @RobertTLange @_yutaroyamada @shengranhu @j_foerst @hardmaru

Introducing Hyperagents: an AI system that not only improves at solving tasks, but also improves how it improves itself. The Darwin Gödel Machine (DGM) demonstrated that open-ended self-improvement is possible by iteratively generating and evaluating improved agents, yet it relies on a key assumption: that improvements in task performance (e.g., coding ability) translate into improvements in the self-improvement process itself. This alignment holds in coding, where both evaluation and modification are expressed in the same domain, but breaks down more generally. As a result, prior systems remain constrained by fixed, handcrafted meta-level procedures that do not themselves evolve. We introduce Hyperagents – self-referential agents that can modify both their task-solving behavior and the process that generates future improvements. This enables what we call metacognitive self-modification: learning not just to perform better, but to improve at improving. We instantiate this framework as DGM-Hyperagents (DGM-H), an extension of the DGM in which both task-solving behavior and the self-improvement procedure are editable and subject to evolution. Across diverse domains (coding, paper review, robotics reward design, and Olympiad-level math solution grading), hyperagents enable continuous performance improvements over time and outperform baselines without self-improvement or open-ended exploration, as well as prior self-improving systems (including DGM). DGM-H also improves the process by which new agents are generated (e.g. persistent memory, performance tracking), and these meta-level improvements transfer across domains and accumulate across runs. This work was done during my internship at Meta (@AIatMeta), in collaboration with Bingchen Zhao (@BingchenZhao), Wannan Yang (@winnieyangwn), Jakob Foerster (@j_foerst), Jeff Clune (@jeffclune), Minqi Jiang (@MinqiJiang), Sam Devlin (@smdvln), and Tatiana Shavrina (@rybolos).

@sethkarten Thanks for answering my questions! This is a cool project, it would be great if coding agents could make it easier for everyone to work on (currently) slow complex environments

@RyanSullyvan Fair point. I appreciate your nuanced comments

@sethkarten If you’re saying that these environments have better reward design thats great, but I think an exact optimized copy is a valuable first step. Both to compare to prior work and to demonstrate that the system can faithfully recreate environments.

@sethkarten Sorry I should have been more specific. In Figure 4 for PokeJAX, the training curves look very different (the optimized env scores 60 elo higher). I actually didnt see the gray line for Red but yes that’s also very different. It seems like the envs aren’t the same for training

@akshitwt You might be interested in this paper by @akarshkumar0101 which used foundation models to search for more interesting artificial life simulations arxiv.org/abs/2412.17799

introducing a new, very fun, LLM benchmark- the Game-of-Life Bench! the rules are simple: given an 8x8 grid following Conway's game of life rules, the goal is to create an initial pattern with at most 32 cells that can last the longest number of turns before dying/repeating. some results to highlight (with caveats detailed below): - gpt 5.1 lasts the longest with a 106 step run - claude models are really bad at this! they refuse to reason about this task and score < 25 points - deepseek r1 is the best open model with 102 steps. why? because i wanted to create a benchmark that has (i think) no practicality, but is still fun to look at, cheap, and still measures something interesting. i also am a big fan of the game of life. its absurdly simple rules leading to intractability is extremely cool to me. also, i saw a lot of work with LLMs trying to "predict" the next state in Conway's game of life, I think game-of-life bench is more fun because it's pretty open ended and only asks the LLM for the initial state. I also think this could be an RL env? but idk why you would ever train on this task haha i don't think this is a "serious" benchmark because it doesnt measure anything practical, but i still think it's a hard benchmark exactly because you can't predict what happens with your initial state many turns into the future; this is why i was initially expecting all LLMs to be bad at it, but turns out, some are clearly better than the others (the ordering may surprise you!) reminder: this is still a work-in-progress; (1) i am gpu-poor so could only do 10 runs for each model, even though total running cost is relatively low. maybe with some more credits i can run more seeds for each model. (2) i handpicked models which i think are at the frontier right now, plus some others that were on my mind. so, if you'd like to see a model on here, let me know. (3) i currently only do an 8x8 grid because i thought that by itself would be pretty hard for current LLMs, but of course we can increase grid sizes! (4) the coolest thing is, i dont think we can calculate the max possible number of states (yay undecidability!) you can go without repeating, so this is essentially a no-ceiling task, which is pretty cool! again, i did this mostly out of a desire to make LLMs do something fun. if this keeps me entertained for a few more days, i'd likely release a blog post on it. if it keeps me entertained for a week (and someone sponsors me), i'll put more work into it :P lastly, this is fully open sourced, so feel free to run this on your own!

🧵 Introducing Xray-Visual (XRV): Scaling Unified Vision Models to 26 Billion Samples. 1/ Do standard vision encoders like SigLIP or DINO generalise to out-of-distribution (OOD) data at scale? While these models dominate academic leaderboards, we observe significant performance degradation when they are confronted with complex, real-world distributions. Introducing Xray-Visual (XRV).

Checkout our work on controlling image editing via sliders in SoTA models appearing at #cvpr2026. Bonus: our method does not require paired data for sliders! ↘️

Can large language models (LLMs) act as the imagination of a reinforcement learning (RL) agent? We found that if you let an LLM "dream" - not by hallucinating pixels, but by writing executable Python code - it can create an open-ended curriculum that drives progress in complex, long-horizon worlds. Introducing Dreaming in Code (DiCode). 🧵👇

@alexUnder_sky It's starting to look like we'll fully automate coding before LLMs can make it to Mine's End.

@RyanSullyvan nethack

We're quickly approaching the point where LLMs can code more complex games than they can play. It's funny to think that games, which are designed to be easily learnable and where task-specific AI already excels, might be the hardest domain for general intelligence.

Memory is probably the biggest challenge for building practical AI agents. Thrilled to share our work exploring a shift from manually defining memory for each domain → enabling agents to design better memory mechanisms for themselves. Meta-learning memory designs unlocks agents that can learn to continually learn across various tasks. Absolute joy working with @yimingxiong_ and @jeffclune on this!! code: github.com/zksha/alma paper: arxiv.org/pdf/2602.07755

Following our previous work on ADAS (x.com/shengranhu/sta…) and DGM (x.com/SakanaAILabs/s…), ALMA is a step toward AI-generating algorithms, including AI with continual learning. Looking ahead, we envision agentic systems that learn to improve all aspects of their agentic system, including their memory (i.e., combining ADAS/DGM and ALMA), learning to continually learn while solving problems in ever-changing real-world environments! The work was led by @yimingxiong_ with excellent mentorship by @shengranhu. Congrats to both on the excellent work! 7/7 🚀✨🎉🤖🧠🌍🔬 arxiv: arxiv.org/pdf/2602.07755 github: github.com/zksha/alma website: yimingxiong.me/alma

We're looking for Research Scientists and Engineers to join the AI team at @iconicgamesio in London. n London. We have diverse positions including: Model Optimization/Efficient Inference, Open-Endedness/Reinforcement Learning, and Generative Vision & Multimodal Foundational Models. Apply here: ats.rippling.com/en-GB/iconic/j… Learn more about Iconic: iconicgames.io We're a small, growing team with high ownership, crafting the minds that inhabit and shape new worlds for interactive entertainment. If you're into games, blueprinting consciousness, or simply building personas that transmit and evoke feelings and emotions beyond what any chatbot assistant could ever do, this is your place. These openings are also suitable for final-year PhD students with experience in the relevant field. Any questions, my DMs are open.

📢 New PhD Position 📢 We (@_rockt, @borruell, and I) are looking for a PhD student to work at the intersection of open-endedness and game design. The student will be part of the @UCL_DARK lab and funded by @iconicgamesio and UCL. See this doc for a more detailed description of the research direction and candidate expectations: docs.google.com/document/d/1Z7… To apply, please complete this form by January 15: docs.google.com/forms/d/16JGfS…

Check out our new Digital Red Queen work! Core War is a programming game where assembly programs fight against each other for control of a Turing-complete virtual machine. We ask what happens when an LLM drives an evolutionary arms race in this domain. We find that as you run our DRQ algorithm for longer, the resulting programs become more generally robust, while also showing evidence of convergence across independent runs - a sign of convergent evolution!

Tomorrow/Sunday 10:15–10:35 Keynote Talk 3 (Jeff Clune) MindGames workshop if you are interested. I'll try to make it fun and controversial! mindgamesarena.com