Joseph Suarez 🐡

Releasing PufferLib 4.0: Train agents in seconds

Reinforcement learning research with Joseph Suarez x.com/i/broadcasts/1…

Reinforcement learning research with Joseph Suarez x.com/i/broadcasts/1…

If you're seeing tons of notifications for my streams: internet here has been blipping out multiple times per hour for 1-5 seconds. @X doesn't allow you to reconnect and instead launches a new stream + post

Reinforcement learning research with Joseph Suarez x.com/i/broadcasts/1…

Reinforcement learning research with Joseph Suarez x.com/i/broadcasts/1…

@vikhyatk it doesn't work. hparam tuning algorithms do work though

too much time is being spent making optimizers marginally faster. what we really need is hparam-free optimizers

Reinforcement learning research with Joseph Suarez x.com/i/broadcasts/1…

Reinforcement learning research with Joseph Suarez x.com/i/broadcasts/1…

Reinforcement learning research with Joseph Suarez x.com/i/broadcasts/1…

Reinforcement learning research with Joseph Suarez x.com/i/broadcasts/1…

@j_foerst No longer! PufferLib is ground-up CUDA as of 4.0

RL has largely been a consumer of a deep learning toolkit that was developed for supervised learning. In our recent work we explore RL specific hierarchical state representations that allow agents to overcome issues with low quality demonstration data.

@yacineMTB + 5.0 will be a short update - coming soon!

Pufferlib is insane. You can train neural networks to play games out of the box if you have a CUDA GPU. Like breakout, Atari games, continuous action space problems. You can go to the website right now and they have neural nets running in wasm

@DanAdvantage yeah you deserved that flame

yesterday i told gpt-5.5 exxxxxtra thinking fast in codex to use up all my monthly subscription to make the latest pufferlib sota breakthrough even better, and it silently determined that the problem was simply "too hard" the actual pr is below notice the maze is not a maze...

Core optimization improvements to PufferLib today: - MinGRU h x 3h projection layer -> orthogonalize the 3 slices separately in Muon - Replace NS with Polar Express - mup scaling makes it easier on our sweeps to tune learning rate jointly with model size - Aurora update on rectangular matrices (note MinGRU is square after splitting it into slices)

@BovardDT There are a few folks in the puffer discord doing the competition! They have likely already ported the env to C and will be training for several hundred years in simulation. It's much more beginner friendly than you would expect, but not exactly notebook material

@jsuarez lots of people on the forums asking how to get started with an RL solution. A puffer.ai starter notebook would be very well received I suspect!

Orbit Wars just hit 3k teams! (and the self-reported RL team just took the lead). Still over a month left for folks who want to jump in! kaggle.com/competitions/o…

After a few hours of working on this, I have been unable to get it to work in PufferLib for RL. It is far less stable and far more brittle than simple cosine decay, even across different timestep budgets

Sold. This guy rocks

Reinforcement learning research with Joseph Suarez x.com/i/broadcasts/1…

Reinforcement learning research with Joseph Suarez x.com/i/broadcasts/1…

Reinforcement learning research with Joseph Suarez x.com/i/broadcasts/1…