Joost Huizinga

Thrilled to share "Multi-task curriculum learning in a complex, visual, hard-exploration domain: Minecraft"! Curriculum learning plus a dynamic exploration bonus enables agents to obtain items far up the Minecraft tech tree. Paper: arxiv.org/abs/2106.14876 youtu.be/MFDudOvn3oc

@risi1979 These are some beautifully modular and regular networks :D

Earliest work where a neural network is used to create the architecture of a neural network (i.e. controller-based NAS)? We showed it can be done with ES-HyperNEAT in 2012, which determines both the architecture and weights of a neural network by an evolved hypernetwork.

@faoliehoek @y0b1byte @AdrienLE @joelbot3000 @kenneth0stanley @jeffclune However, this is clearly not true if you are a learning agent in a dynamic environment. So, in general, I agree that Go-Explore does not solve the exploration/exploitation trade-off, but side-steps it instead.

@faoliehoek @y0b1byte @AdrienLE @joelbot3000 @kenneth0stanley @jeffclune With respect to point 2, I think that Go-Explore effectively takes the perspective that, once you have explored the environment very well, you can then exploit what you have learned pretty much indefinitely, so the reward missed during exploration can be ignored.

Today's notes on "First Return, Then Explore" (nature.com/articles/s4158…) by @AdrienLE @Joost_Huizinga @joelbot3000 @kenneth0stanley and @jeffclune. Read to learn about detachment and derailment in the RL setting. notion.so/First-return-t…

@A3braxas @jeffclune @AdrienLE @joelbot3000 @kenneth0stanley Agreed. QD has been a strong inspiration for the algorithm.

@jeffclune @AdrienLE @Joost_Huizinga @joelbot3000 @kenneth0stanley Go-explore is awesome! QD too.

Thrilled to share that "First return, then explore" appears today in Nature! Go-Explore solves all unsolved Atari games*, ending a long quest by the field that began in Nature. Led by @AdrienLE & @Joost_Huizinga w @joelbot3000 @kenneth0stanley & myself. nature.com/articles/s4158… 1/

@anh_ng8 @jeffclune @AdrienLE @joelbot3000 @kenneth0stanley Thank you very much Anh! When we originally submitted the paper I did not expect that we would get in, but I am very happy to be proven wrong.

@jeffclune @AdrienLE @Joost_Huizinga @joelbot3000 @kenneth0stanley Amazing!!! Congratulations @jeffclune @AdrienLE @Joost_Huizinga and everyone!

@ZoubinGhahrama1 Thank you so much, especially for providing us the opportunity to pursue this work while we were all at Uber AI Labs!

Exciting new paper in Nature on RL exploration - its great to see this work that started at Uber AI Labs out now! Congratulations to the team.

@AwokeKnowing @jeffclune @Tkaraletsos @AdrienLE @joelbot3000 @kenneth0stanley Agreed. In addition, I think that regularization by something like a hypernetwork or indirect encoding could help in learning a representation that captures the important general features rather than learning each game separately.

@Joost_Huizinga @jeffclune @Tkaraletsos @AdrienLE @joelbot3000 @kenneth0stanley i agree, (though perhaps with a carefully designed curriculum could help coax the representation to a general form). i sort of expect there 'is' enough info in the 50 or so games, but 'high score'/novelty is not enough to learn it. perhaps task of 'recreating them'

@Jallafsen @joelbot3000 @kenneth0stanley @jeffclune @AdrienLE Thank you very much, and I love the art as well. Ella Maru Studio did a great job on that (also, I promise we are not sponsored).

@Joost_Huizinga @joelbot3000 @kenneth0stanley @jeffclune @AdrienLE Congrats, Joost and the rest of the team! Great work! And well deserved, I think this is one of the most intriguing concepts to appear in RL in recent years. I also love the cover art with the robot playing computer games. Who made that?

Honored to say that our paper “First return, then explore”, has been published in the journal nature. Massive thanks to my co-authors @joelbot3000, @kenneth0stanley, @jeffclune and especially to my fellow first-author @AdrienLE! nature.com/articles/s4158…

@v_vassiliades @jeffclune @AdrienLE @joelbot3000 @kenneth0stanley Thank you very much!

@jeffclune @AdrienLE @Joost_Huizinga @joelbot3000 @kenneth0stanley Congratulations to the team!!

@AwokeKnowing @jeffclune @Tkaraletsos @AdrienLE @joelbot3000 @kenneth0stanley My intuition is that 50+ games is not enough for few-shot learning. My guess is that it is still easier to fit each game separately than it is to learn a general representation that would work for many Atari games. That said, happy to be proven wrong.

@jeffclune @Tkaraletsos @AdrienLE @Joost_Huizinga @joelbot3000 @kenneth0stanley based on your intuition, would a model trained on 'all games' successfully (same model plays 50+ games) have a chance at playing new game with few-shot learning? or must the frames/representations be worked into the distributed representation in order for meaningful predictions?

@samsinai Interesting, especially since one of the inspirations for Go-Explore was MAP-Elites, which is an evolutionary algorithm that keeps an archive of good "genomes", picks one, mutates it, and adds it back to the archive if it is better or novel.

Cool result, simple but powerful! This is familiar biologists: Gene duplication, keep a copy of a good solution, explore with the copy till you find another good answer. A formal analysis can be found here, the "regeneration process" cuts discovery time: journals.plos.org/ploscompbiol/a…

@IHaveSweaters @joelbot3000 @kenneth0stanley @jeffclune @AdrienLE Thank you very much!

@Joost_Huizinga @joelbot3000 @kenneth0stanley @jeffclune @AdrienLE Congrats Joost :)

@togelius @AdrienLE @joelbot3000 @kenneth0stanley @jeffclune Thank you very much for the kind words!

This is truly important work, reshaping how we think of exploration in RL, and I'm happy to see that it has been published in the world's most prestigious scientific journal. Congrats to @AdrienLE @Joost_Huizinga @joelbot3000 @kenneth0stanley @jeffclune

@savvyRL @AdrienLE @joelbot3000 @kenneth0stanley @jeffclune Thank you very much and happy birthday! A day of celebrations!

Woohoo Nature!!! 🎉🎉 Go go-explore so proud and happy for you all @Joost_Huizinga @AdrienLE @joelbot3000 @kenneth0stanley @jeffclune Annnnd you are welcome it was my birthday wish that it gets published today 😬

Finally, the free-to-read version of the paper can be found here: tinyurl.com/Go-Explore-Nat…

However, for me, the most important take-away of the paper is its incredibly simple principles for exploration: (1) remember where you have been and (2) first return, then explore. Those are the principles that enable Go-Explore to explore so deeply and effectively.

@oh_that_hat Same for me. While not working, I sometimes think about the latest AI problem that I want to solve, but usually I think about something completely different and unrelated.

This is a beautiful state to be in. But I worry that I don’t do this...I’m not constantly thinking about these problems. In fact I’m mostly thinking about other random things, and occasionally it segways into thoughts about AI/work...is that a bad sign?