Aaron Lemke

Inspired by @karpathy's practice of tinkering with smaller GPTs to gain intuition, I've been training small neural nets to play this simple 2d game I created. On the left is a visualization of the neural net firing in real time. Built with @claudeai, @threejs, and @WebGPU and using JAX from @GoogleDeepMind. I’m piping the neuron activation into @Ableton as MIDI to trigger Serum 2.

Minimal editor UI for @threejs, with dynamic UI contrast enforcement so you can always see the UI regardless of what's behind it. I made a simple game object system which supports full voice control of all object parameters via Gemini Live API. This makes iterating WAY easier. We're now in the era of personal software. Anyone can build their dream interface. Made with @claudeai, @webgl_webgpu and Gemini Live from @GoogleDeepMind

Casually rendering ~a hundred million triangles, in the browser. All thanks to @sea3dformat's cool Nanite-style renderer for @threejs. I've added GLB-to-meshlet and PBR rendering to it, very exciting to see where this is going!

Go buddy go! made with @threejs @webgl_webgpu

@memoakten @webgl_webgpu @threejs @GoogleDeepMind The OG! Love these little guys ❤️

@webgl_webgpu @aaronlemke @threejs @GoogleDeepMind Very Karl Sims Evolved Virtual Creatures vibes! youtu.be/JBgG_VSP7f8

New Morphologies! RL creatures made with @threejs @webgl_webgpu, JAX and MuJoCo from @GoogleDeepMind

Intelligent tentacles. Made with @WebGPU and @threejs

Tentacle agent. Made with @threejs @WebGPU, training done in MuJoCo using JAX from @GoogleDeepMind

Thing can quickly get out of hand... Built with @WebGPU and @claudeai

What would a fluid sim platformer look like? Continuing my Navier-Stokes side quest. Compute shader built with @WebGPU and @claudeai

@executiondeth @runpod @GoogleDeepMind @threejs @WebGPU Very cool!

@aaronlemke @runpod @GoogleDeepMind @threejs @WebGPU That’s awesome I did this w a b200 from @runpod

A new RL agent I trained on a @runpod A100. Built with JAX from @GoogleDeepMind , @threejs, and @WebGPU. I can toggle between autonomous and user controlled. Very cool that you basically get a full animation controller for free with RL.

@badjano @runpod @GoogleDeepMind @threejs @WebGPU Very cool!

@aaronlemke @runpod @GoogleDeepMind @threejs @WebGPU cs.stanford.edu/people/karpath…

One of my favorite parts of reinforcement learning is the emergent behavior these systems can exhibit. Mr. Noodle here learned to wedge the end of his tail into the ground, creating a temporary anchor point that lets him push against the floor hard enough to move forward. Built with @threejs , @WebGPU , MuJoCo and JAX from @GoogleDeepMind

@laplacesdust @runpod @GoogleDeepMind @threejs @WebGPU Also been impressed how far @threejs can be pushed into AAA rendering territory!

@aaronlemke @runpod @GoogleDeepMind @threejs @WebGPU Love to see this stuff, the Nvidia Isaac lab is great but their visualizations look really bad in demos

@laplacesdust @runpod @GoogleDeepMind @threejs @WebGPU Not bad, its using MuJoCo under the hood.

@aaronlemke @runpod @GoogleDeepMind @threejs @WebGPU How are the physics accuracy?

@laplacesdust @runpod @GoogleDeepMind @threejs @WebGPU I keep hearing abotu Isaac I will check it out. Also Genesis?

@paws4puzzles @runpod @GoogleDeepMind @threejs @WebGPU For now browser only but i would love to see these kinds of morphologies enter the real world in the coming months.

@aaronlemke @runpod @GoogleDeepMind @threejs @WebGPU I'd bet emergent biomechanics eats keyframe animation soon. JAX + WebGPU solves the 'works on my cluster' problem. You planning sim-to-real or keeping it browser-only?

Continuing my journey into Reinforcement Learning, now moving to 3D. I started by training the @UnitreeRobotics Go1 robot dog from @GoogleDeepMind’s MuJoCo Playground, then brought the policy into my own WebGPU viewer and started mutating the creature from there. Stay tuned for more Stack: MuJoCo MJX + JAX + Brax PPO for training, exported policy running live in-browser.

I believe this took about six or seven hours to train on an A100. The inputs were X and Yvelocity, is grounded, various raycasts in different directions, so it can sense the platforms around it, how many collectibles are left, the time, the cool down on its boost, and location of nearest collectible

@aaronlemke @karpathy @claudeai @threejs @WebGPU What are the inputs to the network? (I can't quite see in the video) How long did this take to train, and how was it trained?

Inspired by @karpathy's practice of tinkering with smaller GPTs to gain intuition, I've been training small neural nets to play this simple 2d game I created. On the left is a visualization of the neural net firing in real time. Built with @claudeai, @threejs, and @WebGPU and using JAX from @GoogleDeepMind. I’m piping the neuron activation into @Ableton as MIDI to trigger Serum 2.

@MAMware @karpathy @claudeai @threejs @WebGPU Just running locally on my MacBook Pro. But I used an A100 in the cloud via @vast_ai to train it.

@aaronlemke @karpathy @claudeai @threejs @WebGPU what tech stack is need to run it?