Chen Lu

Composer 2 is pretty good!

@kevinyang 🔥 congrats!

We raised $6.5M to build the agent for professionals. When your reputation is on the line, you need an agent that's reliable, secure, and one step ahead. Try it now at serif.ai

LTM-2-Mini is our first model with a 100 million token context window. That’s 10 million lines of code, or 750 novels. Full blog: magic.dev/blog/100m-toke… Evals, efficiency, and more ↓

@yapdianang Hey Dian Ang! Don't think there were too many surprises because this was a learning project. I did learn that convolutions are hard, and you really should worry about memory transfers

@_Chen_Lu_ @karpathy @Si_Boehm Chen!!! Wow

I wrote a UNet diffusion model in pure CUDA: github.com/clu0/unet.cu This project was inspired by @karpathy 's llm.c (github.com/karpathy/llm.c). I also learnt a lot about CUDA kernels from @Si_Boehm 's Matmul blog (siboehm.com/articles/22/CU…). (1/3)

@MaheshaGodekere Read Simon's blog, it's terrific

@_Chen_Lu_ @karpathy @Si_Boehm Awesome....I want to learn CUDA kernals....any advice?

@RisingSayak @karpathy @Si_Boehm images are 64x64, using fp32, not sure if that's what you're asking?

@_Chen_Lu_ @karpathy @Si_Boehm Also, what resolution are you benchmarking at? A transformer would be very nice too.

@iman2_718 @karpathy @Si_Boehm Thanks for the suggestion!! It looks promising for reducing the memory reloads for the convolutions, let me check it out

@_Chen_Lu_ @karpathy @Si_Boehm Have you seen docs.nvidia.com/cuda/cublasdx/ ? (to fuse blas into your kernel)

@naklecha @karpathy @Si_Boehm Thanks @naklecha ! Excited to see what's next from aaaaaaaaaa!

@_Chen_Lu_ @karpathy @Si_Boehm you are cracked omg

@TiggerSharkML @karpathy @Si_Boehm Ah that should be doable already with the kernels in Andrej's llm.c, since DiT uses the same components You'd probably want to rewrite some kernels to get best performance, otherwise you will be doing a bunch of slow tensor permutes. Definitely would be cool to see though

@_Chen_Lu_ @karpathy @Si_Boehm How about DiT in cuda

@ChrisChoy208 @karpathy @Si_Boehm Ah thanks for spotting the cudaMalloc in the training loop Chris! I thought I had moved all of them outside the loop😅 Almost all of the memory is otherwise allocated before the training loop, so hopefully this is not affecting the times too much.

@_Chen_Lu_ @karpathy @Si_Boehm This is impressive! You might want to check out memory allocators. cudaMalloc is an expensive operator and replacing it with some allocators that cache allocation could improve timing!

@eliebakouch @karpathy @Si_Boehm Thanks elie!

@_Chen_Lu_ @karpathy @Si_Boehm congrats it's super nice 🔥

@karpathy Thanks Andrej! Big fan of your work 😊

unet.cu Let's go!! 🚀 :)

The main targets for optimization are the forward and backward passes of convolutions, which are currently written in a matmul-like fashion. (3/3)

Most of the effort so far was spent getting the whole model to work. There is still a lot of room for optimization. Current per iteration timings on a single RTX 4090: - this repo: 143ms - PyTorch: 66ms - PyTorch with torch.compile: 59ms (2/3)