max fu

@jaygala223 1. I use your app by the way! I'm trying to learn Telegu on the side with it! 2. I haven't thought about triton. I was thinking of PTX instead. But yes, go for a PR!

@maxxfuu This is so cool 😍 Are you planning to add dsl support? Happy to open a PR for adding triton support

Day 9/90 of Inference Engineering I built a visualization tool for CUDA kernels and C code to view the buffers for each step through. Check out visualkernels.com Yesterday, I was terramogged by a simple implementation of a basic matmul between three matrices in 1D vector form. There were essentially three 1D vectors, each vector representing a corresponding matrix (A, B, C). In the implementation, each matrix is just just a pointer to the first element of a 1D array. You are to compute the output matrix through A^T @ B and store it in C. This morning I built a visualizer to help me see and understand why the triple for-loops work the way they do. In the coming days, I’m confident I can make sense as well as backpropagation! As for visualkernels.com, it does not include a built-in compiler. It's just an interpreter; the visualization shows exactly what the code does, bugs included. The main thing I wanted to visualize is matmul with a triple for-loop between a sequential program to that of a parallel one. Expect bugs and other inconsistencies within visualkernels.com. I’ll polish this mini-project in the coming days. As for the rest of today, I’ll be trying to understand backpropagation in C and CUDA .

Yes there's vim

@0xluffy you’re gonna have the most anabolic protein farts🤯

4 american dollars for 365kcal 55g protein

@adelbucetta Give me more!

@maxxfuu that's still just scratching the surface of what's possible with gpu-accelerated simulations

Day 8/90 of Inference Engineering I built a visual interactive simulation of warp scheduling on a GPU SM. It models one SM with 8 resident warps, and all context is held within the 256 KB register file. You can click to inject a cache miss and then watch the stalled warp park with a live countdown. The scheduler moves to the next ready warp on the same tick. Showing that nothing is saved or restored. I'm pretty sure it's roughly 400 cycles measured on Volta. I built this to show how warp switching is free, which is how GPUs hide memory latency with parallelism instead of large caches. I am also struggling really hard to write the backward pass for the MNIST classifier in C. It's easy to understand math. But when it comes to implementing the matmul where each of the matrices is represented as a 1D vector; the implementation becomes so much more difficult. I starred at the for-loop and traced the code while visualizing each step in my head. I want to finish this properly so I can write CUDA again!

@okwilkins I’m taking your advice! Will update you when I succeed 💯😤

@maxxfuu Cool stuff! I also struggled with flattening matrices/tensors too. Going through the first 6 chapters of PPMP and doing the exercises there helped me to build the mental model needed. It becomes much easier.

@misraetel @dwarkesh_sp @_sholtodouglas a collab i didn’t know i needed!

Huge fun training the @dwarkesh_sp podcast folks as well as a lone, wandering @_sholtodouglas. These guys CRUSHED! And only one person died! Success!

NN Backward Pass: youtube.com/watch?v=m27Tt_… My Github Repo: github.com/maxxfuu/CUDA-K…

Day 7/90 of Inference Engineering I built a PyTorch MNIST classifier so that I can practice going through the stages of benchmarking, profiling, and optimizing in CUDA. Anyways, check out this animation that I made. It resembles the H100 SXM5 Module with 132 SM, 8 GPC, and 5 HBM3! For the next 7 days, I want to:  - Finish this mini MNIST CUDA project  - Build a Transformer with CUDA kernel for inference, with CUDA-Python binding pipeline - Go through chapters 1-6 within PMPP Cheers to the first week!

Some teaser results for a new quantization method we've been cooking up🧑‍🍳 GLM 5.2 is getting even faster

@highperfcache @skydotcs @pangram are your dms open, (@highperfcache)? would love to probe ur brain on what you know about high performance cache (not a joke)

@skydotcs @maxxfuu @pangram receiving one of these gotta be depressing lol

Day 6/90 of Inference Engineering I wrote a CUDA kernel for 1D Convolution, just getting the reps in for writing unoptimized boring but correct CUDA! I also read PagedAttention (2312.07104), watched 1 short lecture on GPU Memory, and hack squatted 455lb x6 and 495lb x3 as a top set! Heres what I've learned about PagedAttention: PagedAttention reduces internal fragmentation and solves external fragmentation. During the decode phase, the KV cache manager actually hands out one cache block at a time to store output tokens. If the output tokens overflows the cache block, the KV cache manager assigns a new cache block to hold the remaining tokens that didn't fit into the previous cache block. What ends up happening is that the VRAM is used effectively such that each cache block is filled before new cache block is allocated on the VRAM. In a naive KV cache implementation, memory is reserved up front for the maximum sequence length, which might not ever be used, meaning there is just a massive chunk of VRAM that's not touched. PagedAttention reduces internal fragmentation by ensuring memory is never wasted between each decode sequence. To make usage of memory even more efficient, any unique prompts that share the same prefix tokens, the KV cache manager allows them to share the same cache block, however, it writes to a new cache block starting from where the new tokens differ within the two prompts. While each decode phase effectively writes to random parts of the VRAM, the Block table is able to provide an abstraction that makes KV cache seem like a contiguous memory to the model. Making the allocation process as simple as checking if a cache block is full or not. This continuous practice of using cache blocks to allocate just enough memory and using a block table to allocate non-contiguous free scattered memory on the VRAM reduces internal fragmentation and solves external fragmentation. --- Watched a video on GPU Memory: youtube.com/watch?v=Zrbw0z… I read this blog that made absolutely cemented everything, written by (@hamzaelshafie): hamzaelshafie.bearblog.dev/paged-attentio…

@highperfcache @skydotcs @pangram haha, i was sad at first because i thought that he thought my writing sounded too robotic. but then again, each reflection takes about 1 hour, so it’s high quality human slop. this is the best yall are gonna get 😌

@prathamgrv @TensorTonic super excited for you!!!

career update: my time at my dream job, Microsoft, has come to an end. i'm going full time as founder at @tensortonic. started as a pet project for myself in a private repo. now it powers thousands of people writing ML and GPU code daily, closing in on 50K users. i went all in because the ceiling kept moving. inbounds from universities, edtech platforms, companies wanting it for their engineers. people proudly sharing on linkedin, X, instagram. none of which i sought out, which made the decision feel easy. is it a kaggle competitor? leetcode for ML? a cloud IDE? it's been called all three, and i'm not in a hurry to explain it. more soon.

@maharshii yo chill...

i have nothing to post, i think i fell off deleting my account in 3…. 2….

@SzymonOzog_ oh my god i didn’t know you’re simon oz on yt!

Saturday reading: "What happens when you run a CUDA kernel" - very cool blogpost on the details about the CPU<->GPU communications required for lauching a kernel

@skydotcs @pangram im putting my reputation on the line and documenting everything. theres a higher chance of you seeing human slop than AI slop LOL

@pangram @maxxfuu noice