levi

Day 129/365 of GPU Programming Lecture 6 of CS336 was a nice review of memory hierarchy in GPUs (bank conflicts on shared memory vs memory coalescing on HBM), occupancy calculations, PyTorch profiling, Triton, and PTX. Always helps going through material again and again, especially when there are topics like profiling or Triton that I'm still quite new to.

Day 128/365 of GPU Programming Continuing Stanford's CS336 class (Language Modeling from Scratch). Onto lecture 5 on GPUs and TPUs today! Quite similar to last year's lecture which I watched during the AMD challenge but provided a bit more detail on the TPU this time around drawing parallels between Nvidia's GPU architecture and Google's TPUs, touches on prefill/decode disaggregation and also dives into MXFP8/MXFP4 (+ associated issues with transposing) And speaking of softmax, good reminder in the lecture on how lower precision depends on the operations you're optimizing for.

Day 125/365 of GPU Programming One thing I'm still struggling to understand is why softmax? What is it about the softmax function that made it survive/thrive for this long? What is it about exp() compared to another positive, monotonic, differentiable function that is so sticky? So studying softmax functions in a bit more depth today, taking a look at optimizations via SFUs on Nvidia GPUs and listening to the GOATs (Andrew Ng, Hinton, etc) explain the reasoning behind softmax as a primary choice. If anyone has good resources that dive into softmax and softmax alternatives, please share!

Day 127/365 of GPU Programming Since I always wanted to tie this back to my understanding of GPUs, I've been trying to learn more about Special Function Units (SFUs) today but it's been more difficult than expected to find good public resources around the topic. One good read on the topic was this paper titled Design and Verification of an open-source SFU model for GPGPUs, which specifically looks at sin x, cos x, log2 x, 2 ^x and 1/√x. The Modal glossary provides is a nice definition but is quite light on the mechanics of the actual unit. If anyone has some resources that dive deeper into SFUs, please share!

Day 124/365 of GPU Programming Since I've been studying state space models and hybrid attention architectures recently, I'm spending today using some of these models and seeing what kind of inference optimizations I can play around with. Mainly trying out Qwen 3.5 and looking at Qwen 3.5 with its gated attention layers. Also first time working with Hugging Face, which has been fun. Really nice how easy they make comparing models and downloading weights. Will probably spend the next few days looking into different ways to minimize inference latency, reading through existing solutions and trying out different things on my local machine with whatever models I can fit.

Day 125/365 of GPU Programming One thing I'm still struggling to understand is why softmax? What is it about the softmax function that made it survive/thrive for this long? What is it about exp() compared to another positive, monotonic, differentiable function that is so sticky? So studying softmax functions in a bit more depth today, taking a look at optimizations via SFUs on Nvidia GPUs and listening to the GOATs (Andrew Ng, Hinton, etc) explain the reasoning behind softmax as a primary choice. If anyone has good resources that dive into softmax and softmax alternatives, please share!

Introducing Aurora, a new optimizer for training frontier-scale models. We train Aurora-1.1B, which achieves 100x data efficiency on open-source internet data. Despite having 25% fewer parameters, 2 orders of magnitude fewer training tokens, and using fully open-source internet-only data, Aurora matches Qwen3-1.7B on several benchmarks. Aurora was developed after identifying a major failure mode that can occur under Muon, an increasingly popular optimizer that has shown strong gains over Adam(W). We find that Muon can cause a huge percentage of neurons to effectively die early in training, reducing effective network capacity so that many parameters no longer meaningfully contribute to network outputs. By redistributing update energy more uniformly across neurons while preserving Muon’s stability properties, Aurora prevents neuron death and recovers substantial model capacity. What makes this work especially exciting is that it points toward a broader direction for ML research: better optimizers may not come purely from elegant mathematical abstractions, but from understanding and addressing the concrete dynamics and pathologies that emerge inside real training systems.

i never quite understood mamba, while reading about a jane street post on positional embedding, it suddenly made sense - and, i think it is part of a general trend that nothing is explained well until it must be explained to a mass audience (in this case myself)

Day 122/365 of GPU Programming Continuing to learn about state space models (SSM), especially the mamba model family. I find them a bit more difficult to understand than Transformers, so trying to build up a clearer picture progressively from earlier related versions like S4 and understanding the motivations (in particular, hardware related) behind their existence. Also insane how one person (Tri Dao) can be behind so many of the interesting systems papers of the last few years. Really inspirational.

Day 123/365 of GPU Programming Another day, another attempt at understanding SSMs from first principles. Gaining a real intuition for them (and their hardware implications) has been harder than expected, so I'm just taking a closer look at the foundational state space model papers (Hippo, H3, Mamba family, S4/5, et cetera) today to see if I can understand their genealogy and the rationale behind their evolutions better. If anyone has specific blog posts or code that helped them get a better sense of the problem space, I'd love to know!