sankalp

New in-depth blog post time: "Inside TPU and GPU Clusters: The Anatomy of Collective Communication". If you want to deeply understand the core primitives behind scaling the training / inference for MoEs and dense transformers, going a level below FSDP, expert parallelism, data parallelism, model/tensor parallelism this might be a fun read. I cover: * TPU cluster topology: (super)pods, slices, DCN, PCIe, ICI * All-Gather: 1D/2D rings, and path algo (lots of visuals so should be crystal clear how these work even if you're not a perf engineer) * Reduce-Scatter (which is the dual of AG) and All-Reduce * All-to-All (used to dispatch tokens to target experts in MoEs) * NVIDIA GPU cluster topology (reference DGX architecture): nodes, scalable units, fat tree * GPU collectives within the node: rings, trees (log2 steps), and SHARP (in network compute unit) * GPU collectives across nodes, hierarchical algorithms over InfiniBand etc. I was heavily inspired to do this deep dive after reading the excellent Scaling book by an excellent group of people @jacobaustin132 @_sholtodouglas @reinerpope and others! What originally started as "let me maybe just make four figures covering All-Gather, Reduce-Scatter, All-Reduce, and All-to-All so I can understand them better, it shouldn't take more than a day, right, right?" somehow turned into this 40 figures later. Along the way, I realized that the collective algorithms only really make sense once you understand the underlying hardware topology. TPUs were a bit easier to reason about, but I couldn't skip GPUs, I love them too much. Rings are cool, but I also wanted to understand tree algorithms. But also SHARP, and fat trees, and hierarchical collectives. :') So the scope slowly expanded, and little by little, this blog post came to fruition. Just a side-quest. Hope you like it! :) --- Also a big thank you to my friends for reviewing the blog and providing feedback: * @ArunDemeure (prev GPU/AI stuff at Magic, GPU architect at Apple and Imagine, my llm.c buddy!) * @axel_s_feldmann (making GPUs go brrr at Jane Street, we met for the first time at @marksaroufim's excellent GPU mode event) * @pranjalssh (ex xAI GPU wizard, one of two people who inspired my original matmul blog!)

new essay: the more you write, the more you begin to see i started writing personal essays last year because i was worried i had lost my ability to think deeply after years in a consuming corporate job. what i did not expect was that writing consistently would change my whole life, what i noticed, how ideas connected, and how i made sense of my experience this essay is about writing as a net for catching thoughts before they disappear, the abundance that appears when you follow an idea for long enough, and the tension between paying closer attention to your life and turning it all into material read here: open.substack.com/pub/velvetnois…

new essay: the more you write, the more you begin to see i started writing personal essays last year because i was worried i had lost my ability to think deeply after years in a consuming corporate job. what i did not expect was that writing consistently would change my whole life, what i noticed, how ideas connected, and how i made sense of my experience this essay is about writing as a net for catching thoughts before they disappear, the abundance that appears when you follow an idea for long enough, and the tension between paying closer attention to your life and turning it all into material read here: open.substack.com/pub/velvetnois…