Kimbo

New York to impose the country’s first statewide moratorium on data centers. nbcnews.com/news/us-news/n…

New in-depth blog post time: "Inside TPU and GPU Clusters: The Anatomy of Collective Communication". If you want to deeply understand the core primitives behind scaling the training / inference for MoEs and dense transformers, going a level below FSDP, expert parallelism, data parallelism, model/tensor parallelism this might be a fun read. I cover: * TPU cluster topology: (super)pods, slices, DCN, PCIe, ICI * All-Gather: 1D/2D rings, and path algo (lots of visuals so should be crystal clear how these work even if you're not a perf engineer) * Reduce-Scatter (which is the dual of AG) and All-Reduce * All-to-All (used to dispatch tokens to target experts in MoEs) * NVIDIA GPU cluster topology (reference DGX architecture): nodes, scalable units, fat tree * GPU collectives within the node: rings, trees (log2 steps), and SHARP (in network compute unit) * GPU collectives across nodes, hierarchical algorithms over InfiniBand etc. I was heavily inspired to do this deep dive after reading the excellent Scaling book by an excellent group of people @jacobaustin132 @_sholtodouglas @reinerpope and others! What originally started as "let me maybe just make four figures covering All-Gather, Reduce-Scatter, All-Reduce, and All-to-All so I can understand them better, it shouldn't take more than a day, right, right?" somehow turned into this 40 figures later. Along the way, I realized that the collective algorithms only really make sense once you understand the underlying hardware topology. TPUs were a bit easier to reason about, but I couldn't skip GPUs, I love them too much. Rings are cool, but I also wanted to understand tree algorithms. But also SHARP, and fat trees, and hierarchical collectives. :') So the scope slowly expanded, and little by little, this blog post came to fruition. Just a side-quest. Hope you like it! :) --- Also a big thank you to my friends for reviewing the blog and providing feedback: * @ArunDemeure (prev GPU/AI stuff at Magic, GPU architect at Apple and Imagine, my llm.c buddy!) * @axel_s_feldmann (making GPUs go brrr at Jane Street, we met for the first time at @marksaroufim's excellent GPU mode event) * @pranjalssh (ex xAI GPU wizard, one of two people who inspired my original matmul blog!)

New in-depth blog post time: "Inside TPU and GPU Clusters: The Anatomy of Collective Communication". If you want to deeply understand the core primitives behind scaling the training / inference for MoEs and dense transformers, going a level below FSDP, expert parallelism, data parallelism, model/tensor parallelism this might be a fun read. I cover: * TPU cluster topology: (super)pods, slices, DCN, PCIe, ICI * All-Gather: 1D/2D rings, and path algo (lots of visuals so should be crystal clear how these work even if you're not a perf engineer) * Reduce-Scatter (which is the dual of AG) and All-Reduce * All-to-All (used to dispatch tokens to target experts in MoEs) * NVIDIA GPU cluster topology (reference DGX architecture): nodes, scalable units, fat tree * GPU collectives within the node: rings, trees (log2 steps), and SHARP (in network compute unit) * GPU collectives across nodes, hierarchical algorithms over InfiniBand etc. I was heavily inspired to do this deep dive after reading the excellent Scaling book by an excellent group of people @jacobaustin132 @_sholtodouglas @reinerpope and others! What originally started as "let me maybe just make four figures covering All-Gather, Reduce-Scatter, All-Reduce, and All-to-All so I can understand them better, it shouldn't take more than a day, right, right?" somehow turned into this 40 figures later. Along the way, I realized that the collective algorithms only really make sense once you understand the underlying hardware topology. TPUs were a bit easier to reason about, but I couldn't skip GPUs, I love them too much. Rings are cool, but I also wanted to understand tree algorithms. But also SHARP, and fat trees, and hierarchical collectives. :') So the scope slowly expanded, and little by little, this blog post came to fruition. Just a side-quest. Hope you like it! :) --- Also a big thank you to my friends for reviewing the blog and providing feedback: * @ArunDemeure (prev GPU/AI stuff at Magic, GPU architect at Apple and Imagine, my llm.c buddy!) * @axel_s_feldmann (making GPUs go brrr at Jane Street, we met for the first time at @marksaroufim's excellent GPU mode event) * @pranjalssh (ex xAI GPU wizard, one of two people who inspired my original matmul blog!)

1/ Still looking for a minimalist, high-performance framework for agentic RL research? Meet Molt — an agentic-first, PyTorch-native reinforcement learning framework with roughly 9K lines of RL code for 700B models. ⭐ github.com/NVIDIA-NeMo/la…