PyTorch

Normalization layers often introduce memory-bound bottlenecks in large language models and recommendation systems due to unique hardware tiling requirements. To address this, Meta has been working on techniques to fuse normalization operations directly with GEMM and Attention kernels. By overlapping CUDA Core normalization math with Tensor Core execution pipelines, notable speedups can be unlocked. Multiple novel ideas and techniques have been introduced to tackle the key blocker to norm-related fusions: the tiling difference. These include Lazy Pre-Norm, Multi-CTA Norm Fusion, and FlashNormAttention. Testing on real-world RecSys traffic and NVIDIA B200 hardware shows these techniques can hide up to 90% of normalization latency behind GEMM kernels. This yields up to a 35% latency reduction for full Attention blocks with pre-norm, post-norm, and residual connections. This milestone shows what is possible when you combine algorithmic insights with low-level hardware optimization. We are moving closer to a world of free normalization, where developers get full modeling power without performance compromises. Read the complete blog, link in comments 👇