Reuben Stern

We have enjoyed collaborating with @thinkymachines on some of the attention backend that supports this impressive work. Congrats to everyone involved!

@Simon_Vt Thanks, Simon!

good work from friends at colfax as always research.colfax-intl.com/dynamic-persis…

We have a new blog about cluster launch control (CLC) on NVIDIA Blackwell GPUs! CLC is a powerful tool for dynamically scheduling work across the GPU, both within- and accross-kernels. research.colfax-intl.com/dynamic-persis…

All my homies love CLC

@Simon_Vt @GPU_MODE @marksaroufim Jack has a special gift for exposition!

@ReubenConducts @GPU_MODE @marksaroufim Best talk on CuTe Layout algebra imo :)

My colleagues Jack Carlisle and Jay Shah gave a fantastic lecture for @GPU_MODE yesterday on our categorical foundations for CuTe layout algebra! They were joined by Cris Cecka, the inventor of CuTe, and @marksaroufim as moderators. Bravi tutti! youtu.be/MVh_guNbWMA?si…

@charles_irl pmpp and the rising sea are the only two books anyone could ever need!

I am the very model of a modern engineer-general, I have citations technical, go-to-market, and spiritual, I know the emperors of Rome and algebras categorical, I'm well-acquainted too with databases relational, I've written CUDA kernels memory and compute-maxxical...

@drisspg gemm+epi op+fused quant is big

Same question but O(new operators needed) Obviously we all want flex linear attention, but what else? :SOL quant kernels - do we need to add some common fusions for them in eager? What about jitted Gemm+epilogue without full compile machinery?

@drisspg the answer of course is "more than you think; not as much as you want"

“You aren’t a researcher” ohh yeah? I’m researching the limits of how much butter and bread the human body can consume. we are not the same.

🚀 Linear Attention is unlocking million-token context windows by dropping computational complexity from O(N^2) to O(N), but software is increasingly bottlenecking the hardware. Meet cuLA (CUDA Linear Attention): hand-written kernels using CuTe DSL & CUTLASS C++ to extract maximum performance on NVIDIA GPUs. A drop-in replacement for FLA designed to push hardware to its absolute limits.

Thank you to the companies and open-source communities behind Kimi K2.5, Ray, ThunderKittens, PyTorch, and more. We'd also like to thank Fireworks and Colfax for their collaboration and partnership.

PyTorch 2.11 is now available, featuring 2,723 commits from 432 contributors since PyTorch 2.10. This release prioritizes performance scaling for distributed training and next-generation hardware architectures. Highlights include a FlashAttention-4 backend for FlexAttention on Hopper and Blackwell GPUs, Differentiable Collectives for distributed training, and performance optimizations for Intel GPUs via XPU Graph. This release also delivers comprehensive operator expansion for Apple Silicon (MPS) and RNN/LSTM GPU export support. 🖇️ Read the PyTorch 2.11 release blog and release notes: pytorch.org/blog/pytorch-2… #PyTorch #OpenSource #AIInfrastructure

The frontier has increasingly shifted to hybrid models - from Qwen to Kimi-Linear and now with NVIDIA's Nemotron-3 Super - that rely on a strong linear sequence model. Today we release Mamba-3, the most powerful linear model to date. x.com/_albertgu/stat…

@drisspg @gaunernst it's pretty close, but FA-4 is missing in particular some of the inference optimizations that FA-3 has, such as cuda graphability via dynamic scheduling metadata. coming soon, though!

@gaunernst FWIW its already there but that being said I would argue that they are at different levels of code maturity / feature coverage

While all the hype has been around FA4 there is still alot of users of FA3 and hopefully this will help to make that easier :) And similarly you can use this with SDPA: #L21" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorc…

It's been great working on the FA-4 backend to FlexAttention -- check out this blog post to learn more!

New blog post on introspection for interpretability, and why I think training models to self-explain is a promising frontier for interpretability research:

Friends at @colfaxintl released an excellent work on the mathematical foundation of CuTe layouts. CuTe layouts are central to the modern programming models on NVIDIA GPUs. You can (almost) ditch C++, but you cannot ditch CuTe. In fact, you can (almost) ditch C++ because of a thing called Python CuTe DSL. And to use that, you must know CuTe layouts. Despite their role, CuTe layouts are highly unintuitive. I think you can only make sense of CuTe layouts with some mathematical guarantees about their peculiar behaviors. Colfax friends provided that 👇 It's amazing that these best things are free.

@YuriSulyma not with that attitude you can't

@YuriSulyma i'm so proud of you

Amazing, Matt! Combined with one of my favorite acts of opera ever 😃