Eric Schreiber

NVIDIA’s CuTe layouts are gaining traction. I wanted to see why everyone loves them. The basics were easy, but intermediate resources beyond matrix algebra were scarce. So I wrote a blog post sharing my journey, building up to a GEMM kernel that can beat cuBLAS 🧵

@tugot17 @UniofOxford So cool! congrats 🙌🏻

I will give a lecture on "LLM Inference Economics" at @UniofOxford in July 🇬🇧

@TomaszSternal Huge congrats 🙌🏻

Thanks to my wonderful collaborators, our three papers got accepted to #ICML2026 🎉 Huge thank you to the team and see you in Seoul 🇰🇷!

At ICLR. Let’s connect and chat: hardware, CUDA, architecture, pre/post-training, and whatever’s got you excited.

@tugot17 🫡 Great as always. Why do you think there are not a lot more models that perform continued pre-training to sparsify their attention?

deepdive into the economics of DeepSeek Sparse Attention (DSA) and how it affects the profit margins of serving a Claude-Code-like products link in the thread 1/x

Nice one! I can tell you put a lot of effort into this post. I’ve started reading it and will need some time to go through it all :) I did it the opposite way, started with CuTe (bit of self-promotion: have als a blogpost) and now looking into the mxfp8 and fp4 stuff.

@yacinelearning @jonashubotter Reasoning trace length increasing is usually a good proxy for a healthy GRPO run. This method, to me, produces a strong instruct model that performs well without verbose reasoning (which is fantastic). However, for ood cases, preserving backtracking likely still matters.

@yacinelearning @jonashubotter Don’t get me wrong, I love the paper. However, I see some weaknesses in the method. Generalization may be challenging because reasoning traces are heavily reduced, pushing the model to jump straight to the correct answer (arxiv.org/pdf/2603.24472).

I've been studying this paradigm for the past few weeks guys and I get this feeling that this is it

@SzymonOzog_ you realise just how much you‘re standing on the shoulders of giants

This was a great read. Man computers are fun

@willccbb Thought this recent work from @jonashuebotter was pretty cool: For SFT: arxiv.org/pdf/2601.19897 For RL: arxiv.org/pdf/2601.20802

when is lifelong in-the-weights continual learning gonna be solved?

@karpathy @maxbittker For me it helped to create a human-opus-interaction.txt file for outputs and interaction, telling the model an absurd and unrealistic goal (kernel duration, target loss …) and not to come back to me until it has achieved it. Prolonged the loop significantly

sadly the agents do not want to loop forever. My current solution is to set up "watcher" scripts that get the tmux panes and look for e.g. "esc to interrupt", and send keys to whip if not present. Need an e.g.: /fullauto you must continue your research! (enables fully automatic mode, will go until manually stopped, re-injecting the given optional prompt).

From @karpathy's autoresearch .md

@maharshii When I started trying it out last summer it was awful. Since this year, given some initial ideas, it‘s been pretty neat. Also running in a loop to improve an implementation works quite well too

i wonder how does Claude / Codex perform in autonomously writing or optimizing CuTeDSL kernels with little to no “human in the loop” and will that be faster than a human doing it, has anyone tried this yet?

@tri_dao Once I got the hang of CuTe I'm loving it as well. The compile time is amazing! But the entry barrier feels huge. Feels like you need to know CUDA and have a PhD in math before you can even begin

I’m unreasonably excited about the fact that we wrote everything in Cute-DSL, embedded in Python. Installing / “compiling” now takes seconds instead of minutes / hours (looking at you, C++ templates). Try pip install fa4!

The FA4 paper is finally out after a year of work. On Blackwell GPUs, attention now goes about as fast as matmul even though the bottlenecks are so different! Tensor cores are now crazy fast that attn fwd is bottlenecked by exponential, and attn bwd is bottlenecked by shared memory bandwidth.  Some fun stuff in the redesigned algorithm to overcome these bottlenecks: exponential emulation with polynomials, new online softmax to avoid 90% of softmax rescaling, 2CTA MMA instructions that allow two thread blocks to share operands to reduce smem traffic.

Feedback very welcome.
Blog: open.substack.com/pub/schreibere… Code & Profiles: github.com/ericschreiber/…

The final kernel reaches 116% of cuBLAS on A100s for 2048×2048 BF16 matrices. Though to be fair, the specific lead at 2048 is more likely due to them underperforming rather than to my ingenuity

Last week we explored NVIDIA's CuTe layouts. Today, we put that theory into practice. Part 2 is out now! Most CuTe examples skip straight to highly optimized code without explaining the reasoning. Join me as we build a MM kernel with CuTe, that can beat cuBLAS in certain cases 🧵

@elliotarledge check out verda.com

Does anyone have spare H100s or any decent compute sitting around? My providers are at capacity.