Paul

Speedrunning GPT-2 is now routine thanks to @karpathy. But can we speedrun GPT3-175B? We attempted to match accuracy on a <$10K budget; while we didn't quite reach it, our first results show that quality data, engineering, and native FP4 can get close. Details in 🧵

What’s the best model you can train in a day if someone hands you a pile of Blackwell GPUs? You can try out yourself On April 9 in Paris, @GPU_MODE + @verdacloud + @sestercegroup are hosting a GPU hackathon with a bunch of GPUs to run on and even more of them for the winners.

@iamgrigorev github.com/IST-DASLab/llmq does some of that

I am thinking of writing the next blogpost about these topics: Optimizing training throughout with FP8 I will show how to write FP8 kernels How to implement DDP How to implement FSDP, with distributed Muon How to implement TP Gradient accumulation Gradient checkpointing I think using “How to scale your model” on a real consumer gpus connected with pcie and writing that from scratch on pure PyTorch would be really useful

🚀 We are releasing state-of-the-art post-training quantization (PTQ) algorithms for Microscaling FP4, together with kernels: - First study focused on MXFP4/NVFP4 PTQ for LLMs - New Micro-Rotated (MR) format and GPTQ algorithm - QuTLASS GPU kernels with up to 3.6x speedups.

@antferdom @JackMonas @DataCrunch_io @scannell_aidan @rmwu36 @yizhao0313 @gpupoorftw @JackMonas, I'm just checking the submission deadline, the ICCV workshop page says 20.09, but is it now 27.09?

@JackMonas @DataCrunch_io Truly important challenge to understand in-depth video world models problems! Making explicit mentions to the main team contributors as well, @Scannell_aidan in compression track, and @Rmwu36 and @Yizhao0313 sampling. Lead by @edchangy and @gpupoorftw

New leader for the Compression track in the ICCV 1X World Model Challenge! Submission from @DataCrunch_io @antferdom Final deadline to submit solutions is Sep. 27 AoE

👇Retweet this tweet, private message me your WaveSpeedAI account, and I'll give you guys $3 credits!🤩Only the first 100 people can get it

❗️ We just expanded our capacity of B200 SXM6 180GB servers – available in our Cloud Platform. The best thing is… With DataCrunch, you can deploy the Blackwell platform without approvals. Just sign in and select the instance type: cloud.datacrunch.io/?utm_source=x&…

Also pretty cool to see open source community building on top of each other!

Link to blog: datacrunch.io/blog/multi-hea…

A new paper just dropped from Tri Dao(🐐)'s lab! arxiv.org/abs/2505.21487 Here is my hot take!

The paper also suggests Group Tied Attention (GTA), which works in the opposite direction and draws inspiration from MLA, incorporating those techniques into GQA.

Well, the paper suggests a hybrid method. What about using MLA and adding groups?

First of all, a confession! In the blog titled 'Multi-Head Latent Attention: Benefits in Memory and Computation', we didn't tell the whole story—the benchmarking on a single GPU. In reality, for DeepSeek V3-style models, parallelization is needed.

Instead, one must make a copy of the latent component across GPUs, which feels wasteful.

🆕 Inference API for FLUX.1 Kontext [max] & [pro] are now available on DataCrunch! We are an infrastructure partner of @bfl_ml for Kontext, a suite of generative flow matching models for text-to-image and image-to-image editing. Learn more: datacrunch.io/managed-endpoi…

🚨 Summer Inference by Symposium AI is happening next Wednesday, June 4, at 16:00-22:00. 🇫🇮 This event will bring together 250 AI engineers, researchers, and founders under one roof in Helsinki. 🔗 You can still grab one of the last remaining seats: lu.ma/x5hhj79x