Nathan

Our research team has 9 papers at ICML next week! Spanning the full stack from frontier agents to GPU kernels, we're excited to share what our researchers and collaborators have been working on. If you're in Seoul for ICML come meet our team or catch a talk! 🧵

Excited to release PKB Parallel Kernel Bench, led by Willy Chan and Nathan Paek @asplencmnt!! A benchmark of mostly net-new multi-GPU kernel problems (solutions are independently useful for real-world workloads).

LLMs write fast single-GPU kernels. Ask for a multi-GPU one and they fall apart. ParallelKernelBench measures how they fail by benchmarking against 87 problems pulled from real codebases including Megatron-LM, DeepSpeed, DeepEP, TensorRT-LLM, NeMo-RL. New research from Willy Chan @asplencmnt @simonguozirui @simran_s_arora and @realDanFu

Super grateful to my coauthor Willy Chan, collaborator @simonguozirui, and awesome mentors @simran_s_arora and @realDanFu! And huge thanks to @togethercompute for making the project happen! Check out PKB here: Blog 🌐: together.ai/blog/parallelk… Paper 📜: alphaxiv.org/abs/2606.paral… GitHub 💻: github.com/togethercomput… HuggingFace 🤗: huggingface.co/datasets/toget…

That said, a few models did find solutions faster than the original repo code, producing net-new kernels! The example below speeds up a real vision workload.

Excited to release ParallelKernelBench (PKB), a benchmark for measuring LLMs’ ability to write fast multi-GPU kernels! 😀 Multi-GPU kernel generation compounds several hard problems: - a large parallelism design space - a new communication axis to optimize - and hardware-specific decisions around communication mechanisms Existing kernel-generation benchmarks mostly target single-GPU workloads, so we built PKB to cover real-world multi-GPU workloads (many of which do not have existing optimized solutions). 🧵👇