Marc Sun retweetledi
When you run a @PyTorch model on a GPU, the acutal work is executed through kernels. These are low-level, hardware-specific functions designed for GPUs (or other accelerators).
If you profile a model, you'll see a sequence of kernel launches. Between these launches, the GPU can sit idle, waiting for the next operation. A key optimization goal is therefore to minimize gaps between kernel execution and keep the GPU fully utilized.
One common approach is `torch.compile`, which fuses multiple operations into fewer kernels, reducing overhead and improving utilization.
Another approach is to write custom kernels tailored to specific workfloads (e.g., optimized attention or fused ops). However, this comes with significant challenges:
> requires deep expertise in kernels writing
> installation hell
> integration with the model is non-trivial
To address this,@huggingface introduces the `kernels` library.
With this one can:
> build custom kernels (with the help of a template)
> upload them to the Hub (like models or datasets)
> integrate them to models with ease
Let's take a look at how the transformers team use the kernels library to integrate it into the already existing models. (more in the thread)
English
