Luka Ribar
10 posts
@luka_ribar
Research Scientist at Graphcore. Previously PhD & MEng at the University of Cambridge.
Thrilled to announce that our SparQ paper has been accepted to #ICML2024! ✨🎉 For those who can't wait, we'll also be at the ME-FoMo & PML4LRS workshops next week at @iclr_conf in Vienna. Keen to chat with anyone interested in efficient attention. twitter.com/savelichic/sta…
SparQ Attention: Bandwidth-Efficient LLM Inference paper page: huggingface.co/papers/2312.04… Generative large language models (LLMs) have opened up numerous novel possibilities, but due to their significant computational requirements their ubiquitous use remains challenging. Some of the most useful applications require processing large numbers of samples at a time and using long contexts, both significantly increasing the memory communication load of the models. We introduce SparQ Attention, a technique for increasing the inference throughput of LLMs by reducing the memory bandwidth requirements within the attention blocks through selective fetching of the cached history. Our proposed technique can be applied directly to off-the-shelf LLMs during inference, without requiring any modification to the pre-training setup or additional fine-tuning. We show how SparQ Attention can decrease the attention memory bandwidth requirements up to eight times without any loss in accuracy by evaluating Llama 2 and Pythia models on a wide range of downstream tasks.
