sky
15.2K posts
Day 6/90 of Inference Engineering I wrote a CUDA kernel for 1D Convolution, just getting the reps in for writing unoptimized boring but correct CUDA! I also read PagedAttention (2312.07104), watched 1 short lecture on GPU Memory, and hack squatted 455lb x6 and 495lb x3 as a top set! Heres what I've learned about PagedAttention: PagedAttention reduces internal fragmentation and solves external fragmentation. During the decode phase, the KV cache manager actually hands out one cache block at a time to store output tokens. If the output tokens overflows the cache block, the KV cache manager assigns a new cache block to hold the remaining tokens that didn't fit into the previous cache block. What ends up happening is that the VRAM is used effectively such that each cache block is filled before new cache block is allocated on the VRAM. In a naive KV cache implementation, memory is reserved up front for the maximum sequence length, which might not ever be used, meaning there is just a massive chunk of VRAM that's not touched. PagedAttention reduces internal fragmentation by ensuring memory is never wasted between each decode sequence. To make usage of memory even more efficient, any unique prompts that share the same prefix tokens, the KV cache manager allows them to share the same cache block, however, it writes to a new cache block starting from where the new tokens differ within the two prompts. While each decode phase effectively writes to random parts of the VRAM, the Block table is able to provide an abstraction that makes KV cache seem like a contiguous memory to the model. Making the allocation process as simple as checking if a cache block is full or not. This continuous practice of using cache blocks to allocate just enough memory and using a block table to allocate non-contiguous free scattered memory on the VRAM reduces internal fragmentation and solves external fragmentation. --- Watched a video on GPU Memory: youtube.com/watch?v=Zrbw0z… I read this blog that made absolutely cemented everything, written by (@hamzaelshafie): hamzaelshafie.bearblog.dev/paged-attentio…
