cache appreciator

@skydotcs python is a beautiful language wdym

in the python mines

@maxxfuu @skydotcs @pangram yeah sure! (i possibly don’t know much lol)

@highperfcache @skydotcs @pangram are your dms open, (@highperfcache)? would love to probe ur brain on what you know about high performance cache (not a joke)

Day 6/90 of Inference Engineering I wrote a CUDA kernel for 1D Convolution, just getting the reps in for writing unoptimized boring but correct CUDA! I also read PagedAttention (2312.07104), watched 1 short lecture on GPU Memory, and hack squatted 455lb x6 and 495lb x3 as a top set! Heres what I've learned about PagedAttention: PagedAttention reduces internal fragmentation and solves external fragmentation. During the decode phase, the KV cache manager actually hands out one cache block at a time to store output tokens. If the output tokens overflows the cache block, the KV cache manager assigns a new cache block to hold the remaining tokens that didn't fit into the previous cache block. What ends up happening is that the VRAM is used effectively such that each cache block is filled before new cache block is allocated on the VRAM. In a naive KV cache implementation, memory is reserved up front for the maximum sequence length, which might not ever be used, meaning there is just a massive chunk of VRAM that's not touched. PagedAttention reduces internal fragmentation by ensuring memory is never wasted between each decode sequence. To make usage of memory even more efficient, any unique prompts that share the same prefix tokens, the KV cache manager allows them to share the same cache block, however, it writes to a new cache block starting from where the new tokens differ within the two prompts. While each decode phase effectively writes to random parts of the VRAM, the Block table is able to provide an abstraction that makes KV cache seem like a contiguous memory to the model. Making the allocation process as simple as checking if a cache block is full or not. This continuous practice of using cache blocks to allocate just enough memory and using a block table to allocate non-contiguous free scattered memory on the VRAM reduces internal fragmentation and solves external fragmentation. --- Watched a video on GPU Memory: youtube.com/watch?v=Zrbw0z… I read this blog that made absolutely cemented everything, written by (@hamzaelshafie): hamzaelshafie.bearblog.dev/paged-attentio…

@tnm @armoredmeatbal why specifically git for this?

@armoredmeatbal it’ll be faster

our work on git will go to upstream, in the C programming language, so long as they are useful generally. so, a public fork.

@skydotcs @maxxfuu @pangram receiving one of these gotta be depressing lol

@maxxfuu hmmm prolly not but @pangram !

@LFauexProd does this not still happen? cameras just got better at processing it

Crazy that there was an era of photography where in most pictures a phenomenon would occur that turned everyone's eyes glowing red

@FangYi11101 @Andrew_Semenza considering the exact minute of the text it’s probably Switzerland’s red card vs Argentina

@Andrew_Semenza ?? Your place got robbed?

@codybrown @FangYi11101 @RokoMijic No they’re quite accurate, Tuas is quite literally a vast industrial area. You could even add another 30min-1hr if they choose to go through immigration on a busy day.

@FangYi11101 @RokoMijic you are really reaching and also now just making up random stuff about "marshlands/industrial areas." No one who goes to NS thinks its in Singapore but its nice that its close. If Waterloo was an hour from NYC, they'd make reference to that as well... but it's ~13hrs by bus.

Balaji’s Network School does everything it can to avoid mentioning that it’s actually based in Malaysia. Too much aura loss.

@injaneity have you considered writing it by hand

looking for a good skill or plugin for generating wikis of very condensed information (like technical documentation). anyone have some recommendations?

@brianchew Haha we were at the airport around the same time!

Time check 07:15! 15 hours flight to touchdown in SFO!

@FangYi11101 Their interview tool definitely supports drawing lol

This also happened to me. First thing they asked was “你会说普通话吗?” for a role in the Bay Area

@ItsCuthulhu @unusual_whales Are you not a grad student? It can definitely cost more under certain high volume use cases

"AI can cost more than human workers now," per Axios

@injaneity schedule this year is very ai-skewed, apart from the cpython talk which i would find the most interesting

anyone been to previous editions of pycon singapore? curious if its useful / interesting

@sarahookr i’ve never seen a cyclist inside funan

First time I have seen a cyclist lane go through a mall. How does this work? Speed is set to 10km/h

@dedene @weswinder their subsidies will eventually run out too

It’s so dumb. I get that the subsidies can’t keep going like this forever, but this will just drive everyone to use OpenAI or the Chinese model providers. You can only lose developer goodwill so many times. I begin to suspect the PR/Marketing department of Anthropic is being run by Haiku

anthropic just downgraded claude code subs programmatic usage previously used your full rate limits which was something like a 5-10X discount vs api costs now ALL programmatic usage is capped at $20/$100/$200 in api costs (a single opus session can easily burn $20) not good

@andimgladofit @Sketchapotamus @FangYi11101 is this quite literally you aphyr.com/posts/342-typi…

@Sketchapotamus @FangYi11101 if i was going to use some boring shizo language id just use brainfuck. the point is for me to spend 1.5 hours talking about lambda calculii

interview candidate tmr wants to do the technical round in C++ fml

@RJPJR @bryanrbeal just checked, USD 600 back then for a one way journey, good times

@RJPJR @bryanrbeal i’ve done this on premium economy for <1k back in 2024

@bryancsk just dropped to my knees in a heartland mall parking lot

An astonishing artefact of many contradictions. How should a Singaporean think of this object, both so foreign and so familiar?

@eshear i love twitter

It’s an honor just to be nominated

@FangYi11101 the difference between them is maybe a few dedicated weekends