Filip Morawiec

i never quite understood mamba, while reading about a jane street post on positional embedding, it suddenly made sense - and, i think it is part of a general trend that nothing is explained well until it must be explained to a mass audience (in this case myself)

Day 122/365 of GPU Programming Continuing to learn about state space models (SSM), especially the mamba model family. I find them a bit more difficult to understand than Transformers, so trying to build up a clearer picture progressively from earlier related versions like S4 and understanding the motivations (in particular, hardware related) behind their existence. Also insane how one person (Tri Dao) can be behind so many of the interesting systems papers of the last few years. Really inspirational.

Day 118/365 of GPU Programming Spent some time looking more into GPU fabric and learned a little bit about how modern GPU clusters move data across NVLink, NVSwitch, PCIe, RDMA and similar interconnects. My current understanding is that once a model no longer fits on one GPU, performance might be less about whether a single GPU can do some computation and more about how much data to move between GPUs and how to avoid moving too much? In decode a new query can be really small (e.g. just one token or a small # of tokens) but the KV cache can be huge because it stores KVs for the entire previous context. And since online softmax can be computed in pieces where each shard keeps only small local summary stats like the local max score, local normalizer and locally weighted value sum, those summaries can be merged later into the same result as full attention. So seems like it’s possible to compute exact long context attention by leaving the KV cache distributed across the GPU fabric, sending small query blocks to the KV shards and merging only tiny softmax stats instead of moving raw KV back to one GPU? Not sure but maybe someone who's more knowledgeable can enlighten me on how this works exactly one day.

The Google turboquant paper is making ML folks in this decade discover JL lemma and interact with math folks (which is cool). It appears more cool and mysterious if you do not read ML papers from the 90s and early 2000s :)

why does pre-norm work better than post-norm in transformers? i've been diving into transformer architecture (n'th time again) and noticed something interesting this time- almost all the implementations I have seen uses the "pre-norm" variant (normalizing before the sublayer then adding residual) consistently outperforms the original "post-norm" design (adding residual then normalizing). the code difference is simple: • post-norm: output = norm(x + sublayer(x)) • pre-norm: output = x + sublayer(norm(x)) but why does this seemingly small change allow for much deeper and more stable transformer training? i get that it helps gradient flow, but i'm curious about deeper intuitions, I mean what's the pin point maths behind this?!