Fergus Finn

The problem ig is ‘the kV transfer issue being trivial’ is functionally equivalent to ‘infinite HBM’ which enables so much other stuff One good arg I think is that mooncake etc. are the reason why your two axes (reprefill+load balance or kV cache+kv-aware) aren’t actually orthogonal - you could rebalance (kinda like EPLB, which is analogous) your HBM kV caches to satisfy both goals while users or agents are idle (tool calls, sleep)

@finn_fergus ya, i spoke with some people here and it seems my argument's a bit reductionist. as in, i agree that > sparse attn hits both sides but if the kv transfer issue were trivial to solve, you could have a large kv / no need for sparsity imo

one of the most unintuitive problems to solve for in ml inference is the routing problem imo. and i claim that this, and this alone, is responsible for the birth of every flavor of sparse attention (SWA, NSA, DSA, etc). to set up why routing birthed sparse attention: you have multiple nodes, and you have multiple requests. 1) can do pure load-balancing: request 1 goes to node 1 request 2 to node 2 (node 1 is busy) request 3 to node 3.... this would be perfect if the model did not depend on the kv cache. but it does, and so this is a terrible system, since you do prefill over the same tokens many times. on the other extreme: 2) can do pure cache-aware routing. request 1 goes to node 1 ( kv cache stored on node 1) request 2 goes to node 1 (the kv cache is on node 1) request 3 goes to ... node 1 (the kv cache ...) this would be perfect if the system only depended on the kv cache. but it does not. decode time makes the requests queue, so this is also a terrible system. and so it seems that the two pull in opposite directions (at least if we run both the cache aware and load based routing to their extreme case) this is analogous to a somewhat dysfunctional engineering team. if ticket i+1 comes up, do you give it to the person who has context on ticket i, or do you route it to someone who has to onboard? the former is probably working on something else, but could finish it quickly. the latter can start on it immediately, but will take a while. you have to do a balance of the 2. and so the solution becomes a calculation that the router has to do: what is the a) estimated queue time b) number of uncached tokens c) speed of prefill (processing said uncached tokens) and it assigns the total=a+b/c to each node, picking the node with the least total time at a per request level. but, imo, this is a patch that exists because the kv cache stays on the node it was computed on, because we don't have the ability to keep one shared kv cache / one shared storage system for every node within a cluster... but i'm obviously wrong. mooncake (and others) exists. maintaining a shared kv pool is a decade old idea... but then: 1) you're still limited by the fact that you can't have a remote / distributed kv pool. you need every single block in the kv to be present on the node you're doing decode on (due to HBM speed and decode being mem bound), so you must pay the price of doing the transfer. 2) the global pool can only try to transfer some of the kv to a node, but cannot guarantee, because of evictions, and long transfers at scale. even if transfers were guaranteed, even if evictions never happened, you still need (b)/(c) for the 'redo prefill' vs 'wait for kv transfer' arbitrage. so then: - what is the point of maintaining global kv pool, if only to track where distributed kv blocks are and do said arbitrage, if it cannot be used to directly draw kv blocks from? - can this only be solved when network bandwidth equals that of HBM (ie unsolvable problem)? and, on the premise that the answer to both points above is yes, the solution becomes to need the least amount of kv blocks per request, and this is why every open source lab is racing to show off a new and shiny sparse attention flavor.

@SzymonOzog_ Thanks for sharing!

Saturday reading: "What happens when you run a CUDA kernel" - very cool blogpost on the details about the CPU<->GPU communications required for lauching a kernel

Full post, with the tricks: blog.doubleword.ai/what-happens-w…

So why do checkpoints & restores run slow? ~70% of the time is on the host: allocating & zeroing the staging buffer. Handing libcuda a pre-zeroed buffer, and deferring unmapping it speeds up a checkpoint cycle by a factor of 4.

New post: reverse-engineering NVIDIA's cuda-checkpoint utility to force it to run faster. There's a feature in the NVIDIA's driver that freezes a running CUDA process into host memory, and later restores it exactly where it was. We use it to speed up cold starts for inference servers. It's surprisingly slow!

Full post, with data: blog.doubleword.ai/speculating-on…

We explore accept-length distributions across content & drafter type, and argue — about a week too late, thanks to DSpark — that engines ought to pick draft lengths dynamically from the drafter's own confidence

New post on the Doubleword blog: what MoE routing data says about the trade-off between batching and speculative decoding.

What if attention were code? We show that many attention heads in transformer LMs can be replaced by human-readable Python programs. Swap them in and the model barely notices. See our experiments here: Explaining Attention with Program Synthesis [arxiv.org/abs/2606.19317]

@charles_irl Thanks for sharing!

@norpadon very cool! i guess the load balancing comes in as a pretraining objective because of wideEP inference and training efficiency, but it could probably be weakened to just spatial load balancing even at pretrain time

@finn_fergus There is a very exciting line of work focused on making MoE activations temporally stable: see arxiv.org/abs/2604.20156 for example This is something that we are also actively working on

One of the reasons Mixture of Experts models are a huge pain in the ass is that they make roofline analysis way harder

@norpadon one interesting empirical thing re. spec. dec: speculative token runs are more correlated in which experts they activate than random batching, so you claw a little bit of the MoE pain back

@norpadon in case interesting: i wrote up something similar here, it does indeed make rooflines much harder. blog.doubleword.ai/economics-of-s…

New post: a CUDA kernel in excruciating detail. We trace a simple kernel from nvcc to SASS, through libcuda to the driver, down to the warps and back to the output. blog.doubleword.ai/what-happens-w…

I guess the right q is what's better for fixed verification cost though. In principle trees ought to be strictly better (lines being degenerate trees) if the drafter is calibrated well enough that some algo can branch when uncertain (like arxiv.org/pdf/2604.12989). But then maybe the calibrations the hard bit

As always, it’s a lot more nuanced than this. In spec decoding, the predominant cost is target verification, the chain vs tree tradeoff is well known, and at production concurrencies you’re often better off training a better draft model than doing tree drafting.

In our latest Doubleword blog we analyse 18 different benchmarks to try and figure out when open source LLMs will finally close the gap with their closed source cousins: blog.doubleword.ai/frontier-os-llm