Neehal Tumma

Some say Gated DeltaNet > Mamba-2. Others say Mamba-2 > Gated DeltaNet. But what if Gated DeltaNet = Mamba-2? 👀 Well maybe not exactly — but with least-squares preconditioning, we show that they reduce to the same recurrence! We use this lens to design PDN, PGDN, and PKDA: preconditioned delta-style recurrences that outperform their unpreconditioned counterparts at scale 📈 📄 Paper: arxiv.org/abs/2604.21100 w/ @loo_noel @liquidai 💻 Code: github.com/ntumm120/preco…

@jeff_rey_chiu Can Anthropic give me some compute?

Some say Gated DeltaNet > Mamba-2. Others say Mamba-2 > Gated DeltaNet. But what if Gated DeltaNet = Mamba-2? 👀 Well maybe not exactly — but with least-squares preconditioning, we show that they reduce to the same recurrence! We use this lens to design PDN, PGDN, and PKDA: preconditioned delta-style recurrences that outperform their unpreconditioned counterparts at scale 📈 📄 Paper: arxiv.org/abs/2604.21100 w/ @loo_noel @liquidai 💻 Code: github.com/ntumm120/preco…

@YifeiZuoX Agreed, I think that would be a super interesting way to push beyond soft max attention. Also Ive read your LLA paper, cool stuff!

@ntumm120 Nice work. Preconditioning actually works for Softmax Attention as well, not just for Linear Attention family.

@VaibhavBerlia Yeah we looked into this a bit, Woodbury solve is expensive though

@ntumm120 Nice - curious whether a low-rank + diagonal preconditioner would recover more of the gap without blowing up the kernel cost

@skippperD Means a lot man 😂

Not quite sure what this means but if Neehal says it, it’s big

(14/N) Next up: we’re working on updating our kernels for the latest flash-linear-attention stack, including the newer GDN Tilelang and FlashKDA kernels, so stay tuned! And if you made it this far, thanks for reading :)

(13/N) But we think this only scratches the surface. PDN/PGDN/PKDA are just three points in a larger design space: different recurrence families + different preconditioners. There is a huge optimization literature on preconditioning that can be translated into recurrence design. The bigger message: preconditioning is a useful new lever for designing linear recurrences.

Today, we release LFM2.5-VL-450M, a vision-language model built for real-time reasoning on edge devices. It processes a 512×512 image and returns structured outputs in ~240ms on-device.