Mickey Atwal
1.3K posts
@MickeyAtwal
Gurinder Singh “Mickey” Atwal. Machine learning and biomedical scientist. Lapsed physicist. 🇬🇧 in 🗽
Flash Invariant Point Attention １．FlashIPA introduces a linear-scaling reformulation of Invariant Point Attention (IPA), a core algorithm in protein and RNA structure modeling. It achieves SE(3)-invariant geometry-aware attention with dramatically reduced memory and runtime, enabling training on sequences with thousands of residues. ２．IPA has been widely used in structural biology models like AlphaFold2, ESMFold, and FoldFlow, but its O(L²) scaling in sequence length severely limits training on long biomolecules. FlashIPA overcomes this with a factorized attention mechanism that leverages FlashAttention for efficient GPU usage. ３．FlashIPA maintains the geometric inductive bias of IPA by encoding pairwise spatial information via low-rank factorized representations, avoiding materializing full pairwise tensors. This preserves structural accuracy while dramatically reducing I/O costs. ４．In benchmark tests, FlashIPA matches or outperforms original IPA on validation tasks, while requiring significantly less GPU memory. It reduces memory usage by over 90% at length 512 and enables batch sizes up to 39× larger compared to IPA on the same hardware. ５．Integrating FlashIPA into models like FoldFlow and RNA-FrameFlow showed faster convergence, better scaling, and extended generation capacity. For proteins, sc-RMSD scores improved when trained with FlashIPA, especially when trained on full-length data without truncation. ６．For RNA generation, FlashIPA enabled training and inference on sequences over 4000 nucleotides—impossible with standard IPA. Models trained on a single GPU with FlashIPA performed comparably to IPA trained on 4 GPUs, demonstrating cost efficiency. ７．FlashIPA runs up to 30× faster than IPA for long RNA sequences and scales linearly in both memory and runtime. This opens the door to modeling large protein complexes or long RNAs previously out of reach due to hardware limitations. ８．Despite using approximate factorized representations, FlashIPA retains SE(3) invariance and maintains modeling fidelity. Loss curves and self-consistency scores validate its effectiveness in both protein and RNA generative tasks. ９．FlashIPA is designed for easy drop-in replacement, with an interface similar to existing IPA modules. It is compatible with standard biomolecular modeling pipelines and paves the way for efficient, scalable geometric deep learning. １０．Future improvements may include extending FlashIPA to support arbitrary head dimensions and exploring fully linear attention mechanisms. This would push biomolecular modeling even further toward large-scale and real-time applications. 💻Code: github.com/flagshippionee… 📜Paper: arxiv.org/abs/2505.11580 #GeometricDeepLearning #ProteinFolding #RNA3D #FlashAttention #InvariantPointAttention #AlphaFold #ComputationalBiology #FlashIPA
“The Billion Cells Project will help clarify our understanding of the fundamental biology underpinning human health + disease while supercharging efforts at the intersection of AI & biology.” - @JCoolScience Learn more about the project czi.co/4hBZ5kM
A new Science study presents “Evo”—a machine learning model capable of decoding and designing DNA, RNA, and protein sequences, from molecular to genome scale, with unparalleled accuracy. Evo’s ability to predict, generate, and engineer entire genomic sequences could change the way synthetic biology is done. Learn more in this week's issue: bit.ly/3OsmUPr
