Morris Yau

@akyurekekin @maojiayuan @jacobandreas @StefanieJegelka @joshtenenbaum

Oral at Neurips 2025! The optimization of linear attention admits nearly perfect theoretical characterization. Hope our work inspires new perspectives in Transformer learning and scaling architectural choices. arxiv.org/abs/2410.10101

Huge thanks to an incredible group of collaborators! @sharut_gupta @vneoncourse @jacobandreas @StefanieJegelka Kazuki Irie

See our paper for why this training diagram is "all you need" to understand modern RNN's (parallelizable training, constant time decode).

Transformers: ⚡️fast to train (compute-bound), 🐌slow to decode (memory-bound). Can Transformers be optimal in both? Yes! By exploiting sequential-parallel duality. We introduce Transformer-PSM with constant time per token decode. 🧐 arxiv.org/pdf/2506.10918

🙏 Huge thanks to @akyurekekin @maojiayuan @joshtenenbaum @StefanieJegelka @jacobandreas

🤔 When increasing the number of parameters in a "real" transformer by a fixed budget, should we add more heads or more layers? Our synthetic data experiments reveal that adding heads is more effective than deepening the network.

🧐 Is there a learning algorithm that rapidly finds the best fit transformer parameters to any dataset? arxiv.org/pdf/2410.10101

I suppose this begins my shameless journey of self promotion.