Parallel Software and Systems Group

🧠 From early programming lessons to scaling #AI on supercomputers, Abhinav Bhatele (@bhatele) shares his path and research at @UofMaryland's Department of Computer Science. Read more: go.umd.edu/Bhatele-4-2026

Congratulations to Dr. Joy Kitson for successfully defending her PhD dissertation last Friday (April 10)! That makes her the fifth PhD student to graduate from our group. Dissertation title: "Scaling Agent-based Epidemic Diffusion on HPC Clusters"

On Mar 25, we set a personal record with 2 PhD defenses in a day! Congrats to @onurcankur & @jhdavis_josh for their excellent work: Onur's title: "Longitudinal Data Analytics of HPC Systems & Applications" Josh's title: "LLM-driven Development of Performant & Portable GPU Codes"

The Distributed Deep Learning tutorial is happening right now in Room 123. #SC25 #DistributedAI #AxoNN #YALIS All the tutorial material is on github: github.com/axonn-ai/distr…

We have arrived in St. Louis for @Supercomputing 2025. Learn more about our group's research through the various talks, panels and tutorials below. We will also be at the @UofMaryland booth (3123) at various times. #SC25 #UMDCS #HPC #AI #MLSys #AI4Science

Come work with us! All of our PhD students will be @Supercomputing, don’t feel shy if you want to chat.

We are on a roll, second successful dissertation defense in a week (March 28)! Congratulations to @siddharth_3773 on becoming the second PhD graduate from PSSG!! Dissertation title: "Optimizing Communication in Parallel Deep Learning on Exascale-class Machines" #HPC #AI #HPC4AI

We present our very first newly minted Dr. Daniel Nichols (@DanielNichols10), who successfully defended his dissertation today. Congrats & best wishes for a bright future ahead!! Dissertation Title: "On Learning Behaviors of Parallel Code and Systems Across Modalities" #HPC #AI

We have a postdoc opening in our group on the intersection of HPC and AI, specifically on developing and applying Code LLMs to HPC software development. Please help us spread the word: umd.wd1.myworkdayjobs.com/en-US/UMCP/det… Position will remain open until filled. #HPC #AI

Poster session in progress … if you miss us and want to chat, you can find more information on pssg.cs.umd.edu.

We show that our method, Loki, achieves better accuracies compared to other sparse attention methods for short-context and long-context tasks across a range of models. Paper: arxiv.org/abs/2406.02542 Code: github.com/hpcgroup/loki #NeurIPS2024 #HPC #AI

Can we get away w/ reducing attention keys to a lower-dimensional space to optimize compute during inference? @prajwal1210 & @siddharth_3773 investigated using PCA on key vectors & found that the rank of attention keys is much lower than the full dimensionality. #NeurIPS2024

Come see our #NeurIPS2024 poster today on" Loki: Low-rank Keys for Efficient Sparse Attention" Paper: arxiv.org/abs/2406.02542 Code: github.com/hpcgroup/loki Loki is a sparse attention method that reduces the computational and memory costs of LLM inference! By exploiting the low-dimensional space of key vectors in self-attention, Loki achieves faster attention computation without compromising model performance.

Want to learn about our findings around the low-rank structure of attention keys, and how to use it to improve sparse attention? Come to our poster at #NeurIPS2024 tomorrow (Friday) at the East Exhibit Hall from 11-2 (Poster #2000)!

Excited to present “Be Like a Goldfish: Don’t Memorize!” led by @ahans30 at NeurIPS, East Building #4709. In this work, we propose a simple yet effective loss function to reduce verbatim memorization. Stop by the poster and grab some goldfish crackers while you’re at it!

🏆 Researchers from the @UofMaryland received the HPC Innovation Excellence Award from @Hyperion_HPC for their work on AxoNN—a scalable framework that’s pushing the limits of Large Language Model (LLM) training! 🚀 💫 bit.ly/3CsmjKM 🏆 bit.ly/3V9HFTw

We are honored to receive an HPC Innovation Excellence Award for work done by @siddharth_3773 on scaling AxoNN to large-scale GPU supercomputers. #SC24 #HPC #AI

We had a lot of fun presenting a tutorial on Distributed Deep Learning on GPGPU clusters at #SC24. We still have more t-shirts and stickers, come to one of our talks or to the UMD booth. #HPC # AI

A sneak peek into what PSSG is going to be up to this week at @Supercomputing in Atlanta. Come and talk to us, we'll be giving away t-shirts, buttons and lots of stickers! #SC24 #HPC #AI