Ritvik Gupta

Check out some really interesting results from our study! Despite the wave of FMs in specialized domains such as genomics, satellite imaging, and time series, most of these specialized FMs are often no better--and sometimes even worse--than simple supervised baselines.

Excited about our NeurIPS'25 tutorial Data Privacy, Memorization & Copyright in GenAI with Cooper (co-founder, GenLaw) & Joe (represents OpenAI, Stability in all US copyright litigations) We bring together ML researchers, with those who understand its legal implications. Pls RT

Happy to see a time series workshop @ NeurIPS 2025 motivated in part by our search for BERT moments in specialized foundation models (🧵here: x.com/khodakmoments/…)

Excited to present our work at #ICLR2025! Join us on Thursday, April 23 (Afternoon Session) to chat about the current state of foundation models in specialised domains 🧬📈 Hall3+2B #40 Joint work with @ZongzheX2001 @WenduoC @musishen @JunhongShen1 @atalwalkar @khodakmoments

I have some news to share! @datadoghq is forming a new AI research lab, and I'm excited to announce that I've joined as Chief Scientist to lead this effort. Datadog has a great work culture, lots of data and compute, and is committed to open science and open sourcing. Our team is working on ambitious research areas grounded in real-world challenges in cloud observability and security, with three current areas of focus: 1. Observability Foundation Models for forecasting, anomaly detection, and multi-modal telemetry analysis (logs, metrics, traces, etc.). 2. Site Reliability Engineering (SRE) Agents to detect, diagnose, and resolve production incidents. 3. Production Code Repair Agents that leverage code, logs, and runtime data to identify and fix performance issues. On a personal note, I'm thrilled to work with Oli and Alexis again (we worked together in the early aughts before they co-founded Datadog). I’m also excited that Datadog, as part of its expansion in AI, is partnering with CMU (note: I will continue to work part-time at CMU and maintain my research activities). Datadog is actively hiring out of our NYC office!

🧵1/ Introducing ScribeAgent 🤖! Using extensive 𝗿𝗲𝗮𝗹-𝘄𝗼𝗿𝗹𝗱 𝘄𝗲𝗯 𝘄𝗼𝗿𝗸𝗳𝗹𝗼𝘄 𝗱𝗮𝘁𝗮, we at @mldcmu and @ScribeHow have adapted 𝗴𝗲𝗻𝗲𝗿𝗮𝗹-𝗽𝘂𝗿𝗽𝗼𝘀𝗲 𝗼𝗽𝗲𝗻-𝘀𝗼𝘂𝗿𝗰𝗲 𝗟𝗟𝗠𝘀 into 𝘀𝗽𝗲𝗰𝗶𝗮𝗹𝗶𝘇𝗲𝗱 𝘄𝗲𝗯 𝗮𝗴𝗲𝗻𝘁𝘀, outperforming traditional agents that rely on proprietary models like GPT-4 and o1-preview. Learn more about our breakthroughs and findings👇

wake up babe, new AI term just dropped

Turns out that strong baselines still matter in 2024! While there have been a TON of specialized FMs introduced recently, we show that leading FMs in genomics, satellite imaging, and time series are generally no better (and often worse than) much simpler supervised baselines!

Which model is best for coding? @CopilotArena leaderboard is out! Our code completions leaderboard contains data collected over the last month, with >100K completions served and >10K votes! Let’s discuss our findings so far🧵

Specialized Foundation Models Struggle to Beat Traditional Supervised Learning Baselines 1. This study provides a rigorous comparison between specialized foundation models (FMs) and traditional supervised learning approaches, revealing that well-tuned CNNs and simple autoregressive models often outperform large-scale FMs across genomics, satellite imaging, and time series domains. 2. Through extensive benchmarking on over 30 tasks, including the Nucleotide Transformer benchmark and satellite imagery datasets, the researchers found that traditional models like Wide ResNet and UNet consistently match or surpass the performance of specialized FMs, despite the FMs’ extensive pretraining on massive datasets. 3. Key finding: In genomics tasks, DASHA, an optimized CNN workflow with architecture search, outperformed FMs on histone modification tasks, challenging the assumption that transformer-based FMs are universally superior. 4. The study introduces an open-source automated tuning workflow, DASHA, which efficiently combines neural architecture search and hyperparameter optimization, demonstrating that resource-intensive FMs are not always necessary for peak performance in specialized domains. 5. Results also show that a simple autoregressive model, with minimal computational demands, matched or outperformed large FMs on time series forecasting tasks. This underscores the importance of strong, well-tuned baselines in FM evaluations. @atalwalkar @JunhongShen1 @WenduoC @ritvikgupta199 📜Paper: openreview.net/pdf?id=wgBYYUj… #FoundationModels #MachineLearning #Bioinformatics #CNN #TimeSeries #SatelliteImaging

📉Time Series FMs hold promise, but still struggle to outperform well-tuned statistical baselines👇 We need better evals, and more research to unlock their full potential🧠. Let's makes today's TSFMs the dumbest we'll every have!📈

SCiPS-QA now live at - huggingface.co/datasets/zorps…

(1/6) Recommender systems shape our digital experiences, filtering much of what we see online. But how should we measure their influence? We provide a unified causal framework to think through this question and develop metrics to audit recommender systems. arxiv.org/abs/2409.13210

Want to scale RL with your shiny new GPU? 🚀 In our ICML24 Oral we find that RL algorithms hit a barrier when data is scaled up. Our new algorithm, SAPG, proposes a simple fix. It scales to 25k envs and solves hard tasks where PPO makes no progress. sapg-rl.github.io 1/n

@dutta_oshin Explore our paper for an in-depth understanding of TVA-Prune and its impact on deploying LLMs on resource-constrained devices. Paper: openreview.net/forum?id=cqhAz… Code available on request. Joint work with @dutta_oshin @lawraga #AI #MachineLearning #DeepLearning #LLMs #ICML2024 [8/8]

🧵 Excited to introduce our latest paper on Token Dependency-Aware Variational Information Bottleneck-based Pruning for LLMs (TVA-Prune), presented at the Efficient Systems for Foundational Models II Workshop at ICML 2024. 👇[1/n]

@dutta_oshin By preserving token dependencies across model layers, TVA-Prune addresses a critical challenge in structured pruning, ensuring robust performance and efficient inference. 🔗🔬[7/n]

@dutta_oshin TVA-Prune adheres to user-defined sparsity criteria, making it a versatile and adaptable solution for various deployment scenarios, especially on resource-constrained devices. 🖥️✨[6/n]

@dutta_oshin We demonstrate the effectiveness of TVA-Prune on pre-trained LLMs, including variants of LLaMA-7B (also LLaMA 3 8B) and Mistral-7B. The results show superior performance and higher inference speedups compared to existing pruning methods. 📊🔍[5/n]

@dutta_oshin Our method introduces a post-pruning adaptation step that adjusts dimensions to match the block sizes of GPU tensor cores. This step ensures optimal parallelism during inference, leading to significant speedups. ⚙️⚡[4/n]

@dutta_oshin TVA-Prune efficiently removes redundant heads, intermediate dimensions, and global token representations, all on a single GPU. This structured approach ensures that the essential parts of the model are preserved, enhancing both efficiency and performance. 🚀🔧[3/n]