Nitin Kedia

Drop 13/14: The 30B and 105B models, benchmarks, and HF links will all come. But today it is a drop about people. About how our team of just 15 folks gave it their all to do what many doubted as not doable - ie train usefully large, globally competitive models from scratch in India. This team of 15 has now firmly launched @sarvam into its second innings. Yes, we can! @_mohit_singla @anand_404 @kediaharshit9 @AashaySachdeva @sumanthd17 @ArpitDwivedi100 @HarveenChadha @rkal4 @sushil_khyalia @ManavSinghal157 @sohampetkar missing in the pictuere - @selfawareatom @AnnaUpreti Anand @MeghMakwan33973 Utkarsh

TokenWeave – Efficient Compute-Communication Overlap for Distributed LLM Inference. Why? Even with highspeed NVLink on H100 DGX, communication overhead for distributed LLM inference can be > 20 %! Can we recover this overhead? (1/10)

1/7 🧵 MoEs: A tale of expectation vs reality Marketing: "Only compute the expert parameters you need!" Reality: Batch 16 requests → ALL experts activate At serving time (vLLM/TGI), arithmetic intensity: AI ≈ (num_tokens * top_k) / total_experts In simpler terms: Your decode arithmetic intensity scales inversely with expert count 🤔 #MoE #LLMs #ChatGPT #Claude #vllm #AI #ML

@Google has silently but surely developed an edge over @OpenAI. Long context processing seems to be the key to Google's AI strategy. NotebookLM is a prime example of what long context processing can unlock. In our latest paper, we talk about how systems can be built to support multi-million context length matching Google's capabilities. In case you missed the paper, here is the NotebookLM generated podcast! Podcast: notebooklm.google.com/notebook/764f5… Arxiv: arxiv.org/abs/2409.17264

Are you getting the performance you paid for from your LLM provider? Benchmark it using Metron. It is one our biggest learning while working on LLM Inference for the last year at @MSFTResearch and @gtcomputing when we shipped Chunked Prefill at OSDI'24 and Vidur @MLSysConf.

Excited to present Sarathi-Serve at OSDI'24. Learn how chunked prefills make your llm chat buttery. @usenix @ChatGPTapp @vllm_project

[LG] Vidur: A Large-Scale Simulation Framework For LLM Inference arxiv.org/abs/2405.05465 - This paper presents Vidur, a high fidelity and easily extensible simulator for large language model (LLM) inference, along with a benchmark and search suite. - Vidur models the performance of LLM operators using a combination of experimental profiling and predictive modeling, and evaluates end-to-end inference performance for different workloads. - It estimates metrics like latency, throughput, model FLOPs utilization, memory utilization, etc. with high accuracy. - Vidur addresses challenges unique to simulating LLM inference like finer time granularity, varying iteration times, and cascading errors. - It uses insights like architectural uniformity of LLMs, operator triaging, and automated profiling for parallelism strategies to achieve fidelity. - Vidur-Search uses Vidur to automatically identify optimal cost-effective deployment configurations meeting performance constraints.

Made with🩷from @gtcomputing and @MSFTResearch India AI Infra Team. Folks behind: @agrawalamey12, @jayashree2912, Ashish, @nipunkw, Bhargav, @ramaramjee and @alsched.

Vidur is a tool 🛠️. Use it how you want! We need your contributions to add more devices (@AMD GPUs anyone) and more models and architectures (go MoE @mistalai). Code: github.com/microsoft/vidur (n/n)

We at @MSFTResearch and @GeorgiaTech believe that running LLM's shouldn't be so expensive 💵 So, we built a tool 🛠️ that will enable you to run it cheaper, make it cheaper. Introducing Vidur👳🏽, the first LLM Inference System simulator. #mlsys #vllm #llm #llama #gpt

1/ LLM inference systems are like high-performance engines ⚙️—complex, powerful, and full of intricate settings. Efficiently deploying them to maximize GPU performance is a challenge typically tackled by experts at orgs like @OpenAI and @AIatMeta 🚀. 🧵