Chinnadhurai Sankar

Meta just dropped MobileLLM-Pro on Hugging Face a 1B foundational language model in the MobileLLM series, designed to deliver high-quality, efficient on-device inference across a wide range of general language modeling tasks two variants of the model: A pre-trained base model along with quantized checkpoints for CPU and accelerator inference, as well as an instruction tuned version, showing competitive performance against models in the this size range on tasks like tool calling, question answering, rewriting and summarization MobileLLM-Pro base achieves impressive pre-training results, outperforming Gemma 3 1B and Llama 3.2 1B by on average 5.7% and 7.9% respectively on reasoning, knowledge, and long-context retrieval benchmarks. This performance is achieved by pre-training on less than 2T fully open-source tokens

1/n Introducing CoSMoEs 🪐, a set of Compact Sparse Mixture of Experts at on-device scale 📱(arxiv.org/abs/2503.00245). In CoSMoEs, we explore how to enable Sparse Mixture of Experts for on-device inference, focusing on quality, memory, and latency. This work is done with my amazing co-authors @AkshatS07 @erniecyc @Chinnadhurai @Ahhegazy77 and @AdithyaSagarSci

Inference-time procedures (e.g. Best-of-N, CoT) have been instrumental to recent development of LLMs. The standard RLHF framework focuses only on improving the trained model. This creates a train/inference mismatch. Can we align our model to better suit a given inference-time procedure? We answer this affirmatively, check out the thread below.

Nice to see @ai4bharat showcase its work before @satyanadella! AI4Bharat is here to push the boundaries of open-source AI/ML/NLP for Indian languages. To the moon! 🚀🚀🚀 @srija_anand and @MiteshKhapra looking spiffy! :)

Introducing Content-Adaptive Tokenizer (CAT) 🐈! An image tokenizer that adapts token count based on image complexity, offering flexible 8x, 16x, or 32x compression! Unlike fixed-length tokenizers, CAT optimizes both representation efficiency and quality. Importantly, we use just captions (no pixels!) to guide tokenization, enabling adaptive representation for text-to-image generation. Big shout out to collaborators @AIatMeta: @violet_zct @liliyu_lili @LukeZettlemoyer @imisra_ @michiyasunaga @kushal_tirumala Paper: arxiv.org/abs/2501.03120 More details in 🧵

Want to know how 𝐫𝐞𝐰𝐚𝐫𝐝 𝐦𝐨𝐝𝐞𝐥 𝐠𝐞𝐧𝐞𝐫𝐚𝐥𝐢𝐳𝐚𝐛𝐢𝐥𝐢𝐭𝐲/𝐜𝐫𝐨𝐬𝐬-𝐥𝐢𝐧𝐠𝐮𝐚𝐥 𝐚𝐥𝐢𝐠𝐧𝐦𝐞𝐧𝐭 relates to 𝐚 𝐰𝐨𝐫𝐥𝐝-𝐟𝐚𝐦𝐨𝐮𝐬 𝐅𝐫𝐞𝐧𝐜𝐡 𝐟𝐨𝐨𝐝 𝐜𝐫𝐢𝐭𝐢𝐜? Listen to the 2min podcast generated by NotebookLM on @zhaofeng_wu's #EMNLP2024 paper!

A must-read paper on understanding LLM reasoning!

@arvind_io @AIatMeta @OpenAI Congrats 🎉

Excited to join @AIatMeta! The past 4.5 years at @OpenAI,working on embeddings, GPT-3 & 4,API and ChatGPT, have been career highlights. Now, I'm thrilled to work on the next generations of Llama and contribute to its impact on the developer ecosystem and billions of users!🚀 1/2

Chernoff bounds characterize large deviations of a RV (from its mean). On the other hand, they are outperformed by the simple Markov's inequality when considering small deviations of a *non-negative* RV. Can we get the best of both worlds? 🧵

Zyphra is proud to release Tree Attention, a fast inference method for extremely large sequence lengths • 8x faster inference speed vs. Ring Attention • 2x less peak memory • low data communication volumes Paper: arxiv.org/abs/2408.04093 Code: github.com/Zyphra/tree_at… A 🧵

🚨New paper!🚨 Self-Taught Evaluators - Llama 3-70B trained w/ synthetic data *only* - Iteratively finds better judgments in training - Best LLM-as-a-Judge model on RewardBench (88.3, 88.7 w/ maj vote) - Outperforms bigger models or human labels arxiv.org/abs/2408.02666 🧵(1/4)

🚨New paper!🚨 Meta-Rewarding LMs - LM is actor, judge & meta-judge - Learns to reward actions better by judging its own judgments (assigning *meta-rewards*) - Improves acting & judging over time without human labels ... beats Self-Rewarding LMs arxiv.org/abs/2407.19594 🧵(1/6)

@apsarathchandar @polymtl @ChandarLab Congratulations 🎉

I am happy to announce that I have been promoted to Associate Professor with tenure at @polymtl. This is an achievement for the entire @chandarlab! I want to thank my awesome students, without whom this would not have been possible! 1/n

Check out our demo at the Google booth, starting now!

@AlbalakAlon @ucsbNLP @ucsantabarbara @synth_labs Congrats Alon!!

With all of the excitement of the past few months, it's time for a career update: 🎉I graduated with my PhD from @ucsbNLP @ucsantabarbara 🥳and joined @synth_labs 🎊to drive open-science collaborations and push the boundaries on data strategies for synthetic data 👇I'm at #ICLR!

@ravi_iitm @iitmadras @rbc_dsai_iitm @cerai_iitm @IBSE_IITM @ai4bharat @DSAI_IITM Congrats! Looks very promising.

Starting on an exciting journey at the new Wadhwani School of Data Science and AI. The video summarizes the motivation for starting the school. Come join us in enriching the Data Science and AI ecosystem at @iitmadras. @rbc_dsai_iitm @cerai_iitm @IBSE_IITM @ai4bharat @DSAI_IITM

@peizNLP Congrats, Pei!

PhDone!!!! 👨‍🎓 08/2019-04/2024 What a journey 🥳🚞 I especially feel lucky to share this once-in-a-life-time moment with people I love ❤️ . And seeing my passion-driven research efforts being acknowledged by researchers I deeply admire 🌞!! Special thanks to my awesome committee members @xiangrenNLP @jay_mlr @toby_mintz @jieyuzhao11 Looking back when I made the decision to pursue a PhD 5 years ago. Just feel like hugging the me who strived through the years and giving him the biggest high five🫸🫷 Excitedly and confidently striding into the future journey 🏔️!

📣 Exciting news! @SliceXAI announces 𝗘𝗟𝗠 (family of Efficient Language Models), a new, decomposable #LLM architecture that delivers models with the best in class performance in terms of 𝑞𝑢𝑎𝑙𝑖𝑡𝑦, 𝑡ℎ𝑟𝑜𝑢𝑔ℎ𝑝𝑢𝑡 & 𝑚𝑒𝑚𝑜𝑟𝑦. 🔗 Blog 👉 medium.com/sujith-ravi/in…

Check out Zhaofeng's work from his internship with us! TL;DR A reward model trained on language S preference data could be used to align a language T LLM. This sometimes works even better than using a reward model trained on language T preference data.

Hi #NLProc!! If you are at #EMNLP2023 and are excited about the Novel Ideas in Learning-to-Learn through Interaction, join us in the exciting series of invited talks and a line up of presentations. In-person attendees can join us in **Leo** at the venue. cs.mcgill.ca/~pparth2/nilli…