Olivia Simin Fan
97 posts
@Olivia61368522
🎓Ph.D.@EPFL_en-MLO|| https://t.co/QGwaUTkuyY.@UMich. || https://t.co/QGwaUTkuyY.@sjtu1896. ML&LLM research🧐 NOT a physicist.
Heading to Rio for ICLR? 🇧🇷🌴 Excited to share our ICLR 2026 Oral paper 🎉 Communication is becoming a real bottleneck in large scale training. As clusters scale up, a natural question is: If communication is limited/expensive, how should we allocate the communication "budget" over training ? Paper: openreview.net/forum?id=zrFnw…
Long in the making, finally released: Apertus-8B and Apertus-70B, trained on 15T tokens of open data from over 1800 languages. Unique opportunity in academia to work on and train LLMs across the full-stack. We managed to pull off a pretraining run with some fun innovations, ...
Super excited to share our work on data influence cascade in decentralized learning, just accepted by #ICLR2025! 🎉 Data quality is crucial for LM training. But can we quantify the importance of data in a fully decentralized learning system? 🤔 Here’s a surprising insight: the same data can contribute very differently depending on where it resides in the communication graph! 👉 openreview.net/pdf?id=2TIYkqi… 🎯 Key Finding: We show that in decentralized training, data contribution doesn't stay local—it spreads across the communication graph like "ripples in water".💦 We define this phenomenon as Influence Cascade 🌊. This cascade means your data's influence can extend far beyond your local node. But how do we measure it? Through theoretical analysis, we derive a mathematical form for data contribution in fully decentralized learning: Surprising, it depends on: 1️⃣ Data itself (e.g., quality and diversity) 2️⃣ The curvature of loss landscape 3️⃣ Communication graph structure Beyond theoretical contributions, DICE (our approach) also holds great potential to address critical challenges, such as: 1️⃣Identifying new suitable collaborators 🤝 2️⃣Detecting free-riders 🚶♂️ 3️⃣Mitigating malicious behaviors 🔒 Joint work with Fengxiang @FengxiangFHe, Wenhao, and my advisor Can Wang. Huge thanks to my amazing collaborators! 🙌 📄 Interested in the details? Check out our paper! 👉 openreview.net/pdf?id=2TIYkqi… We hope this work may contribute to the practical realization of scalable, autonomous, and reciprocal decentralized learning ecosystems!✨
I saw a slide circulating on social media last night while working on a deadline. I didn’t comment immediately because I wanted to understand the full context before speaking. After learning more, I feel compelled to address what I witnessed during an invited talk at NeurIPS 2024 by Professor Rosalind Picard. I deeply respect Professor Picard’s scholarship and contributions to the field. However, her comments during the talk reflected a deeply troubling and racist view of Chinese scholars. This was not just inappropriate but also profoundly disheartening. First, it was entirely unnecessary to mention the student’s nationality when discussing an incident of cheating. The point about academic integrity could have been made without emphasizing nationality. Yet, Professor Picard chose to highlight it. This choice perpetuates harmful stereotypes about Chinese scholars and reflects a broader bias against Asians, often rooted in the assumption that we “work hard, avoid conflict, and don’t push back.” This needs to change. Asians, like everyone else, have the right to speak out and demand accountability when racism occurs. We will ensure that being racist against Asians has consequences, including here, Professor Picard. What made this incident worse was how it unfolded during the Q&A session. A Chinese attendee asked a professional and thoughtfully articulated question. She began by thanking Professor Picard for her talk and posed this question: Are you calling out the student’s nationality because you find most Chinese scholars honest, and the fact that the cheating student was Chinese is rare? Is that why you emphasized nationality? This was a generous and high-EQ question, offering Professor Picard an opportunity to reconsider or clarify her comments. Unfortunately, she doubled down instead. Professor Picard reinforced her remarks by quoting the student’s excuse —that ethics wasn’t taught in their school—and generalized this as a broader issue with Chinese education. This statement is both factually incorrect and deeply offensive. There are glaring logical flaws in this argument: 1.If the student cheated, why would their excuse about ethics education be taken at face value? A serious scholar would investigate the claim before making it a central part of their argument. 2.Even if the student’s school didn’t teach ethics (which is false for schools in China), other sources like family and community often instill strong ethical values. Ignoring this nuance is careless and reinforces stereotypes. What is most heartbreaking is that Professor Picard couldn’t even acknowledge something as simple as: “Most Chinese scholars are honest and upright.” Instead, she focused on the singular exception and added, “Of course, with this one exception in this case” in her response. I regret that this happened at NeurIPS. I regret that this happened in my research community—a place I have cherished and contributed to for over 14 years. I regret that this happened at MIT, an institution of excellence and aspiration for many Chinese scholars. Racism has no place in academia, and incidents like this tarnish the principles of inclusion and respect that we, as a global research community, should uphold. I hope NeurIPS and the broader academic community take this as a wake-up call to address the biases and systemic issues that enable such comments to go unchallenged. We must do better. @MIT_CSAIL @NeurIPSConf
Certain words like "showcasing", "pivotal", "realm" see large increases in frequency after #ChatGPT launch. These words are also more likely to be used by LLMs in scientific writings than by humans. 2/n
💥Hello world! The CHATS lab just got 3x smarter! 💥 Say Hi to my first-ever batch of PhD students🥳🥳🥳 @simon_ycl @jcz12856876 @JiayiZhang0427 – these brave souls signed up for endless research🧐, questionable coffee☕️, nice and cold days in Boston🌇
Could LLMs tune its data preference to generalize better with various targets? Introduce DoGE — A lightweight, effective and robust framework to reweight training data sources (domains) to enhance generalization. Arxiv: arxiv.org/abs/2310.15393 🧵1/N
Could LLMs tune its data preference to generalize better with various targets? Introduce DoGE — A lightweight, effective and robust framework to reweight training data sources (domains) to enhance generalization. Arxiv: arxiv.org/abs/2310.15393 🧵1/N
