Margaret Li @ Neurips ‘25

We nearly drove ourselves insane trying to reproduce scaling laws papers 📉 So of course we wrote a paper about it 😵‍💫 1/9

To appear at #ICLR2025 Thanks to the coauthors @snehaark @LukeZettlemoyer who descended into this madness with me Arxiv: arxiv.org/pdf/2502.18969 Code: github.com/hadasah/scalin… Checkpoints: huggingface.co/misfitting/mis… 9/9

(4) How are we optimizing the fit? (Loss? Optimizer? Initialization?) 📈: Do we need to perform a grid search for our initializations? How many points should we try? What happens if we initialize from a hypothesized law (e.g. chinchilla)? 8/9

We nearly drove ourselves insane trying to reproduce scaling laws papers 📉 So of course we wrote a paper about it 😵‍💫 1/9

@maxforbes omg ty, a good figure is the only kind of eye candy I want but on a serious note, H/T to @universeinanegg , I think he was the first one of us to bring up sankey diagrams

@margs_li These Sankey diagrams are 🔥

RLHF-aligned LMs excel at long-form generation, but how? We show how current models rely on anchor spans ⚓: strings that occur across many samples for the same prompt, forming an implicit outline, viz below.

@HJCH0 We're trying to minimize the influence of, e.g., vocab distribution on our long-term generation diversity metrics p=0.9 for RLHF models reflects common practice. For Base models, we truncate using the p which most closely matches the RLHF model statistics More info in the paper!

@margs_li Interesting! Quick question, why do you use different p values for the base model vs the RLHF model?

Of course, there’s much more in the paper than we could fit in a tweet thread! Paper: nlppapers.notion.site/Predicting-vs-… And thanks to all my amazing co-authors: @WeijiaShi2, @ArtidoroPagnoni, @PeterWestTM, and @universeinanegg!

We use span-alignment algorithms from Bioinformatics to quantify the implicit outline that RLHF’d models use. It turns out that even when using truncated sampling to compensate for differences in diversity, aligned models exhibit significantly more overlap than their base LMs.

Yes, the sneak peek is a joke, generated by @AnthropicAI 's Claude. The message is not though! We're super excited to discuss modular / sparse LLMs and how we train them ☺️

Sneak peek: "My fellow AI practitioners, I come to you today to spread the good news of embarrassingly parallel training of expert models. Too often we limit ourselves to single monolithic models. No more I say! The path to AI enlightenment is through specialization."

New paper alert!! ✨ Translate to Disambiguate: Zero-shot Multilingual Word Sense Disambiguation with Pretrained Language Models (PLMs) ✨ We evaluate how well PLMs translate words in context and then leverage this prompting setup to perform zero-shot WSD on 18 languages! 1/n

Sharing our project on 1) accelerating and 2) stabilizing training for large language-vision models 1) Towards accelerating training, we introduce SwitchBack, a linear layer for int8 quantized training which matches bfloat16 within 0.1 for CLIP ViT-Huge arxiv.org/abs/2304.13013

@suchenzang @jefrankle @MosaicML differentiating b/t this and just having, e.g., a code model, a c4 model, a papers model, etc., because I'm interested in how the different restrictions play with each other / want some more careful curation and comparable results on the same tasks

@suchenzang @jefrankle ❤️ @suchenzang for the callout, @jefrankle lowkey wanna do this, but would also love it from @MosaicML: this expt w/ each model trained on diff data subsets from a giant heap of everything w/ diff quality/code/lang filters, domain, dedup, etc -- super curious how they'd compare

Would anybody be interested in a couple dozen 1B, llama-style (waaaay past Chinchilla) language models trained on different data mixes? I don't know if this question has been well-studied before.