Manan Dey

Thrilled our global data ecosystem audit was accepted to #ICLR2025! Empirically, we find: 1⃣ Soaring synthetic text data: ~10M tokens (pre-2018) to 100B+ (2024). 2⃣ YouTube is now 70%+ of speech/video data but could block third-party collection. 3⃣ <0.2% of data from Africa/South America. 1/

Testing LLMs' reasoning skills is tough—human evaluations are expensive, data contamination is common, and LLM judges can be biased. We propose StructTest, the first benchmark that checks how well LLMs follow complex instructions and create structured outputs. It uses a rule-based evaluator that’s easy to adapt to new tasks. StructTest is unbiased, cheap, hard to cheat and highly scalable. By testing structured outputs in areas like Summarization, Code, HTML, and Math—and evaluating 17 top LLMs—StructTest proves to be a challenge even for models like Deepseek-V3/R1 and GPT-4o. It’s also highly correlated with ChatBot Arena (~93%) and MMLU (>96%), making it a solid way to measure reasoning skills. Code & Data: github.com/SparkJiao/Stru… Paper🔗: arxiv.org/abs/2412.18011

✨New Report✨ Our data ecosystem audit across text, speech, and video (✏️,📢,📽️) finds: 📈 Rising reliance on web, synthetic, and YouTube data. 🛑 80%+ datasets carry hidden restrictions. 🌍 Relative representation in languages and creators has not improved for 10+ yrs. We're delighted to see this study covered by @Melissahei in the @techreview: bit.ly/49GEjxv 1/

@Muennighoff @Stanford @karpathy Congrats, Niklas!

Excited to start a PhD in AI @Stanford today🌲 Grateful for help from many people! In the LLM era, many rightly questioned me doing a PhD, but the points in @karpathy's great PhD Guide still hold i think. Regardless feel free to reach out if you have extra H100s😁 (or to collab!)

✨New Preprint ✨ How are shifting norms on the web impacting AI? We find: 📉 A rapid decline in the consenting data commons (the web) ⚖️ Differing access to data by company, due to crawling restrictions (e.g.🔻26% OpenAI, 🔻13% Anthropic) ⛔️ Robots.txt preference protocols are ineffective These precipitous changes will impact the availability and scaling laws for AI data, affecting coporate developers, but also non-profit and academic research. 🔗 dataprovenance.org/consent-in-cri… 1/

@KassnerNora @DeepMind Congratulations, Nora!

📢 Update 📢: Super excited to share that I've joined Google @DeepMind as a Research Scientist. I’ll be working as part of the Language team! 🥳

Introducing: 💫StarCoder StarCoder is a 15B LLM for code with 8k context and trained only on permissive data in 80+ programming languages. It can be prompted to reach 40% pass@1 on HumanEval and act as a Tech Assistant. Try it here: shorturl.at/cYZ06r Release thread🧵

Announcing a holiday gift: 🎅SantaCoder - a 1.1B multilingual LM for code that outperforms much larger open-source models on both left-to-right generation and infilling! Demo: hf.co/spaces/bigcode… Paper: hf.co/datasets/bigco… Attribution: hf.co/spaces/bigcode… A🧵:

✨Our work "How sensitive are translation systems to extra contexts? Mitigating gender bias in Neural Machine Translation models through relevant contexts" got accepted at the Findings on EMNLP 2022!✨ Joint work with @manandey and our awesome mentor @koustuvsinha 🎉

@koustuvsinha @MetaAI Congratulations, Koustuv!

🚨 Life update: today I start as a Research Scientist in NLP/Speech at Fundamental AI Research @MetaAI NYC 🎉 Good to be back! 😇

BLOOM is here. The largest open-access multilingual language model ever. Read more about it or get it at bigscience.huggingface.co/blog/bloom hf.co/bigscience/blo…

New paper alert! 🎉 Turns out you can reduce the gender biases your translation models just using relevant contexts, purely during inference! Checkout this cool work led by @evolvedeve and @manandey! arxiv.org/abs/2205.10762 [1/4]

🧐🕵️I am looking for the best possible open source tool to do memory profiling! I would like to know what part of my python code is causing these memory usage spikes that don't necessarily come from the Python interpreter. Looking forward to reading your recommendations! 🤗

We are releasing PromptSource, a toolkit for creating, sharing, and using natural language prompts. We used it to create the largest open-source collection of English prompts: 2,000 prompts for 170 datasets! 📄 arxiv.org/abs/2202.01279 💻 github.com/bigscience-wor…

Tokenization—the least interesting #NLProc topic? Hell no! We, members of the @BigScienceW tokenization group are proud to present: ✨Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP✨ arxiv.org/abs/2112.10508 What's in it? [1/10]

We’ve seen crazy interest in T0++ (pronounced "T Zero Plus Plus"), and almost 10’000 queries to the model since we announced it 3 days ago. Probably the most hilariously decisive prediction from the model (courtesy of @_philschmid): 1/N

First modeling paper out of BigScience is here! T0 shows zero-shot task generalization on English natural language prompts, outperforming GPT-3 on many tasks, while being 16x smaller! Model: huggingface.co/bigscience/T0pp Repo: github.com/bigscience-wor… Paper: arxiv.org/abs/2110.08207

@koustuvsinha Thanks a lot, Koustuv for being an amazing mentor and your guidance throughout! 😀

Do drop by and support the exciting research done by two amazing young scientists Shanya & Manan! Glad to be their mentor :) #NeurIPS2020 #NLProc

Hi #NeurIPS2020! I and @manandey will be presenting our poster on *Evaluating Gender Bias in NLI* at the Workshop on Dataset Curation and Security today (11th Dec) at 2:30 PM EST. Drop by if you're around :) cc: @koustuvsinha Gather Town (Poster 19) neurips.gather.town/app/A4yaHmXq3U…

I'm really happy to share that our work on evaluating gender bias in NLI systems has been accepted at #NeurIPS2020 Workshop on Dataset Curation and Security. Joint work with amazing collaborators @manandey and @koustuvsinha. More details coming soon!