Instruction Workshop, NeurIPS 2023

197 posts

Instruction Workshop, NeurIPS 2023 banner
Instruction Workshop, NeurIPS 2023

Instruction Workshop, NeurIPS 2023

@itif_workshop

The official account of the 1st Workshop on Instruction Tuning and Instruction Following (ITIF), colocated with NeurIPS, in December 2023.

New Orleans Joined Ağustos 2023
29 Following247 Followers
Instruction Workshop, NeurIPS 2023 retweeted
Shayne Longpre
Shayne Longpre@ShayneRedford·
Excited to see our Economies of Open Intelligence work highlighted in Chp. 1 of @StanfordHAI's #AIIndex2026! We release tons of info on the open model ecosystem, using 🤗 HF data. Thank you @russellwald and team!
Shayne Longpre tweet media
English
1
3
12
645
Instruction Workshop, NeurIPS 2023 retweeted
Shayne Longpre
Shayne Longpre@ShayneRedford·
We just released the Google Research Blog for ATLAS 🗺️! Check out for: 1) Multilingual scaling and data mixing laws for 100s of languages 2) "Curse of Multilinguality" modeling 3) Cross-lingual transfer scores 🌎 research.google/blog/atlas-pra…
Shayne Longpre tweet media
English
1
5
17
656
Instruction Workshop, NeurIPS 2023 retweeted
Shayne Longpre
Shayne Longpre@ShayneRedford·
The open AI race is becoming more competitive, and more international. Check out @Melissahei's excellent coverage of our new study in the @FT! ft.com/content/931c82…
English
4
8
15
9.5K
Instruction Workshop, NeurIPS 2023 retweeted
Shayne Longpre
Shayne Longpre@ShayneRedford·
Who is winning the open AI race? Our new study "Economies of Open Intelligence" maps 2.2B @huggingface downloads across 851k models (2020→2025). 1) Power is rebalancing (US big tech ↓; China + community ↑) 2) Models got big & efficient (MoE, quant, multimodal surge) 3) Intermediaries now matter (adapters/quantizers steer usage) 4) Transparency is slipping /🧵
Shayne Longpre tweet mediaShayne Longpre tweet media
English
9
26
92
30.1K
Instruction Workshop, NeurIPS 2023 retweeted
Shayne Longpre
Shayne Longpre@ShayneRedford·
TL;DR: Seven lessons on collaboration in AI research (slides below). Presented last week as a keynote at @Cohere_Labs Connect 2025. Last week I gave a keynote on Collaboration in AI Research at Cohere Labs Connect 2025, alongside Cohere co-founder @1vnzh, Head of Cohere Labs @mziizm, and Chief AI Officer @jpineau1. I’m grateful for the invitation—and for the many collaborators whose work shaped this talk. Slides: a practical roadmap for starting an interdisciplinary research collaboration. In brief: 1. Ideate clearly. Share a one-pager with potential collaborators, advisors, and critics. 2. Run a pre-mortem. Find your “reviewer #2” early and invite their critiques to sharpen direction. 3. Recruit widely. Don’t let geography or institution limit you; add complementary skills, senior advisors, and junior contributors. 4. Structure incentives. Be explicit about authorship, credit, visibility, and impact; let people own their piece. 5. Communicate early and often. Relationships may be the most durable outcome. Align early on authorship contingencies, timeline, cadence, and responsibilities. 6. Document decisions. Keep a living record of links, choices (and why), and who did what—details slip in larger teams. 7. Focus on infrastructure & visuals. Beyond relationships, open-source tools and clear figures/tables travel the farthest and longest. I also folded in lessons from advisors, mentors, and collaborators including @alex_pentland , @sarahookr, @percyliang, @BlancheMinerva, @PeterHndrsn, @RishiBommasani, @sayashk, @AnkaReuel, and @RobertMahari and @TobinSouth.
Shayne Longpre tweet mediaShayne Longpre tweet mediaShayne Longpre tweet media
English
3
9
33
3.2K
Instruction Workshop, NeurIPS 2023 retweeted
Sneha Kudugunta
Sneha Kudugunta@snehaark·
A new, tractable approach to study scaling laws for larger data mixtures compared to prior art. We achieve significantly better fit ($R^2=0.98$) on multilingual data mixtures with ~50 languages.
Sneha Kudugunta tweet media
English
1
7
8
2.7K
Instruction Workshop, NeurIPS 2023 retweeted
I-Hung Hsu
I-Hung Hsu@IHung_Hsu·
Have you ever wondered how to build target-language LMs most efficiently? 🤨 Will finetuning from other multilingual LMs help? No curse of multilinguality?? See the snapshot below for the answer. What about transferring data to help? What languages help? See our new paper! 👇
I-Hung Hsu tweet media
Shayne Longpre@ShayneRedford

📢Thrilled to introduce ATLAS 🗺️: scaling laws beyond English, for pretraining, finetuning, and the curse of multilinguality. The largest public, multilingual scaling study to-date—we ran 774 exps (10M-8B params, 400+ languages) to answer: 🌍Are scaling laws different by language? 🧙‍♂️Can we model the curse of multilinguality? ⚖️Pretrain from scratch or finetune from multilingual checkpoint? 🔀Cross-lingual transfer scores for 1444 lang pairs? 1/🧵

English
1
7
11
1.5K
Instruction Workshop, NeurIPS 2023 retweeted
Florian Brand
Florian Brand@xeophon·
@ShayneRedford What an insane banger and amazing work
English
0
1
6
412
Instruction Workshop, NeurIPS 2023 retweeted
Marzieh Fadaee
Marzieh Fadaee@mziizm·
Many gems in this paper for those who want to systematically scale multilingual learning with fewer arbitrary decisions. Great job Shayne et al.
Shayne Longpre@ShayneRedford

📢Thrilled to introduce ATLAS 🗺️: scaling laws beyond English, for pretraining, finetuning, and the curse of multilinguality. The largest public, multilingual scaling study to-date—we ran 774 exps (10M-8B params, 400+ languages) to answer: 🌍Are scaling laws different by language? 🧙‍♂️Can we model the curse of multilinguality? ⚖️Pretrain from scratch or finetune from multilingual checkpoint? 🔀Cross-lingual transfer scores for 1444 lang pairs? 1/🧵

English
0
2
42
4.4K
Instruction Workshop, NeurIPS 2023 retweeted
Niklas Muennighoff
Niklas Muennighoff@Muennighoff·
To scale data-constrained LLMs, repeating & denoising objectives can help. Another solution: Add multilingual data. But what languages help & how much? Below a snapshot for this at 2B scale, e.g., Chinese can hurt English while Indonesian may help.
Niklas Muennighoff tweet media
Shayne Longpre@ShayneRedford

📢Thrilled to introduce ATLAS 🗺️: scaling laws beyond English, for pretraining, finetuning, and the curse of multilinguality. The largest public, multilingual scaling study to-date—we ran 774 exps (10M-8B params, 400+ languages) to answer: 🌍Are scaling laws different by language? 🧙‍♂️Can we model the curse of multilinguality? ⚖️Pretrain from scratch or finetune from multilingual checkpoint? 🔀Cross-lingual transfer scores for 1444 lang pairs? 1/🧵

English
1
5
21
2.4K
Instruction Workshop, NeurIPS 2023 retweeted
Cody Blakeney
Cody Blakeney@code_star·
Exciting to see someone do a study so elaborately and at such scale. I really like this grid looking at transfer synergy. Also this result is cool. Intuitive that model capacity helps, but great to see an empirical result.
Cody Blakeney tweet media
Shayne Longpre@ShayneRedford

Q2: Which languages actually help each other during training? And how much? 🌟Answer: We measure this empirically. We built a 38×38 transfer matrix, or 1,444 language pairs—the largest such resource to date. We highlight the top 5 most beneficial source languages for each target language. 4/

English
1
4
20
1.8K
Instruction Workshop, NeurIPS 2023 retweeted
Shayne Longpre
Shayne Longpre@ShayneRedford·
📢Thrilled to introduce ATLAS 🗺️: scaling laws beyond English, for pretraining, finetuning, and the curse of multilinguality. The largest public, multilingual scaling study to-date—we ran 774 exps (10M-8B params, 400+ languages) to answer: 🌍Are scaling laws different by language? 🧙‍♂️Can we model the curse of multilinguality? ⚖️Pretrain from scratch or finetune from multilingual checkpoint? 🔀Cross-lingual transfer scores for 1444 lang pairs? 1/🧵
Shayne Longpre tweet mediaShayne Longpre tweet media
English
7
42
155
24.5K
Instruction Workshop, NeurIPS 2023 retweeted
Shayne Longpre
Shayne Longpre@ShayneRedford·
Copyrighted 🚧, private 🛑, and sensitive ☢️ data remain major challenges for AI. FlexOlmo introduces an architectural mechanism to flexibly opt-in/opt-out segments of data in the training weights, **at inference time**. (Prior common solutions were to filter your data once before training, and hope you got it right.) Highly recommend checking out the incredible work, led by @WeijiaShi2, @AkshitaB93, @notkevinfarhat, @sewon__min 1/
Shayne Longpre tweet media
English
1
9
39
4.1K
Instruction Workshop, NeurIPS 2023 retweeted
Shayne Longpre
Shayne Longpre@ShayneRedford·
Come say hello at ICLR! 👋 Here's where you can find me: Friday: Data-centric AI Social! lu.ma/rmyoy2vw Saturday: Multimodal Data Provenance poster (3 pm, Hall 2B #494) Sunday: MLDPR Workshop (3 pm) [mldpr2025.com]—I'll talk about challenges to AI data collection and access Monday: Building Trust in LLMs workshop (10:30 am) [building-trust-in-llms.github.io/iclr-workshop/]—I'll talk about third-party AI evaluation and flaw reporting Monday again: Sci-FM Workshop (2 pm) [open-foundation-model.github.io]—I'll talk about AI data access again Come say hi! Attaching the works below.
Shayne Longpre tweet media
English
1
6
24
2.1K
Instruction Workshop, NeurIPS 2023 retweeted
Shayne Longpre
Shayne Longpre@ShayneRedford·
Thrilled our global data ecosystem audit was accepted to #ICLR2025! Empirically, we find: 1⃣ Soaring synthetic text data: ~10M tokens (pre-2018) to 100B+ (2024). 2⃣ YouTube is now 70%+ of speech/video data but could block third-party collection. 3⃣ <0.2% of data from Africa/South America. 1/
Shayne Longpre tweet media
English
4
21
75
15.3K
Instruction Workshop, NeurIPS 2023 retweeted
Shayne Longpre
Shayne Longpre@ShayneRedford·
What are 3 concrete steps that can improve AI safety in 2025? 🤖⚠️ Our new paper, “In House Evaluation is Not Enough” has 3 calls-to-action to empower independent evaluators: 1️⃣ Standardized AI flaw reports 2️⃣ AI flaw disclosure programs + safe harbors. 3️⃣ A coordination center for transferable AI flaws affecting many systems. 1/🧵
Shayne Longpre tweet media
English
5
29
59
12K
Instruction Workshop, NeurIPS 2023 retweeted
Shayne Longpre
Shayne Longpre@ShayneRedford·
I compiled a list of resources for understanding AI copyright challenges (US-centric). 📚 ➡️ why is copyright an issue? ➡️ what is fair use? ➡️ why are memorization and generation important? ➡️ how does it impact the AI data supply / web crawling? 🧵
Shayne Longpre tweet media
English
2
7
18
1.5K
Instruction Workshop, NeurIPS 2023 retweeted
Shayne Longpre
Shayne Longpre@ShayneRedford·
I wrote a spicy piece on "AI crawler wars"🐞 in @MIT @techreview (my first op-ed)! While we’re busy watching copyright lawsuits & the EU AI Act, there’s a quieter battle over data access that affects websites, everyday users, and the open web. 🔗 technologyreview.com/2025/02/11/111… 1/
Shayne Longpre tweet media
English
3
9
36
4.4K

Discover

@StanfordHAI @russellwald @Melissahei @FT @huggingface @Cohere_Labs @1vnzh @mziizm