Frederick Riemenschneider

How and when do multilingual LMs achieve cross-lingual generalization during pre-training? And why do later, supposedly more advanced checkpoints, lose some language identification abilities in the process? Our #ACL2025 paper investigates.

The world’s largest NLP conference with almost 2,000 papers presented, ACL 2025 just took place in Vienna! 🎓✨Here is a quick snapshot of the event via a short interview with one of the authors whose work caught my attention. 🎥 Watch: youtu.be/GBISWggsQOA #acl2025NLP #acl2025

ACL paper: aclanthology.org/2023.acl-long.… Models: #language-models" target="_blank" rel="nofollow noopener">github.com/Heidelberg-NLP… Read more: cl.uni-heidelberg.de/nlpgroup/news/… Morphological Analysis Demo: huggingface.co/spaces/bowphs/… Machine Translation Demo: huggingface.co/spaces/bowphs/ Best Thesis Award: gscl.org/en/activities/…

I am honored to receive the 2025 #GSCL Best Thesis Award at #KONVENS in Hildesheim for my Master’s thesis, which investigates multilinguality and develops language models for Ancient Greek and Latin. Thank you to my mentors and collaborators. I look forward to what comes next.

Looking at Bruegel's Tower of Babel in Vienna makes you wonder: How can multilingual language models overcome the language barriers? Find out tomorrow! 📍 Level 1 (ironic, right?), Room 1.15-1 🕐 2 PM #ACL2025NLP

How and when do multilingual LMs achieve cross-lingual generalization during pre-training? And why do later, supposedly more advanced checkpoints, lose some language identification abilities in the process? Our #ACL2025 paper investigates.

@crestonbrooks As for loss patterns: Unfortunately, detailed loss analyses are challenging with BLOOM since only 6 checkpoints were published. I'm currently investigating potential grokking phenomena in more controlled toy settings where we can track loss curves more comprehensively.

@crestonbrooks Thanks for the interesting questions! Regarding concept curriculum: We examined relatively "general" concepts and couldn't identify a clear pattern in which concepts get "translated" first. However, we did observe that different languages follow distinct patterns.

Read the full paper here: arxiv.org/pdf/2506.01629 Reach out if you have any questions or if you are attending ACL and want to say hi. 🙋

This phenomenon has a visible effect on text generation: In BLOOM-560m, activating 'earthquake' neurons derived from Spanish data at checkpoint 10,000 generates Spanish text. At checkpoint 400,000, the same method yields English text!

Read the full paper: aclanthology.org/2025.findings-… Work by @crestonbrooks, Johannes Haubold, Charlie Cowen-Breen, Jay White, Desmond DeVaul, me, Karthik Narasimhan, and Barbara Graziosi

Our work brings new computational methods to a field traditionally dominated by manual scholarship, potentially accelerating the discovery of textual errors that have remained hidden for centuries.

What did Aristotle actually write? We think we know, but reality is messy. As Ancient Greek texts traveled through history, they were copied and recopied countless times, accumulating subtle errors with each generation. Our new #NAACL2025 findings paper tackles this challenge.