Bryan Li

Do LLMs' reasoning abilities come from training on code🤔? Many think so, but how does this hold across languages🌐? We study the interplay of code and reasoning in our recent work (#acl2024). 📃arxiv.org/abs/2403.02567 🗃️github.com/amazon-science… 1/6 🧵

@jaseweston @YungSungChuang @yangli625 @dongwang218 @intrepidvagrant @LukeZettlemoyer @endernewton @sainingxie @scottyih @ShangwenLi1 @Hu_Hsu Super impactful work, look forward to trying out the embeddings! The finding that multilinguality is effective across all langs + English aligns with our findings in our ACL24 paper on complex reasoning: x.com/bryanlics/stat…

🌿Introducing MetaCLIP 2 🌿 📝: arxiv.org/abs/2507.22062 code, model: github.com/facebookresear… After four years of advancements in English-centric CLIP development, MetaCLIP 2 is now taking the next step: scaling CLIP to worldwide data. The effort addresses long-standing challenges: (1) large-scale non-English data curation pipelines are largely undeveloped, and (2) the curse of multilinguality, where English performance often degrades in multilingual CLIP compared to English-only CLIP. With a complete recipe for worldwide CLIP—spanning data curation, modeling, and training—we show that English and non-English worlds can mutually benefit and elevate each other, achieving SoTA multilingual performance. Join the Meta booth at #ACL2025 to learn more. (1/3)

I'm in Vienna this week to present our poster on the robustness of RAG systems to multilingual contexts at #ACL2025NLP! 🗓️ Poster Session | Wednesday, July 30, 16:00 - 17:30 📍 Hall 4/5 @aclmeeting

In a world of geopolitical conflicts, how can AI help us navigate? Our #ACL2025-F work studies RAG robustness across 49 languages. TL;DR: 📈 boost robustness w/ multilingual RAG, 🤔 take care w/ low-resource citations 📜arxiv.org/abs/2410.01171 🤗huggingface.co/datasets/borde… 1/4 🧵

@mingyang2666 @aclmeeting Super cool work! I'll be presenting a poster, on the other end of cross-lingual inconsistency from RAG: arxiv.org/abs/2410.01171 Hope to chat at ACL!

I'll be at @aclmeeting next week to present this paper! 🗓️ Poster Session | Wednesday, July 30, 11:00–12:30 📍 Hall 4/5 Happy to grab a coffee and chat! ☕

This is the final paper of my PhD! Thanks to my many @upennnlp collaborators: @samarhdr, Chris, and the 7 wonderful students who I was fortunate to mentor. Please look out for our poster at ACL 2025 in Vienna. 4/4 🧵

We study cross-lingual robustness over 4 LLMs and 2 IR models. We find A) multilingual RAG performs best; B) LLM’s citations varies widely across langs. Our further experiments investigate aspects of cross-lingual RAG from IR to LLM explanations. 3/4 🧵

@yong_zhengxin Really thorough work on multilingual reasoning! A quick self-promotion of our xSTREET dataset arxiv.org/abs/2403.02567… (ACL 2024), which has annotations for the intermediate reasoning steps for STEM problems.

📣 New paper! We observe that reasoning language models finetuned only on English data are capable of zero-shot cross-lingual reasoning through a "quote-and-think" pattern. However, this does not mean they reason the same way across all languages or in new domains. [1/N]

@mykocyigit Congrats! Data contamination is v relevant these days with bigger and bigger training corpora

Our work got accepted to ICML! Looking forward to sharing more about this project with everyone this summer!

🚀 How well can LLMs know you and personalize your response? Turns out, not so much! Introducing the PersonaMem Benchmark -- 👩🏻‍💻Evaluate LLM's ability to understand evolving persona from 180+ multi-session user-chatbot conversation history 🎯Latest models (GPT-4.1, GPT-4.5, o4-mini, Llama-4, Gemini 2.0, Deepseek-R1, Claude-3.7) all struggle in personalization! 🎨7 personalization skills tested in 15 scenarios 🌟Realistic long-context evaluation up to 1M tokens 👇 Check out what we discovered… (1/6)

TL;DR - translation pairs > bilingual terminologies, generation especially boosts translations for small LLMs Our ablations highlight the need for more challenging domain-adapted MT datasets with modern LLMs. Thanks to collaborators Jiaming, @ebriakou & @ColinCherry!

Externally retrieving knowledge empowers LLMs for domain-adapted MT ⚖️🩺. But how is knowledge best represented, and how viable is generating it from an LLM itself? Our @GoogleAI paper investigates these questions through a careful experimental setup 📜. arxiv.org/abs/2503.05010

@_reachsumit Great work! Nice to see a pipeline approach to multilingual QA generation in 2025. Reminds me of our EMNLP 2023 work arxiv.org/abs/2304.12206 (my last paper without LLMs 😅)

Few-Shot Multilingual Open-Domain QA from 5 Examples Leverages large-scale self-supervised pre-training using WikiData followed by fine-tuning on LLM-generated synthetic data from just 5 examples per language, outperforming existing few-shot baselines. 📝arxiv.org/abs/2502.19722

We share Code-Guided Synthetic Data Generation: using LLM-generated code to create multimodal datasets for text-rich images, such as charts📊, documents📄, etc., to enhance Vision-Language Models. Website: yueyang1996.github.io/cosyn/ Dataset: huggingface.co/datasets/allen… Paper: arxiv.org/pdf/2502.14846 Code: github.com/allenai/pixmo-…

🚨 LLMs must grasp implied language to reason about emotions, social cues, etc. Our @GoogleDeepMind paper presents the Implied NLI dataset. Targeting social norms 🌎 and conversational dynamics 💬, we enhance LLM understanding of real-world implication! arxiv.org/abs/2501.07719

@bryanlimy 繁体字真的next level 🤯

We'll be presenting this at the NLP for Wikipedia workshop @emnlpmeeting. This is ongoing work, and we'd love to hear feedback from the community! A shout-out to my collaborators Fiona and Adwait for their amazing first paper efforts, @samarhdr, and Chris. 4/4 🧵

Using cross-lingually aligned queries, we analyze responses in a RAG setting. Responses can be "flipped" by varying passages' linguistic composition. We thus find these systems to be far from cross-lingually robust, as certain viewpoints can be amplified over others. 3/4 🧵

RAG enables LLMs to access external info 📖. But when this info is multiple languages 🌐, can LLMs reconcile differing viewpoints 🧐? We introduce BordIRlines, a dataset to study the robustness of cross-lingual RAG. 📃arxiv.org/abs/2410.01171 🗃️ huggingface.co/datasets/borde… 1/4 🧵