Ivan Vulić

RAG and in-context learning are the go-to approaches for integrating new knowledge into LLMs, making inference very inefficient We propose instead 𝗞𝗻𝗼𝘄𝗹𝗲𝗱𝗴𝗲 𝗠𝗼𝗱𝘂𝗹𝗲𝘀 : lightweight LoRA modules trained offline that can match RAG performance without the drawbacks

Automating Multi-Agent Design: 🧩Multi-agent systems aren’t just about throwing more LLM agents together. 🛠️They require mastering the subtle art of prompting and agent orchestration. Introducing MASS🚀- Our new agent optimization framework for better prompts and topologies!

We achieved the first instance of successful subword-to-byte distillation in our (just updated) paper. This enables creating byte-level models at a fraction of the cost of what was needed previously. As a proof-of-concept, we created byte-level Gemma2 and Llama3 models. 🧵

🚀Let’s Think Only with Images. No language and No verbal thought.🤔 Let’s think through a sequence of images💭, like how humans picture steps in their minds🎨. We propose Visual Planning, a novel reasoning paradigm that enables models to reason purely through images.

We created Approximate Likelihood Matching, a principled (and very effective) method for *cross-tokenizer distillation*! With ALM, you can create ensembles of models from different families, convert existing subword-level models to byte-level and a bunch more🧵

We've got plenty of exciting ideas flying around, so consider applying to carve them further with us!

📣Happy to (pre-)release my Fleurs-SLU benchmark to evaluate massively multilingual spoken language understanding on SIB & Belebele. Work done at @Mila_Quebec with @davlanade @gg42554 @licwu Datasets: huggingface.co/datasets/WueNL… huggingface.co/datasets/WueNL… Details to follow👇

Thrilled to share our updated paper: "UNDIAL: Self-Distillation with Adjusted Logits for Robust Unlearning in Large Language Models" We propose a new robust LLM unlearning method via Self-Distillation on Adjusted Logits (UNDIAL). 📄 Paper: arxiv.org/pdf/2402.10052

Do you DARE? Introducing a multiple-choice VQA benchmark ✨DARE✨ with: - 4 main robustness evaluation ⛓️ - 5 diverse categories 🧩 - Extensive analysis of 4 widely used VLMS 🤖

Introducing 🪓Segment any Text! 🪓 A new state-of-the-art sentence segmentation tool! Compared to existing tools (and strong LLMs!), our models are far more: 1. efficient ⚡ 2. performant 🔝 3. robust 🚀 4. adaptable 🎯 5. multilingual 🗺

As someone who spent years working in multilingual NLP, I am so happy that we're finally seeing (L)LMs and (N)MT systems working in tandem towards the shared cause. The idea in this work is so simple & sweet, and yet it moves! 🌍🌏🌎

Which output is better? [A] or [B]? LLM🤖: B❌ [B] or [A]? LLM🤖: A✅ Thrilled to share our preprint in addressing preference biases in LLM judgments!🧑‍⚖️We introduce ZEPO, a 0-shot prompt optimizer that enhances your LLM evaluators via fairness⚖️ 📰Paper: arxiv.org/abs/2406.11370

Excited to introduce TopViewRS: VLMs as Top-View Spatial Reasoners🤖 TopViewRS assess VLMs’ spatial reasoning in top-view scenarios🏠just like how you read maps🗺️ Spoiler🫢GPT4V and Gemini are neck-and-neck, each excelling in different setups but neither even close to us humans

Introducing Zero-Shot Tokenizer Transfer (ZeTT) ⚡ ZeTT frees language models from their tokenizer, allowing you to use any model with any tokenizer, with little or no extra training. Super excited to (finally!) share the first project of my PhD🧵

Adapters are just a great way to share/benefit from new capabilities without handing around the kitchen sink. Congrats to the AdapterHub folks for adding support for quantized training (Q-LoRA and friends).

If we align LLMs through preferences, perhaps we should also evaluate them the same way (and respect transitivity)? The answer is: yes, we should. The trick, however, is how to make evaluation tractable. If you are into the whole "LLM-as-Judges" line of work, check this paper!

🚨 A belated update: Our survey on "Modular Deep Learning" has been published in TMLR. Check out the updated version: openreview.net/forum?id=z9EkX…

I am still looking for PhD students starting in September 2024! The deadline to apply for the CDT in NLP is the 11th of March. If you wish to do research in modular and efficient LLMs, here are some highlights of my lab's research from the past year ⬇️🧵

Think globally, act locally? Well, we were thought-experimenting whether LLMs would understand people from different places around our hometowns better than we ever might... And then we have eventually decided to make an actual (non-thought) experiment out of these thoughts! 👇👇

We scaled sparse fine-tuning (SFT) to LLMs (such as Llama 2) by making it both parameter- and memory-efficient! (q)SFT instruction tuning performance is often better than (q)LoRA with comparable speed and memory load. Paper: arxiv.org/abs/2401.16405 Code: github.com/AlanAnsell/peft (SFT PEFT) github.com/ducdauge/sft-l… (experiments) @AlanAnsell5 @licwu @h_sterz @annalkorhonen