Noah Lee

⁉️Why do reward models suffer from over-optimization in RLHF? We revisit how representations are learned during reward modeling, revealing “hidden state dispersion” as the key, with a simple fix! 🧵 Meet us at @icmlconf! 📅July 16th (Wed) 11AM–1:30PM 📍East Hall A-B E-2608

@shangbinfeng Congrats Shangbin!

Now accepted at ICML 2025

🏆Glad to share that our BiGGen Bench paper has received the best paper award at @naaclmeeting! x.com/naaclmeeting/s… 📅 Ballroom A, Session I: Thursday May 1st, 16:00-17:30 (MDT) 📅 Session M (Plenary Session): Friday May 2nd, 15:30-16:30 (MDT) 📅 Virtual Conference: Tuesday May 6th, 20:30-21:00 (MDT) I'd like to appreciate our coauthors @scott_sjy Ji Yong Cho @ShayneRedford @chaechaek1214 @dongkeun_yoon @gson_AI Yejin Cho @shafayat_sheikh @jinheonbaek @suehpark @ronalhwang @Jinkyung_Jo Hyowon Cho @haebinshin_ @sylee_ai @hanseok_oh @nlee288 @itsnamgyu @joocjun @miyoung_ko @yoonjoo_le2 @hyungjoochae @jay_shin @jang_yoel @SeonghyeonYe @billyuchenlin @wellecks @gneubig Moontae Lee @Kyungjae__Lee @seo_minjoon! It wouldn't have been possible without everyone's feedback and hard work 😀

🌏"When" and "Why" can we use English reward models (RMs) in other languages? Late news, but I'm happy to share that a paper co-led with @nlee288, studying the conditions for cross-lingual transfer in RMs, has been accepted to #NAACL2025!🎉 Main insights in the thread🧵

Glad to share that BiGGen Bench has been accepted at @naaclmeeting! See you at New Mexico🤗

Please visit Riverfront Hall 10:30 AM tmr for our poster presentation of ORPO!

@emnlpmeeting @nlee288 Lastly, my recent work with @nlee288 is on the track to generalizing RLHF to different languages and tasks! 📎Paper: arxiv.org/abs/2410.18027 I would also love to chat about the multilingual alignments and reward modeling mechanisms for generalizing RLHF🙌

As more and more attention shifts back to on-policy RL for LLM post training, thx o1, (away from just using DPO-like methods for alignment) it's been clear we need a better reward model ecosystem. The good news is, we're starting to get a lot of evals. True progress only comes with good evals. The most recent two papers that caught my eye are multilingual RM work (more on this topic soon from me 🤫). 1. MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Models; Son et al. 2. Cross-lingual Transfer of Reward Models in Multilingual Alignment; Hong at all I've been arguing this is needed since April 2023 (blog links to back this up below), and was why I built RewardBench, so in some ways "mission accomplished," but our models really are so far behind closed labs. We need to be exploring how to use models like llama 405B as a reward model. Will open up a lot of doors.

🎉Thrilled to share that two papers, including ORPO, have been accepted at #EMNLP2024 Main Track! Can't wait to attend my first NLP conference😀 1⃣ ORPO: arxiv.org/abs/2403.07691 2⃣Stable Language Model Pre-training by Reducing Embedding Variability: arxiv.org/abs/2409.07787

🤔How can we systematically assess an LM's proficiency in a specific capability without using summary measures like helpfulness or simple proxy tasks like multiple-choice QA? Introducing the ✨BiGGen Bench, a benchmark that directly evaluates nine core capabilities of LMs.

Margin-aware Preference Optimization for Aligning Diffusion Models without Reference Modern alignment techniques based on human preferences, such as RLHF and DPO, typically employ divergence regularization relative to the reference model to ensure training stability. However, this often limits the flexibility of models during alignment, especially when there is a clear distributional discrepancy between the preference data and the reference model. In this paper, we focus on the alignment of recent text-to-image diffusion models, such as Stable Diffusion XL (SDXL), and find that this "reference mismatch" is indeed a significant problem in aligning these models due to the unstructured nature of visual modalities: e.g., a preference for a particular stylistic aspect can easily induce such a discrepancy. Motivated by this observation, we propose a novel and memory-friendly preference alignment method for diffusion models that does not depend on any reference model, coined margin-aware preference optimization (MaPO). MaPO jointly maximizes the likelihood margin between the preferred and dispreferred image sets and the likelihood of the preferred sets, simultaneously learning general stylistic features and preferences. For evaluation, we introduce two new pairwise preference datasets, which comprise self-generated image pairs from SDXL, Pick-Style and Pick-Safety, simulating diverse scenarios of reference mismatch. Our experiments validate that MaPO can significantly improve alignment on Pick-Style and Pick-Safety and general preference alignment when used with Pick-a-Pic v2, surpassing the base SDXL and other existing methods. Our code, models, and datasets are publicly available via

Introducing MaPO, a memory-efficient technique for aligning T2I diffusion models on preference data 🔥 We eliminate the need to have a reference model when performing alignment fine-tuning. Code, models, datasets, and paper are up at: mapo-t2i.github.io 1/7

In Feb, our CEO @dvilasuero built a 7K version 💡Idea ✅Big labs use multiturn preference data ❌The OSS AI community had single-turn prefs Misunderstood by many, @jiwoohong98 @nlee288 showed its great for ORPO, thanks! Now it's used by +100s models twitter.com/jiwoohong98/st…

ICYMI, a couple days ago, we released 🪁 Zephyr 141B fine-tune of Mixtral 8x22B using ORPO with Capybara DPO A fruitful collaboration between @kaist_ai, @huggingface and us @argilla_io 👏🏻 Special kudos to @_lewtun @jiwoohong98 @nlee288 and @krasul huggingface.co/HuggingFaceH4/…

Welcome Zephyr 141B to Hugging Chat🔥 🎉A Mixtral-8x22B fine-tune ⚡️Super fast generation with TGI 🤗Fully open source (from the data to the UI) huggingface.co/chat/models/Hu…

Try our Zephyr-ORPO-141B-A35B😃 Now it is in HuggingChat, thanks to @huggingface 🔥Mixtral-8x22B + ORPO🔥

@altryne Thank you for having us! 🙂

Great collab w/ @huggingface and @argilla_io !!! Also note ORPO has been officially integrated in the TRL Library with the v0.8.2 release. github.com/huggingface/trl

Data: hf.co/datasets/argil… ORPO paper: hf.co/papers/2403.07… Code recipe to replicate: github.com/huggingface/al… Amazing collaboration between @argilla_io, @kaist_ai and @huggingface. Kudos to @jiwoohong98 @nlee288 @alvarobartt and @_lewtun 🔥

Aligning a diffusion model on preference data WITHOUT a reference model could be nice no? So, @krasul and I are ideating the use of ORPO to align SDXL 1.0 on PickAPic. Diffusion ORPO with LoRA 💫 Code and model ⬇️ huggingface.co/sayakpaul/diff…