Keming (Luke) Lu

Will be at #ICLR2025 next week to present this interesting work from @kakakbibibi! Can’t wait to discuss more about the next stage of LLM/VLM & RL

Life update: Following my recent graduation, I've joined the Bytedance Seed Edge team to pursue this research direction further. Although this post was written last year, my conviction in this approach has only strengthened (many ideas here echo compelling recent writings from the legendary Rich Sutton and Shunyu, such as the need for rewards to help models evolve beyond the classic finite-context learning paradigm). In the near term, my focus will be on making individual agents evolvable, with the next phase involving connecting and scaling these evolvable agents. I'm incredibly excited about the potential achievements in this direction and welcome connections, discussions, or collaboration. I'll also be attending ICLR next week; please send a DM if you'd like to chat. Welcome to the second half of AGI 😉

I guess yes if scaling law holds and is still the right path towards AGI. It’s too risky to recklessly run into training huge models? (small models are necessary for debugging btw lol

I have to retweet this one… verl is evolving so fast. Just can’t wait to see what’s next

context distillation + rejection sampling almost solve all challenges this week… Highly recommend！

It is so offensive for including racial background in such claim as moving out the racial info could still clearly show the argument…

Impressive work from Chujie 🌟 His benchmark will definitely set a new landmark in this topic

@JustinLin610 GRPO/PPO+(model-based/rule-based) verifier

Can anyone tell me your guess of the implementation of reinforcement finetuning?

Andrej’s tweet is the right way to think about it right now but I totally believe that in one or two years we will start relying on AI for very challenging decisions like diagnosing disease under limited information. Key thing to note here is that big decisions can be viewed as a tree of individual reasoning steps and RL on chain of thought seems like a feasible way for AI to do any single step pretty well and probably recover if there is a mistake In addition, with better scaffolding like improvements in retrieval, browsing, and long context management, AI will be able to leverage its inherent advantages over humans like not getting tired or distracted, having nearly infinite memory, and not being clouded by emotions. So i think we will reach the “magical AI feeling” soon :)

What does it mean to think, to question, to understand? Note: This is the pronunciation of QwQ: /kwju:/ , similar to the word “quill”. Blog: qwenlm.github.io/blog/qwq-32b-p… Model: huggingface.co/Qwen/QwQ-32B-P… Demo: huggingface.co/spaces/Qwen/Qw… Something that can reason, making math problem solving and coding much better. A lot of limitations by the way. Forgive us for a lot of things undone. We will make them happen soon.

@Xidong_Feng @GoogleDeepMind Cong!!

Thrilled to share that I’ll be joining @GoogleDeepMind as a Research Scientist with the Discovery Team! It’s a dream come true—8 years ago, I watched AlphaGo live in high school, and now I am lucky enough to be part of this incredible journey. Can’t wait to discover what’s next!

Otter has been accpeted by EMNLP2024 as a main conference paper👏

Neat project👏 You should definitely try this one if you are looking for challenging math benchmarks!

@JustinLin610 fight on🥹

sleep no more🥝

Deep RL is finally back out of the Trough of Disillusionment

New survey: Towards a unified view of preference learning for LLMs! 🧠 LLMs are powerful, but aligning them with human preferences is key. This survey breaks down existing alignment strategies into four components: Model, Data, Feedback, and Algorithm. 🔗 This unified view reveals connections between different methods and opens doors for synergistic solutions. 🚀 Explore the challenges and future directions of aligning LLMs with human preferences. Paper: arxiv.org/abs/2409.02795 Github: github.com/kbsdjames/awes… Notion Blog: aeolian-agenda-626.notion.site/Towards-a-unif… #LLMs #AI #PreferenceLearning #Survey #MachineLearning

Cool！Look forward to the data👏

Today we are thriiled to announce the release of Qwen2-VL! Specifically, we opensource Qwen2-Vl-2B and Qwen2-VL-7B under Apache 2.0 license, and we provide the API of our strongest Qwen2-VL-72B! To learn more about the models, feel free to visit our: Blog: qwenlm.github.io/blog/qwen2-vl/ GitHub: github.com/QwenLM/Qwen2-VL HF: huggingface.co/collections/Qw… ModelScope: modelscope.cn/organization/q… Qwen2-VL is the latest version of our vision language models built upon Qwen2. It consists of the following features: SoTA understanding of images of various resolution & ratio: Qwen2-VL achieves state-of-the-art performance on visual understanding benchmarks, including MathVista, DocVQA, RealWorldQA, MTVQA, etc. Understanding videos of 20min+: Qwen2-VL can understand videos over 20 minutes for high-quality video-based question answering, dialog, content creation, etc. Agent that can operate your mobiles, robots, etc.: with the abilities of complex reasoning and decision making, Qwen2-VL can be integrated with devices like mobile phones, robots, etc., for automatic operation based on visual environment and text instructions. Multilingual Support: to serve global users, besides English and Chinese, Qwen2-VL now supports the understanding of texts in different languages inside images, including most European languages, Japanese, Korean, Arabic, Vietnamese, etc.

I try Qwen2Strawberry

Model merging is a popular research topic with applications to LLM alignment and specialization. But, did you know this technique has been studied since the 90s? Here’s a brief timeline… (Stage 0) Original work on model merging dates back to the 90s [1], where authors showed that taking an average of neural network parameters yields a model that performs similarly to averaging the output of multiple neural networks (i.e., an ensemble). (Stage 1) Averaging along the training trajectory. Several works in the mid-to-late 2010s explore the idea of taking an average of model checkpoints throughout the training process. This can be done via an exponential moving average [2] or by just averaging specific checkpoints during training [3] and improves training stability / performance / generalization in certain cases. (Stage 2) Linear mode connectivity [4, 5] is a research topic–coming from research on neural network pruning / sparsity–that is highly related to model merging. Linear mode connectivity shows us that multiple neural networks (finetuned from the same base model) have a linear path of non-increasing loss between them in the parameter space. Put simply, interpolating between two finetuned models yields another model that also performs well. [6] studies this topic in the context of LLMs. (Stage 3) Weight averaging. Based on findings in linear mode connectivity research, we began to see several papers that directly average the parameters of several neural networks obtained in separate training runs (i.e., “model soups”) [7, 8, 9]. This strategy–although simple–was found to have several benefits in terms of model performance and generalization capability, somewhat similarly to creating an ensemble of models from several separate finetuning runs. (Stage 4) Better merging approaches. After initial explorations of model merging, we began to see several research papers that propose more specialized / effective merging techniques, such as: - Using task vectors to merge models [10]. - The TIES [11] or DARE [12] strategies for reducing interference during merging. These works show that model merging is highly effective for improving performance and combining the capabilities of multiple models. However, performance depends on the strategy we use for merging, which can be optimized to reduce conflicts / interference. (Stage 5) LLM-specific research. More recently, we have seen a widespread adoption of model merging within LLM research to combine LLM capabilities [13], improve the alignment process [14], or train better reward models [15]. One especially notable application of model merging in recent research is the use of WARP [14]–a state-of-the-art model merging technique for improving LLM alignment–for aligning Gemma-2 [16].