Yining Lu

RLVR is powerful — but how do you train with multiple rewards effectively? 🤔 🎯GDPO (not GRPO) is coming. We introduce Group reward-Decoupled Normalization Policy Optimization (GDPO), a new multi-reward RL algorithm that consistently improves per-reward convergence over GRPO across a wide range of tasks. (1/n)

We implement this as a Jupyter-like REPL environment: The key idea is to put the user’s prompt in a Python variable and give the LLM a REPL loop where it can try to understand the prompt, without directly reading the whole content. The “root” LM interacts with the environment by writing code and looking at the outputs of each cell, and can recursively call LM calls inside this REPL environment to navigate its context. This is far more general than any “chunking” strategy. We argue that you should let the LLM itself decide how to best poke around, decompose, and recursively process the long prompt.