Yunhao (Robin) Tang

Maybe to one's surprise, taking KL estimates as `kl_loss` to minimize does *not* enforce the KL. This implementation, however, is quite common in open source RL repos and recent research papers. In short: grad of an unbiased KL estimate is not an unbiased estimate of KL grad.

@canaesseth @xidulu Indeed! Sorry the post misses the nuance in this regard. It was mostly referring to some very specific recent RL implementations.

@xidulu @robinphysics If you are using the "reparameterization trick" you'll be fine as in that setting the expectation of the gradient of the unbiased loss is the gradient of the KL. The original post misses a bit of nuance in that the statement is sometimes true.

Maybe to one's surprise, taking KL estimates as `kl_loss` to minimize does *not* enforce the KL. This implementation, however, is quite common in open source RL repos and recent research papers. In short: grad of an unbiased KL estimate is not an unbiased estimate of KL grad.

@y0b1byte Thanks so much for the kind words!

another good one!

Taking the k3 estimate as an example (from John's popular blogpost joschu.net/blog/kl-approx…). Contrary to popular practice, differentiating the estimate as a loss ends up enforcing the reverse-KL, but only incidentally. See more details: arxiv.org/pdf/2506.09477

Announcing Magistral, our first reasoning model designed to excel in domain-specific, transparent, and multilingual reasoning.

It was refreshing to see the impact that small algorithmic changes have on the system performance. While the “double-sided” PPO/GRPO clipping is dominant in the literature, we argue that a single-sided clipping akin to IMPALA fits the design of distributed training more.

Introducing LlamaRL, a distributed RL framework for training LLM at scale. LlamaRL is highly modular, Pytorch-native, customizes optimization of actors/learners to max out throughput, and adjusts for systemic off-policyness to stabilize training arxiv.org/pdf/2505.24034

@c_voelcker @charlinelelan Many thanks @c_voelcker for the kind words! Very glad that our past investigation can be of help to your exciting new study here, look forward to reading in more details!

Big shout-out to @robinphysics and @charlinelelan whose amazing works on analyzing training dynamics in RL were hugely inspirational and foundational to our study

Eventually, humans will need to supervise superhuman AI - but how? Can we study it now? We don't have superhuman AI, but we do have LLMs. We study protocols where a weaker LLM uses stronger ones to find better answers than it knows itself. Does this work? It’s complicated: 🧵👇

@PandaAshwinee Thanks! Sorry completely missed the reply here... Indeed H2 is quite surprising. I think it's mainly bc contrastive losses don't work well w/ offline data. That is pi(y_w) / pi(y_l) can increase while both pi(y_w) and pi(y_l) are low. If we change to Bo2, H2 is less prominent.

@robinphysics Interesting work, nice to see the evaluation of the 5 hypotheses. I'm a bit surprised to see the results of Hypothesis 2. Any more details to share regarding that?

Online interaction is probably a defining property of RL. But with the rise of offline algo, it is not clear if the “online” bit of RL is necessary for RLHF. We hypothesis test the causes of the perf gap between online and offline alignment. arxiv.org/pdf/2405.08448… Details in🧵

Thanks @_akhaliq for promoting our work! Unlike regular RL where golden r(s,a) are available and online is generally deemed better than offline, in RLHF this is less clear. Complementary to some concurrent work, we investigate causes to the perf gap between online vs. offline.

The findings ought to be taken with a grain of salt due to limitations in our experimental setups. But hopefully this investigation contributes to a better understanding of RLHF practices. Finally, very grateful to my collaborators @GoogleDeepMind on this fun project!

Some takeaways: - There is something more to online than wider coverage of response generation - Offline training improves policy is a much more implicit way than online (discriminative vs. generative abilities) - The gap persists across wider variants of algos and network sizes