Kianté Brantley

Does LLM RL post-training need to be on-policy?

Prefix RL for multi-turn agentic tasks. Additionally the difficulty of each turn is measured and intermediate rewards are assigned (potentially using a model).

LLM has been struggling to solve search and optimization at scale when feedback is stochastic. We propose a simple solution, POLCA, using text embedding with “provable” guarantee. Excited to see the first theoretically correct work of LLM optimization. Kudos to @XuanfeiRen

Learning from feedback is instrumental but human preference data can be expensive. How much reward supervision could we get from raw web text instead, without human labels? Our latest work, built on a year of incredible effort by @fjxdaisy, advances pure RLHF training across multiple models and tasks. Unsupervised Reward Modeling: split web docs into (prefix, true continuation); treat mismatched continuations in-batch as negatives; train w/ BT loss + score-centering. Findings: 📝 steady gains on RewardBench v1/v2 using just 11M tokens of math web text 📝 transfers across backbones (Llama-3.2 1B/3B, Qwen2.5 3B/7B Instruct) 📝 improves Best-of-N selection (math + safety) and provides a usable reward for GRPO policy optimization 📝 acts as a mid-training procedure that helps further RLHF 📑Paper: arxiv.org/abs/2603.02225 🌐Project: jingxuanf0214.github.io/reward-scaling Joint work with @lisali126, @ZhentingQi, @zdhnarsil, @xkianteb, @ShamKakade6

Attention NYC undergrads: Applications are open for our 13th annual Data Science Summer school at Microsoft Research NYC! Apply here by April 14th: bit.ly/3pCQENh

I built an interactive JavaScript thingy to study the two faces of KL divergence. timvieira.github.io/blog/interacti… I have wanted this since 2009. Thank you, Claude Code, for helping me get there!

@xtimv has dropped a new blog post!

students continue to be eager to learn, absorb all i can teach them and improve every week. this semester is one of the most rewarding semesters so far. some of us here feel unnecessarily, incorrectly and prematurely so defeated that we send out wrong messages of defeatism. stop! it’s just you who feel defeated and depressed. don’t impose it on others who are up and coming, and seriously much better than us.

I'm reading this stuff in my hotel room right now btw

📢✨We are thrilled to announce the 2026 recipients of #KempnerInstitute Research Fellowships: Jieneng Chen, Marco Fumero, Audrey Huang, Raja Marjieh, Yinuo Ren and Anne Wu! Learn more: bit.ly/3NfzB2Y @anne_youw @marco_fumero @Yinuo_Ren @RajaMarjieh

Going to do a more technical deep dive on our enterprise knowledge agents and how we train them with RL. Overall we found that simple, yet principled off-policy RL works at scale for complex agentic tasks with hundreds of steps of tool use and context management. Here are the key takeaways from our 80 page technical report. (1) RL does not just sharpen base model's distribution. We see test-time scaling improves consistently over the iterations of the RL training. Skills learned during RL transfers to unseen prompts and agent learns to solve prompts where base model has zero accuracy under pass@16. (2) Multi-task RL generalizes really well Simple mixing of training data from multiple tasks works well and allows multi-task RL scale beyond your in-distribution training tasks. We found that multi-task RL just works better than multi-expert distillation. (3) End-to-end RL for tools and context management works best. We skipped mid-training, and directly trained everything end-to-end using RL at scale (2m tokens per gradient computation). Models learned to use vector database tools and context compression at the same time.

(1/7) RL-finetuned VLMs report steady gains on visual reasoning benchmarks, but whether those improvements are robust in practice is still unclear—especially given ongoing grounding failures and hallucinations. Our preprint shows that simple controlled perturbations can cause substantial accuracy drops—and even when the final answer is right, the chain-of-thought is often wrong or inconsistent even in the presence of grounding signals. Work done during my internship at Apple last summer!

Does LLM RL post-training need to be on-policy?

Can confirm: OAPL is great for real-world tasks. x.com/xkianteb/statu…

tested and this works super well.. congrats @xkianteb!!

Thanks @murefil! Our team spent a lot of time ensuring this algorithm makes sense. I am glad it is working for you!

Meet KARL, an RL'd model for document-centric tasks at frontier quality and open source cost/speed. Great for @databricks customers and scientists (77-page tech report!) As usual, this isn't just one model - it's an RL assembly line to churn out models for us and our customers 🧵

Specifying rewards has always been a pain ☹️Perhaps not such a pain going forward? Robometer proposes a methodology for training general purpose reward models that work out of the box across new robots/tasks/environments! The key - find a good way to use negative data :) Once you do this - you can enable fun downstream applications like - efficient real-world RL, world model steering, offline RL, failure detection and more. We release code and models for you to play around with it yourself! Website: robometer.github.io Paper: arxiv.org/abs/2603.02115 Code: github.com/robometer/robo… Special shout out to our incredible postdoc @Jesse_Y_Zhang for corraling a bunch of awesome collaborators to make this happen!

If you aren’t following @owenoertell, you’re missing out on a rising star in RL

really fun project! you don't need to be on-policy to do RL! off-policy is just as good (and more sample efficient)!

Very proud of our @databricks RL team members!