Brace(Hanyang) Zhao

@yule_gan Interesting work! Curious that will the conclusion hold for models beyond Qwen 2.5, e.g. Qwen3? I remember there's supurious rewards paper last year whose observations also only hold for Qwen 2.5. Could this be the Qwen 2.5's magic?

Simply adding Gaussian noise to LLMs (one step—no iterations, no learning rate, no gradients) and ensembling them can achieve performance comparable to or even better than standard GRPO/PPO on math reasoning, coding, writing, and chemistry tasks. We call this algorithm RandOpt. To verify that this is not limited to specific models, we tested it on Qwen, Llama, OLMo3, and VLMs. What's behind this? We find that in the Gaussian search neighborhood around pretrained LLMs, diverse task experts are densely distributed — a regime we term Neural Thickets. Paper: arxiv.org/pdf/2603.12228 Code: github.com/sunrainyg/Rand… Website: thickets.mit.edu

Did not think before that my favorite two topics - RL and Diffusion - can meet in a such interesting way :)

@xidulu Believe me that getting team matched twice and then both rejected could feel worse 😭

Every intern application season, I had more papers, more citations, more internship experiences, and offers from more prestigious places than last year; However one thing that doesn't change is that Google student researcher program NEVER gives me any sort of interview LOL

@hamishivi self-reinforcement learning

4.6 writes a plan and then says 'Excellent plan' to itself lmao

@QPHutu I guess it is much easier to choose a proper clip ratio to start with like 0.2 in common practice than this absolute difference delta? Any idea of the range of delta and is delta sensitive?

This time we should say goodbye to PPO/GRPO for real 👋 PPO is a great algorithm in classical RL settings. However, it is fundamentally flawed in LLM regime due to the large, long-tailed vocabulary.💔 Checkout our paper for more details👇

Great work! We did something quite similarly for image diffusion almost one year ago 🤣 In Rich Preference Optimization (arxiv.org/pdf/2503.11720), we train diffusion models by generating better images conditional on feedbacks (like your second turn answer), yet we use dpo instead of online rl.

RL on LLMs inefficiently uses one scalar per rollout. But users regularly give much richer feedback: "make it formal," "step 3 is wrong." Can we train LLMs on this human-AI interaction? We introduce RL from Text Feedback, with 1) Self-Distillation; 2) Feedback Modeling (1/n) 🧵

Now I am really curious: Why Anthropic is still hiring engineers ...

Cool work! We had an earlier quite similar exploration in utilizing VLMs to improve Diffusion Models as well in RPO (arxiv.org/pdf/2503.11720), but instead we create synthetic preference pairs by revising upon original image from critiques and revising suggestions provided by VLMs!

Vision-language models are getting better every day. Can we use them to improve image compression? Yes! For my internship, working w/ @GoogleDeepMind, @GoogleResearch, we designed VLIC, a diffusion autoencoder post-trained with VLM preferences. Our preprint is out today! A🧵:

Really miss the good old times of RL and Diffusion Models: Both domains are with interesting math involved, and cool experiments that can be done like RL gym envs or image generation tasks to verify the ideas. Recent research and papers are fundemenatlly different flavors....

Happy moments of the past 2025: received a grant from @thinkymachines 🥹 Hope to build something cool in the new year!

Applications change, but the principles are enduring. After a year's hard work led by @JCJesseLai, we are really excited to share this deep, systematic dive into the mathematical principles of diffusion models. This is a monograph we always wished we had.

Introducing Tinker: a flexible API for fine-tuning language models. Write training loops in Python on your laptop; we'll run them on distributed GPUs. Private beta starts today. We can't wait to see what researchers and developers build with cutting-edge open models! thinkingmachines.ai/tinker

LoRA makes fine-tuning more accessible, but it's unclear how it compares to full fine-tuning. We find that the performance often matches closely---more often than you might expect. In our latest Connectionism post, we share our experimental results and recommendations for LoRA. thinkingmachines.ai/blog/lora/

@aviral_kumar2 Sounds interesting! We also did some investigation in utilizing diffusion models structure for value function design in arxiv.org/pdf/2502.01819, not sure whether some parameterizations can be correlated!

🚨🚨New paper on core RL: a way to train value-functions via flow-matching for scaling compute! No text/images, but a flow directly on a scalar Q-value. This unlocks benefits of iterative compute, test-time scaling for value prediction & SOTA results on whatever we tried. 🧵⬇️

This paper clearly reminds myself of our ICLR 2025 RainbowPO (arxiv.org/pdf/2410.04203) paper last year, in which we did similar things for DPO variants. Enjoy reading (and doing) such works which could help elucidating the design space.

ByteDance is exploring diffusion LLMs too! 👀 Seed Diffusion Preview: a blazing-fast LLM for code, built on discrete-state diffusion. With 2,146 tokens/sec inference on H20 GPUs, it outpaces Mercury & Gemini Diffusion, while matching their performance on standard code benchmarks. New SOTA on the speed–quality Pareto frontier. 🚀

An advanced version of Gemini with Deep Think has officially achieved gold medal-level performance at the International Mathematical Olympiad. 🥇 It solved 5️⃣ out of 6️⃣ exceptionally difficult problems, involving algebra, combinatorics, geometry and number theory. Here’s how 🧵

Will be presenting this paper on Wednesday (July 16) 4:30 p.m.-7:00 PDT p.m. at West Exhibition Hall B2-B3 W-706! Love to chat about Diffusion Models, RL, LLMs or anything else!