Leitian Tao@NeurIPS 2025

Hybrid Reinforcement (HERO): When Reward Is Sparse, It’s Better to Be Dense 🦸‍♂️ 💪 📝: arxiv.org/abs/2510.07242 - HERO bridges 0–1 verifiable rewards and dense reward models into one 'hybrid' RL method - Tackles the brittleness of binary signals and the noise of pure reward models -> better results! ✔️ Stratified normalization anchors dense scores within verifier groups ✔️ Variance-aware weighting emphasizes harder, high-variance prompts ✔️ Stable + informative rewards, no drift 📈 Results: 🔥 +11.7 pts vs RM-only, +9.2 pts vs verifier-only on hard-to-verify reasoning tasks 🔥 Generalizes across Qwen and OctoThinker models 🔥 Works well when training with easy-to-verify/hard-to-verify/mixed samples. Hybrid reward → stable, dense, reliable supervision, advancing reasoning RL 🧵(1/5)

Hybrid Reinforcement (HERO): When Reward Is Sparse, It’s Better to Be Dense 🦸‍♂️ 💪 📝: arxiv.org/abs/2510.07242 - HERO bridges 0–1 verifiable rewards and dense reward models into one 'hybrid' RL method - Tackles the brittleness of binary signals and the noise of pure reward models -> better results! ✔️ Stratified normalization anchors dense scores within verifier groups ✔️ Variance-aware weighting emphasizes harder, high-variance prompts ✔️ Stable + informative rewards, no drift 📈 Results: 🔥 +11.7 pts vs RM-only, +9.2 pts vs verifier-only on hard-to-verify reasoning tasks 🔥 Generalizes across Qwen and OctoThinker models 🔥 Works well when training with easy-to-verify/hard-to-verify/mixed samples. Hybrid reward → stable, dense, reliable supervision, advancing reasoning RL 🧵(1/5)

Collecting large human preference data is expensive—the biggest bottleneck in reward modeling. In our #NeurIPS2025 paper, we introduce latent-space synthesis for preference data, which is 18× faster and uses a network that’s 16,000× smaller (0.5M vs 8B parameters) than text-based synthesis methods. 📄arxiv.org/abs/2509.26074 🧵Thread below -------------------------------------------------------- Instead of generating and annotating new text, our approach — LENS (Latent EmbeddiNg Synthesis) — learns to expand preference datasets directly in embedding space. We train a variational autoencoder on existing preference data, then sample new latent vectors to synthesize diverse, high-quality preference pairs — all without text generation or extra human labeling. This simple shift from text space → latent space makes reward modeling dramatically more efficient while preserving semantic consistency. 📊 Results: - 18× faster data generation - 16,000× smaller augmentation model (0.5M vs 8B parameters) - On HH-RLHF and TL;DR benchmarks, LENS achieves large improvements over text-based augmentation baselines. - Generalizability: Works across different LLM backbones and shows the strongest gains in low-data regimes. 📚 Theoretical insights We show that latent-space synthetic pairs preserve preference ordering within a provable bound, and that augmentation with LENS improves the generalization error of reward models. In other words — it’s not just faster; it is theoretically grounded. While LENS is motivated by efficiency in preference data synthesis, its potential extends beyond. For example, LENS opens doors for: - Low-resource alignment: For languages, domains, or communities with limited human preference data, LENS can expand training sets in embedding space, helping close the data gap in pluristic alignment efforts. - Personalized reward modeling: Generating synthetic preferences in latent space tailored to individual user preferences or styles. Massive credit to @LeitianT for leading the effort and @xuefeng_du for mentorship. Couldn’t have done this without this stellar team!

Collecting large human preference data is expensive—the biggest bottleneck in reward modeling. In our #NeurIPS2025 paper, we introduce latent-space synthesis for preference data, which is 18× faster and uses a network that’s 16,000× smaller (0.5M vs 8B parameters) than text-based synthesis methods. 📄arxiv.org/abs/2509.26074 🧵Thread below -------------------------------------------------------- Instead of generating and annotating new text, our approach — LENS (Latent EmbeddiNg Synthesis) — learns to expand preference datasets directly in embedding space. We train a variational autoencoder on existing preference data, then sample new latent vectors to synthesize diverse, high-quality preference pairs — all without text generation or extra human labeling. This simple shift from text space → latent space makes reward modeling dramatically more efficient while preserving semantic consistency. 📊 Results: - 18× faster data generation - 16,000× smaller augmentation model (0.5M vs 8B parameters) - On HH-RLHF and TL;DR benchmarks, LENS achieves large improvements over text-based augmentation baselines. - Generalizability: Works across different LLM backbones and shows the strongest gains in low-data regimes. 📚 Theoretical insights We show that latent-space synthetic pairs preserve preference ordering within a provable bound, and that augmentation with LENS improves the generalization error of reward models. In other words — it’s not just faster; it is theoretically grounded. While LENS is motivated by efficiency in preference data synthesis, its potential extends beyond. For example, LENS opens doors for: - Low-resource alignment: For languages, domains, or communities with limited human preference data, LENS can expand training sets in embedding space, helping close the data gap in pluristic alignment efforts. - Personalized reward modeling: Generating synthetic preferences in latent space tailored to individual user preferences or styles. Massive credit to @LeitianT for leading the effort and @xuefeng_du for mentorship. Couldn’t have done this without this stellar team!