Peihao Wang

Interestingly, we revealed a duality: 🔵 Training-time alignment ≈ amortized parameter-space optimization 🔵 Test-time optimization ≈ latent space sampling From a classical statistical inference lens, these two are tightly connected, just operating over different spaces.

We formulate decoding as an optimization problem: find responses that maximize a differentiable reward subject to being sampled from an LLM . Gradients are backpropagated into the model’s hidden states, steering inference into a form of test-time training.

Latent space reasoning via looped transformers has gained attention lately. It is rooted in optimization unrolling , where each loop implicitly models a GD step on hidden states. Our ICLR paper studied what if we explicitly run GD in latent space at test time?

@liliang_ren Congrats. Looking forward to your new chapter.

Life update: Actually I joined Thinking Machines Lab yesterday. Feels grateful for my friends and managers at Microsoft for their flexibility, trust and support. And super excited to see what I can build next at TML!

"One static model does not fit all." Reminds me of the old parametric vs. non-parametric regression debate. Nice to see scalable generative weight-space models finally taking shape.

Are multi-agent systems necessary? Here is a great new paper addressing this. The big assumption most AI devs make today is that more agents lead to better performance. But here is the overlooked reality: most multi-agent systems are homogeneous. All agents typically share the same base LLM, differing only in prompts, tools, and positions in the workflow. This raises a compelling question of whether a single agent can simulate these workflows through multi-turn conversations. This new research investigates this across seven benchmarks spanning coding, mathematics, QA, domain-specific reasoning, and real-world planning. A single agent with KV cache reuse can match the performance of homogeneous multi-agent workflows while reducing inference costs. The cost advantage comes from shared KV cache across agent interactions, avoiding redundant prefill computation. Because homogeneous agents possess identical reasoning capabilities and differ only in specialized instructions, a single agent can role-play these agents sequentially, exploiting the workflow's task decomposition without needing separate model instances. Building on this finding, the researchers propose OneFlow, an algorithm that automatically designs workflows optimized for single-agent execution. OneFlow uses a dual meta-LLM architecture (Creative Designer + Critical Reviewer) with Monte Carlo Tree Search to discover streamlined workflows with comprehensive system prompts and fewer total agents. OneFlow with single-agent execution achieves 92.1% on HumanEval, 81.4% on MBPP, 93.3% on GSM8K, matching or exceeding multi-agent baselines while significantly reducing cost. Single-LLM methods cannot capture truly heterogeneous workflows where agents use different base models, since KV caches cannot be shared across different LLMs. These results position single-LLM implementation as a strong baseline for MAS research. The authors suggest that the real opportunity lies in developing heterogeneous systems where model diversity benefits outweigh coordination costs. Paper: arxiv.org/abs/2601.12307 Learn to build effective AI agents in our academy: dair-ai.thinkific.com

@zhiwen_fan_ Thx Zhiwen! Glad that I finally made some progress chasing your excellence.

Huge congratulations, Peihao🫡 @peihao_wang

@VITAGroupUT It won't be possible without the team I'm working with.

🎉 Huge congratulations to PhD student Peihao Wang (@peihao_wang ) on two major honors: 🏆 2025 Google PhD Fellowship in Machine Learning & ML Foundations 🌟 Stanford Rising Star in Data Science Incredibly proud of Peihao's outstanding achievements! 🔶⚡

Thank you, @JeffDean. Really honored to join the 2025 class of Google PhD Fellows! Excited to carry forward the inspiration to explore the AI frontier where logics and physics meet algebra and geometry.

This work is so special to me. I first touched cryo-EM as a junior - couldn’t believe a neural net could predict bio structure from extremely low SNR, unposed images. with so many AI progress in these 5 years, scaling laws make AI-driven protein discovery feel real

We already introduced #LightGaussian last year to accelerate the rendering speed of 3DGS. In our CVPR'25 paper, SteepGS, we go further by demystifying and improving density control during 3DGS optimization — making training more efficient and reliable. Project Page: vita-group.github.io/SteepGS/

@ccccrs_0908 Congrats 🎊 so proud of you

Excited to share that I have been awarded NVIDIA fellowship! 🎉 Immensely grateful for the recognition and support - this inspires me to continue advancing research in LLM efficiency and AI security. blogs.nvidia.com/blog/graduate-…

Layer-wise routers are surprisingly redundant in current MoE. Check out Read-ME for the system-friendly MoE refactorization technique with system co-design!

🚀 Our NeurIPS '24 work, Large Spatial Model (LSM), is here! LSM performs semantic 3D reconstruction in just 0.1s, processing unposed data via feed-forward 3D reconstruction. 👉It leverages large-scale 3D datasets with minimal annotations, defining a 3D latent space. We are continuously exploring how this explicit 3D representation can further enhance reasoning and robotic learning. 🔗 Try our online Gradio demo with your own data at largespatialmodel.github.io #NeurIPS2024 #3DReconstruction

Train one - Get many🚀! Check more details about Flextron at cairuisi.github.io/Flextron/

Introducing Score identity Distillation with Long and Short Guidance (SiD-LSG), our data-free solution to distill Stable Diffusion models into one-step text-to-image generators, achieving a COCO2014 zero-shot FID of 8.15. Excited to share the code and checkpoints with the community! Code: github.com/mingyuanzhou/S… Paper: arxiv.org/abs/2406.01561 #Diffusion #Distillation #StableDiffusion @ZhendongWang6 @UnderGroundJeg @haihuang_ml

Tired of training varying-size LLMs to fit various GPU memory and latency requirements? Check out Flextron! Our new ICML (Oral) paper shows how to train one model deployable across GPU series. Learn more: cairuisi.github.io/Flextron/🚀

The Flextron-Llama2-7B model family demonstrates superior MMLU performance compared to both open-source models (including Pythia, OpenLLaMA-v2) and existing post-hoc compression methods (including Sheared-LLaMA, SliceGPT, LLM-Pruner, Compresso, LaCo).

Managing long context is challenging due to quadratic attention memory usage. But what if we could compress growing context information into a fixed-size memory? 🤔 Check out our new ICML paper: "LoCoCo: Dropping In Convolutions for Long Context Compression"! 1/3

Training 3D foundation models? In our CVPR2024 work, we propose a new concept that directly enhances 2D prediction’s view consistency via image based rendering. It generalizes to many 2D foundation models in zero shot and transfers their success to 3D at little training cost.