Yu Lei

🤖Co-training is everywhere (sim↔real[e.g. GR00T, LBM], human↔robot[e.g. PI, EgoScale], even non-robot data[e.g. PI, LBM). But why does it work? How can we improve it further? Taking sim-and-real imitation learning in diffusion/ flow-based models as the test bed, we performed a rigorous mechanistic analysis, drawing on theoretical insights and multi-layered experiments. 😮Key insight: it’s all about representations. - Alignment → enables transfer - Discernibility → enables adaptation ⚖️Both are necessary — it's better to have more aligned representations, but the model must be able to discern the domains. We term this as structured representation alignment. ⬇️Let’s take a deep dive into that: Paper: arxiv.org/pdf/2604.13645 Website: science-of-co-training.github.io

Dexterous hands vary widely—so do tactile modalities. 🖐️🌈 Our vision on tactile human-to-robot transfer: 🔓 Not tied to specific hardware ♻️ Reuse human tactile demos across embodiments Presenting TactAlign, a cross-sensor tactile alignment for cross-embodiment policy transfer.

Gemini Omni is a major leap in world understanding & multimodal editing! It can take photos, video & audio and build entirely new scenes. Over time it’ll be able to handle any input & any output - starting w/ video You can even give it your own videos & iterate on your ideas:

I will be in Vienna in two weeks to give a keynote at #ICRA2026. I'll share our recent progress on building generalist humanoid robots and show some of the latest results. Check out my talk on June 3: 2026.ieee-icra.org/program/keynot…

Introducing HRM-Text. An ultra-lean 1B-parameter reasoning language model designed to deliver strong general performance with a fraction of the data, compute, and infrastructure. Trained on just 40B structured tokens, HRM-Text achieves competitive performance while using ~1/1000 of the training data of comparable models. The kicker? The full model trains in roughly one day on a $1,000 budget. This opens the door to a new generation of AI that is powerful, accessible, and radically easier to adapt. Theories and research concepts once deemed too expensive to test are officially back in the game. Sapient Intelligence invites you to help us shape a new paradigm for general intelligence.

@wenhaocha1 @thoma_gu Deeply supervised(aligned) network

@thoma_gu after chatting with mingyang. he think in pixel-space we could apply multiple representation alignment. and i think its true for both repa-like or fd-loss-like style. I totally agree repr is super important! just say pixel is not equals to no repr (?

In my view, pixel space matters differently for VLMs and generative models (including UMMs). For VLMs we always fine-tune end-to-end, so I don't see a fundamental advantage to going encoder-free. For generative models, new features often hit a VAE/RAE ceiling; font rendering is the canonical example. And since end-to-end fine-tuning is off the table there, I suspect that on any product roadmap requiring sustained iteration, pixel may even be cheaper than latent. That's my take.

We taught two F.03 robots to clean a room and make a bed in under 2 minutes - fully autonomous.

@alpercanbe A culture.

just a glimpse of the world we are building

Love this.

Open-sourcing the whole package here! The last piece of our SONIC open-source, data collection, gr00t VLA post-training, inference just hit the repo! Train your Autonomous policies on G1 Whole-body with SONIC and gr00t N1.7! 🧑‍💻Code: github.com/NVlabs/GR00T-W… 📑Docs: nvlabs.github.io/GR00T-WholeBod…

GR00T-VisualSim2Real is now open source! VIRAL and DoorMan are now available with training code, simulation assets, and the full recipe for bringing visual sim-to-real loco-manipulation skills to your own humanoids. Repo: github.com/NVlabs/GR00T-V…

Just had a single-author paper accepted to #RSS2026! arxiv.org/abs/2604.21456 Motivated by growing interest in differentiable world models and physics simulators, we ask whether there is a unified principle for combining sampling-based global “exploration” with gradient-based local “exploitation” in trajectory and policy optimization with differentiable dynamics. By viewing control through the control-as-inference lens—recasting optimization as sampling from an unnormalized Boltzmann distribution defined by an energy function—Tempered Sequential Monte Carlo (TSMC) naturally integrates importance sampling with gradient-based Hamiltonian Monte Carlo. The key idea behind TSMC is to define a tempering path that gradually transforms an easy-to-sample prior into a complex, multi-modal posterior—or equivalently, deforms a convex energy landscape into a nonconvex one (graduated non-convexity)! We implement TSMC for both trajectory and policy optimization. On small- to medium-scale problems, it appears broadly applicable and compares favorably with state-of-the-art baselines. Excited to explore whether TSMC can scale to large-scale planning with complex, high-dimensional dynamics!

@VectorWang2 Like it!!!

Building up a mini Tidybot for <$2k in total. More dynamic (realtime no speed up) and more robust Plugged into upcoming X-bot universe with OpenClaw agentic applications. github.com/TidyBot-Servic…

@jiazhi_yang2024 Congrats jiazhi!

🤗RISE got accepted to RSS 2026! Can't wait to see everybody in Sydney! 💻Code: github.com/OpenDriveLab/R…

@MingchenZhuge Thanks for inviting me for the talk and the panel discussion! It was super fun! Talk slides in yuandong-tian.com/talks/rsi_work…. Thanks for promoting my book as well 😄

@kevin_zakka @pabbeel congrats Kevin!!

Gave my PhD dissertation talk on Friday! It's been an incredible journey made possible by the best advisor who believed in me and gave me the freedom and support to explore. Thank you @pabbeel! And thank you to everyone who came to support and share this milestone with me 🙏

@tongzhou_mu We just say the same words🙂

this is my dream

My dream of robot learning research.

Teaching a robot a new task typically means stopping operations, collecting teleoperated demonstrations, and retraining. That process takes hours at a minimum. We wanted to know if we could collapse it to seconds — from a single human demo, on the fly, no retraining required. Early research preview: we can.

Introducing LDA, a latent world action foundation model that, for the first time, unifies the utilization of heterogeneous embodied data across simulation and reality, humans and robots, and varying levels of action quality and annotation. By breaking long-standing data silos in embodied intelligence, LDA enables the field, much like GPT did for language, to benefit continuously from scaling data, marking the transition into a new era of scalable learning. #Galbot #Robotics #Innovation #AI #Technology #Humanoid #WorldModel

@TX_Leo_Wang Thanks, Leo!

That’s real research! Great work