Jianfei Yang
141 posts
Jianfei Yang
@Jianfei_AI
Assistant Professor @NTUsg Prev Researcher @Harvard @UCBerkeley @UTokyo_News U30 @Forbes
Singapore Katılım Şubat 2024
223 Takip Edilen1.2K Takipçiler
𝐎𝐧𝐞 𝐨𝐟 𝐭𝐡𝐞 𝐦𝐨𝐬𝐭 𝐞𝐱𝐜𝐢𝐭𝐢𝐧𝐠 𝐬𝐡𝐢𝐟𝐭𝐬 𝐢𝐧 𝐫𝐨𝐛𝐨𝐭𝐢𝐜𝐬 𝐫𝐢𝐠𝐡𝐭 𝐧𝐨𝐰 𝐢𝐬 𝐭𝐡𝐚𝐭 𝐫𝐨𝐛𝐨𝐭𝐬 𝐚𝐫𝐞 𝐥𝐞𝐚𝐫𝐧𝐢𝐧𝐠 𝐧𝐨𝐭 𝐨𝐧𝐥𝐲 𝐟𝐫𝐨𝐦 𝐝𝐚𝐭𝐚, 𝐛𝐮𝐭 𝐚𝐥𝐬𝐨 𝐟𝐫𝐨𝐦 “𝐢𝐦𝐚𝐠𝐢𝐧𝐞𝐝” 𝐟𝐮𝐭𝐮𝐫𝐞𝐬.
✨𝐖𝐨𝐫𝐥𝐝 𝐦𝐨𝐝𝐞𝐥𝐬 𝐚𝐫𝐞 𝐦𝐚𝐤𝐢𝐧𝐠 𝐭𝐡𝐢𝐬 𝐩𝐨𝐬𝐬𝐢𝐛𝐥𝐞 𝐚𝐧𝐝 𝐭𝐡𝐞 𝐟𝐢𝐞𝐥𝐝 𝐢𝐬 𝐦𝐨𝐯𝐢𝐧𝐠 𝐢𝐧𝐜𝐫𝐞𝐝𝐢𝐛𝐥𝐲 𝐟𝐚𝐬𝐭.
World models, predictive representations of how environments evolve under actions, are quickly becoming one of the central building blocks of modern robotics. They allow robots not only to act, but also to imagine, predict, plan, simulate, and evaluate future outcomes before taking actions in the real world.
What makes this field especially exciting is how rapidly it is evolving. In just a short time, we have seen the rise of foundation-scale robotic video generation, controllable simulation, learned physics, and world-guided robot policies. But at the same time, the literature has become highly fragmented across architectures, paradigms, and embodied applications.
To help the community keep up, our MARS lab organized and led a comprehensive survey together with an amazing group of researchers, including @HaoranGeng2 , @ZeYanjie, @pabbeel, @JitendraMalikCV, @jiajunwu_cs, @du_yilun, @liuzhuang1234, @mapo1 , @philiptorr , @oier_mees Tatsuya Harada, across @UCBerkeley @Stanford, @Harvard @Princeton @ETH @UniofOxford @UTokyo_News @MSFTResearch.
The survey reviews how world models are used for robot policy learning, planning, reinforcement learning, simulation, navigation, autonomous driving, and large-scale embodied video generation, while also summarizing datasets, benchmarks, evaluation protocols, and future research directions.
📖 “World Model for Robot Learning: A Comprehensive Survey”
Paper: arxiv.org/abs/2605.00080
Project: ntumars.github.io/wm-robot-surve…
Updated Github: github.com/NTUMARS/Awesom…
We will also continuously maintain the repository to keep track of newly emerging papers, benchmarks, and resources for the community.
#EmbodiedAI #RobotLearning #WorldModel #PhysicalAI #Robotics #FoundationModels
English
𝐀𝐟𝐭𝐞𝐫 𝐕𝐋𝐀𝐬, 𝐰𝐨𝐫𝐥𝐝 𝐦𝐨𝐝𝐞𝐥𝐬 𝐚𝐫𝐞 𝐛𝐞𝐜𝐨𝐦𝐢𝐧𝐠 𝐭𝐡𝐞 𝐧𝐞𝐱𝐭 𝐛𝐢𝐠 𝐭𝐡𝐢𝐧𝐠 𝐢𝐧 𝐫𝐨𝐛𝐨𝐭 𝐥𝐞𝐚𝐫𝐧𝐢𝐧𝐠 — 𝐚𝐧𝐝 𝐭𝐡𝐞 𝐩𝐚𝐜𝐞 𝐢𝐬 𝐛𝐫𝐞𝐚𝐭𝐡𝐭𝐚𝐤𝐢𝐧𝐠 🚀 𝐒𝐨 𝐰𝐞 𝐰𝐫𝐨𝐭𝐞 𝐚 𝐬𝐮𝐫𝐯𝐞𝐲.
World models, predictive representations of how environments evolve under actions, have become one of the most important building blocks in modern robot learning. They power policy learning, planning, simulation, evaluation and data generation. And with the advent of large-scale generative video models, the field is moving faster than ever.
To help the community keep up, we wrote a comprehensive survey together with @pabbeel, @JitendraMalikCV, @jiajunwu_cs, @du_yilun, @mapo1, @philiptorr, @Jianfei_AI and many others 📖
"World Model for Robot Learning: A Comprehensive Survey"
Paper: arxiv.org/pdf/2605.00080
Project: ntumars.github.io/wm-robot-surve…
@UCBerkeley @Stanford @Harvard @ETH @Microsoft @UniofOxford @NTUsg
English
@AvivTamar1 Yeah deterministic policy via diffusion is interesting!
English
@Jianfei_AI Nice! We actually had a rejected ICLR paper with a similar idea some time ago:
openreview.net/forum?id=iKd99…
but didn't follow up unfortunately. Happy to see this direction is picking up!
English
Excited to share a piece of work that I'm personally very proud of 👇
Our paper "Action-to-Action Flow Matching (A2A)" has been accepted to RSS-2026.
What's the idea?
Instead of generating robot actions from random noise (slow), we start from past actions and directly map to the next one via flow matching.
Result:
⚡ single-step inference
⚡ great success rate
⚡ closer to real-world control speed
From diffusion-style "slow thinking" → to instant action. Very excited about this step toward execution-speed embodied intelligence.
🔗 Project page: lorenzo-0-0.github.io/A2A_Flow_Match…
🔗 Paper link: arxiv.org/abs/2602.07322
English
@siddancha Super cool! Let me read your work carefully! Would love to meet up during RSS’26 if you attend!
English
Really cool work, and congrats on RSS'26! 👏 You might find our recent work from CoRL'25 relevant: x.com/siddancha/stat… (streaming-flow-policy.github.io) . SFP is also an "action-to-action" flow matching policy which treats the **action trajectory as the flow trajectory**, in contrast to your work that does "action-chunk-to-action-chunk" denoising. But like Sec. 4.3.2 of your paper, we also add a small amount of Gaussian noise to past actions!
Also keep an eye out for a neat SDE-based generalization of SFP from @haroldsoh also at RSS'26!
Siddharth Ancha@siddancha
Diffusion/flow policies 🤖 sample a “trajectory of trajectories” — a diffusion/flow trajectory of action trajectories. Seems wasteful? Presenting Streaming Flow Policy that simplifies and speeds up diffusion/flow policies by treating action trajectories as flow trajectories! 🌐 streaming-flow-policy.github.io 🧵 1/15
English
Jianfei Yang retweetledi
One static model does not fit all😭
We just dropped our latest work: Functional Neural Memory. Instead of static models, we generate custom "parameters" for every single input.
✅Prompt your model anytime
✅Instant personalization
✅Better instruction following
✅Flexible & dynamic memory (w/o memory bank✌️)
(🧵1/6)
English
Thrilled to share that our NTU MARS Lab paper “RM-RL: Role-Model Reinforcement Learning for Precise Robot Manipulation” has been accepted to ICRA 2026 @ieee_ras_icra 🎉🤖
🔗 Project Page: lnkd.in/gRK-TmvX
📄 Paper: arxiv.org/abs/2510.15189
“When three people walk together, there must be a role model whom I can learn from.”(三人行，必有我师焉)
—— Confucius, The Analects
Inspired by this wisdom, we asked:
Can a robot learn from its own “role model” — without relying on costly human demonstrations?
In high-precision tasks (e.g., millimeter-level cell plate placement), traditional RL is data-hungry and unstable in the real world. Our Role-Model RL (RM-RL) introduces a simple but powerful idea:
✅ During online interaction, select the best action under similar states as a role model
✅ Automatically label peer samples
✅ Reuse them in offline supervised updates
✅ Unify online exploration + offline efficiency
The results in real-world experiments are exciting:
• 🔹 53% improvement in translation accuracy
• 🔹 20% improvement in rotation accuracy
• 🔹 100% success rate in precise shelf placement (with pretraining)
• 🔹 Faster and more stable convergence than standard RL
No human teleoperation. No massive dataset collection.
Just structured self-improvement, guided by the best example in the room.
English
Thrilled to co-organize ScaleBot @CVPR 🤖🚀
We warmly welcome the community to join the first workshop on Scalable Robot Learning Systems. It’s especially exciting to have such an outstanding group of speakers:
• Joel Jang (Nvidia GEAR) @jang_yoel
• Sergey Levine (UC Berkeley & Physical Intelligence) @svlevine
• Jason Ma (Dyna Robotics) @JasonMa2020
• Chuan Wen (Shanghai Jiao Tong University) @ChuanWen15
Looking forward to lively discussions, new collaborations, and seeing your submissions in Denver! 🙌
Sijin Chen (CH3COOK)@ch3cook_csj
📢 CVPR 2026 Workshop Call for Papers: ScaleBot @CVPR ! 🤖 Join the FIRST Workshop on Scalable Robot Learning Systems at #CVPR2026 in Denver, June 3/4! We’re bringing together researchers/engineers from CV, NLP, robotics & beyond to build scalable learning systems for general-purpose robots. Let’s unlock real-world robot generalization! 🚀 🔗 Website url: scalebot-workshop.github.io 🌟 Keynote Speakers (Tentative): • Joel Jang (Nvidia GEAR) @jang_yoel • Sergey Levine (UC Berkeley & Physical Intelligence) @svlevine • Jason Ma (Dyna Robotics) @JasonMa2020 • Chuan Wen (Shanghai Jiao Tong University) @ChuanWen15 📌 Topics We Love (not limited to!): • Robot data acquisition/strategies; • Data pyramids; • VLMs/VLAs; • World models; • Dual-system architectures; • Fair evaluations & more. ⏰ Two Submission Tracks - Don’t Miss Out! ✅ Track 1 (Proceedings): Original research | DDL: March 1, 2026 (AoE) ✅ Track 2 (Non-Proceedings): WIPs, datasets, tech reports, recent work | DDL: April 14, 2026 (AoE) 📝 Submit via OpenReview, see scalebot-workshop.github.io for more detailed guidelines! 📧 Questions? Reach us at: scalebot@googlegroups.com Retweet to tag robotics peers – let’s accelerate real-world general-purpose robots! 🤖✨ #Robotics #AI #MachineLearning #CVPR #ScaleBot2026 #ScalableAI
English
@yongleluo @iclr_conf @karpathy @cvondrick @zhenyuxue281636 It seems that the bug does not come from ICLR but openreview….
English
I didn't expect @iclr_conf to be so unprofessional. Is this what happens with vibe coding?
@karpathy @cvondrick @zhenyuxue281636
太草台班子了，这次的审稿人估计会睡不好了。
中文
@jd92wang @amazon @GoogleDeepMind @Google It really helped a lot! Hope to have the chance to collaborate in the future. :)
English
@Jianfei_AI @amazon @GoogleDeepMind @Google That's a long time ago! Glad to know that my tutorial helped!
English
Winning these two prestigious grants from @amazon and @GoogleDeepMind @Google in the first year of my career is a true relief. I would like to acknowledge all of my friends and students who provided significant support during this time! Stay tuned and more to come:)
cdsp.wm.edu/about/news-eve…
English
@Ed__Johns Congratulations! I ever made a podcast for the authors regarding the conference version. It’s impressive!
English
I'm very excited to finally announce one of the most ambitious projects we've worked on — which makes the front cover of Science Robotics today:
☀️ Learning a Thousand Tasks in a Day ⭐️
Everyday tasks — like those below — can now be learned from a single demonstration each...
English
I will join Northwestern University Computer Science as an Assistant Professor in Fall 2026!
I am actively recruiting PhD students and seeking collaborations in robotics, human-robot interaction, brain-computer interfaces, cognitive science, societal impact of AI & automation, and AI for art & design.
Please see the recruitment announcement on my personal website, and feel free to reach out!
English
🚀 Excited to share that 3 papers from our NTU MARS Lab have been accepted to the top AI conference, AAAI-26 @RealAAAI , advancing the frontier of multimodal embodied AI!
1️⃣ Mask2IV introduces an interaction-centric video generation framework that serves as a world-model engine for robot learning, producing controllable human-object and robot-object interaction videos without dense annotations.
2️⃣ ZOMG enables zero-shot, open-vocabulary human motion grounding, automatically decomposing motion sequences into semantically meaningful sub-actions without labels, paving the way for scalable, annotation-free motion understanding.
3️⃣ mmPred pioneers radar-based human motion prediction in the dark, leveraging a diffusion-based architecture to achieve robust and privacy-preserving perception for challenging low-light and occluded environments, which lays a foundation for robotic perception at private homes.
Very happy that AAAI-26 will be held in Singapore, and we warmly welcome everyone to visit NTU’s MARS Lab when you’re here!
English
Agreed... I don’t believe the “general-purpose robot” will have a sudden “ChatGPT moment”.
Nils@broodsugar
TL;DR: The Chinese are building programmable robots at scale already, and are looking for distributors to help them get the robots ready for specific tasks. They sacrifice some of the upside and share some of the risk. They wish to sell the robot to distributors, today. The Americans are trying to leapfrog the era of programmable robots by going straight for general intelligence robots that can do any job well without any integration. They want to do all hardware and software inhouse, and handle distribution themselves. They wish to rent the robot directly to the end customer, tomorrow.
English
What do students and startup teams need these days to do humanoid robotics research and GTM, apart from solid AI and control?
Apparently… filmmaking skills! 🎥😂
Behind the scenes of our crew filming in the test arena: keeping a safe distance from the robot while debating camera angles like Spielberg.
Maybe it’s time to add “Cinematography for Robotics” to the syllabus next semester 😆
English