Yang Fu

Check out our recent work 𝐒𝐑-𝟑𝐃: a unified framework that brings 𝟑𝐃 𝐬𝐩𝐚𝐭𝐢𝐚𝐥 𝐫𝐞𝐚𝐬𝐨𝐧𝐢𝐧𝐠 and 𝐫𝐞𝐠𝐢𝐨𝐧-𝐥𝐞𝐯𝐞𝐥 𝐩𝐫𝐨𝐦𝐩𝐭𝐢𝐧𝐠 into VLMs for powerful cross-view understanding!

How do we make dexterous hands handle both power and precision tasks with ease? 🫳👌🫰 We introduce Power to Precision (💪➡️🎯), our new paper that optimizes both control and fingertip geometry to unlock robust manipulation from power grasp to fine-grained manipulations. With simplified finger motions and augmented fingertips, the hand can perform diverse motions from pinching a nut🔩 to handling a pan🍳. Check the demos below🎥.

@hanwenjiang1 Huge congrats!

🏆 RayZer is selected as a BEST PAPER CANDIDATE @ICCVConference ! I will present RayZer at Wild3D workshop on Monday morning and the oral session on Tuesday afternoon

Test-Time Training (TTT) is available in video generation now! We can directly generate complete one-minute video, with great temporal and spatial coherence. We created more episodes of Tom and Jerry (my favorite cartoon in childhood) with our model. test-time-training.github.io/video-dit/ Enjoy more in 🧵

[1/n] We are releasing M3 (#ICLR2025): a Gaussian Splatting method that builds LMM memories for arbitrary scenes. 🔥 [Efficient] 16 degrees in each Gaussian primitive for one LMM. 🔥 [Alignment] The rendered features are directly in the source LMM embedding space.

Meet our first general-purpose robot at @DexmateAI dexmate.ai/vega Adjustable height from 0.66m to 2.2m: compact enough for an SUV, tall enough to reach those impossible high shelves. Powerful dual arms (15lbs payload each) and omni-directional mobility for ultimate versatility. More videos showcasing Vega in action coming this week! Stay tuned to see this robot partner.

Can we bring human-like Touch to robots🤖? Introducing our CoRL work on 3D-ViTac. Humans rely on both vision 👁️ and touch 🫳 for complex tasks. With combined visual-tactile sensing, robots can now tackle challenging tasks, like precise in-hand reorientation, fragile objects grasping. Website: binghao-huang.github.io/3D-ViTac/ #Robotics #CoRL2024 #Touch #tactile #AI #ML

Very excited to get this out: “DVT: Denoising Vision Transformers”. We've identified and combated those annoying positional patterns in many ViTs. Our approach denoises them, achieving SOTA results and stunning visualizations! Learn more on our website: jiawei-yang.github.io/DenoisingViT/

TTT could model long sequences with linear time complexity. It's a drop-in upgrade for any sequence modeling operators like self-attention. It has been super fun to work on TTT with the amazing team! Code is available: github.com/test-time-trai…

🌟Introducing "🤖SpatialRGPT: Grounded Spatial Reasoning in Vision Language Model" anjiecheng.me/SpatialRGPT SpatialRGPT is a powerful region-level VLM that can understand both 2D and 3D spatial arrangements. It can process any region proposal (e.g., boxes or masks) and provide answers to complex spatial reasoning questions. 🧵(1/n)

We introduce🌟Editable Image Elements🥳, a new disentangled and controllable latent space for diffusion models, that allows for various image editing operations (e.g., move, resize,  de-occlusion, object removal, variations, composition) jitengmu.github.io/Editable_Image… More details🧵👇

Want to obtain time-consistent dynamic mesh from monocular videos? Introducing: Dynamic Gaussians Mesh: Consistent Mesh Reconstruction from Monocular Videos liuisabella.com/DG-Mesh/ We reconstruct meshes with flexible topology change and build the corresp. across meshes. 🧵(1/n)

I have been cleaning my daughter's mess for more than two years now. Last weekend our robot came to home to do the job for me. 🤖 wholebody-b1.github.io Our new work on visual whole-body control learns a policy to coordinate the robot legs and arms for mobile manipulation. See how the legs bend on grasping objects on the ground based on visual inputs autonomously. (It is NOT teleoperation!) We again adopt a Sim2Real approach and our policy generalizes to beaches, forests, and streets in San Diego. 👇🧵

If you look at the best image/video diffusion models, they are still not able to get the hands quite right, especially when interacting with objects ✍️ We present HOIDiffusion #CVPR2024, a way to generate accurate and realistic hand-object interaction images, in diverse poses, scenes, and styles. mq-zhang1.github.io/HOIDiffusion/ This is not just for synthesizing beautiful pixels, but also a way to generate infinite pairs of hand-object images and 3D GTs data.

Thrilled to see how generalizable NeRFs are helping real robots understand their surroundings. Check our latest project, GeFF, which enhances reconstructions from our ICLR24 work ScenePriors (oasisyang.github.io/neural-prior/) with feature fields distilling from 2D foundation models.

We have seen a lot of legged robots doing navigation in the wild. But how about mobile manipulation in the wild? I have been pushing the direction of learning a unified, efficient, and dynamic 3D representation of scenes (for navigation) and objects (for manipulation) for the past two years. And now we have GeFF --- our large-scale, generalizable feature field, that combines the speed of a feed-forward neural network with the rich semantics from Foundation Models, to handle dynamically changing scenes, and enable open-ended, language-grounded scene and object understanding. geff-b1.github.io

@bowenc0221 @Tesla @OpenAI Congrats!

Today is my last day @Tesla autopilot. It was a pleasant one and a half years: working with talented people, learning how to build good product, etc. But my journey to AGI will not stop, I will soon join @OpenAI post-training team to build multimodal models.

Let’s think about humanoid robots outside carrying the box. How about having the humanoid come out the door, interact with humans, and even dance? Introducing Expressive Whole-Body Control for Humanoid Robots: expressive-humanoid.github.io See how our robot performs rich, diverse, and expressive motions in the real world 👇🧵

Check our COLMAP-Free 3D Gaussian Splatting results on Sora videos. It seems we have an opportunity to get infinite 3D data.

Tired of seeing synthetic objects go round and round, round and round, round and round? Introducing the WildRGB-D dataset! We collect a dataset of real-world RGB-D objects in 360 under cluttered scenes. To download: github.com/wildrgbd/wildr… Website: wildrgbd.github.io ✅ It comes with GT depth, GT camera pose, and 360 views. ✅ A lot of objects : 20k videos, 8.5k objects, 46 classes. ✅ It is cluttered, there is no clean background. ✅ We have single-object, multi-object, and HAND-Object videos. ✅ Supporting multiple downstream tasks: camera pose estimation, view synthesis, 3D reconstruction, 6D pose estimation.