Jad Abou-Chakra

1/ Real2Sim and Sim2Real are well-known paradigms in robotics🤖. Is there another paradigm where sim and the real world are more tightly interlinked? We introduce Real-IS-Sim: a framework featuring an always-in-the-loop, correctable simulator for behavior cloning policies.

Grateful to @NVIDIA for featuring our work on their Tech Blog: Physically Embodied Gaussians, an interpretable real time world model for #robotics using Warp and #GaussianSplatting. developer.nvidia.com/blog/building-… @krshnrana @nikoSuenderhauf @QUTRobotics

@siddancha Thats right

@jadachakra Makes sense! So you wouldn't expect this shaking in offline mode when there are no visual corrections, correct?

1/ Real2Sim and Sim2Real are well-known paradigms in robotics🤖. Is there another paradigm where sim and the real world are more tightly interlinked? We introduce Real-IS-Sim: a framework featuring an always-in-the-loop, correctable simulator for behavior cloning policies.

@siddancha Great question! Two reasons: 1. The cameras are not perfectly calibrated - therefore the tblock’s optimal position that would reduce the photometric loss is always contested. 2. Visual forces are somewhat naively implemented and cause oscillations because there is no damping.

@jadachakra @jadachakra Really cool and exciting work!!👏👏 One question: in the bottom right corner of this video, the block seems to shake in-place even when the robot is not touching it. Do you know why that is? realissim.rai-inst.com/static/videos/…

🙏 Supported by @rai_inst #NeuralRendering #PhysicsSim #Robotics #ComputerVision #GaussianSplatting

🚀 Embodied Gaussians is now opensource! 🔥 📂 GitHub: github.com/bdaiinstitute/… ✨ Key Features: 1️⃣ Gaussian Splatting (GSplat) + Physics (Warp) 💥 2️⃣ Parallelizable environments! ⚡ 3️⃣ Render virtual cameras in parallel 📸 🔜 Coming Soon (hopefully): 🌀 Deformables!

6/ Videos and paper are available at realissim.rai-inst.com and arxiv.org/abs/2504.03597

5/ This project was completed during my internship at @rai_inst (formerly Boston Dynamics AI). Special thanks to my team—Lingfeng Sun, @krshnrana, Brandon May, Karl Schmeckpeper, Maria Vittoria Minniti, and Laura Herlant for their support during this project.

@qunomaan Thank you @qunomaan :) The artifacts are largely due to the use of only 5 viewpoints during initialization. Parts of the objects may be unobserved at the start. A better initialization procedure and a means of correcting modeling errors on the fly are on our future hitlist.

@jadachakra Very cool work! We are working on something similar. Is there any particular reason the predicted visual states have a lot of artifacts?

1/ What is a good representation of the physical world for a robot🤖? We believe it is one that is: initialized easily, 3D, physically constrained, correctable with new observations (in realtime), and capable of being forward simulated (faster than realtime).

5/ More videos and details are available at the project page (embodied-gaussians.github.io) and in the paper (arxiv.org/abs/2406.10788). A special thanks to @krshnrana, Feras Dayoub, @nikoSuenderhauf, and @QUTRobotics for their support during this project.

4/ Particles collide with each other and are affected by external forces and kinematic constraints. The particles transport their linked Gaussians along with them. The system runs in realtime @ 30Hz with only 3 cameras. It works on both deformables and rigid objects.

📌How can we scale imitation learning to long-horizon, multi-object tasks & generalise (spatial & intra-category) from only 10 demos? ✨Introducing Affordance-Centric Policy Decomposition 💡Relative task-frame diffusion policies that self-chain for seamless operation 🧵👇