Aurel Arnold

@metrox_eth @SHOW_ROBOTICS Glad it was helpful, keep us posted!

@aurel_arnold @SHOW_ROBOTICS Thanks, your setup notes saved me. Without them I'd still be dragging the leader through honey 😅 Script's done, button not wired yet, testing later today.

Update from the @SHOW_ROBOTICS workshop - Basic VR teleoperation unlocked, still need to implement 5 sec wrist calibration (@aurel_arnold technique) - Path to first ACT stacking demo is now clear: leader-follower teleop + VR teleop + synthetic data in MuJoCo + DAgger correction at the end - Still waiting on Dynamixel servos to build the Koch 1.1 leader and compare demo quality - RSI loop now has a name, and no it's not Jarvis 😁. It's Iris (Iterative Recursive Improvement System), talks with a neural voice when I'm in the headset

First day at the factory, we arrived at 6 Pm and spent the evening set up our workstations + arms and then talking to the owner. Some thoughts: - Didn’t think to ask whether they have wifi. They actually do and it’s 150mbps download. We shouldn’t assume this is representative. - We noticed that we were missing a USB stick and a HDMI cable. Bought both on the way, but should create a full checklist for next time. - We could save travel time if we leave a workstation + arms with a company. Working with a company closer to Zurich would also solve this (just put everything in a car), but we have a special relation with this company which makes it worth. For example: we were completely alone at the factory. This was only possible because the owner trusts us. That gave us extra time we wouldn’t have had somewhere else. - We‘re in the center of the shop floor. Will be fun talking to all employees walking past our workstations tomorrow.

We‘re off to Poland where we‘ll be spending the week working from an electronic factory floor. With us: two 5090 workstations, two bimanual arm setups, a Meta Quest, spare 3D parts and tools.

@jaikr_x Not directly. I wrote the WebXR page + IK myself, using mujoco and LeRobot as OS building blocks.

@aurel_arnold Did you use something OS for VR teleoperation?

Collected 3h of data in ~5h wall time with VR teleop today. Honestly feels way better than leader arms. There's never anything in your way even with weird motions, and being able to leave an arm frozen when not in use is huge. What do you think of the setup? Tried to make it as comfortable as possible ;)

I use WebXR in the Quest’s built-in browser, which gives you controller poses you can stream over a websocket to your host. Then you need IK to map the controller poses to joint angles. If it’s a 6-DoF arm + gripper, it makes sense to separate the joints for position and orientation.

@aurel_arnold I must try this, I have Quest 2 and Quest pro, how do you interface it ?

@a_andlyu For VR, implementing IK to work reliably took me the longest

@aurel_arnold Whats the difficulty to set it up?

@ck_thrillseeker Felt quite good but concentration dropped drastically at around hour 4. For sessions this long, I need to take more regular breaks next time.

@aurel_arnold For me, freezing an arm freely is the real win. Moving naturally without clutter beats leader arms hands down. How does it feel over longer sessions?

@max_pe2002 What do you mean by that?

@aurel_arnold thats because most of robotics research use leader arms wrong

@iamRezaSayar Fair point but my neck wasn’t too bad, my back felt way worse!

@aurel_arnold looks nice! wasn't your neck feeling uncomfortable after 5 hours of this?

@nurvai_ai Thanks! Will do when start working on UMI integration.

@aurel_arnold Looks nice. UMI-style data collection definitely speeds things up. In our experience, reliability depends a lot on the gripper setup. Some grippers can tire operators out quickly. Feel free to message us if you want to compare notes.

It’s quite impressive what's possible with teleop, but I imagine UMI-style data collection would make this so much easier and faster. Does anyone have experience with UMI on parallel-jaw grippers? Any tips welcome!

@Moky40685179 @abhishek_viv Keep us updated, would be very interesting!

@abhishek_viv @aurel_arnold yes i am considering opensourse all my works!

Yes, that’s cardboard holding up our camera there. A new stand should arrive soon!

Small sidequest: The faucet in our office had a broken part. I grabbed the only measuring tool available (a tape measure) and eyeballed the dimensions, modelled and printed it. Every dimension right on the first try. Faucet working again!

@jonastbrg Need to look into that, thanks!

We are using IsaacTeleop github.com/NVIDIA/IsaacTe… for that, works quite well with the quest, I'll PR our setup soon with Keyboard interactions as well, to update the policy_runner.py in IsaacLab-Arena you just need to add a tiny stop condition. huggingface.co/docs/lerobot/h… Is also cool!

Now I know why it didn’t feel natural. Two issues: 1. DLS over all 6 joints couples position and orientation. Wrist motions lead to elbow swings. 2. WebXR reports controller pose somewhere at the palm, but you naturally rotate around your wrist. Fixed with Pieper-style decoupling + a 5-sec in-VR wrist-twist calibration.

@cyrux004 @DominiqueCAPaul I started by reading the controller pose with WebXR in the browser and streamed it to my workstation. Then you need IK, since you don’t get joint angles like with the leader.

@DominiqueCAPaul @aurel_arnold I have a leader-follower setup and thinking about moving to bimanual setup (with weaker leader motors like you mentioned). I have the original Quest. whats the easiest way to start on teleoperation

We‘re gonna be doing DAgger style data collection soon. But the leader arms can’t carry their own weight and would need to follow the leaders at all time. That feels messy, so @aurel_arnold implemented a teleop setup with the Quest in the last 1.5 days. First configuration is working now, but not feeling 100% natural yet. Hard to describe why.

@davehpitt This should help with general smoothness of the policy in my understanding, but how about the jumps when handing over control back to the policy?

@aurel_arnold Cool stuff. You should try train-time real-time chunking if you haven't! Helps a lot with smoothness

DAgger-style intervention now works with the Quest. I sometimes get ugly “jumps” when handing over control to the policy though. Caused by OOD states the ACT policy hasn’t learned smooth actions for yet?

@PrimeErwin I wouldn’t call it much experience but I went with WebXR on the Meta Quest (browser), host on the same LAN with a self-signed cert, FastAPI server, mujoco for IK with Pieper-style decoupling.

@aurel_arnold What's the best stack for VR teleop from your experience?