Michael Hoffman

Can GPT-4 teach a robot hand to do pen spinning tricks better than you do? I'm excited to announce Eureka, an open-ended agent that designs reward functions for robot dexterity at super-human level. It’s like Voyager in the space of a physics simulator API! Eureka bridges the gap between high-level reasoning (coding) and low-level motor control. It is a “hybrid-gradient architecture”: a black box, inference-only LLM instructs a white box, learnable neural network. The outer loop runs GPT-4 to refine the reward function (gradient-free), while the inner loop runs reinforcement learning to train a robot controller (gradient-based). We are able to scale up Eureka thanks to IsaacGym, a GPU-accelerated physics simulator that speeds up reality by 1000x. On a benchmark suite of 29 tasks across 10 robots, Eureka rewards outperform expert human-written ones on 83% of the tasks by 52% improvement margin on average. We are surprised that Eureka is able to learn pen spinning tricks, which are very difficult even for CGI artists to animate frame by frame! Eureka also enables a new form of in-context RLHF, which is able to incorporate a human operator’s feedback in natural language to steer and align the reward functions. It can serve as a powerful co-pilot for robot engineers to design sophisticated motor behaviors. As usual, we open-source everything! Welcome you all to check out our video gallery and try the codebase today: eureka-research.github.io Paper: arxiv.org/abs/2310.12931 Code: github.com/eureka-researc… Deep dive with me: 🧵

Unity quietly released a new hands package that introduces a hands subsystem enabling native hand tracking via OpenXR in vanilla Unity. It also fully supports Meta’s aim extension that adds far rays and pinch. The sample hand mesh is getting fixed soon and xri samples are coming

some progress on SunflowerOS — making the VRChat mirror real!! This is through the lens on Magic Leap 2 w/ StereoKit.

I've been noodling with ReadyPlayerMe avatars in StereoKit, and sometimes coding is truly delightful :D I'm... guessing I nailed the head position part! Welcome to how I normally see the world I guess?

ICYMI: Start your on-demand #MRDevDays viewing journey with @its_graceh and @3DBrent introducing #MRTK3. msft.it/6013bAE19

Excited to announce the first update to the MRTK3 public preview. We're here to make MRTK better, faster, and cleaner, and chew bubble gum. And we're all out of bubble gum. Come join us! #mrtk #mrtk3 github.com/microsoft/Mixe…

Wait...switching from testing my Unity app on a HoloLens 2 vs. a Quest 2 is now as easy as just plugging in a different headset and hitting play? Awesome! youtu.be/LI6lyW9TG9o?t=… I used to get lost trying to remember all the config changes... #mrtk3 #openxr #mrdevdays

Welcome to the world MRTK3, & welcome to Twitter MRTK PM Grace Hsu! Action items: 1. Download MRTK3 (linked here👇) 2. Follow Grace @its_graceh 3. Check out more MR Dev Days 🧵 docs.microsoft.com/en-us/windows/…

Nailing down async texture load priority, diffuse first, then the others! This is ~200mb of png decoding.

Happy birthday #HoloLens. Today you are 5 years old [)-)

Microsoft announces HoloLens 2 linkedin.com/pulse/microsof…

Forth Dimension Displays and VividQ Collaborate on Holographic 3D Mixed Reality Headset that offers true depth perception. The article provides a good overview of why this is relevant. lnkd.in/gq8wWvU

What a perfect application of MR, truly adding precision and efficiency to a skilled task. lnkd.in/gBgwZeF

Edge compute - This Qualcomm blog post is the best description I've seen yet of why and how edge cloud/compute + 5G will revolutionize the miniaturization of XR devices. At first, these edge compute devices will be deployed in bus…lnkd.in/gxFqDxD lnkd.in/gsCCZH9

#ar #vr #mr lnkd.in/g4F8Rey

My new novel, FALL, OR DODGE IN HELL, goes on sale in June 2019 in the USA. Info on tour & signed copies to follow. In the meantime, here’s the link to pre-order and learn more. harpercollins.com/9780062458711/…

Army orders 100,000 Microsoft HoloLens MR devices! #microsoft #army #mixedreality #hololens #mr #xr #objecttheory lnkd.in/dQ-4b8B