DenTechs

@ThatRyth both at head level, ~90 degrees apart.

@TRDenTechs what's the best setup and placement for two v2s? this has me very intrigued to buy into but unsure if I have the room for two and no idea how to go at setup 🤔

finally time to tease this… i've spent months building a brand new app for full body tracking in VRChat using Kinects or iPhones with custom ML models. steam page dropping soon😉

@xerxesses @yacineMTB To my knowledge all of the existing Kinect for VR apps use the built in Kinect sdk for tracking, not custom algorithms. Take a look at my follow up tweet for a comparison

@TRDenTechs @yacineMTB Wasn’t this already done before? Still great work though

@JEthenoobgamer yeah, amethyst/driver4vr/etc use the built in kinect sdk tracking which is quite old, they dont have their own tracking algorithms.

@TRDenTechs So... The quality was mostly cause of software? I remember amethyst being decent (ish) as long as you stayed within view, this looks a lot better on the legs. Nice Will probably try it out when it's out.

I wasn't planning on showing this yet but the response to the last post was incredible, so here's a side by side with Amethyst / Driver4VR's tracking in VRchat. Steam page coming soon😄

@burricoad dual kinect is better at handling occlusion, but outside of that the tracking quality is roughly the same

@TRDenTechs Using kinect v2 1x is it better or worse than this?

@yacineMTB i have a synthetic data pipeline for depth maps using blender. synthetic data didn't work well for RGB pose detection as its hard to get enough variety, so im using real datasets for that.

@TRDenTechs Are you training models for this? This honestly seems like, very solvable with the set up you have

@Spaghardo Fully local. No required data collection.

@TRDenTechs Does it upload data to a server or work natively offline?

@Starlitblitzz This is just the 3d preview, the actual app works with SteamVR like any other FBT solution. More demos to come soon

@TRDenTechs Out of curiosity, how does this software work?? I can’t get myself to figure it out somehow… regardless, it’s neat asf!! As someone who primarily uses the Kinect360 with Amethyst, I just want to figure this out so I can be prepared whenever this program comes out!!

@WitoTV I'm not sure my approach would be ideal for that, but I do have some friends working on optitrack-like projects so you may be more interested in those. (Notably Seneral's aster track and i3llamas wip project)

@TRDenTechs Yea yea, Its mostly if one were to train a model to just look for that specific object and its properties (rotation, position, etc, etc). Think of it as using the camera output and feeing it to separate model. World is kind of craving for a open source Vicon like solution...

@WitoTV Not with the current model architecture, it's specifically built for pose estimation

@TRDenTechs Do you think it would be possible to do object tracking as well in the future?

@ByteFuserUK The tracking latency itself is fairly low, under 50ms, but due to the inherent noisiness of ML tracking it needs pretty strong filtering to reduce jitter. I'm still tuning it to balance smoothness vs latency.

@TRDenTechs Proper impressed - I've been waiting for someone to crack decent FBT without those bloody Vive pucks. Point cloud density looks solid. What's the latency like?

@Brucebuypower I do plan on adding BVH recording/exporting. I'm not sure if it'll be done for the initial release, but it is on my radar.

@TRDenTechs Can this also be used for animation data like to put into something like Blender?

@zmiennys @LhymeFruit It's a set of custom trained models based on recent research papers. As far as I know, nothing like this exists off the shelf.

@TRDenTechs @LhymeFruit Can you share which tracking ML did you use? Are there MediaPipe equivalent but supporting depth data?

@XRTerra_ It should work on Intel GPUs but I dont have one to test with. Also, because i'm a solo dev and dont use linux much i dont plan on supporting it officially.

@TRDenTechs Sick, though I'd assume anyone remotely interested in this has at least a 3070 tier GPU already so it shouldn't be too bad. Also, Intel? Will this also be cross platform? A lot of Linux furries have insanely beefy computers but might not have full body tracking gear yet

@XRTerra_ Yes, TensorRT for rtx 30 series and up, ONNX for all other gpus (yes AMD is supported). The gpu usage is fairly noticable, but there are options for lower resolution or smaller ML models for lower end PCs.

@TRDenTechs Assuming this is GPU accelerated?

@LhymeFruit that's because all existing "kinect for vr" apps use the Kinect SDK for tracking, which uses old and poor performing tracking techniques. the actual sensor quality of the V2 is much higher.

@TRDenTechs That's strange because I've had better experiences with a 360 on all the other tracking software for the Kinect

@wicho_scruffy the tracking will be more accurate and handle more varied poses such as crossed legs, sitting, laying down, etc at the cost of more GPU usage.

@TRDenTechs Good to know thanks! I'm currently using my 360 Kinect with amethyst Is there a benefit to using your solution compared to amethyst?

@LhymeFruit If you have both I would recommend using the kinect V2 as it performs better

@TRDenTechs No Kinect v2 and Kinect 360? Damn I'll just use the 360 then

@sparkie237 supprted devices are: 1x kinect v2 2x kinect v2 1x kinect 360 1x iphone

@TRDenTechs kinect azure?

@wicho_scruffy supprted devices are: 1x kinect v2 2x kinect v2 1x kinect 360 1x iphone

@TRDenTechs I have an og kinect how many would I need?