Daniel DeTone

@yesitsarmin yes, the main limitation is the 2D detector here, but there are tons of better models (SAM3, VLMs) if you have the compute. for very cluttered scenes it doesn't work as well

@ddetone this is very cool, can it work for any arbitrary objects, like stuff on a table, and can it work with stereo camera?

Today we release Boxer, a new lightweight approach that lifts open-world 2D bounding boxes to *metric* 3D: facebookresearch.github.io/boxer/ Here we show Boxer in action on an egocentric sequence captured from smart glasses:

@ElioenaiSiqCst Yes, I didn't show any examples of that but we trained on a massive internal-only Quest3 dataset

@ddetone Works In meta quest?

@BlueAquilae great question! I would not expect it to work well here, we would need to re-train it with a full 9 DoF representation. but feel free to try it out anyway, I'd be curious

@ddetone Would this work in space station since you specified it rely on 'gravity hints' or will the model get lost in orientation?

@CleverBetTips The National?

@ddetone Your name is competing with one of the great albums of the last 30 years 😀

@haodongli00 One limitation I found using both of those models is the runtime. For detecting 1000+ text prompts with SAM3 it takes 20+ sec per image. SAM3D also takes ~15 sec per object, so running on large datasets can be expensive. OWLv2 runs at ~30ms and Boxer takes ~20ms

@ddetone Great work! @ddetone Also doing something very similar, using SAM3, SAM3D and many other powerful tools! 😎

@nickkarpov Feel free to file a GitHub issue if you have any problems! Will do my best to answer them quickly

@ddetone Awesome work, going to use this

For more details, check out the arxiv paper here: arxiv.org/abs/2604.05212

BoxerNet runs FAST 🔥🔥, taking roughly 20ms on a 4090 with bfloat16 for ALL prompts in an image (e.g. 30 boxes in parallel)

Giving Claude new skills is the closest thing I’ve felt to this Matrix moment

Check out our recent work ShapeR, a generative model which gets high quality object 3D meshes from Aria glasses

For those interested in 3D perception, check out the Sonata pre-trained backbone. I don’t think I’ll ever co-author another paper that triples performance (22% -> 72%) on a commonly used benchmark (linear probe scannet 3D sem seg). Released with a permissive license too

📢Sonata: Self-Supervised Learning of Reliable Point Representations📢 Meet Sonata, our"3D-DINO" pre-trained with Point Transformer V3, accepted at #CVPR2025! 🌍: xywu.me/sonata 📦: github.com/facebookresear… 🚀: github.com/Pointcept/Poin… 🔹Semantic-aware and spatial reasoning representations learned with no label; 🔹3x linear probing accuracy (from 21.8% to 72.5%) on ScanNet; 🔹2x data efficiency performance with only 1% of the data compared to previous approaches; 🔹As always, establish new SOTA results across indoor and outdoor 3D perception tasks. Our author team: @HengshuangZhao, @jstraub6, @rapideRobot, @ddetone, @NinjaDuncan, @TianweiS, @Christopher_Xie, @NanYang719.

The first generation of Aria glasses have made a big impact in the research community, can't wait to see all the new possibilities these will unlock meta.com/blog/project-a…

@ZMurez @gazorp5 heres more details on the EVL model you were asking about

cc @ZMurez remember 3D surfaces? :)

What is the right benchmark for a 3D Egocentric Foundation model? We recently open-sourced a small, high quality egocentric benchmark consisting of 1) 3D surfaces 2) 3D objects. We released a simple 3D CNN baseline model called EVL: projectaria.com/research/efm3d/ Try to beat our model!