Daniel DeTone

@weikaih04 @allen_ai @uwcse @AIatMeta Very cool! Is this conditioning on the semi dense depth point cloud or RGB only?

We updated @allen_ai @uwcse 's WildDet3D for end-to-end 3D localization on Project Aria using the great codebase from @ddetone @AIatMeta's Boxer. Same AriaLoader, same CSV schema, same view_fusion / view_tracker - try WildDet3D in Proj Aria and do world coordinates-based detection! Code: github.com/allenai/WildDe…

The biggest impact of the Segment Anything line of work was not the actual image segmentation, but rather the flood of paper titles with the name “Any” in them. Cmon folks, let’s just call this generalization and move on!!

@MattNiessner I agree that metric 3D is critical! It’s a compressed, minimal representation. Playing devils advocate — humans also operate on projections of the 3D world and we are able to operate pretty well

Large foundation models have made enormous progress in modeling language, images, and video. These systems can generate highly realistic outputs and capture complex statistical structure in data. However, they still operate on projections of the world, text sequences and 2D pixel grids, rather than the world itself. The real world is not a sequence of text tokens or frames; the real world is inherently anchored in 3D metric space, and dynamics across time. Objects occupy space and persist over time. They interact according to physical laws. Any model that aims to support real-world intelligence, e.g., for robotics, simulation, design, or spatial computing, must capture this structure. This is where current approaches fall short. While most video models can generate visually plausible frames, they often lack a consistent notion of the underlying scene due to limited context windows. As a result, geometry drifts, scale is ambiguous, objects appear and disappear, and interactions are not physically grounded. The model produces superficial appearance without a persistent world representation. For many downstream applications, this is not enough. The first step toward addressing this is modeling 3D space and keeping it consistent. A model should recover a coherent spatial representation of the scene, including layout, geometry, and scale. This not only allows the environment to be rendered from new viewpoints but also, more critically, reasoned about in metric space. If a model cannot produce a stable 3D representation, it is not grounded in the physical world, and it will fail to model the world due to its inefficient contextual memory. However, 3D is only the beginning. A truly useful world model must also be temporally and physically consistent. It should not only reconstruct a scene, but also simulate it, predicting how it evolves, how objects interact, and what happens under intervention. Eventually this requires moving beyond static representations toward models that capture dynamics and causality. I believe that generative approaches are highly compelling in this context, as they can be trained on large-scale data in a self-supervised fashion. In particular, comprehensive 3D world modeling is a highly-promising path forward, since richer environmental representations directly enable deeper and more effective learning of physical reality. Crucially, such generation enforces consistency: for instance, to generate a scene across viewpoints, a model must implicitly recover its underlying 3D structure. To generate it over time, it must capture its dynamics. This forces the model to internalize the latent state of the world, including geometry, scale, materials, motion, and physical behavior. This also highlights a limitation of purely abstract representations. High-level embeddings or action-centric models can be effective for specific tasks, but without the ability to model and simulate the world, they will eventually remain incomplete. They compress observations, but do not fully model the underlying process that generates them. The next generation of AI systems should therefore move beyond text and pixels, and toward physically-grounded world models: models that represent space, maintain consistency over time, and enable simulation and interaction. This is the missing layer between the physical and digital world, which will ultimately enable AI systems not just to observe the world, but to understand and operate within it.

New blog post about Boxer is live on the Project Aria website

@neural_avb I was there in Barcelona! Epic

My favourite piece of Schmidhuber lore is when he challenged Ian Goodfellow during a NIPS presentation on GANs Live in public Deep Learning drama peaked here. You have seen nothing like this.

@Capsbrr Ah sorry, I thought you meant on Quest cameras, not running on the ML model on Quest hardware. I don’t think this model can run in realtime on Quest. Though it could probably be distilled significantly with further effort and maybe work

@ddetone How does Boxer run in real-time on the Quest 3, when it takes 20 ms to run on a *RTX 4090*? The hardware specs are worlds apart. Im genuinely curious.

Today we release Boxer, a new lightweight approach that lifts open-world 2D bounding boxes to *metric* 3D: facebookresearch.github.io/boxer/ Here we show Boxer in action on an egocentric sequence captured from smart glasses:

@Capsbrr yes

@ddetone I would also be curious: Does it run in real-time on the meta quest 3?

@weikaih04 that would be great! we didn't train on much on outdoor data, I would expect a big boost the WildDet3D dataset training for outdoors

Just check out the boxer model — the latency (20–40ms) and the generalization are also pretty impressive. Huge congrats to Daniel and the other authors! If you're interested in open-world 3D detection for outdoor/in-the-wild scenarios, also check out our WildDet3D 👇 github.com/allenai/WildDe…. Thinking about training Boxer with WildDet3D data to do 30fps in-the-wild 3D tracking.

Cool showcase from @_satyam_ai running Boxer on RGB video using COLMAP for poses + pointcloud and GeoCalib for gravity estimation

@_satyam_ai @pesarlin GeoCalib looking solid 💪

@_satyam_ai Amazing!

Meta recently open-sourced Boxer, a model that lifts 2D bounding boxes into 3D oriented bounding boxes (OBBs) for scene understanding. The catch? It was designed for Aria AR glasses, not regular cameras. So I built a pipeline to make it work with any phone video. The hard part: Boxer expects gravity from Aria's IMU. COLMAP doesn't know "up" from "down." Had to estimate gravity from camera poses and rotate the entire reconstruction. @Meta #ComputerVision #3DReconstruction #MetaAI #SceneUnderstanding

@_satyam_ai the gravity estimate looks a little bit off. another idea could be to run this per frame and take the global 3D average: github.com/cvg/GeoCalib

@ddetone Yes, feeding in the COLMAP sparse point cloud as depth input along with RGB frames. For the example video, it had around ~14K points.

@yesitsarmin yes, the main limitation is the 2D detector here, but there are tons of better models (SAM3, VLMs) if you have the compute. for very cluttered scenes it doesn't work as well

@ddetone this is very cool, can it work for any arbitrary objects, like stuff on a table, and can it work with stereo camera?

@ElioenaiSiqCst Yes, I didn't show any examples of that but we trained on a massive internal-only Quest3 dataset

@ddetone Works In meta quest?

@BlueAquilae great question! I would not expect it to work well here, we would need to re-train it with a full 9 DoF representation. but feel free to try it out anyway, I'd be curious

@CleverBetTips The National?

@ddetone Your name is competing with one of the great albums of the last 30 years 😀

@haodongli00 One limitation I found using both of those models is the runtime. For detecting 1000+ text prompts with SAM3 it takes 20+ sec per image. SAM3D also takes ~15 sec per object, so running on large datasets can be expensive. OWLv2 runs at ~30ms and Boxer takes ~20ms

@ddetone Great work! @ddetone Also doing something very similar, using SAM3, SAM3D and many other powerful tools! 😎

@nickkarpov Feel free to file a GitHub issue if you have any problems! Will do my best to answer them quickly

@ddetone Awesome work, going to use this

For more details, check out the arxiv paper here: arxiv.org/abs/2604.05212

BoxerNet runs FAST 🔥🔥, taking roughly 20ms on a 4090 with bfloat16 for ALL prompts in an image (e.g. 30 boxes in parallel)