Hugues Bruyère

Mixed Reality + Diffusion prototype as a tool for exploring concepts, styles, and moods by transforming real-world surroundings into alternate realities. #MixedReality #MR #AI #StableDiffusion #Quest3

Carveout Explorations #02 — Coiffure Mile End⁣⁣ ⁣⁣ Another space explored through Carveout: a small coiffure salon in my neighborhood.⁣⁣ ⁣⁣ Once again what I enjoy about these scenes is seeing familiar places as something halfway between a reconstruction and a dataset... a place oscillating between a real location, a point cloud and a collection of addressable objects.⁣⁣ ⁣⁣ #GaussianSplatting #3DGS #segmentation #pointcloud

@Dusanwriter Not only the view layer. Carveout is the pipeline that identifies and segments objects in the rendered views, then connects that information back to the 3DGS scene. The interface is currently how I explore and visualize those relationships.

@smallfly Carveout is the view layer?

@Dusanwriter Thank you! Historic or heritage locations feel like a particularly meaningful context for this.

@smallfly “Addressable objects” love this framing. Doing same with historic/heritage locations. Keep sharing! So inspired

@JohannesTscharn The accuracy depends on the specific characteristics of the scene. I’m currently working on making it more robust across a wider range of environments, while adding more user control throughout the pipeline so the quality can be more consistent across different types of scenes.

@smallfly It’s just impressively accurate. Would love to “remix” a place by changing some parts and replacing them with others. In a way this makes captured locations… modular.

@heyapple78 You should but the capture step would be longer.

@smallfly could you achieve this level of clarity with a phone scan?

Radio Hovsep Four years ago, I captured this unique place that defines my neighborhood. I went back this weekend to say hello to Mr. Hovsepian and see if I could once again capture his store, but this time using the PortalCam. Just like the previous time, Mr. Hovsepian was welcoming, curious, and interested in my process. Joseph Hovsepian, Montreal’s “radio doctor,” has been repairing radios, turntables, and other electronics here in the city’s Mile End neighborhood since the 1960s. Capturing these spaces feels like a small way to preserve their memory and contribute to a collective cultural heritage; snapshots of everyday life and community. This new capture is much better than the one I previously did. Using the PortalCam makes it easier to achieve good results in only 20 minutes. Now that reaching this level of quality and ease of capture is possible, I want to create a more complete “experience” around Radio Hovsep and Joseph Hovsepian... Raw capture explored in real-time using SuperSplat. #montreal #GaussianSplatting #3DGS #pointcloud #portalcam

@EthanGoodhart @XGRIDS_OFFICIAL ‘s Portal Cam.

@smallfly How’d you record the splat?

Carveout Explorations #01 ⁣ I built Carveout to experiment with object understanding in 3D Gaussian Splat scene with segmentation down to the Gaussian level. ⁣ Part of the appeal, however, isn’t the object detection itself, but the visual experience it creates.⁣ ⁣ The scene becomes covered with labels and metadata. As I move through walls, the 3DGS reconstruction gives way to a point cloud, revealing the space from the outside while the objects remain identified and anchored in space. ⁣ ⁣ There is something satisfying about seeing a familiar place oscillate between reconstruction, abstraction and data visualization.⁣ ⁣ 3D GS rendering done using SparkJS.⁣ ⁣ #GaussianSplatting #3DGS #segmentation #pointcloud

@AlexTolson91 Thanks. Yes the Portal Cam is great. The capture could have been better though.

@smallfly Great capture! The PortalCam is a great piece of hardware

New capture of my neighborhood… this time, YM Fruiterie, the dépanneur at the corner of my street. A small store, but packed floor to ceiling with products, making it a dense scene to capture and reconstruct. This 3DGS was captured with the PortalCam in about 20 minutes. Once more, there is something I enjoy about preserving these everyday places. The kind of spaces you pass through regularly without necessarily paying attention to all the details. I now know exactly where the apple sauce is. #GaussianSplatting #3DGS #pointcloud #portalcam

@rosejn It all starts from…

@smallfly Cool idea! How does it work?

I have been working on a project I named Carveout, for which Splat Analyzer was the catalyst. I loved the visual experience of exploring a 3D Gaussian Splat scene covered with labels and metadata, but also the possibilities that came with it: being able to search a scene, identify objects, and eventually edit or manipulate parts of the capture itself. When I started testing Splat Analyzer on some of my own captures, the results were mixed, so I wanted to better understand how this object detection and labeling worked, and how it could be improved. That research led me to SAM 3 (instead of OWLv2) and eventually down a much larger rabbit hole as the question became: How can segmentation masks be propagated all the way down to the Gaussian level instead of stopping at object detections in rendered images? That was the moment I started from a clean slate: Carveout. The pipeline renders a series of synthetic views from an existing 3DGS scene, uses SAM 3 to segment objects, then accumulates evidence across views (using a FlashSplat-inspired mask lifting approach implemented on top of gsplat) to determine which Gaussians belong to which objects. Those labeled Gaussians are then grouped into 3D instances with centroids, bounding boxes, and metadata. The constraints for this tool were simple: it had to work on existing captures. No retraining. No language features baked into the splats. Still early, but already producing promising results across a variety of scenes. #GaussianSplatting #3DGS #segmentation #pointcloud

@Ferhan_XRB I might make the repo public once I have worked on some UI to have an overview of the different steps in the pipeline.

This is next-level! It might be useful for our upcoming SIGGRAPH Hack since the hackers will have access to the XGRIDS Portal Cam! Let us know when you have any links to share so we can add them to the hack resources! For those who would like to also contribute to the Splat Analyzer and other World Model kits, here is the git directory: github.com/SensAIHackadem…

@bolayer I might later on, once I have worked on some UI.

@smallfly This is so cool! Do you open source by chance? Great work.

Thanks! The boxes don’t do the segmentation, they’re derived from it. Each object is first resolved as a set of individual Gaussians (2D masks accumulated across many views, each Gaussian going to whichever object best explains it). So two objects whose boxes overlap are still cleanly separated because the separation happens per-Gaussian, not per-box.

@smallfly Amazing! The accuracy of the bounding boxes seems more than high enough to properly segment the objects. How do you handle overlapping objects?

@ashokM93 This capture was done using the Portal Cam.

@smallfly Great work! The splat quality is top, could you share more info on how you achieved it?

@rgilman33 I will be sharing more as I go.

@smallfly Have been waiting for this, will be following closely!

There is something magical about playing with the camera’s FOV inside a 3DGS scene.