Hugues Bruyère

Mixed Reality + Diffusion prototype as a tool for exploring concepts, styles, and moods by transforming real-world surroundings into alternate realities. #MixedReality #MR #AI #StableDiffusion #Quest3

@hkchrism Updating this experiment is on my todo list 😶‍🌫️ …

@smallfly Put this in some XR glasses over the world their front facing cameras are capturing and you've got something.

New iteration on Parallel Timelines (working title). FLUX.2-Klein 4B now runs at a satisfactory real-time speed, balancing performance and visual quality. Hand tracking controls the split, letting me define where the transformed output appears; a more gestural and playful approach than the previous eye-tracked version. #ai #realtime #realtimeai #StableDiffusion

@iSaif_ai This is the fp8 version, running in a custom app built using HF Diffusers.

@smallfly Can you tell us more details?

@jasonkneen No, there isn’t. Not yet, at least.

@smallfly is there a repo?

@ShadowMade_AI No, I haven’t tried on macOS.

@smallfly have you ran this on M4 pro mac silicon yet? Very impressive indeed

@hbcbh1999 Yes, I’m getting slightly better performance with a 5090.

@smallfly incredible - will a better gpu reduce the lagging?

Iteration on the Parallel Timelines mirror. Improved real-time performance with FLUX.2-Klein 4B + optimizations; here running on a RTX 4090. MediaPipe handles live tracking and drives the split. #ai #realtime #realtimeai #stablediffusion

I have only tried StreamDiffusionV2 once, and only briefly, but I did not manage to get outputs that were as good as the original version of StreamDiffusion. Therefore, I did not look into it further. In terms of performance, it may be faster, but Flux is impressive for image editing and can work with nine-square inputs.

@smallfly is it better than StreamDiffusionV2?

@slop_swap Here it’s running on a RTX 4090, therefore 24GB.

@smallfly Really impressive. How much VRAM for this speed in real time?

Starting a new series of real-time img2img diffusion explorations. First experiments combining FLUX.2-Klein with MediaPipe for live tracking-driven transformations. #ai #realtime #realtimeai #stablediffusion

@jamesckane Just tweaked splats budget and foveated rendering settings; quite aggressively. 550k splats budget for a 17M splats scene. My AVP is a M1.

@smallfly Awesome! Did you add any custom settings for the LOD system for Vision Pro specifically? I am experimenting w/ the M5 and tried to do similar, but I’m getting worse perf than this despite lowering some settings. That said I’m running game logic and gltf too - just need to profile

1st go at #Spark v2.0.0-preview’s LoD sys. The same scene (18M GS) rendered across desktop, MBP M1, iPhone, and Apple Vision Pro. This scene is challenging, as some areas have a dense concentration of splats. The screen recording isn't helping the AVP. #3DGS #GaussianSplatting

@sparkjsdev The v2.0.0-preview’s LoD rendering and streaming system (including the .rad format) is amazing!

@JoanieLemercier Thanks.

@smallfly beautiful !

A few stills from the previous video. X’s compression is not very kind to point clouds.

@DanielleFong 🙌😶‍🌫️

this is incredible.