Jiaming Sun

Check out our collaboration with Corridor Crew on 4D Gaussian Splatting: youtu.be/X8yRlA7jqEQ?si… Huge thanks to @SirWrender and the entire @CorridorDigital team for making this happen. It’s easily one of the best explainer videos on Gaussian Splatting we’ve seen. Also check out our website at 4dv.ai. From jazz performances, to the first wedding ever captured in 4D, to a cute little cat, to a REAL rocket launch, we’re constantly pushing the boundaries of what 4D capture can be and bringing 4D creation to everyone. We believe 4D content is the new, and ultimately the most powerful form of visual media. Unlocking it means solving the entire pipeline end to end. That’s what we’re building at 4DV.ai: a full-stack toolkit for 4D content creation. Beyond the specialized 4D representations for data processing discussed in the video, we’re also building mobile capture systems with PTZ camera rigs, non-linear 4D editing tools, and playback and distribution workflows across the web, Unity, Blender, UE5, Houdini, and other platforms. More to come soon. Stay tuned.

F u t u r e

4D Gaussian Splatting is cool

Want to use Depth Anything, but need metric depth rather than relative depth? Thrilled to introduce Prompt Depth Anything, a new paradigm for accurate metric depth estimation with up to 4K resolution. 👉Key Message: Depth foundation models like DA have already internalized rich geometric knowledge of the 3D world but lack a proper way to elicit it. Inspired by the success of prompting in LLMs, we propose prompting Depth Anything with metric cues to produce metric depth. This method proves to be very effective when using a low-cost lidar (e.g., iPhone's LiDAR), which is widely available, as prompts. We believe the prompt can generalize to other forms as long as scale information is provided. Prompt Depth Anything offers 1⃣A series of models for iPhone lidars. 2⃣4D reconstruction from monocular videos (captured with iPhone). 3⃣Improved generalization ability for robot manipulation, e.g. Training on cans but generalizing on glasses. 4⃣More detailed depth annotations for the ScanNet++ dataset. The first author is our excellent intern @HaotongLin. Paper: huggingface.co/papers/2412.14… Huggingface: huggingface.co/papers/2412.14… Project Page: promptda.github.io Code: github.com/DepthAnything/…

A fascinating lecture on using 2D physical methods to store and recreate a radiance field!

@JohnEMotionFX Is it based on a MLP that maps joint rotations to per-vertex deformation? Any paper reference would be really nice!

UE 5.1 will ship with a ML Deformer framework to build custom models and a new model we call Neural Morph Model. This allows high quality deformations in real-time. Character below uses ~1 mb of ML memory and takes below 100 microsecs on CPU and 40 micros on GPU. #ue5 #mldeformer

Incredible progress, this comes much sooner than I expected.

"Voyage through Time" is my first artpiece using #stablediffusion and I am blown away with the possibilities... We're crossing a threshold where generative AI is no longer just about novel aesthetics, but evolving into an amazing tool to build powerful, human-centered narratives

Check out our SIGGRAPH paper on modeling multiple closely interacting people with few cameras!

Happy to share our #CVPR2022 paper—PlanarRecon 😄 PlanarRecon can detect and reconstruct the 3D planes from a posed monocular video. Project page: neu-vi.github.io/planarrecon/

Joint work with Xi Chen, @QianqianWang5, @zhengqi_li, @ElorHadar, @XiaoweiZhou5 and @Jimantha. It has been a pleasure working with this amazing group of people and thanks for all your insight and support!

Glad to share our work “Neural 3D Reconstruction in the Wild” in SIGGRAPH 2022! We show that with a clever sampling strategy, neural-based 3D reconstruction can be better and faster than COLMAP. Check out the project page at: zju3dv.github.io/neuralrecon-w/.

No CAD models, no instance or category-specific training, all you need is a strong generic feature matcher like SuperGlue or LoFTR. Check out our CVPR 2022 paper on object pose estimation without CAD models!

NeRFusion: Fusing Radiance Fields for Large-Scale Scene Reconstruction abs: arxiv.org/abs/2203.11283 project page: jetd1.github.io/NeRFusion-Web/

It is not the "smartest" people who succeed in science. It takes intrinsic motivation, perseverance, the drive to stay active and hands on, a strong support network of mentors, and the curiosity to keep learning. Nice summary in Nature: nature.com/articles/d4158…

SelfRecon: Self Reconstruction Your Digital Avatar from Monocular Video abs: arxiv.org/abs/2201.12792 project page: jby1993.github.io/SelfRecon/

I get a lot of reviews that say my work is not novel and I bet I'm not alone. It's always frustrating because I see novelty where the reviewer doesn't. Rather than rebut every critique, I've written a blog post to help reviewers think about novelty. perceiving-systems.blog/en/news/novelt…

Groundbreaking results!

AI is clearly lacking this kind of intuitive spatial/physics reasoning, but I don't see enough research on what for me is the biggest challenge: building general, *efficient* composable 3D world models from real-time vision + sensors. My views: arxiv.org/abs/1803.11288 #SpatialAI