Liam Schoneveld

📢Pix2NPHM: Learning to Regress NPHM Reconstructions From a Single Image📢 We directly regress neural parametric head models (NPHMs) from a single image — fast, stable, and significantly more expressive than classical 3DMMs such as FLAME. Face tracking & 3D reconstruction are often limited by the representational capacity of PCA-based face models. By lifting NPHMs to a first-class reconstruction primitive, we enable more accurate geometry, richer expressions, and finer animation control. Pix2NPHM obtains fast and reliable NPHM reconstructions on real-world data. Inference-time optimization against surface normals and canonical point maps can further increase fidelity. Key to successful and generalized training of our ViT-based network are: (1) large-scale registration of existing 3D head datasets, and (2) self-supervised training on vast in-the-wild 2D video datasets using pseudo ground-truth surface normals. Finally, we show that geometry-aware pretraining on pixel-aligned reconstruction tasks significantly outperforms generic visual pretraining (e.g., DINO-style features) in terms of generalization. 🌍simongiebenhain.github.io/Pix2NPHM 🎥youtu.be/MgpEJC5p1Ts Great work by @SGiebenhain, @TobiasKirschst1, @liamschoneveld, Davide Davoli, Zhe Chen

🐑🐑 SHeaP inference code is out ! 🐑🐑For all your real-time head pose and expression tracking desires! Check it out at: 🤗 HuggingFace spaces: huggingface.co/spaces/nlml/sh… 📟 Github: github.com/nlml/sheap

@Michael_J_Black @MattNiessner 😍 This approach can still be improved a lot, I think

@MattNiessner Nice. I’ve been wanting to replace the old photometric loss with splatting. Results look great.

📢 SHeaP: Self-Supervised Head Predictor Learned via 2D Gaussians 📢 Given a single input image, we predict accurate 3D head geometry, pose, and expression. Previous works (e.g. DECA, EMOCA) use differentiable mesh rasterization to learn a self-supervised head geometry predictor via a photometric reconstruction loss. We borrow these ideas, but our key insight is to replace the mesh rendering with 2D Gaussian Splatting. This leads to much higher accuracy of the underlying predicted geometry and thus more gradient signal during training. 🌍 nlml.github.io/sheap/ 🎥 youtu.be/vhXsZJWCBMA/ Great work by @liamschoneveld @_davidedavoli_ @jiapeng_tang

📢 Our new paper - SHeaP - is out! 📢 TLDR: self-supervised head tracking and geometry (FLAME) prediction, learned via photometric loss with a 2D gaussian splatting renderer. See more: 🌍 nlml.github.io/sheap/ 🎥 youtu.be/vhXsZJWCBMA/

@minchoi Are its predictions in a local or world coordinate system?

Read more at their project page: m-usamasaleem.github.io/publication/Ge…

This is GenHMR. New AI research for 3D human modeling. Turn a single image into a lifelike 3D human model. Handles tricky poses, occlusions, & depth issues with ease. 10 examples: 1. Chasing

Great work by @jiapeng_tang and co ! Gaussian Avatars from just a handful of input images, by leveraging a multi-view diffusion prior !

@dome_271 Classifier-free guidance always seemed like some weird hack to me. There must be a more mathematically elegant solution out there, waiting to be found.

Do you all think there is something still fundamentally suboptimal in diffusion models? Just the reason that we need to rely on cfg-style sampling so much seems weird. Not using cfg just still looks very bad.

@camo2572 @LabAgainstWar @AlboMP @RichardMarlesMP @SenatorWong I don’t think it would be that hard to come up with a better plan than spending $360b for offensive nuclear subs we are not even contractually guaranteed to receive? I feel like literally any plan is better than that one.

@LabAgainstWar @AlboMP @RichardMarlesMP @SenatorWong What’s your plan then to defend our island nation. Do you honestly think we abandon AUKUS we will be left alone because we are a nice country . I’m really curious when I hear this demand to scrap AUKUS but no plan forward

Australia must withdraw from AUKUS. Labor Against War calls on @AlboMP @RichardMarlesMP @SenatorWong to make Australia safe again.

@1111nonbeliever @janusch_patas They could release the code but you wouldn’t get very far without their data and compute 😅

@janusch_patas No code released?

URAvatar: Universal Relightable Gaussian Codec Avatars Contributions (cited): (1) We introduce a universal relightable avatar prior model learned from hundreds of dynamic performance captures with a multi-view and multi-light system. (2) We build a drivable head avatar from a phone scan that can be rendered and relit with global light transport in real-time. (3) A capture system and evaluation protocol to measure the accuracy of relighting under continuous illuminations.

@MartinGTobias Most of these measures make sense to me? As someone working in tech, I think it’s well worth spending a little money to encourage more women into the field.

Your government at work with your money.

@finbarrtimbers Perhaps limiting the representation space via the limited size of the codebook forces the network to better compress what's really important in the images.

I don’t understand quantized image tokens (VQ-VAE style). Why would we ever want to use them vs continuous visual features?

@lvminzhang Is there a paper accompanying this code somewhere?

@janusch_patas Thanks!

@liamschoneveld Ups, did not remove the expiration date. You are welcome :) discord.gg/NqwTqVYVmj

I've started a Discord server for hacking on Gaussian Splatting, discussing radiance field papers, the latest view-independent episode, or just hanging out. If you're interested, feel free to join (invitation link in comments)!

@AlboMP Wow that was easy! Now could you quickly do gambling ads and mining companies paying no royalties!?

Our plan to legislate a minimum age for social media will support parents and protect children.

@senbmckenzie Are you suggesting we should raise teacher salaries? Good on you!

Big build projects blow out by $10.1 billion in the recent budget driven by CFMEU corruption that has been a major contributor to rising inflation. This Labor Government rewards their mates at the expense of everyday Australians. Study and work hard for years to earn a decent wage? Take a 3-day course, enter the CFMEU and become a stop-go worker and make $200k? That $4.2 million that the CFMEU chucked in Labor coffers at the last election sure paid off! Australia cannot take another 3 years of Labor and their mates draining tax-payer money that could be funding more infrastructure, more hospitals, and more services for all of the citizenry.

@levelsio @whittomd Wow LEDs! What futuristic technology

@whittomd But the same happened in Korea and they keep updating, same with China x.com/Rainmaker1973/…

Japan is not futuristic Japan is a fossil stuck in 1990 99% of Westerners still don't get this Futuristic Asia is China, Korea, Vietnam, etc No Asian rates Japan for modernity at all

@techchildrights @laion_ai Coming from a human rights organization, I am sure you appreciate the importance of transparency. Without @laion_ai's ongoing AI transparency efforts, we would know very little about the data going into these models.

🇦🇺NEW: The personal photos of Australian children are being secretly used to build powerful AI tools. Others are then using these tools to create malicious deepfakes, putting even more children at risk of serious harm. hrw.org/news/2024/07/0…

@Saboo_Shubham_ @laion_ai This definitely wouldn’t work as well on papers that haven’t had 1000s of blog and Reddit etc posts written about it though

Claude 3.5 Sonnet transformed a research paper into an interactive learning dashboard in just 30 seconds. It goes beyond the capabilities of GPT-4o, Gemini Pro, Llama and other existing LLMs. Education will never be the same again with AI.

@dome_271 I actually had this problem a long time ago when trying to use ConvNets to generate audio. Perhaps looking at audio generative model literature may help as high frequency details are perhaps even more important in that domain.

Does anyone else feel like diffusion models have a hard time generating high frequency details? Any experience / thoughts / ideas / pointers on that manner? We have observed often that our models don't generate high frequencies in images and have found it hard improving it.