CMU Center for Perceptual Computing and Learning

Amazing work from a CMU alumn! #goSmithHall!!! linkedin.com/posts/hwkang_l…

20+ new papers from CMU at CVPR 2026!!! vision.cs.cmu.edu/papers.html

Smith Hall and Scaife Hall unite! 3D human pose estimation for improving physical therapy!

#goSmithHall !!!

New model from Meta, SAM 3D Body, powered by people from Smith Hall (@kkitani,@jinkuncao, David Park, Jyun-Ting Song) of course! #goSmithHall Introducing SAM 3D: a New Standard for 3D Object & Human Reconstruction ... youtu.be/B7PZuM55ayc?si… via @YouTube

Super excited to share the release of SAM 3D. It's been a year in the making. Two models for lifting object and people to 3D!

TWO Best Paper Awards at ICCV Generating Physically Stable and Buildable Brick Structures from Text Ava Pun*, Kangle Deng*, Ruixuan Liu*, Deva Ramanan, Changliu Liu, Jun-Yan Zhu Spatially-Varying Autofocus Yingsi Qin, Aswin C. Sankaranarayanan, Matthew O'Toole #goSmithHall

A closed door looks the same whether it pushes or pulls. Two identical-looking boxes might have different center of mass. How should robots act when a single visual observation isn't enough? Introducing HAVE 🤖, our method that reasons about past interactions online! #CORL2025

Excited to share recent work with @kaihuac5 and @RamananDeva where we learn to do novel view synthesis for dynamic scenes in a self-supervised manner, only from 2D videos! webpage: cog-nvs.github.io arxiv: arxiv.org/abs/2507.12646 code (soon): github.com/Kaihua-Chen/co…

✨New edition of our community-building workshop series!✨ Tomorrow at @CVPR, we invite speakers to share their stories, values, and approaches for navigating a crowded and evolving field, especially for early-career researchers. Cheeky title🤭: How to Stand Out in the Crowd🙋? Details & context here: sites.google.com/view/standoutcv

New work on unifying 2D and 3D vision-language models from CMU and Meta!

ASAP learns diverse, agile, whole-body humanoid motions via learning a residual action model from the real world to align sim and real physics, enabling motions that were previously difficult to achieve. It has two stages: Stage 1 pretrains a phase-based motion tracking policy to mimic human motions in simulation. Stage 2 rolls out such a policy in real to collect data, learns a residual action model to compensate for the dynamics mismatch, and finally fine-tunes the pretrained policy with the learned residual model. ASAP is fully open-sourced: agile.human2humanoid.com ASAP is not just for sim2real. It provides a general framework to align physics in training and deployment environments. To facilitate smooth transfer between different simulators, we also released HumanoidVerse, a multi-simulator humanoid learning framework: github.com/LeCAR-Lab/Huma… A key design principle of HumanoidVerse is the separation and modularization of simulators, tasks, and algorithms, allowing switching between simulators and tasks with minimal effort. We support training & evaluation in multiple simulators (IsaacGym, IsaacSim, Genesis) by changing only ONE command line: +simulator= Led by @TairanHe99 @WinstonGu_ @_wenlixiao @Yuanhang__Zhang. Collaborations with the @nvidia GEAR lab led by @yukez and @DrJimFan.

Should have recorded our reactions when the first successful siuuu happened! 🎉 Collecting and learning real world data will be incredibly important for humanoids moving forward, and we have just took our first step ASAP🫡

🎵✨Excited to share our #NeurIPS2024 paper on personalized music video generation! We combine multimodal AI with identity protection to let listeners be co-creators, generating custom music videos that reflect both music and themselves. 🎥🔒 arxiv.org/abs/2502.02610 #CreativeAI

Excited to present new work on using diffusion priors for video amodal segmentation and content completion! with @kaihuac5 (lead author) and @RamananDeva arXiv: arxiv.org/abs/2412.04623 project page: diffusion-vas.github.io

Spot-on tips for faculty applicants from RI postdoc @unnatjain2010. Big congrats to him and UC Irvine! 🎉

Excited to share that I'll be joining University of California at Irvine as a CS faculty in '25!🌟 Faculty apps: @_krishna_murthy, @liuzhuang1234 & I share our tips: unnat.github.io/notes/Hidden_C… PhD apps: I'm looking for students in vision, robot learning, & AI4Science. Details👇

Excited to finally release our NeurIPS 2024 (spotlight) paper! We introduce Run-Length Tokenization (RLT), a simple way to significantly speed up your vision transformer on video with no loss in performance!

Can my robot cook my food, rearrange my dresser, tidy my messy table and do so much more without ANY demos or real-world training data? Introducing ManipGen: A generalist agent for manipulation that can solve long-horizon robotics tasks entirely zero shot, from text input! 1/N

1/ Happy to share VADER: Video Diffusion Alignment via Reward Gradients. We adapt foundational video diffusion models using pre-trained reward models to generate high-quality, aligned videos for various end-applications. Below we generated a short movie using VADER 😀, we used ChatGPT to write a script and an off-the-shelf AI music generator to generate the sound. Our code & weights are open-sourced: vader-vid.github.io