Duygu Ceylan
43 posts
Congratulations to @omidtaherii on defending his PhD summa cum laude on the topic of “Modeling Dynamic 3D Human-Object Interactions: From Capture to Synthesis”. Omid’s work on the GRAB dataset, GRIP, GOAL, and more has changed the field of human-object interaction.
Generative Rendering: Controllable 4D-Guided Video Generation with 2D Diffusion Models paper page: huggingface.co/papers/2312.01… Traditional 3D content creation tools empower users to bring their imagination to life by giving them direct control over a scene's geometry, appearance, motion, and camera path. Creating computer-generated videos, however, is a tedious manual process, which can be automated by emerging text-to-video diffusion models. Despite great promise, video diffusion models are difficult to control, hindering a user to apply their own creativity rather than amplifying it. To address this challenge, we present a novel approach that combines the controllability of dynamic 3D meshes with the expressivity and editability of emerging diffusion models. For this purpose, our approach takes an animated, low-fidelity rendered mesh as input and injects the ground truth correspondence information obtained from the dynamic mesh into various stages of a pre-trained text-to-image generation model to output high-quality and temporally consistent frames. We demonstrate our approach on various examples where motion can be obtained by animating rigged assets or changing the camera path.
Generative Rendering: Controllable 4D-Guided Video Generation with 2D Diffusion Models paper page: huggingface.co/papers/2312.01… Traditional 3D content creation tools empower users to bring their imagination to life by giving them direct control over a scene's geometry, appearance, motion, and camera path. Creating computer-generated videos, however, is a tedious manual process, which can be automated by emerging text-to-video diffusion models. Despite great promise, video diffusion models are difficult to control, hindering a user to apply their own creativity rather than amplifying it. To address this challenge, we present a novel approach that combines the controllability of dynamic 3D meshes with the expressivity and editability of emerging diffusion models. For this purpose, our approach takes an animated, low-fidelity rendered mesh as input and injects the ground truth correspondence information obtained from the dynamic mesh into various stages of a pre-trained text-to-image generation model to output high-quality and temporally consistent frames. We demonstrate our approach on various examples where motion can be obtained by animating rigged assets or changing the camera path.
Pix2Video: Video Editing using Image Diffusion abs: arxiv.org/abs/2303.12688 project page: duyguceylan.github.io/pix2video.gith…
Happy to announce our paper got accepted at #CVPR2023! We generate cinemegraphs from images by animating clothes using a CycleNet trained without real data. This is the result of my internship at #AdobeResearch. Website: hbertiche.github.io/CycleNet/ arXiv: arxiv.org/abs/2303.08639