Slavina Ivanova
1.6K posts
@slavinarium
F500 Consultant • Sr Product Designer by day, AI Explorer by night • Sharing daily AI learnings, prompts and insights
Matryoshka Diffusion Models paper page: huggingface.co/papers/2310.15… Diffusion models are the de facto approach for generating high-quality images and videos, but learning high-dimensional models remains a formidable task due to computational and optimization challenges. Existing methods often resort to training cascaded models in pixel space or using a downsampled latent space of a separately trained auto-encoder. In this paper, we introduce Matryoshka Diffusion Models(MDM), an end-to-end framework for high-resolution image and video synthesis. We propose a diffusion process that denoises inputs at multiple resolutions jointly and uses a NestedUNet architecture where features and parameters for small-scale inputs are nested within those of large scales. In addition, MDM enables a progressive training schedule from lower to higher resolutions, which leads to significant improvements in optimization for high-resolution generation. We demonstrate the effectiveness of our approach on various benchmarks, including class-conditioned image generation, high-resolution text-to-image, and text-to-video applications. Remarkably, we can train a single pixel-space model at resolutions of up to 1024x1024 pixels, demonstrating strong zero-shot generalization using the CC12M dataset, which contains only 12 million images.
Join us for the Generative AI Friday 𝕏 Space to hear the latest updates with some of our friends: @saana_ai @spacecasetay @Salmaaboukarr @LinusEkenstam & many more. Don't miss the best Gen AI conversation on 𝕏. Set your reminder 👇 twitter.com/i/spaces/1yoKM…
