Liangke Gui

Here’s how it works: 🔵 Design your world and character using text and visual prompts. 🔵 Nano Banana Pro makes an image preview that you can adjust. 🔵 Our Genie 3 world model generates the environment in real-time as you move through. 🔵 Remix existing worlds or discover new ones in the gallery.

Our new native image generation and editing is state-of-the-art, and ranked #1 in the world. And we're rolling it out for free to everyone today. You’ve got the tools. Now go bananas. Ideas & inspiration in the 🧵below.

Glad to share Seaweed-7B, a cost-effective foundation model for video generation. Our tech report highlights the key designs that significantly improve compute efficiency and performance given limited resources, achieving comparable quality against other industry-level models. To unleash the power of the foundation model, Seaweed-7B further enables a wide range of downstream applications including image-to-video generation, human video generation, subject-consistent video generation, video-audio joint generation, long video generation and storytelling, real-time generation, super-resolution generation, camera controlled generation. Check out our webpage and report for more details: Webpage: seaweed.video Paper: seaweed.video/seaweed.pdf It's a wonderful journey of the last year. Thanks to all teammates for their contributions, sincerely.

Check out our latest work CameraCtrl II. By carefully collecting and processing data and introducing as little inductive bias as we can, users are allowed to explore the generated world with appealing dynamics and consistency. Together with extension and distillation, CameraCtrl II can support ultra-fast interaction and long-term exploration. Homepage: hehao13.github.io/Projects-Camer…

The effectiveness of Video LMMs can be enhanced from DPO training using language model reward, which leverages detailed video captions as proxies for video content, leading to cost-effective preference optimization for video LMM alignment. twitter.com/RuohongZhang/s…

See more in the paper here: arxiv.org/abs/2205.09256 Or grab the code and models to play around yourself: github.com/guilk/VLC Joint work with Qiuyuan Huang, Alex Hauptmann, @ybisk, and @JianfengGao0217

produces surprisingly interpretable patch alignments on concepts not in COCO or ImageNet-1K

What if we don’t need supervised pretraining for vision-language models? We find that unsupervised visual representations (e.g., MAE) are actually better initializations for language and vision.

Additionally, our explicit knowledge integration improves interpretability of model predictions in our analysis. Paper: arxiv.org/abs/2112.08614 Code: github.com/guilk/KAT (Work w/ Qiuyuan Huang, Alex Hauptmann, @ybisk and @JianfengGao0217)

This provides a nice place to investigate the trade-offs of information already available in model weights vs extracted from structured sources.

Can multimodal transformers leverage explicit knowledge in their reasoning? What’t the role of explicit vs implicit knowledge in visual tasks like OK-VQA? Check out our KAT (A Knowledge Augmented Transformer for vision-and-language) paper at #NAACL2022.