Shuyang (Kevin) Sun

We have an important result to share: if you reduce multiple dense vision tasks into a single RGB-image-prediction task, fine-tuning a strong image generator (in our case Nano Banana Pro) matches or beats all specialized models for monodepth, normals, and semantic segmentation.

vision🍌 is here vision-banana.github.io if you got into computer vision the way I did, starting with pixel-level labeling tasks like segmentation, edges, depth, or surface normals, you’ll probably feel the same seeing these results -- something big has quietly shifted, and it’s going to change how we approach these problems for good 🧵

Check out the website and the paper: vision-banana.github.io arxiv.org/abs/2604.20329

Vision Banana is also advised and supported by Oliver Wang, Saining Xie, Howard Zhou, Kaiming He, Thomas Funkhouser, Jean-Baptiste ALAYRAC, and Radu Soricut. N/N

Are we finally witnessing the GPT-3 moment for computer vision? We just dropped Vision Banana 🍌 , a vision foundation model that seamlessly unifies generation and perception by treating all vision tasks as just another image generation problem. 1/N #googledeepmind #nanobanana

Huge shout out to my amazing collaborators: @vgabeur , @ShangbangLong , @songyoupeng , @PaulVoigtlaend1 , Yanan Bao, Karen Truong, Zhicheng Wang, Wenlei Zhou, @jon_barron , Kyle Genova, Nithish Kannen, Xue Ben, Yandong Li, Mandy Guo, Suhas Yogin, Yiming Gu, and Huizhong Chen.

Vision Banana achieves state-of-the-art performance under the zero-shot transfer setting across both 2D and 3D vision tasks, proving that generative vision models are the ultimate generalist foundation. 2/N

🚀 Excited to announce Vision Banana 🍌 and our new paper: “Image Generators are Generalist Vision Learners”. We turn Nano Banana Pro into a state-of-the-art visual generation and understanding model. 🖼️ Check out our gallery at vision-banana.github.io 🧵 (1/N) continue ⬇️

Gemini is back

Great work from our student researcher Jiageng Mao @PointsCoder to enable scalable robot learning by imitating AI-generated videos.

What if you could not only watch a generated video, but explore it too? 🌐 Genie 3 is our groundbreaking world model that creates interactive, playable environments from a single text prompt. From photorealistic landscapes to fantasy realms, the possibilities are endless. 🧵

kNN-CLIP: Retrieval Enables Training-Free Segmentation on Continually Expanding Large Vocabularies Zhongrui Gui, Shuyang Sun, Runjia Li et al.. Action editor: Yu-Xiong Wang. openreview.net/forum?id=ZSqP1… #forgetting #vocabularies #memory

@csaba_botos @fhuszar @oiwi3000 Life advice: don't jump into the river while wearing a gown😅

@fhuszar @oiwi3000 Thank you :) this was the perfect ending of one of the biggest chapters in my life. To advertise the alleged demos: botcs.github.io/dopanet botcs.github.io/label-delay/de… Which is going to be turned into a series of YouTube tutorials over the summer!

Great day in Oxford joining forces with @oiwi3000 as co-examiners for the DPhil viva of @csaba_botos Congrats Csaba for the great thesis and amazing live demos.

Project page: torrvision.com/clip_as_rnn/ Code: github.com/kevin-ssy/CLIP… CPU Demo: huggingface.co/spaces/kevinss… work done with @RunjiaLi , @philiptorr , @laoreja001 and Siyang Li from University of Oxford and Google DeepMind

I will be presenting our paper "Clip as RNN: Segment Countless Visual Concepts without Training Effort" at @CVPR poster #341, today (June 20th) at 10:30 am. If you're interested in open-vocabulary segmentation and multi-modal learning, please come and chat with me!

After almost a decade, I have made the decision to leave OpenAI.  The company’s trajectory has been nothing short of miraculous, and I’m confident that OpenAI will build AGI that is both safe and beneficial under the leadership of @sama, @gdb, @miramurati and now, under the excellent research leadership of @merettm.  It was an honor and a privilege to have worked together, and I will miss everyone dearly.   So long, and thanks for everything. I am excited for what comes next — a project that is very personally meaningful to me about which I will share details in due time.

Unfortunately, I won't be able to attend ICLR in person due to visa issues. However, if you're keen on using synthetic data for training, I'd love for you to visit our poster Real-fake on Friday morning!

You find some training data sources were compromised, potentially causing #backdoors, #bias, #mislabels etc. Can you remove its influence from previously trained #ML models instead of stopping their use? New work with @AmyPrb @AmartyaSanyal @ponguru @OxfordTVG studies this🧵👇

It's also a great journey collaborating with Jianhao @DYDYYDYYYD. Please broadcast and reach out if you're interested! @_akhaliq @OxfordTVG

Another paper on using synthetic data for better image recognition. Distribution matching may not be all you need but it's very helpful to bridge the real-synthetic domain gap.