Zineng Tang

🚨Announcing Zebra-CoT, a large-scale dataset of high quality interleaved image-text reasoning traces 📜. Humans often draw visual aids like diagrams when solving problems, but existing VLMs reason mostly in pure text. 1/n

CoT transformed text reasoning. What about multimodal? 🤔 Check out our new dataset of interleaved text and image reasoning traces. We also show interesting visual CoT examples generated inherently by the model finetuned on our dataset!

Excited to share our new work! DOVE 🕊️: a dynamic vision encoder that adapts token count to image complexity. Fewer tokens, same fidelity—outperforming fixed-length AEs tokenizer on classification & VLM tasks! Arxiv: arxiv.org/abs/2506.03643 Web: dove-encoder.github.io/dove-encoder/ #AI #CV

Big thanks to my undergrad intern Lingjun for delivering such impressive work, and to Rudy for the thoughtful co-advising!

🔥 DOVE uses 68 % fewer tokens but has better FID than VQGAN/TiTok and +10–12 pts on VQA/ImageNet/CIFAR. It achieves significantly stronger performance on classification, probing, and VLM tasks. DOVE also brings emerging properties—PCA heatmaps reveal sharper segmentation.

@yzy_ai ❤️

Another great work by @ZinengTang , generative contrastive learning for vision+language.

@jiayi_pirate ❤️

Zineng makes things work. And it's entirely done in Berkeley under resource constrain

@ziqiao_ma ❤️

Every Zineng paper is on my reading list :)

Also thanks to, @LongTonyLian, Seun Eisape (seuneisape.github.io), @XDWang101, @roeiherzig, @Yalatweets, @alsuhr, and @trevordarrell for their great efforts!

TULIP achieves state-of-the-art performance across multiple vision and vision-language benchmarks. It significantly improves zero-shot classification on ImageNet-1K, enhances fine-grained object recognition, and boosts multimodal reasoning scores. Compared to existing methods, TULIP shows up to a 3× improvement on MMVP and a 2× boost in fine-tuned vision tasks.

We are thrilled to announce TULIP! 🌷 tulip-berkeley.github.io A state of the vision language encoders coupled with generative model for stronger representation learning.

Automating AI research is exciting! But can LLMs actually produce novel, expert-level research ideas? After a year-long study, we obtained the first statistically significant conclusion: LLM-generated ideas are more novel than ideas written by expert human researchers.

We announced Phi 3.5 series today! 1️⃣ Multilingual Mini 3.8B: huggingface.co/microsoft/Phi-… 2️⃣ MoE 16x3.8B (active 6.6B): huggingface.co/microsoft/Phi-… 3️⃣ multi-frame vision LLM: huggingface.co/microsoft/Phi-…

CoDi-2 is selected as #CVPR2024 Highlight. Come joint us in today’s poster session Arch 4A-E #314 in 5pm to 6:30 pm! codi-2.github.io @yzy_ai @nlpyang @ChenguangZhu2 @mohitban47

Honored and grateful to be named as one of the @UNC permanent distinguished professorships 🙏 100% of the credit goes to my awesome students, postdocs, collaborators, and colleagues+family ❤️

Excited to share that CoDi-2 is accepted to @CVPR In this work, we show that alignment of multimodal inputs to language unlocks ICL and Few-shot prompting ability for multimodal generation. #CVPR2024 @yzy_ai @nlpyang @ChenguangZhu2 @mohitban47

Had a great time speaking at @indoml_sym & meeting students+faculty+researchers (overall, kudos to organizers+volunteers on an excellent program)! 🙂 Talked about our long journey on Multimodal Generative LLMs: (1) Unified/Universal Multimodal Learning for Generalizability, Shared Knowledge, Efficiency: LXMERT ➡️ VL-T5 ➡️ TVLT ➡️ UDOP ➡️ CoDi1+CoDi2 (2) LLM Planning/Programming for Interpretable+Controllable+Faithful Multimodal Generation: VPGen ➡️ DiagrammerGPT ➡️  VideoDirectorGPT (3) Evaluation of Multimodal Generation Models for Fine-grained Skills, Faithfulness, Social Biases: DALL-Eval ➡️ VPEval ➡️ Davidsonian Scene Graph