UW RAIVN Lab

Last year Molmo set SOTA on image benchmarks + pioneered image pointing. Millions of downloads later, Molmo 2 brings Molmo’s grounded multimodal capabilities to video 🎥—and leads many open models on challenging industry video benchmarks. 🧵

📢Applications are open for summer'25 internships at the PRIOR (computer vision) team @allen_ai: Come join us in building large-scale models for: 📸 Open-source Vision-Language Models 💻 Multimodal Web Agents 🤖 Embodied AI + Robotics 🌎 Planet Monitoring Apply by December 11, 2024!

Congrats [Dr. [Dr. Aditya] Kusupati]!!🪆🪆

Very cool to see our RAIVN lab alum Rowan in the latest GPT-4o demo!!🚀🚀

🎉 Very Excited to present our recent work on “Selective🔍 Visual Representations for Embodied-AI🤖” next week at ICLR in Vienna🇦🇹!! 📣📣Important update! Our code and pretrained models are now available through our project website 🌐: embodied-codebook.github.io🚀 👋Come to my poster, say hi, and learn more about our findings! (Poster #111, Session 8, on Friday, May 10th at 4:30 PM)

CLIP models have become a lot better since 2021

Check out🪆MatFormer🪆co-led by @adityakusupati: it’s a simple yet powerful general-purpose architecture with flexibility and elasticity built within. It works across modalities and enables super cool things at web-scale tasks🔥🔥

E) The attention logit growth instability is still present when replacing softmax with pointwise alternatives. Side note: If you're interested in learning more about replacing softmax with a pointwise alternative like relu^2/√seqlen, checkout arxiv.org/abs/2309.08586! (12/15)

Sharing some highlights from our work on small-scale proxies for large-scale Transformer training instabilities: arxiv.org/abs/2309.14322 With fantastic collaborators @peterjliu, @Locchiu, @_katieeverett, many others (see final tweet!), @hoonkp, @jmgilmer, @skornblith! (1/15)

Exciting work with open source code to facilitate research on medium sized language models! 🎉

Check out this cool work led by @DJiafei on collecting robot data without a robot! 🤖🦾

If you are at #CVPR2023, come check out prompting-in-vision.github.io on Monday, June 19 from 9am - 12pm in West room 223-224. Speakers include @sarahmhpratt from RAIVN lab as well as @liuziwei7 @phillip_isola @hyojinbahng @lschmidt3 and @denny_zhou!

Checkout CREPE led by @zixianma02, a new large-scale benchmark for vision-language model!! Drop by the poster at CVPR 2023.

Check out our new work on improving large-scale nearest-neighbor search!

Introducing💃AdANNS: A Framework for Adaptive Semantic Search🕺 TL;DR: Up to 90× faster nearest neighbor retrieval and 2× lower memory cost for web-scale search. Applies to vector search at scale & improves all "retrieval" augmented models! arxiv.org/abs/2305.19435 [1/8]

1/9 I am excited to announce that our workshop "Towards the Next Generation of Computer Vision Datasets" will be happening at ICCV 2023 in Paris. We will feature DataComp submissions, other data-centric papers, and invited talks by experts. datacomp.ai/workshop

Excited to announce DataComp! Let us create stronger models using better-filtered data.

Introducing DataComp, a new benchmark for multimodal datasets! We release 12.8B image-text pairs, 300+ experiments and a 1.4B subset that outcompetes compute-matched CLIP runs from OpenAI & LAION 📜 arxiv.org/abs/2304.14108 🖥️ github.com/mlfoundations/… 🌐 datacomp.ai