Hao Dong

🏹 Job alert: Multiple PhD and Postdoc Positions in Multimodal Learning, Foundation Models and Agentic Systems at ELLIS Institute Finland and Tampere University 📍 Helsinki & Tampere 🇫🇮 ⏰ 31 December 🔗 donghao51.github.io/open-positions/

#PhD #Postdoc #Hiring #MachineLearning #AI #Multimodal #FoundationModels #ELLIS #TampereUniversity #Finland

We're hiring! 𝗣𝗵𝗗 & 𝗣𝗼𝘀𝘁𝗱𝗼𝗰 𝗽𝗼𝘀𝗶𝘁𝗶𝗼𝗻𝘀 in multimodal learning, foundation models and agentic systems. I will be joining the ELLIS Institute Finland and Tampere University as a PI and TTAP! I'm now recruiting PhD students and postdocs to join my research group.

More details on the positions and how to apply: donghao51.github.io/open-positions/ Please feel free to share this post or tag anyone who might be interested!

Core research areas • Multimodal Learning and Generative AI • Reliable and Trustworthy Machine Learning • Generalization, Adaptation, and Continual Learning of Foundation Models and Agentic Systems • Applications in Robotics, Healthcare, Industrial Monitoring, and Beyond

Our Multimodal Intelligence Lab focuses on multimodal learning, generative AI, and vision-language models, with a specific emphasis on how foundation models and agentic systems can safely adapt and generalize to new environments.

Are We Making Progress in Multimodal Domain Generalization? A Comprehensive Benchmark Study arxiv.org/abs/2605.06643 github.com/lihongzhao99/M…

Happy to share that our survey paper on Multimodal Adaptation and Generalization has been accepted by 𝐓𝐏𝐀𝐌𝐈! Advances in multimodal adaptation and generalization: From traditional approaches to foundation models Paper: arxiv.org/abs/2501.18592 Code: github.com/donghao51/Awes…

We review state-of-the-art methods, benchmarks, and applications across classification, segmentation, OOD detection, and more. Finally, we highlight open challenges and future research directions.

🚀 New Survey Paper Alert! 📷 Excited to share our latest work: "Adapting Vision-Language Models Without Labels: A Comprehensive Survey" arxiv.org/abs/2508.05547 github.com/tim-learn/Awes…

In this survey, we introduce the first taxonomy of unsupervised VLM adaptation based on the availability of unlabeled visual data: 1️⃣ Data-Free Transfer 2️⃣ Unsupervised Domain Transfer 3️⃣ Episodic Test-Time Adaptation 4️⃣ Online Test-Time Adaptation

Vision-Language Models, such as CLIP, have demonstrated impressive zero-shot capabilities; however, in real-world deployments, their performance can decline without adaptation. Gathering labeled data is costly, so unsupervised adaptation has emerged as a powerful alternative.

🚀 𝗧𝗵𝗿𝗶𝗹𝗹𝗲𝗱 𝘁𝗼 𝗮𝗻𝗻𝗼𝘂𝗻𝗰𝗲 𝗗𝗣𝗨 was 𝗮𝗰𝗰𝗲𝗽𝘁𝗲𝗱 𝗮𝘀 𝗮 𝗦𝗽𝗼𝘁𝗹𝗶𝗴𝗵𝘁 𝗮𝘁 𝗖𝗩𝗣𝗥 𝟮𝟬𝟮𝟱! 🎉✨ 🌟 𝗗𝘆𝗻𝗮𝗺𝗶𝗰 𝗣𝗿𝗼𝘁𝗼𝘁𝘆𝗽𝗲 𝗨𝗽𝗱𝗮𝘁𝗶𝗻𝗴 (𝗗𝗣𝗨): A Breakthrough for Multimodal 𝗢𝘂𝘁-𝗼𝗳-𝗗𝗶𝘀𝘁𝗿𝗶𝗯𝘂𝘁𝗶𝗼𝗻 𝗗𝗲𝘁𝗲𝗰𝘁𝗶𝗼𝗻 (𝗢𝗢𝗗). Robust AI must identify unfamiliar inputs to avoid costly mistakes—but how do we tackle multimodal inputs like video, optical flow, and audio? 🤖🎬🎧 Our solution, DPU, dynamically updates class prototypes to capture intra-class variations, significantly boosting OOD detection performance. 🚦🔍 🎯 𝗞𝗲𝘆 𝗛𝗶𝗴𝗵𝗹𝗶𝗴𝗵𝘁𝘀: • 💡 First to reveal and tackle 𝘪𝘯𝘵𝘳𝘢–𝘤𝘭𝘢𝘴𝘴 𝘷𝘢𝘳𝘪𝘢𝘣𝘪𝘭𝘪𝘵𝘺 challenges in multimodal OOD detection. • ⚙️ 𝘗𝘭𝘶𝘨-𝘢𝘯𝘥-𝘱𝘭𝘢𝘺, 𝘮𝘰𝘥𝘦𝘭-𝘢𝘨𝘯𝘰𝘴𝘵𝘪𝘤 framework compatible with diverse OOD models. • 📈 Achieves state-of-the-art results, improving Far-OOD detection performance by up to 80%! 🙌 Big thanks to my fantastic collaborators: Li Li, Huixian Gong, Hao Dong, Tiankai Yang, Yue Zhao, from University of Southern California and ETH Zürich. 🔗 Read the full paper: lnkd.in/g43xX2PN 🛠️ Code Available: lnkd.in/ggTyfF55 𝗟𝗼𝗼𝗸𝗶𝗻𝗴 𝗳𝗼𝗿𝘄𝗮𝗿𝗱 𝘁𝗼 𝗱𝗶𝘀𝗰𝘂𝘀𝘀𝗶𝗼𝗻𝘀 𝗮𝘁 𝗖𝗩𝗣𝗥 𝟮𝟬𝟮𝟱! 𝗟𝗲𝘁'𝘀 𝗽𝘂𝘀𝗵 𝘁𝗵𝗲 𝗯𝗼𝘂𝗻𝗱𝗮𝗿𝗶𝗲𝘀 𝗼𝗳 𝗺𝘂𝗹𝘁𝗶𝗺𝗼𝗱𝗮𝗹 𝗔𝗜 𝗳𝘂𝗿𝘁𝗵𝗲𝗿 𝘁𝗼𝗴𝗲𝘁𝗵𝗲𝗿! 💬✨

Advances in Multimodal Adaptation and Generalization: From Traditional Approaches to Foundation Models arxiv.org/abs/2501.18592 github.com/donghao51/Awes…

These components enhance the model’s ability to distinguish unknown class samples during online adaptation by amplifying the entropy difference between known and unknown samples.

To leverage this, we propose two key components: Unknown-aware Adaptive Entropy Optimization (UAE) and Adaptive Modality Prediction Discrepancy Optimization (AMP).

Happy to share our #ICLR2025 paper on Multimodal Open-set Test-time Adaptation! openreview.net/forum?id=hj323…

We introduce an Agree-to-Disagree (A2D) training algorithm, inspired by the Modality Prediction Discrepancy phenomenon. We also introduce a new outlier synthesis algorithm NP-Mix that explores broader feature spaces and complements A2D to strengthen OOD detection performance.

Happy to share our #NeurIPS2024 𝗦𝗽𝗼𝘁𝗹𝗶𝗴𝗵𝘁 paper "MultiOOD: Scaling Out-of-Distribution Detection for Multiple Modalities". In this work, we introduce the first benchmark for Multimodal OOD Detection. paper: arxiv.org/pdf/2405.17419 code: github.com/donghao51/Mult…