HKUST NLP

Great work! @ZhaoweiWang4

Check out our new work on multimodal long-context!

How far are we from Artificial General Intelligence—and what might follow as Artificial Superintelligence? IJCAI Closing Panel opened by James Kwok, #IJCAI2025 Programe Chair. share.google/DbB2Ky20bCc3B0… #Montreal #AI

🎉 Congrats to our students on the new #EMNLP2025 paper acceptances! 1⃣ LLM Natural-Formal Hybrid Reasoning arxiv.org/abs/2505.23703 2⃣ Text-Instructed Image Editing on Medical Domain arxiv.org/abs/2506.01921 3⃣ Knowledge Boundary Aware Multi-Compositional Reasoning arxiv.org/abs/2504.21773 4⃣ End-to-End Optimized Multimodal RewardRAG

Actually, we implemented this kind of capability in our AdaCtrl paper three months ago by injecting difficult-aware tags (i.e., easy, hard, adaptive) to trigger different reasoning behaviors of LLMs. Paper: arxiv.org/pdf/2505.18822

Excited to present our work, MMBoundary, at #ACL2025! Come chat with us at our poster session! 📍 Hall 4/5, Session 12: Poster Session 4 🗓️ Wednesday, July 30 ⏰ 11:00-12:30

If you are at ACL, please talk to our members of @HKUSTKnowComp and @hkustNLP big family

🤯 Multimodal LLMs can be confidently wrong. A single early mistake in perception can lead to a completely incorrect answer. 🚀Introducing our work, MMBoundary, a new framework to make MLLMs aware of their own knowledge boundaries! 🧵 Paper:arxiv.org/abs/2505.23224 #ACL2025

Find us at the poster booth ✨Wed 7/30 11am Hall 4/5 ✨

[1/n] "𝘔𝘢𝘵𝘤𝘩𝘪𝘯𝘨 𝘤𝘶𝘦𝘴 𝘧𝘰𝘳 𝘪𝘥𝘦𝘯𝘵𝘪𝘤𝘢𝘭 𝘰𝘣𝘫𝘦𝘤𝘵𝘴, 𝘥𝘪𝘴𝘵𝘪𝘯𝘤𝘵 𝘢𝘵𝘵𝘳𝘪𝘣𝘶𝘵𝘦𝘴 𝘧𝘰𝘳 𝘶𝘯𝘪𝘲𝘶𝘦 𝘰𝘯𝘦𝘴." Such 𝙘𝙧𝙤𝙨𝙨-𝙘𝙤𝙣𝙩𝙚𝙭𝙩 𝙫𝙞𝙨𝙪𝙖𝙡 𝙧𝙚𝙖𝙨𝙤𝙣𝙞𝙣𝙜 is extremely simple and straightforward for the human cognitive process, but 𝗾𝘂𝗶𝘁𝗲 𝗰𝗵𝗮𝗹𝗹𝗲𝗻𝗴𝗶𝗻𝗴 𝗳𝗼𝗿 𝗰𝘂𝗿𝗿𝗲𝗻𝘁 𝗹𝗮𝗿𝗴𝗲 𝘃𝗶𝘀𝗶𝗼𝗻 𝗹𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗺𝗼𝗱𝗲𝗹𝘀 (𝗟𝗩𝗟𝗠𝘀), especially across multiple images and videos ‼️ 𝙒𝙝𝙮, and 𝙝𝙤𝙬 𝙘𝙖𝙣 𝙬𝙚 𝙞𝙢𝙥𝙧𝙤𝙫𝙚 upon this major shortcoming in future research? 🔍 🔥🚀 Check out our latest release, “𝗩𝗟𝗠𝟮 -𝗕𝗲𝗻𝗰𝗵: 𝗔 𝗖𝗹𝗼𝘀𝗲𝗿 𝗟𝗼𝗼𝗸 𝗮𝘁 𝗛𝗼𝘄 𝗪𝗲𝗹𝗹 𝗩𝗟𝗠𝘀 𝗜𝗺𝗽𝗹𝗶𝗰𝗶𝘁𝗹𝘆 𝗟𝗶𝗻𝗸 𝗘𝘅𝗽𝗹𝗶𝗰𝗶𝘁 𝗠𝗮𝘁𝗰𝗵𝗶𝗻𝗴 𝗩𝗶𝘀𝘂𝗮𝗹 𝗖𝘂𝗲𝘀”! 🚀🔥 🔗 Project page: vlm2-bench.github.io 📑 Paper: arxiv.org/abs/2502.12084 💻 Github: github.com/vlm2-bench/VLM… 🤗 Huggingface: huggingface.co/datasets/Sterz… --- Work led by our amazing team of students Jianshu, Dongyu, and Renjie at The Hong Kong University of Science and Technology. Find us at the poster booth ✨Wed 7/30 11am Hall 4/5 ✨

📢 Join us for the ACL Mentorship Session @aclmeeting #ACL2025NLP #NLProc • Session Link: mentorship.aclweb.org/schedule • Ask Questions: tinyurl.com/y2v2j462 Mentors: • @May_F1_ (@hkust) • @d_aumiller (@cohere) • @vernadankers (@Mila_Quebec) • @ziqiao_ma (@UMichCSE) • @WeijiaShi2 (@uwnlp) • @ZhijingJin (@UofTCompSci) • @OanaIgnatRo (@SantaClaraUniv)

Heading out to #ACL2025 in Vienna with six main/finding papers to present! 🇦🇹✈️🤩 Would love to chat about research on multimodal model reasoning and agent, as well as opportunities in my group @hkustNLP. Please DM if you'd like to meet!

#COLM2025 Our work has been accepted to COLM 2025😊 Looking forward to discussing Scalable Oversight and Synthetic Data with old and new friends in Montréal @COLM_conf ！

Excited to share our new survey on the reasoning paradigm shift from "Think with Text" to "Think with Image"! 🧠🖼️ Our work offers a roadmap for more powerful & aligned AI. 🚀 📜 Paper: arxiv.org/pdf/2506.23918 ⭐ GitHub (400+🌟): github.com/zhaochen0110/A…

Thrilled to share a major milestone: the culmination of a 15-month project, ATLAS—a new benchmark in event graphs and conceptualization! This journey began with Probase in 2012, evolved through ASER (2019), AbstractATOMIC (2022), and AbsPyramid (2023), and now realizes a decade-long dream of building a web-scale knowledge graph integrating entities, events, and a conceptualization framework. It’s also a tribute to Marvin Minsky’s K-Lines Theory. I still vividly recall 2012 when I worked with Haixun, including intern Fangting Xia, spent months exploring event semantic primitives and conceptualization, only to pivot due to limitations. Fast forward to the era of large language models, and we’ve finally brought this vision to life! Someone argued that knowledge graphs are obsolete in the age of large models. We disagree—it’s about having high-quality graphs that scale with these models. That’s why we invested HK$1.5M to process Wikipedia, academic abstracts, and 3% of web data from Dolma, creating a 1B-node graph—the best we could achieve. While entities cover 70% of factual queries, adding event semantic primitives boosts coverage to 90%. This is why text decomposition is gaining traction. Our extracted knowledge graph excels on complex factual queries like HotpotQA, MMLU, and FELM, making it the largest open-source GraphRAG dataset available. This work is the result of 10+ students and nearly 10 collaborators pouring their hearts into it. I’m deeply grateful for their contributions to this meaningful milestone, and I hope they’ve gained as much from the journey as I have. Big shoutout to anyone curious about pushing this further! Parsing the entire web’s data would cost ~HK$40M, and I’d love to collaborate with bold thinkers to make it happen. Paper: arxiv.org/abs/2505.23628 Code & Resources: github.com/HKUST-KnowComp… Project: pypi.org/project/atlas-… Feedback and critiques are welcome! Let’s keep advancing the field together. #KnowledgeGraphs #ATLAS #GraphRAG

We studied both rule-based and model-based verifiers and found that each has unique limitations. Rule-based verifiers are often unreliable, even in math, and are unavailable in many domains. Model-based verifiers can be easily hacked. In our paper, we construct simple adversarial patterns to successfully crack several open model-based verifiers. 🧐Interestingly, as the community shifts toward generative verifiers, we found them to be far more vulnerable than discriminative ones. This suggests that discriminative verifiers might be more robust against reward hacking in RLVR.

Great to see the wonderful series of work that @WangCarrey has been leading at UIUC. We also had a fun collaboration recently together with my incoming PhD student Shijue. Check out our latest release 𝘈𝘥𝘢𝘊𝘵𝘳𝘭: 𝘈𝘥𝘢𝘱𝘵𝘪𝘷𝘦 𝘢𝘯𝘥 𝘊𝘰𝘯𝘵𝘳𝘰𝘭𝘭𝘢𝘣𝘭𝘦 𝘙𝘦𝘢𝘴𝘰𝘯𝘪𝘯𝘨 𝘷𝘪𝘢 𝘋𝘪𝘧𝘧𝘪𝘤𝘶𝘭𝘵𝘺-𝘈𝘸𝘢𝘳𝘦 𝘉𝘶𝘥𝘨𝘦𝘵𝘪𝘯𝘨 🔥🔥🔥 . . . arxiv.org/pdf/2505.18822

🚀 We release MMLongBench: Benchmark for evaluating long-context VLMs. 📊 13,331 examples across 5 tasks: – Visual RAG – Many-shot ICL – Needle-in-a-haystack – VL Summarization – Long-document VQA 📏 Lengths: 8 / 16 / 32 / 64 / 128K 🔍 Benchmarking both thoroughly & effectively!

🚀 Our new survey "From Automation to Autonomy: LLMs in Scientific Discovery" is live! 📜 Paper: arxiv.org/pdf/2505.13259 ⭐ Awesome Github Repo: github.com/HKUST-KnowComp… #LLMs #ScientificDiscovery #AI #NLP #Agent #AIforScience @HKUSTKnowComp @hkustNLP (cont'd)👇