Cor-Paul Bezemer

And one more! The finalists for Best Academic Research are Vishnu Nair, Ian Gauk, Kathrin Gerling, Anna Chen, and Jesse J Martinez. This category sits outside the usual voting process, but you can find links to their papers at gaconf.com/gaconf-awards-… #gamedev #accessibility

GlitchBench: Can large multimodal models detect video game glitches? paper page: huggingface.co/papers/2312.05… Large multimodal models (LMMs) have evolved from large language models (LLMs) to integrate multiple input modalities, such as visual inputs. This integration augments the capacity of LLMs for tasks requiring visual comprehension and reasoning. However, the extent and limitations of their enhanced abilities are not fully understood, especially when it comes to real-world tasks. To address this gap, we introduce GlitchBench, a novel benchmark derived from video game quality assurance tasks, to test and evaluate the reasoning capabilities of LMMs. Our benchmark is curated from a variety of unusual and glitched scenarios from video games and aims to challenge both the visual and linguistic reasoning powers of LMMs in detecting and interpreting out-of-the-ordinary events. We evaluate multiple state-of-the-art LMMs, and we show that GlitchBench presents a new challenge for these models.

My division at @cse_gbg is hiring 1-2 Assistant / Associate Teaching Professors in Software Engineering and/or Interaction Design. Positions will be 100% & permanent from start.

Tomorrow brings day 2 of @ICPEconf #ICPE2022 and a session I've been looking forward to: The #DataChallenge! Organized by @corpaul, @swy351, and myself, and using a dataset from @MongoDB, we invited participants to do something cool with the dataset. And they have! /1

Very excited to see the #ICPE2022 keynote from @Google's John Wilkes on "Building Warehouse-Scale Computers," taking place at the 4/9-4/13 virtual conference. Learn more about this year's keynotes at: ow.ly/U0Nc50IrGb0 Don't forget to register - it's free!