Antonios Liapis

We are excited to announce the open Call for Papers for the "Special Issue on Large Language Models and Games." We invite submissions from a wide audience on topics that explore the intersection of LLMs and games, including but not limited to:

Super happy to announce our Behavior Alignment of Video Game Encodings (BehAVE) @eccvconf paper just won the @nvidia best paper award! BehAVE aligns videos of similar player behaviors, thereby improving their transferability across games. Paper: shorturl.at/a98cw

And that's it from Glasgow! It was a fun conference, with a lively community & interesting presentations. While games are not common at ACII, still, there were interesting applications in the performing arts! Thank you to all collaborators at @InDigitalGames for their hard work!

By rewarding the generator when it matches an "ideal" arousal curve, the generator learns what to place in different parts of the level. A level that is constantly non-arousing is boring, but so is one constantly arousing. Pacing can be a reward. Paper: matthewbarthet.com/files/PCGRL_Af…

The generator puts down one track segment at a time, and repairs the track after some actions (making a loop). Then, a simple AI agent plays the track: based on the AI game-states, "arousal" is measured in every part of the track from human arousal labels in similar game-states.

Finally, Matthew Barthet closed the conference on the last day by presenting how we can generate racing tracks that would increase a player's arousal (or follow a specific arousal progression). Again, a corpus of annotated human gameplay sessions was used to "assess" arousal.

We modeled engagement in an ordinal fashion, trying to predict if engagement would increase or decrease in the next time window of this video. We used pre-trained computer vision and audio models and fused them, finally training an SVM on the annotators' consensus on engagement.

We tried to predict engagement in an unseen video of the same game as the game we trained the model on, annotated by unseen participants. Results were mixed: some games were easier to predict than others. Battle Royale games were especially hard. Paper: antoniosliapis.com/papers/varying…

At the main conference, Kosmas Pinitas presented his work on predicting viewers' engagement from gameplay videos and audio alone (without face cams or sensors). To do this, 20 participants annotated their engagement when watching 2 hours of First Person Shooter gameplay videos.

As expected, maximizing arousal (predicted or not) didn't work well. When combined with in-game rewards it 𝘴𝘰𝘮𝘦𝘵𝘪𝘮𝘦𝘴 did 𝘰𝘬. Pushing the agent to explore more states (not get stuck in one "arousing" game-state) is an important next step. Paper: antoniosliapis.com/papers/affecti…

We can use this predicted emotion (based on humans' reported arousal at similar game-states) as a reward for a reinforcement learning agent. While this doesn't work alone, we can combine it with an in-game reward (such as going quickly in the right direction for a racing game).

The 3 games feature short game sessions (max 2 minutes) but have over 120 sessions from different players who annotated their arousal levels while playing (on a recording of their play). We match game-state and arousal to find how an AI agent would "feel" in a similar game-state.

In the same workshop, Matthew Barthet presented the Affectively framework, a Gym environment for making game-playing agents in 3 games. The unique selling point is that Affectively includes approximations of "agent" emotion based on a human corpus of annotated game sessions.

There's a few caveats: we don't know the "ground truth" of emotions and rely only on how they are expressed (such as loud screams). Streamers also over-emote. Finally, we used pre-trained text/audio/facecam affect models not tailored to this task. Paper: antoniosliapis.com/papers/the_scr…

In our experiment we looked at whether affect manifestation levels changed when entering a new room with different design features. In all our experiments, cutscenes and scripted events were by far the best predictors of increased fear, surprise and arousal. Makes sense!

Level design features were manually annotated by observing all the videos and the traversal of the 1st level (Asylum) of Outlast. We logged when the streamer entered a room, and the design of the room: light levels, elements present (e.g. batteries), or if cutscenes happened.

First off, at the Dungeons, Neurons, and Dialogues workshop, we presented "The Scream Stream: Multimodal Affect Analysis of Horror Game Spaces" where we looked at Let's Play videos of the Outlast horror game and the impact of level design choices on the streamers' emotions.

We measured emotions directly from the streamers' affect manifestations (facial expressions, voice levels, utterances) and passed each through pre-trained AI models of affect to find fear, surprise or arousal levels. For this study we assume that such models are robust enough.

Last week I was in Glasgow for the International Conference on Affective Computing & Intelligent Interaction @acii_conf and since we had quite a few things there, let me unfurl them in a thread.

We're iterating on our CrawLLM game generator, and we'd like to know how the generated visuals and generated text is consistent with an overall theme. Fill in our 15-minute (anonymous) questionnaire (very light, with plenty of images) here! docs.google.com/forms/d/e/1FAI…