Wayne Chi ✈️ ICML

New preprint alert 🚨 Can LLM agents develop video games? We release GameDevBench, the first benchmark evaluating agentic game development in a game engine, Godot. We also present two simple multimodal feedback mechanisms that lead to immediate performance gains. /🧵

🎶How can we learn to follow along to a music performance in a score without direct supervision?🎶 Introducing FuSiLi (Fused Sinkhorn-Localized Similarity)! A multimodal contrastive learning approach that learns precise local alignments using only global supervision. 🧵

Would people like to see K3 evaluated on gamedevbench?

@garywu_eth @OpenAI Working on getting it here :)

@iamwaynechi @OpenAI fable on here?

GPT-5.6 Sol is a frontier model for game development With high reasoning, it's +7.2 pp better than the previous best model on GameDevBench This is the biggest jump I've seen in agentic game development capabilities yet Huge congrats to @OpenAI for a fantastic model!

@MsShannon112 @OpenAI Coming soon!

@iamwaynechi @OpenAI When will you test fable?

@ajs6888 @OpenAI 沒錯沒錯

@iamwaynechi @OpenAI 看起来游戏原型迭代要更卷了

@RonfortMartin @OpenAI It's legit a huge jump in quality

@iamwaynechi @OpenAI 7.2pp jump on agentic game dev is legit. The shift from 'can this model code?' to 'how good is it at iterative game design?' is where we actually measure progress now.

More meaningful to understand true game dev capabilities of GPT-5.6 Sol on the GameDevBench eval than any slop minecraft clones you see on the timeline

Just ran GPT-5.6 Sol on GameDevBench... this model is kind of cracked

At that rate you'll soon be post-economic

Introducing 🎼🫧MulTTiPop, an evaluation dataset of multitrack pop music for automatic music transcription! MulTTiPop contains 3.5 hours (572 segments) of aligned multitrack pop audio and MIDI, and can be previewed at gclef-cmu.org/multtipop 🧵1/7

@ethantsliu Thanks!

@iamwaynechi congrats! 🎉 cool work

LLMs are much less capable judges for code than we might expect. Super excited that our paper on LLM judges for code has been accepted to COLM! In it, we discuss this weakness and provide a pipeline for understanding and diagnosing issues with LLM code preferences. This was led by two wonderful students @RyanShar01 and @AdityaM129. It was an absolute pleasure mentoring them on this project.

Check out the full paper here: arxiv.org/abs/2603.24586

@_albertgu I want to see albert swinging on a latecomer 🦵🏼🦶🏼🦵🏼🦶🏼

Cartesia is hosting an ICML party tomorrow (Thursday) night! it'll go late, but come early or i may have to bounce you again luma.com/gy1b4ryq