Foundations of Cooperative AI Lab (FOCAL) at CMU

New positions at FOCAL! cs.cmu.edu/~focal/index.h…

Shutdown safety valves (giving AI an objective to shut itself down and a means to do so if some capability/access/... is dangerously high) now on arXiv arxiv.org/abs/2603.07315

We (w/ Berker, Hartman, Liu, @C_Oesterheld) introduce a new measure of approximate strategyproofness -- having how many truthful copies of yourself is guaranteed to be at least as effective as misrepresenting your preferences? arxiv.org/abs/2602.22838

Looking forward to presenting "Shutdown Safety Valves for Advanced AI" in the next session at IASEAI'26 in Paris! (Basic idea: give AI a goal of shutting itself down.) cs.cmu.edu/~conitzer/shut…

(3/3) Also thanks to the amazing committee: Fei Fang, Tuomas Sandholm, Ben Levinstein, and Stuart Russell! (There's gotta be a better way to do these zoom pictures though...)

(1/3) Congratulations to my student Caspar Oesterheld @C_Oesterheld (2nd from left) on a successful defense and an important dissertation "New foundational ideas in cooperative AI"!

gave brief closing talk "Cooperative AI: How to set up a contest for environments that aren’t zero-sum" @ Algorithms with a Purpose '26 @CarnegieMellon, where teams competed programming bots to run restaurants, with the option to sabotage your competitor. awap.acmatcmu.com

Our paper in the alignment track of AAAI'26: what if human feedback is unstable over time? @RealAAAI #AAAI2026 arxiv.org/abs/2511.10032

Tomorrow (F) morning at 11am SG time, Emin Berker and @EmanuelTewolde will give a talk at @RealAAAI on our paper below! arxiv.org/abs/2512.16895

Can we always guarantee a representative outcome that is fair to every subgroup, no matter how diverse their preferences are? 🗳️ We are excited to share our new paper at #AAAI2026 that tackles this in multi-winner approval voting! 🧵

our upcoming AAAI @RealAAAI paper on cooperative game theory in multi-winner voting, via an automated reasoning approach (using not satisfiability but mixed-integer linear programming); led by Emin Berker and Emanuel Tewolde! arxiv.org/abs/2512.16895

The videos from this summer's Cooperative AI @coop_ai retreat are online now! Below is mine on "Cooperative AI via Simulations" youtube.com/watch?v=I8ec6t…

Excited to present our position paper in the 4:30pm @NeurIPSConf poster session today! We discuss the problem that AI systems may suspect they're being tested and act differently as a result, & how to approach this with game theory. arxiv.org/abs/2508.14927 neurips.cc/virtual/2025/l…

New paper from our lab, led by UG Xander Heckett: a game-theoretic view of AI safety via debate, where two agents argue for different courses of action and we try to set the rules so that the one that is right will win. arxiv.org/abs/2511.23454

(continuing) I worry too many people think of today's LLM-based chatbots as a good model for studying AGI. Even if they're on the path to AGI, they're not at all what AGI would be like. Here's a chapter I wrote about this recently. philsci-archive.pitt.edu/26351/

article on Nash (a CMU alum!) with a mention of our @FOCAL_lab and the Nash75 conference at Oxford as well cmu.edu/mcs/news-event…

‘Foundations and Frontiers of Cooperative AI’ - with @conitzer (@CarnegieMellon) - youtu.be/nfnbxKWGM50?fe…

Excited that our paper "AI Testing Should Account for Sophisticated Strategic Behaviour" was accepted to the first NeurIPS position paper track! We argue that AI systems may act strategically w.r.t. the possibility they are currently being tested. arxiv.org/abs/2508.14927

I got to give the opening lecture for the 2025 Cooperative AI Summer School and the video is now online! "Foundations and Frontiers of Cooperative AI" youtube.com/watch?v=nfnbxK… @coop_ai

Congratulations to Emin Berker for receiving the Best Talk Award at COMSOC'25! (in the picture with coauthor Edith Elkind; paper arxiv.org/abs/2502.16973 )

Tomorrow at COMSOC, Emin Berker will present our paper on creating clone candidates & strategic nomination (e.g., does the leaderboard rule on Chatbot Arena create incentives to submit/withhold slightly differently fine-tuned versions of the same model?). arxiv.org/abs/2502.16973