Foundations of Cooperative AI Lab (FOCAL) at CMU

New positions at FOCAL! cs.cmu.edu/~focal/index.h…

Two honorable mentions for papers at the ICML AI4GOOD workshop! Paper led by @EmanuelTewolde and Xiao Zhang: ‘CoopEval’ arxiv.org/abs/2604.15267 Paper led by @Akash190104 and @EmanuelTewolde: ‘Do LLMs Take Care of Their Own?’ openreview.net/forum?id=neTpZ…

The AI4Good workshop is today (Korea) @ ICML! @EmanuelTewolde is presenting the CoopEval paper, but also follow-up work with CAIRF fellow Akash Kundu about whether LLM agents cooperate with others that they perceive as similar. openreview.net/pdf?id=neTpZbi… #overview" target="_blank" rel="nofollow noopener">trustworthy-ai-for-good.github.io/#overview

Today (Korea time) at ICML in the 5pm session, Vijay Keswani is presenting our position paper "We Need Practical AI Alignment Methods that Mirror Human Reasoning!" presentation: icml.cc/virtual/2026/p… paper: openreview.net/pdf/a895d4cfe5…

Standing by the CoopEval poster right now at #ICML2026! 📊👋 If you are interested in cooperation and multi-agent safety, game-theoretic mechanisms, or modern LLM agents defaulting to defection, come by board #4008 and let's chat! 🇰🇷🫱🏿‍🫲🏾

@EmanuelTewolde is presenting our CoopEval work at ICML on Wednesday 10:30am session (or catch him at the alignment workshop today)! presentation: icml.cc/virtual/2026/p… arXiv: arxiv.org/abs/2604.15267

Congrats to our recent undergrad+MS grad Jerick Shi on receiving the best paper award at ICML NExT-Game for our paper "When Agents Lie: Premeditation, Persistence, and Exploitation in Repeated Games" w/ Terry J. C. Zhang, Bernhard Schölkopf, @ZhijingJin! openreview.net/forum?id=v8nYI…

our recursive joint simulation paper (w/ Vojta and @C_Oesterheld) accepted to Synthese! TLDR: When players in a game run a simulation of themselves (incl. further subsimulations), that's equivalent to an infinitely repeated game and so allows cooperation. arxiv.org/abs/2402.08128

One of my open math problems apparently got resolved by ChatGPT 5.5 Pro (Ryan O'Donnell prompted it better than I did!), though the proof was so hard for me to read that it seemed easier to just prove it myself. More thoughts on implications for math here: aifails.substack.com/p/even-more-on…

@FOCAL_lab @EmanuelTewolde arxiv.org/abs/2604.15267

@FOCAL_lab member @EmanuelTewolde is presenting his CoopEval work at EPFL on Monday (14:30) and it will be on zoom! (paper link in comment) memento.epfl.ch/event/ai-cente…

(1/4) Can remembering more of the past make AI agents less cooperative? In our new paper, we study LLM agents in repeated social dilemmas. The key variable is not how many rounds they play, but how much prior interaction history they can access when making each decision.

(2/4) Surprisingly, longer recall often degrades cooperation. Across 7 LLMs and 4 repeated social dilemma games, agents with longer histories often shift away from forward-looking cooperation and toward retrospective grievance-tracking.

(3/4) The mechanism is not just “too much context.” It is what the agents remember: replacing histories with synthetic cooperative records restores cooperation, and ablating explicit CoT reasoning often reduces the collapse. We call this the memory curse.

(4/4) Paper link: arxiv.org/abs/2605.08060 Big thanks to my collaborators! @Jack_Litq, @zzzoooeee321, Xin Luo, Haoxuan Zeng, @EmanuelTewolde, Tai Sing Lee, @tonghanwang, @ckingsford, @conitzer

In this paper led by Jerick Shi @Jerick1380, we propose a taxonomy of LLM deception. arxiv.org/abs/2604.04788

(3/3) Also, I don't understand how some people think AGI is just around the corner but the risks are easily manageable! Of course their positions may not be captured accurately here.

(2/3) I'm sure we all have thoughts on our descriptions -- I certainly worry about many other AI risks current and future in addition to scaled misinformation, and I actually think the world is too focused on LLMs-as-chatbots -- but still impressive.

(1/3) Wow, I (and many friends) made it onto this "Mapping AI" map! Does anyone know the methodology? Who is behind it? mapping-ai.org/map

(3/3) You can play with interleaving tokens yourself here! (... works better with better models...) cs.cmu.edu/~focal/alterna…

(2/3) We've also been interested in interleaving tokens for philosophical reasons. This chapter based on a talk I gave at a Duke conference about tests of consciousness discusses how coherent LLM text doesn't necessarily come from any clear unit entity. cs.cmu.edu/~conitzer/LLMc…