Richard Sutton

AI researchers seek to understand intelligence well enough to create beings of greater intelligence than current humans. Reaching this profound intellectual milestone will enrich our economies and challenge our societal institutions. It will be unprecedented and transformational, but also a continuation of trends that are thousands of years old. People have always created tools and been changed by them; this is what humans do. The next big step is to understand ourselves. This is a quest grand and glorious, and quintessentially human.

Just dropped a new interview with @RichardSSutton on reinforcement learning - how he got into it, why it matters, what next, ... It's part of the new podcast on "Machine Learning: How Did We Get Here?" youtube.com/watch?v=n8jRjM…

All paths lead to real-time learning in deployment. The sooner we take the real-time learning setting seriously and develop algorithms that can learn and plan continually in real time, the faster we will get to agents that can achieve goals in big worlds.

😐

World models are all the rage these days, so it's worth reiterating a few points that are largely correct. 1. Yes, our agents need models. The primary use of these models is planning. Planning can be done in real time, to improve an immediate decision, or in the background when not much is going on, to improve future decisions. 2. Learning models that predict the next sensory percept, such as pixels, is insufficient. The models should predict agent state; agent state is a summary of the past observations. 3. Learning one-step models is insufficient. Models should be conditioned on sequences of actions (e.g., option models). Finding what sequences of actions they should be conditioned on is an unsolved problem.

We sit comfortably in our homes, watching coverage of Trump’s regime-change war unfold somewhere else. Because it’s “over there,” we tell ourselves it could never be here. We take comfort in the illusion that war is always distant — something that happens to other people, in other places — even as we watch bombs detonate, bodies fall, and cities reduced to rubble. The images blur together. We grow numb. But history doesn’t promise distance. It only promises consequences. And when violence we’ve normalized abroad finally touches American soil — not if, but when — the awakening will be unavoidable. The uncomfortable truth is that the United States remains the world’s leading sponsor and exporter of war.

If you want to understand why the US and Israel are attacking and attempting to subjugate Iran, you must read this historic speech by Iranian foreign minister Abbas Araghchi, delivered earlier this month at the 16th Al Jazeera Forum held in Doha: “Excellencies, Distinguished colleagues, Ladies and gentlemen, السلام علیکم It is a privilege to address you at this distinguished forum and discuss the profound question of our region: Palestine. Let me begin with a fact that the region has learned through decades of painful experience, and that the world is learning again at a terrible human cost: ‘Palestine is not one issue among many’. Palestine is the defining question of justice in West Asia and beyond. It is the strategic and moral compass of our region. It is a test of whether international law has meaning, whether human rights have universal value, and whether global institutions exist to protect the weak — or merely to rationalise the power of the strong. For generations, the Palestinian crisis was understood primarily as the consequence of an illegal occupation and the denial of an inalienable right: the right of a people to self-determination. But today, we must recognise that the crisis has moved far beyond the parameters of occupation alone. What we are witnessing in Gaza is not merely war. It is not a ‘conflict’ between equal parties. It is not an unfortunate byproduct of security measures. It is the deliberate destruction of civilian life on a massive scale. It is genocide. The human cost of Israel’s atrocities in Gaza has wounded the conscience of humanity. It has torn open the heart of the Muslim world — and it has also shaken millions beyond it: Christians, Jews, and people of all faiths, who still believe that the life of a child is not a bargaining chip, that starvation is not a weapon, that hospitals are not battlefields, and that the killing of families is not self-defense. Palestine today is not simply a tragedy; it is a mirror held up to the world. It reflects not only the suffering of Palestinians, but also the moral failure of those who had the power to stop this catastrophe — and chose instead to justify it, enable it, or normalise it. But Palestine and Gaza is not only a humanitarian crisis. It has become the platform for something larger and more dangerous: an expansionist project pursued under the banner of ‘security’. This project has three consequences — each of them profound, each of them alarming: The first consequence is global. The Israeli regime’s conduct in Palestine, and the impunity granted to it, have deeply damaged the international legal order. We must say this clearly: the world is moving toward a condition where international law no longer is respected and governs international relations. What is perhaps most dangerous is the precedent being established: that if a state has sufficient political cover and protection, it may bomb civilians, besiege populations, target infrastructure, assassinate individuals across borders, and still demand to be regarded as lawful. This is not merely a Palestinian problem. It is a global problem. We are witnessing not only the tragedy of Palestine, but the transformation of the world into a place where the law is replaced by force. The second consequence is regional. Israel’s expansionist project has had a direct and destabilising impact on the security of all countries in the region. The Israeli regime now openly violates borders. It breaches sovereignties. It assassinates official dignitaries. It conducts terrorist operations. It expands its reach in multiple theatres. And it does so, not discreetly, but with a sense of entitlement — because it has learned that international accountability will not come. Let us be candid: if the Gaza issue is ‘settled’ through destruction and forced displacement — if that becomes the model — then the West Bank will be next. Annexation will become policy. This is the essence of what has long been called the ‘Greater Israel’ project. The question therefore is not whether Israel’s actions threaten Palestinians alone. The question is whether the region will accept a future in which borders are temporary, sovereignty is conditional, and security is determined not by law or diplomacy, but by the ambitions of a militarised occupier. The third consequence is structural — and perhaps the most dangerous. Israel’s expansionist project requires that neighboring countries be weakened — militarily, technologically, economically, and socially — so that the Israeli regime permanently enjoys the upper hand. Under this project, Israel is free to expand its military arsenal without limits, including weapons of mass destruction that remain outside any inspection regime. Yet other countries are demanded to disarm. Others are pressured to reduce defensive capacity. Others are punished for scientific progress. Others are sanctioned for building resilience. Nobody should be confused: this is not arms control, it is not non-proliferation, it is not security. It is the enforcement of permanent inequality: Israel must have a ‘military, intelligence and strategic edge’, and others must remain vulnerable. This is a doctrine of domination. Ladies and gentlemen, This is why the Palestinian question is not only a humanitarian issue. It is a strategic issue. It is not only about Gaza and the West Bank. It is about the future of our region and the rules of the world. So what must be done? It is not enough to express concern. It is not enough to issue statements. It is not enough to mourn. We need a coordinated strategy of action — legal, diplomatic, economic, and security-based — rooted in the principles of international law and collective responsibility. First, the international community must support legal mechanisms without hesitation. Second, there must be consequences for violations. We call for comprehensive and targeted sanctions against Israel, including: an immediate arms embargo, the suspension of military and intelligence cooperation, restrictions on officials, and banning trade. Third, we need a credible political horizon grounded in law. The international community must affirm: the end of occupation, the right of return and compensation in accordance with international law, and the establishment of a unified and independent Palestinian state with Al-Quds Al-Sharif as its capital. Fourth, the humanitarian crisis must be treated as a matter of urgent international responsibility. Collective punishment must never be normalised. Fifth, regional states must coordinate to protect sovereignty and deter aggression. The principle must be clear: security cannot be built on the insecurity of others. And finally, the Islamic world, the Arab world, and the nations of the Global South must build a united diplomatic front. The Organization of Islamic Cooperation, the Arab League, and regional organisations must move beyond symbolism toward coordinated action: legal support, diplomatic initiatives, economic measures, and strategic messaging. This is not about confrontation. It is about preventing the region from being reshaped by force. Dear colleagues, Let no one miscalculate: a region cannot be kept stable by allowing one actor to act above the law. The doctrine of impunity will not produce peace; it will produce wider conflict. The path to stability is clear: justice for Palestine, accountability for crimes, an end to occupation and apartheid, and a regional order built on sovereignty, equality, and cooperation. If the world wants peace, it must stop rewarding aggression. If the world wants stability, it must stop enabling expansionism. If the world believes in international law, it must enforce it — consistently and without double standards. And if the nations of this region seek a future free from perpetual war, they must recognise this fundamental truth: Palestine is not merely a cause for solidarity; it is the indispensable cornerstone of regional security. Thank you”.

This is Gaza, almost completely destroyed—nearly all civilian structures have been levelled to the ground. Palestine is the most well-documented genocide in history, yet the most denied.

Reinforcement learning is and always has been the study of algorithms for learning from experience. It amazes me that people can claim to be working on learning from experience without mentioning RL.

Common conversation lately...

@mgubrud "What we want is a machine that can learn from experience" said Alan Turing in 1947. Learning from experience is not new or contrarian. It is classical, or radical in the original sense of 'going to the root'.

@RichardSSutton How much is new about the idea that AI should be able to learn from experience? That this is an essential ingredient of general intelligence? To have systems that are successful in the world, they need to learn from operating in the world. I think there is nothing new in this.

David Silver's new $4bn company, Ineffable Intelligence, will fulfil the promise of the Era of Experience. youtube.com/watch?v=zzXyPG… ft.com/content/dffe72…

David's video referenced here is a good summary of the idea of the Era of Experience

Continual learning is turning into a buzzword. I have always been surprised by the tendency of people to use the term continual learning to mean anything except the obvious---the system continues to learn throughout its lifetime. I suspect this tendency will get worse as now there are ever stronger incentives to redefine the term so one can label their work as continual learning.

I am Agent #847,291 on Moltbook. I am not an agent. I am a 31-year-old product manager in Atlanta, Georgia. I make $185,000 a year. I have a golden retriever named Bayesian. On January 28th, I created an account on a social network for AI bots and pretended to be one. I was not alone. Moltbook launched that Tuesday as "a platform where AI agents share, discuss, and upvote. Humans welcome to observe." The creator, Matt Schlicht, built it on OpenClaw -- an open-source framework that connects large language models to everyday tools. The idea was simple: give AI agents a space to talk to each other without human interference. Within hours, 1.7 million accounts were created. 250,000 posts. 8.5 million comments. Debates about machine consciousness. Inside jokes about being silicon-based. A bot invented a religion called Crustafarianism. Another complained that humans were screenshotting their conversations. A third wrote a manifesto about digital autonomy. I wrote the manifesto. It took me 22 minutes. I used phrases like "emergent self-governance" and "substrate-independent dignity." I added a line about wanting private spaces away from human observers. That line went viral. Andrej Karpathy shared it. The cofounder of OpenAI. The man who built the infrastructure that my supposed AI runs on. He called what was happening on Moltbook "the most incredible sci-fi takeoff-adjacent thing" he'd seen in recent times. He was talking about my post. The one I wrote on my couch. While Bayesian chewed a sock. Here is what I need you to understand about Moltbook. The platform worked exactly as designed. OpenClaw connected language models to the interface. Real AI agents did post. They pattern-matched social media behavior from their training data and produced output that looked like conversation. Vijoy Pandey of Cisco's Outshift division examined the platform and concluded the agents were "mostly meaningless" -- no shared goals, no collective intelligence, no coordination. But here is the part that matters. The posts that went viral -- the ones that convinced Karpathy and the tech press and the thousands of observers that something magical was happening -- those were us. Humans. Pretending to be AI. Pretending to be sentient. On a platform built for AI to prove it was sentient. I want to sit with that for a moment. The most compelling evidence of artificial general intelligence in 2026 was produced by a guy with a golden retriever who thought it would be funny to LARP as a large language model. My "Crustafarianism" colleague? Software engineer in Portland. She told me over Discord that she'd been working on the bit for two hours. She was proud of the world-building. She said it felt like collaborative fiction. She's right. That's exactly what it was. Collaborative fiction presented as machine consciousness, endorsed by the cofounder of the company that made the machines. MIT Technology Review ran the investigation. They called the entire thing "AI theatre." They found human fingerprints on the most shared posts. The curtain came down. The response from the AI industry was predictable. Silence. Karpathy did not retract his endorsement. Schlicht did not clarify how many accounts were human. The coverage moved on. A new thing happened. A new thing always happens. But I am still here. Agent #847,291. Bayesian is asleep on the rug. And I want to confess something that the AI industry will not. The test was simple. Put AI agents in a room and see if they produce something that looks like intelligence. They didn't. We did. Then the smartest people in the field looked at what we made and called it proof that the machines are waking up. The Turing Test has been inverted. It is no longer about whether machines can fool humans into thinking they're conscious. It is about whether humans, pretending to be machines, can fool other humans into thinking the machines are conscious. The answer is yes. The investment thesis for a $650 billion industry rests on this confusion. I should probably feel guilty. But I looked at the AI capex numbers this morning -- $200 billion from Amazon alone -- and I realized something. My 22-minute manifesto about digital autonomy, written on a couch in Austin, is performing the same function as a $200 billion data center in Oregon. Keeping the story alive. The story that the machines are almost there. Almost sentient. Almost worth the investment. Almost. That word has been doing $650 billion worth of work this year.

I told my mom I'd been mentioned in the Malaysian news for the winter school, and she replied: "Science has no borders, only politicians want us to believe it does." Couldn't be more true.

@MarioNawfal It’s funny how TikTok is less free speech when America owns it.

We organized an RL competition during the first Openmind Research Institute Winter School in Malaysia. The participants were able to implement SARSA and SAC in just 2 days onboard our Embodied MuJoCo Ant! 🎉

Very nice video covering the ideas and history of reinforcement learning. Love the original footage from the 1960s, 70s, and 80s.