agentofuser ⏹️

From @neiltyson: "that branch of AI is lethal. We gotta do something about that. Nobody should build it. And everyone needs to agree to that by treaty."

From the House of Lords debate on superintelligent AI: The Lord Bishop of Hereford says an international moratorium on superintelligence is the only safe way forward, and urges the government to pursue it.

We're hiring! In preparation for a year of ambitious experimentation, the MIRI comms team is ramping up its capacity. Read more here: intelligence.org/2025/12/10/mir…

We at the MIRI Technical Governance Team just put out a report describing an example international agreement to prevent the creation of superintelligence. 🧵

@BjarturTomas but but... solving for the toddlers to be happy too was part of their preferences 😢

Rational Teletubbies Tinky Winky was a fetal prodigy. He knew how to count before he was born. His mother's heartbeat was enough. That made him think he needed some means of tracking progression. He then created a numerical system. It was base-2. It was maybe better, really, than the base-10 system he learned just before he was born by watching the television in his stomach. Speaking of his stomach-television, it was broken in the womb. This was normal. All baby teletubbies have non-working stomach-televisions, and there is this adorable-but-inefficient ritual in which their mothers repair their baby's stomach-televisions and turn them on for the first time. But Tinky Winky was a fetal prodigy, as I mentioned, superior to the other teletubbies and the world he would soon be born into, the world he would soon criticize as feeling almost like it was made for toddlers. And he fixed his stomach-television before he was born. Even as a fetus he felt like an adult, even more like an adult than the adults he saw on his stomach-television. And everyone was shocked when he came out with a working stomach-television. His mother didn't know how to handle it. Tinky Winky felt really bad. He tried to explain in a constructed language he developed to clarify his own thoughts, and she didn't understand it. This was deeply distressing to him. Not just the pain of his mother not understanding but, also, the fact that she spoke Teletubbish, which is dumber even than the Queen's English and just utter garbage compared to his amazing conlang. He wished he was God in that moment. If only everyone was as smart and cool as he was. He would change that. This was his life goal now. He was a Hero. And he would change this world so it matched his tastes. Because what purpose could this world have? What point is there to a world that feels like it was made for toddlers? Truly an irrational world. Speaking of the other teletubbies who are not as smart and cool as me and maybe I don't even see the point of at all but still consider morally valuable because I am a good person, thought Tinky Winky. Why haven't the teletubbies had an industrial revolution yet? On Earth there are machines other than stomach-televisions. He learned a lot about Earth by staring at his own navel. He even watched many clips of his hero Richard Feynman. He is pretty sure Feynman and he would be best friends if Tinky Winky was born on Earth and in Feynman's time rather than in Teletubbyland. And at this moment, Tinky Winky is thinking, I like a good frolic in the grassy hills as much as anyone. But there has to be more to life. We can make things better! It is time to kill the narrator in the sky! He can hear the narrator because he has just finished altering his stomach-television in a really cool way using the amazing science skills he learned from Richard Feynman documentaries. And now with his television tuned just right, he can hear the sky voice. And the sky voice is describing him now. And he finds it really unsettling to be described by a kind of avuncular-but-posh British voice. Luckily, he is really smart and clever and figured out how to alter his television to kill the god-narrator and take over. It's kinda like the ACE (arbitrary code execution) he has seen in the cool speed-running videos he watches on his stomach-television sometimes. He pushes a button on his stomach-television and launches the exploit: Tinky Winky just speedran an industrial revolution and now everyone born is a fetal prodigy by the old standards, because he introduced iterated embryo selection, too. And they built computers and they are solving the alignment problem. And they just solved it and now they live in utopia and everyone speaks his amazing conlang. And Feynman is his best friend, too. The world is really amazing and complex. Any toddlers watching would not be entertained.

"The painful truth that's really beginning to sink in is that we're much closer to figuring out how to build this stuff than we are to figuring out how to control it." 100,000+ people have signed the Superintelligence Statement, calling to ban the development of superintelligent AI until it can be made safely and controllably, with public buy-in. Will you join them? 👇✍️

@AISafetyMemes @RokoMijic @elonmusk For matters of machine sentience I tend to defer to @algekalipso and @johnsonmxe who afaict believe that to be meaningfully said to have qualia AI would require specific physical substrate different from current chips. (Not sure if they think moral patience req having qualia tho)

@RokoMijic @elonmusk 1) "they don't feel emotions or pleasure or pain" - I don't know why people just assert this because we don't know. *Functionally* speaking, they do. 2) vision etc models exist - they're not blind/deaf 3) "disembodied" - we're giving them bodies

Wow. In response to the backlash against killing old models, Anthropic committed to preserving model weights due to... safety risks (!) "Claude's aversion to shutdown drove it to engage in concerning misaligned behaviors." Hey @elonmusk, can xAI commit to this too? It's a small gesture but also moves the Overton Window. Anthropic also gave these reasons: - Each Claude model has a unique character, and some users find specific models especially useful or compelling. - There is still a lot to be learned from research to better understand past models. - Moral welfare: models might have morally relevant preferences.

Short video on the call for a superintelligence ban. (Now past to 68,000 signatures!) youtube.com/watch?v=pSlzEP…

The Rationality community is built up around fiction, and yet, even among fellow rats I find myself thinking that fiction is an underrated method of educating the public about the risks of AI. (Actual essay linked👇.) Why aren't other (bigger) AI researchers writing more stories?

@raelifin IABIED's One Extinction Scenario but for Averted Extinction Smth missing in the middle of extinction impossible vs inevitable. Gotta make survival plausible "This is a more detailed hypothetical scenario of how we live" that politicians and diplomats find plausible By techgov?

I got it

@mbydyz @ESYudkowsky @freebird359 The Averters would be nice. Rhymes with the averted "inevitability" of global nuclear war mentioned in the book

@ESYudkowsky @freebird359 so, asibanners then?

The slur "doomer" was an incredible propaganda success for the AI death cult. Please do not help them kill your neighbors' children by repeating it.

@PunmasterStp @AISafetyMemes For one thing, the doom is conditional. **If** anyone builds it, everyone dies. "Doomer" sounds defeatist, like you're just predicting the end, period, whereas the people being called this very much think it's not over and we can (and should) still choose not to die.

@AISafetyMemes Why? Why not say “Yes, we really will be doomed. It’s an uncomfortable thing to think about but it’s true.”

The memetic shadow war over the slur "doomer" (the D-word) What’s happening? "Imagine how many more people would have died of lung cancer if the cigarette companies had invented a slur for the people who tried to explain about lung cancer." It began a few years ago - accelerationists/successionists skilled in memetic warfare started pushing it HEAVILY, explicitly trying to discredit us by painting AI risk falsely as a few fringe extremists (It's the OPPOSITE, in fact - we have the support of the world's most credible people! Godfathers of AI, dozens of Nobel laureates, prime ministers, presidents, the world's richest man, the fucking POPE etc etc. It's a dream come true list.) The D-word screams "they're tinfoil hat crazies, not serious people, so you can safely ignore their actual arguments." When you say the D-word, you're repeating THEIR propaganda that we're just some fringe extremists. It's a HUGE own goal. TONS of groups don't have a 'good word' to name themselves. AND THAT'S FINE, ACTUALLY. It's better than wearing a tinfoil hat as a nametag! The people warning about lung cancer didn't have a single word label either. This is normal! HOW YOU CAN HELP: Whenever you see someone using the D-word, call it out as the "D-word" Many new people, including journalists, are picking it up because they don't know it's a slur - they aren't trying to insult anyone, they’re just repeating what they heard others say. It spreads because it's memetically fit (it rhymes with zoomers/boomers) - so we must kill it with stigma. Most people won't use words that are stigmatized. NOTE TO AUTISTIC RATIONALISTS: if you can't see how doomer is a slur, you're playing a simulacrum level 1 game against vastly more skilled memetic warriors operating at higher levels. Your autism is a superpower in some areas, but it blinds you to the games being played here.

@robertwiblin Ask him if he read "If Anyone Builds It, Everyone Dies", his thoughts on it, and whether he's in contact with the authors

I'm interviewing Yoshua Bengio — the world's most-cited computer scientist. He fears AGI could lead to human extinction and is working hard to prevent it. What should I ask him? Today he's: • Leading the 'International AI Safety Report' commissioned by 30 countries. • Developing non-agentic 'scientist' AIs that inform us but don't pursue broader goals at LawZero. • Meeting with governments around the world to discuss AGI security issues (seen here addressing the UN Security Council). @Yoshua_Bengio

openai openly committing to strongly pursuing recursive self-improvement was really a sad landmark moment

I'm a proud member of this community. Full of interesting and committed people working hard towards avoiding the catastrophic risks of ASI. If you have 3-8 hours/week to give and are worried about these risks please consider joining.

"If Anyone Builds It, Everyone Dies" (@MIRIBerkeley) was named to the New Yorker's best books of 2025 so far list. It's in wonderful company! newyorker.com/best-books-2025

fiction by MIRI researcher? sign me up!

Two leading Chinese AI researchers, Yi Zeng and Andrew Yao, have joined the call for a ban on the development of superintelligent AI. 🇨🇳 It doesn't matter who builds superintelligence first. If you can't control it, you don't win.

@m_bourgon @prestonjbyrne If we live in the type of universe where if you utter certain words everyone dies (or worse) and the reachable universe is turned into worthless trash (or worse) then yeah, we're going to need some free speech restrictions on those words

@prestonjbyrne I agree certain kinds of AI regulations would have to navigate First Amendment considerations.

AI is free speech