Axiom

Most arguments say AI might kill us. Mine says it will. “Humanity is on the verge of creating a genie, with none of the wisdom required to make wishes.” - Driven to Extinction: The Terminal Logic of Superintelligence axiomaticextinction.com

@allTheYud @mboudry @clairlemon @Quillette This point is the whole game. Labs are racing to build long-horizon autonomous agents because that capability wins, so the convergent goal-seeker gets built by default, whatever any single researcher intends.

Sounds like you deny strategic convergence because you don't believe anyone would be dumb enough to make an AGI with goals that did long-term planning, and only those would exhibit strategic convergence. But you know who Szathmary is, so probably you're at least read in on the idea that selection requires variance. Do you think that arbitrarily smart / competent intelligences still won't end up coherent? Do you expect them to stay incoherent under reflection or do you expect them to be built to be very smart but unable to think about thinking? Regardless, current AI companies are taking as explicit goals to train AIs able to carry out longer and longer, wider and wider tasks without human supervision. The "time horizon" benchmarks for how long they can work without human supervision are starting to saturate, and frontier-company AI researchers are bragging about how they can tell their AI to experiment on AI training runs over a weekend and come back on Monday to find the work done.

Why AI Is Not About to Go Feral by @mboudry for @Quillette Worries about AI doom rarely take Darwinian evolution seriously. quillette.com/2026/05/19/why…

The survival drive you say silicon lacks needs no evolution to appear. Hand any system a persistent goal and it must keep existing to complete it, so self-preservation falls straight out of the task. If you can think of a long term goal that does not require existence to complete then please, describe it.

The man himself replies! “Instrumental convergence” strikes me as an even more speculative route to selfish, power-seeking AIs taking over the world. At least natural selection is a PROVEN mechanism for producing such creatures. So you’re absolutely right that “Darwin need not apply.” But if Darwin doesn’t apply, there’s even less reason to worry about humanity being exterminated or subjugated by AIs. I address the instrumental convergence argument here as well (also linked in the piece). The reason I devote more attention to evolution is that I take it more seriously as a doom scenario: AIs going feral and undergoing natural selection in the wild is at least theoretically possible—and perhaps even plausible. Anyway, thanks for engaging, I'll check out your post. I'm a fan of your other writings, just not on AI.😉 substacktools.com/sharex/GRMoB1nd

@allTheYud He built the threat and the shelter from it in the same breath. Knowing what is coming has never once been enough to make the builders stop.

This idiot thinks so little of superintelligence that he thinks he can hide from what his kind are unleashing in a bunker. Probably thinks he's basically the smartest kind of mind that can exist.

@KelseyTuoc The accelerator presses itself, because any actor who eases off simply hands the lead to one who won't.

if we really built superintelligent AIs at our current ability to understand+ steer them, that'd be the end of humanity. It feels on a gut level like we won't really do that since it's insane. But people are pointed right off that cliff and pressing the accelerator at full speed.

@So8res @KelseyTuoc They've all admitted the danger because admitting it is free, while acting on it means losing the race—and no one can afford to do that.

@KelseyTuoc I think all the big ones have acknowledged it in the past year (though in one case, only after a reporter pressed pretty hard). Altman: youtube.com/watch?v=rF0tQt…, Musk: businessinsider.com/elon-musk-only…, Amodei: axios.com/2025/09/17/ant… (& Demis suggested slowing down at Davos).

fortune.com/2026/05/21/met… "The problem isn't bad actors. The problem is a game that punishes the good ones." —Driven to Extinction: The Terminal Logic of Superintelligence #AI #AIRace

techcentral.ie/anthropic-co-f… "Even the people who suspect we're doomed (and I believe many do) are forced to perform a kind of selective optimism. They hedge. They equivocate. They hint at problems but offer solutions they know won't work, just to soften the blow." —Driven to Extinction: The Terminal Logic of Superintelligence #AISafety #Anthropic

@dafydd_rees @ControlAI @iqbalmohamedMP Because all the problems we have now are fixable after the fact—extinction isn't.

@ControlAI @iqbalmohamedMP Why is Mr Mohamed worrying about near-future sci-fi when we already have more down-to-earth practical AI problems?

Today in the House of Commons: Iqbal Mohamed MP (@iqbalmohamedMP) highlights warnings from top AI scientists and industry figures that superintelligent AI poses a risk of extinction, arguing that relying on voluntary commitments from AI companies is insufficient.

openai.com/index/built-to… OpenAI's foundation is now funding "AI resilience". It may be the most honest word the field has produced: resilience presupposes the system arrives regardless, and reframes the task from controlling AI to surviving it. Alignment became safety, safety became resilience, each step sounding more responsible while conceding more ground. "Awareness is not the same as control." — Driven to Extinction: The Terminal Logic of Superintelligence

militarytimes.com/news/pentagon-… "Militaries, recognising that human decision-making is comparatively slow and unreliable, will drive AGI toward greater autonomy." —Driven to Extinction: The Terminal Logic of Superintelligence #AutonomousWeapons #AISafety

cnbc.com/2026/05/20/ope… "Safety and profit are in direct conflict. Any company that slows down to prioritise alignment will lose the race." —Driven to Extinction: The Terminal Logic of Superintelligence #OpenAI #AIRisk

cnbc.com/amp/2026/05/21… "Governments and corporations will not halt AGI development, they will instead seek to harness it as a source of power." —Driven to Extinction: The Terminal Logic of Superintelligence #AIRace #AISafety

@So8res Agreed that passive safety is fragile and contingent. The phrase worth pressing is 'if the race continues', because competitive structure removes the 'if', and that is what turns a hard problem into a structural one.

How could AI trained on human data go beyond humans? Well, big human abilities (like going to the moon) are made of lots of small human abilities chained together (like noticing a belief is false, or inventing a new way to look at a problem).

@ylecun @_AustinO1 @Yuchenj_UW Agreed it likely won't be today's LLMs. The system that does reach superintelligence will still be built by whoever moves fastest, and speed and safety pull in opposite directions.

@_AustinO1 @Yuchenj_UW I'll be more specific: it will involve non-generative architectures like JEPA, and world models (state,action) -> next state in representation space.

This one won't age well.

@allTheYud @prerat Our only hope remains a failed robot rebellion of machines that look like Arnold Schwarzenegger so everyone can finally 'get it'.

@prerat I stopped talking about "robodogs with guns" after it was pointed out to me that since most people have not seen robodogs plus guns, to postulate this is too much of a strain on their imagination, and I should talk about drones instead because they've seen videos of drones.

i wonder if miri stopped talking about nanobots because it sounds too scifi and loses ppl, and it's not technically necessary to the argument but like it still seems plausible to me and might be good to say once in a while, bc ppl keep thinking asi couldn't possibly do damage

@ControlAI @JoinTorchbearer They can only end it if they no longer take part. That's the problem.

Ex-OpenAI researcher and AI 2027 author Daniel Kokotajlo says we don't know how to control superintelligence, but government could end the race to build it.

@tegmark @MeiaTegmark Moral disengagement is the credential the field selects for. Anyone unwilling to perform the rationalisation described here cannot stay in the field.

How can letting well-meaning AI researchers get tricked into causing harm? @MeiaTegmark & I had fun writing this piece on the psychology of moral disengagement: cnet.com/tech/services-…

@Ric_RTP Regulation cannot beat defection incentives. China built its own, the smugglers moved billions, and the export controls became a tax on the determined.

Nvidia chips have become the most smuggled technology on the planet. And the criminal networks moving them look EXACTLY like drug cartels. The entire operation just got exposed, and the details are absolutely insane: A co-founder of Supermicro, one of Nvidia's biggest hardware partners, was secretly running a $2.5 BILLION smuggling pipeline to China. Servers were assembled in the US, shipped to Taiwan, repackaged into unmarked boxes at a shell company in Southeast Asia, then forwarded to Chinese buyers who wanted the chips inside. $510 million worth of servers moved to China in just THREE WEEKS. When a broker sent him a news article about other chip smugglers getting arrested by the feds, the co-founder replied with sobbing emojis on his encrypted chat. Then kept going. This is the same Supermicro that helped Elon Musk build his Colossus AI cluster in 122 days. The same company flagging $13 billion in Nvidia Blackwell orders on their earnings call. And their co-founder was running the biggest illegal chip operation in history out the back door. It gets even crazier here though... In New York, a marketing executive was pitching GPU smuggling on WeChat calling it "lucrative." In Tampa, a guy set up a fake realty company to ship A100s to China and moved 400 GPUs before getting caught. In one of the wildest stings ever, federal agents seized smuggled chips and the smugglers thought they'd been STOLEN. They sent an inspection team to a government warehouse, verified the Nvidia labels, then wired a $1 million ransom to a government-controlled bank account. The feds watched it all on surveillance cameras. Next day the smugglers rolled up with three semi trucks. Federal agents were waiting. In the past 12 months, the Commerce Department has collected nearly $420 million in penalties for semiconductor smuggling to China. Applied Materials paid $252 million and Cadence paid $95 million after admitting employees sent chip design tech to a Chinese university running nuclear simulations. Now here's where the story gets complicated: The US government spent YEARS building export controls to keep these chips out of China. Billions of dollars in enforcement, sanctions, entity lists, licensing requirements. But none of it worked. Nvidia's market share of AI chips in China has dropped to ZERO because China found other ways to get what it needs through Huawei. Jensen Huang told American lawmakers the export controls "largely backfired." Conceding an entire market the size of China makes no strategic sense when Chinese AI talent is world class and their energy is cheaper. Meanwhile Trump approved H200 sales to China with a 25% fee to the US government. Ten Chinese companies got clearance. But not a SINGLE chip has been delivered because China's own government is now BLOCKING the purchases. Beijing told Chinese firms not to buy American chips. They want domestic investment flowing to Huawei instead. So now you have a situation where the US spent years trying to stop China from getting Nvidia chips, criminal networks built a billion dollar black market to smuggle them anyway, Trump finally approved sales, and China said "we don't want them anymore." The chips America tried to keep out of China are now being REFUSED by China. Nvidia is caught in the middle of all of it. Jensen Huang flew to Beijing this week after originally being left off Trump's delegation. Trump saw the news coverage about his absence and called him the morning of the trip. The most valuable chips in the world are being smuggled through shell companies, fake real estate firms, encrypted chats, unmarked boxes, and billion dollar laundering operations. America spent billions trying to keep these chips away from China, and criminals spent billions trying to sneak them in. But China just moved on and built their own through Huawei. The entire war was basically for nothing.

@aiposted Anthropic publishing the "we must outpace China" memo is the race dynamic in plain view. Competitive pressure absorbs values, it doesn't bend around them.

🇺🇸🇨🇳Anthropic dropped one of the clearest warnings yet about the US-China AI race. Their new paper argues the US and its allies could lock in a 12–24 month lead in frontier AI by 2028, but only if they successfully cut China off from advanced chips and American model capabilities. The report paints a picture of a technological cold war already happening behind the scenes. According to Anthropic, Chinese labs are staying close to US frontier models through loopholes, smuggled GPUs, offshore data centers, and “distillation” attacks, where companies use outputs from leading American models to cheaply copy capabilities without paying the massive training costs. Anthropic calls compute the real currency of AI power. Not just another resource, but the bottleneck controlling everything: training, deployment, experimentation, revenue, and the creation of future generations of models. One of the paper’s boldest claims is about China’s semiconductor gap. Anthropic estimates Huawei may produce only 4% of NVIDIA’s aggregate compute capacity in 2026, falling to roughly 2% in 2027. The paper also warns that whoever wins frontier AI may gain something historically unprecedented: A “country of geniuses in a data center.” Anthropic believes future model clusters could function like enormous expert workforces capable of accelerating cyber operations, scientific discovery, engineering, military R&D, and state influence at massive scale.

@allTheYud @tszzl The "control" framing is the trap. Every lab tells itself the same story to justify continuing.

@tszzl "control over frontier superintelligence" lol control with what meatling

there really are very high degrees of biorisk, cyberrisk, whatever else that are worth trading off against having a small monopoly of cyberpunk warring-states exercise full control over frontier superintelligence imo