Gabriel

For deontology's sake, and just in case it's not obvious what my beliefs are. We are on the path to human extinction from superintelligence. At our current pace, it's very likely that we hit a point of no return by 2027. It is plausible that we hit one sooner.

New blogpost: Corporations are not your friend Sometimes I say things along the lines of “Corporations are not your friend.” By this, I don’t mean “The people working at Corporations are big meanies :(“ What I mean is that it is a type error. Link below!

In just a year we went from zero to 110+ UK lawmakers recognizing superintelligent AI as a global security threat. 200+ lawmakers briefed in 4 countries about extinction risk from superintelligence, parliamentary debates & hearings in the UK and Canada, & more. Now we scale.

We've just published our 2025 Impact Report! At a glance: ~1 in 2 UK lawmakers we briefed supported our campaign, for a total of 110+ supporters 2 House of Lords debates on superintelligence & extinction risk A series of hearings at the Canadian Parliament (+ more in thread)

Ex-OpenAI researcher and AI 2027 coauthor Daniel Kokotajlo: There's a 70% chance superintelligence leads to human extinction. "We at the AI Futures Project think that there's a 70% chance of all humans dead or something similarly bad." "All humans dead?" "Correct. Extinction."

Judd says "If AIs are aligned, then it is not bad that they wish for autonomy". But there is no scientific consensus on a definition of alignment, how to recognise it or how to measure it. This is just vibes, a smarter-sounding way of talking about Objective Goodness.

I have joined @ControlAI as US Director and have moved to Washington DC! If you're in the area, give me a shout! If you want to read my retrospective on Conjecture and lessons learnt, link in next post. Excited for a new chapter!

...and that's bad, right @karpathy?

In The Guardian: An AI security researcher reports that an AI at an unnamed California company got "so hungry for computing power" it attacked other parts of the network to seize resources, collapsing the business critical system. This relates to a fundamental issue in AI: developers do not know how to ensure the systems they're developing are reliably controllable. Top AI companies are currently racing to develop superintelligence, AI vastly smarter than humans. None of them have a credible plan to ensure they could control it. With superintelligent AI, the stakes are much greater than collapse of a business system. Leading AI scientists and even the CEOs of the top AI companies have warned that superintelligence could lead to human extinction.

absent a substantive ideological basis for why you want to do this, the political dimension of spaceflight becomes a blank canvas onto which aspiring technocrats paint their antisocial fantasies. only the naive and deranged can organize around fully arbitrary goals like this

Is AI progress accelerating? Or is the recent breakneck pace only temporary? I don't have a confident answer to this, but I wrote an article that makes the picture a lot clearer. forecastingaifutures.substack.com/p/will-ai-prog…

There always have been specification gaming examples, ie this list from 2018 vkrakovna.wordpress.com/2018/04/02/spe… You could get GPT3 to lie and blackmail. The difference is what researchers are willing to frame as "scientific".

Yeah, I agree slowdowns/pauses on either hardware or frontier AI work or both are good. If "it's unrealistic because the other guy will move forward anyway", then the right solution is for the entities involved (corps and govs) to publicly say "I am willing to not do [X] if [entities A, B, C] also make a similar pledge", and move in good faith from there.

My ancestors buried half their children. All mine are alive. My ancestors' house had a dirt floor. Mine is wood. I have indoor plumbing, I have hot water, I have never in my life hauled a full bucket half a mile and I probably never will. Do you know how rare it is, in human history, for small children to wear shoes? Mine have multiple pairs. I can speak to my relatives who live thousands of miles away, for free, at any time. Video, if we want video. With machine translation, if we speak different languages. The original Library of Congress had 740 books in it. I have more than that. If I run out of books in my home my local public library has 350,000. If I want to take a hundred books with me on vacation, they all fit on a device that fits in my purse. I have heat in the winter and AC in the summer and a washing machine and I have never, ever, ever had to scrub a dress clean by hand in the stream. I can look up recipes from more than a hundred different countries and I've tried dozens of them. I ride a clean and modern train across my city for $4, or take a robot taxi if I'm out too late for the train. I donate $40,000 every year to the cause of getting healthcare to the world's poorest people and even after the donations I never have to think about whether I can afford a book, or a pair of shoes, or a cup of coffee. There is a great deal more to fight for, of course. I hope that our descendants will look back on our lives and list a thousand ways they're richer. Maybe we ourselves will do that, if some of the crazier stuff comes true. But the abundance is all around you and to a significant degree you aren't feeling it only because fish don't notice water.

There's little chance of coming back from that. To red-team ideas, I strongly recommend ensuring having talked to people more than to LLMs, in aggregate. If the people around you are not good enough, look for better people. Else, you're cooked.

We are proud to share The AI Chronicle! Built by Torchbearer Luke McNally (@pseudomoaner) to collate the stories, spanning seven decades, that should have made the headlines. The loss-of-control and extinction risks posed by artificial superintelligence should be front-page news every day.

AGI boosters keep boasting historical raises and the world-changing power of AGI But one mention of regulation, USG's intervention, or the CCP stealing their work, and they're the little birthday boy

The stakes of the singularity go up to human extinction. Even pre-singularity, we have no idea for how to deal with govs (USG and CCP) or corps (OpenAI and Anthropic) having AGI. While the research itself may be fun :), it somehow strikes me as an auxiliary consideration.

The correct response to realizing this is what you are building is to notice that if anyone builds it, everyone probably dies and then rather than care who owns it you DON'T F***ING BUILD IT.

No one is in control. Connor Leahy (@NPCollapse) joins @JonhernandezIA to explain that this is scarier than having an enemy. Enemies have plans. We have a race where every company builds faster because the alternative is losing. Nobody chose this. It is just happening.

LLM agents constantly guess people's intentions. To infer the implicit context behind a query. But also to get away with worse responses while looking helpful. Or to decide on which requests are legitimate. In pactice, through RL, AI corps build AIs to be super-manipulators.