Dr Waku

Superintelligence, if we develop it using anything like current methods, would not be under meaningful human control. That's the bottom-line of a new study I've put out entitled Control Inversion (link in second post.) Many experts I talk to who take superintelligence (real, general-purpose, autonomous superintelligence) seriously agree with this. But I wanted to take a deep dive to assess whether I think it's true, and why. Unfortunately, I'm now more convinced than ever. The basic argument is laid out below, but the core implication is worth putting up here: the global race to superintelligence is fundamentally misguided. Companies and countries are rushing to be first, believing whoever builds superintelligence will "grab the prize" of unprecedented power and wealth. This is dangerously wrong. Superintelligent systems would not bestow power on their creators, they would absorb it. Even if superintelligence does not "go rogue" (which it might), humans – including superintelligence's creators – would find themselves sidelined as it makes decisions faster than them, with more complex plans, and with strategic foresight beyond human comprehension. Whether quickly or slowly, losing control of superintelligence would inexorably lead to losing control to superintelligence. If humanity wants to stay in the driver's seat of our civilization, we need to give up this race. So what does the paper say?

EXCLUSIVE: Carl Feynman warns that building AGI likely means human extinction. (Yes, son of Richard Feynman & Al engineer of 45 years.) He's known Eliezer Yudkowsky since the '90s and was initially optimistic about building AGI. Today his P(Doom) is an alarming 43%. Thread 👇