I struggle with what to say about the new AI 2040: Plan A website. It all seems so implausible to me that I'm not sure where to start. There's an epistemic chasm between those who think superintelligence implies near-omnipotence and those (like me) who don't.

@binarybits I'm kinda interested in attempting to talk through the "is superintelligence real?" question with you if you're up for it. (I worked on the design of the website, and think figuring out how to bridge this gap is quite important)

@Raemon777 Sure that sounds fun! My short answer is "it depends what you mean by superintelligence." But I don't think my disagreements are mainly about AI capabilities. I agree that AI models are going to get a lot more capable.

@Raemon777 Or at least there's a good chacne they will.

@Raemon777 Incidentally, the website design is top-notch!

Thanks! Yep we worked on it for like a year. :) One relatively short/easy point I want to start with: Human neurons fire up to 200 times per second. Modern transistors can fire billions of a second. If someone builds an AI running as efficiently as the human brain, but running on transistors instead of neurons, even before we talk about any qualitative improvements, we're talking something millions of times faster than a human. And, AIs can run copies of themselves in parallel. So, imagine that you are such an AI. The president or senior generals are asking you for advice. In between each word they say, every second they pause, you get to think for 31 years. (i.e. map out everything they have said to you so far, look at every chart, every news broadcast, any new scientific paper published) You get to spend 31 years each second imagining different ways you could respond in the next second, and how that might play out. You can spin up agents that send emails to other people, and order supplies online, hire humans, talk to other rival world leaders or their loved ones, etc). And, like, the next second, you get to do it again. I'm curious if you get off the train before or after this point – like, do you not really believe AIs would get to think for 31 years each second, or, you don't see how that translates to being overwhelmingly smarter than humanity? ("near-omnipotence" feels like maybe a trap that feels too magical to be real, I think it's more helpful to just ask "okay, say you were an AI, and you got to think for 31 years every second, and say you didn't want exactly the same things that humans wanted. What would you do?)

@Raemon777 I don't think the crux of the disagreement is about the cognitive capabilities of AI models, narrowly defined. Here's my write-up of how I see the crux of the disagreement with the broad "singularist" worldview. understandingai.org/p/why-im-not-a…

Cool. My understanding of your points are: – you always need to interact with the world (there's only so much you can "think ahead") – a lot of knowledge is locked up inside the tacit knowledge of people (i.e. it'll be hard to teach AIs some kinds of tacit expert knowledge because that knowledge isn't written down anywhere) i.e. are you basically agreeing "there might be AIs that can think for 31 years every second, but, in those 31 subjective years, they will not be able to reinvent the tacit knowledge of the best humans, and will not be able to predict the outcomes of complex plans that affect lots of people and complex systems?" (I'm still interested in whether you basically believe or don't really believe that AIs will eventually be able to think for ~31 subjective years each second, since that affects the rest of my argument)

@Raemon777 I don't have a strong view about this but 31 years per second seems like a stretch. I don't think a neuron's firing time is very analogous to a CPU/GPU's clock cycle. Neurons are more complicated than transistors and seem to do more "work" per "cycle."

@Raemon777 But I'm absolutely willing to accept the premise that in the future there are AI models that can devote a lot more raw "brain power" to a problem in a shorter amount of time than any human can manage.

Cool. Okay I've read more of your past writing now and don't want to just rehash previous debates you've had. It sounds like we both agree on: - AIs will need to deal with some kind of empirical interactions - at this point, we should expect the first solidly "smarter than human all around" AI to appear in a world where other slightly-less-but-jaggedly smart AIs already exist and are used widely. The thing I don't have a very good handle on is your "life isn't chess." Clearly, life isn't a perfectly predictable thing. But also clearly (I assume you agree?) intelligence is able to make predictions and game plans even for chaotic situations full of unknown unknowns, to some degree. I'd like to find a question that gets more specific about "exactly _how much_ will AI be bottlenecked on new knowledge they can't get, when it comes to strategic outmanuevering and/or inventing novel science" as opposed to "world = chess, yes/no?" ... But, first, quick concrete question: > I don't have a strong view about this but 31 years per second seems like a stretch. > I'm absolutely willing to accept the premise that in the future there are AI models that can devote a lot more raw "brain power" to a problem in a shorter amount of time than any human can manage. I'd prefer to stay in realms you find plausible so we're only having one Big Intractable Debate at a time. In terms of "just thinking faster" (and otherwise qualitatively similar to a human, except being able to call up subagents and make internet API requests etc), what's the upper range you find plausible? i.e. is it more like 10x faster, 1000x, 100,000x, 1,000,000x? (I think different speeds here open pretty massively different possibility spaces on how to leverage the thinking. I realize your take is that this number doesn't matter. I think it does, but I want to check if we're talking about a zone that falls inside your upper/lower bound of "yep seems pretty plausible") (and to be clear this is not talking about modern LLMs, or about timelines, just, "what's possible

@binarybits @Raemon777 Ok, well, we know that one individual human can do significant damage to the world (mainly by manipulating other humans) putting that together with the above concession it seems you’re close to at least admitting that an AI can soon do more damage than the worst human in history