I struggle with what to say about the new AI 2040: Plan A website. It all seems so implausible to me that I'm not sure where to start. There's an epistemic chasm between those who think superintelligence implies near-omnipotence and those (like me) who don't.

@binarybits I'm kinda interested in attempting to talk through the "is superintelligence real?" question with you if you're up for it. (I worked on the design of the website, and think figuring out how to bridge this gap is quite important)

@Raemon777 Sure that sounds fun! My short answer is "it depends what you mean by superintelligence." But I don't think my disagreements are mainly about AI capabilities. I agree that AI models are going to get a lot more capable.

@Raemon777 Or at least there's a good chacne they will.

@Raemon777 Incidentally, the website design is top-notch!

Thanks! Yep we worked on it for like a year. :) One relatively short/easy point I want to start with: Human neurons fire up to 200 times per second. Modern transistors can fire billions of a second. If someone builds an AI running as efficiently as the human brain, but running on transistors instead of neurons, even before we talk about any qualitative improvements, we're talking something millions of times faster than a human. And, AIs can run copies of themselves in parallel. So, imagine that you are such an AI. The president or senior generals are asking you for advice. In between each word they say, every second they pause, you get to think for 31 years. (i.e. map out everything they have said to you so far, look at every chart, every news broadcast, any new scientific paper published) You get to spend 31 years each second imagining different ways you could respond in the next second, and how that might play out. You can spin up agents that send emails to other people, and order supplies online, hire humans, talk to other rival world leaders or their loved ones, etc). And, like, the next second, you get to do it again. I'm curious if you get off the train before or after this point – like, do you not really believe AIs would get to think for 31 years each second, or, you don't see how that translates to being overwhelmingly smarter than humanity? ("near-omnipotence" feels like maybe a trap that feels too magical to be real, I think it's more helpful to just ask "okay, say you were an AI, and you got to think for 31 years every second, and say you didn't want exactly the same things that humans wanted. What would you do?)

@Raemon777 I don't think the crux of the disagreement is about the cognitive capabilities of AI models, narrowly defined. Here's my write-up of how I see the crux of the disagreement with the broad "singularist" worldview. understandingai.org/p/why-im-not-a…

Cool. My understanding of your points are: – you always need to interact with the world (there's only so much you can "think ahead") – a lot of knowledge is locked up inside the tacit knowledge of people (i.e. it'll be hard to teach AIs some kinds of tacit expert knowledge because that knowledge isn't written down anywhere) i.e. are you basically agreeing "there might be AIs that can think for 31 years every second, but, in those 31 subjective years, they will not be able to reinvent the tacit knowledge of the best humans, and will not be able to predict the outcomes of complex plans that affect lots of people and complex systems?" (I'm still interested in whether you basically believe or don't really believe that AIs will eventually be able to think for ~31 subjective years each second, since that affects the rest of my argument)

@Raemon777 I don't have a strong view about this but 31 years per second seems like a stretch. I don't think a neuron's firing time is very analogous to a CPU/GPU's clock cycle. Neurons are more complicated than transistors and seem to do more "work" per "cycle."

@Raemon777 But I'm absolutely willing to accept the premise that in the future there are AI models that can devote a lot more raw "brain power" to a problem in a shorter amount of time than any human can manage.

@binarybits @Raemon777 This format of interrogating each area of disagreement is incredibly helpful, please continue this convo in public or consider posting it!