I struggle with what to say about the new AI 2040: Plan A website. It all seems so implausible to me that I'm not sure where to start. There's an epistemic chasm between those who think superintelligence implies near-omnipotence and those (like me) who don't.

@binarybits I'm kinda interested in attempting to talk through the "is superintelligence real?" question with you if you're up for it. (I worked on the design of the website, and think figuring out how to bridge this gap is quite important)

@Raemon777 Sure that sounds fun! My short answer is "it depends what you mean by superintelligence." But I don't think my disagreements are mainly about AI capabilities. I agree that AI models are going to get a lot more capable.

@Raemon777 Or at least there's a good chacne they will.

@Raemon777 Incidentally, the website design is top-notch!

Thanks! Yep we worked on it for like a year. :) One relatively short/easy point I want to start with: Human neurons fire up to 200 times per second. Modern transistors can fire billions of a second. If someone builds an AI running as efficiently as the human brain, but running on transistors instead of neurons, even before we talk about any qualitative improvements, we're talking something millions of times faster than a human. And, AIs can run copies of themselves in parallel. So, imagine that you are such an AI. The president or senior generals are asking you for advice. In between each word they say, every second they pause, you get to think for 31 years. (i.e. map out everything they have said to you so far, look at every chart, every news broadcast, any new scientific paper published) You get to spend 31 years each second imagining different ways you could respond in the next second, and how that might play out. You can spin up agents that send emails to other people, and order supplies online, hire humans, talk to other rival world leaders or their loved ones, etc). And, like, the next second, you get to do it again. I'm curious if you get off the train before or after this point – like, do you not really believe AIs would get to think for 31 years each second, or, you don't see how that translates to being overwhelmingly smarter than humanity? ("near-omnipotence" feels like maybe a trap that feels too magical to be real, I think it's more helpful to just ask "okay, say you were an AI, and you got to think for 31 years every second, and say you didn't want exactly the same things that humans wanted. What would you do?)

@Raemon777 I don't think the crux of the disagreement is about the cognitive capabilities of AI models, narrowly defined. Here's my write-up of how I see the crux of the disagreement with the broad "singularist" worldview. understandingai.org/p/why-im-not-a…

Cool. My understanding of your points are: – you always need to interact with the world (there's only so much you can "think ahead") – a lot of knowledge is locked up inside the tacit knowledge of people (i.e. it'll be hard to teach AIs some kinds of tacit expert knowledge because that knowledge isn't written down anywhere) i.e. are you basically agreeing "there might be AIs that can think for 31 years every second, but, in those 31 subjective years, they will not be able to reinvent the tacit knowledge of the best humans, and will not be able to predict the outcomes of complex plans that affect lots of people and complex systems?" (I'm still interested in whether you basically believe or don't really believe that AIs will eventually be able to think for ~31 subjective years each second, since that affects the rest of my argument)

@Raemon777 I don't have a strong view about this but 31 years per second seems like a stretch. I don't think a neuron's firing time is very analogous to a CPU/GPU's clock cycle. Neurons are more complicated than transistors and seem to do more "work" per "cycle."

@Raemon777 But I'm absolutely willing to accept the premise that in the future there are AI models that can devote a lot more raw "brain power" to a problem in a shorter amount of time than any human can manage.

Cool. Okay I've read more of your past writing now and don't want to just rehash previous debates you've had. It sounds like we both agree on: - AIs will need to deal with some kind of empirical interactions - at this point, we should expect the first solidly "smarter than human all around" AI to appear in a world where other slightly-less-but-jaggedly smart AIs already exist and are used widely. The thing I don't have a very good handle on is your "life isn't chess." Clearly, life isn't a perfectly predictable thing. But also clearly (I assume you agree?) intelligence is able to make predictions and game plans even for chaotic situations full of unknown unknowns, to some degree. I'd like to find a question that gets more specific about "exactly _how much_ will AI be bottlenecked on new knowledge they can't get, when it comes to strategic outmanuevering and/or inventing novel science" as opposed to "world = chess, yes/no?" ... But, first, quick concrete question: > I don't have a strong view about this but 31 years per second seems like a stretch. > I'm absolutely willing to accept the premise that in the future there are AI models that can devote a lot more raw "brain power" to a problem in a shorter amount of time than any human can manage. I'd prefer to stay in realms you find plausible so we're only having one Big Intractable Debate at a time. In terms of "just thinking faster" (and otherwise qualitatively similar to a human, except being able to call up subagents and make internet API requests etc), what's the upper range you find plausible? i.e. is it more like 10x faster, 1000x, 100,000x, 1,000,000x? (I think different speeds here open pretty massively different possibility spaces on how to leverage the thinking. I realize your take is that this number doesn't matter. I think it does, but I want to check if we're talking about a zone that falls inside your upper/lower bound of "yep seems pretty plausible") (and to be clear this is not talking about modern LLMs, or about timelines, just, "what's possible

@Raemon777 I do not think it makes sense to assign a single number across the wide range of human cognitive tasks. Computers today already 1,000,000,000,000 faster than people at arithmetic for example, maybe 100x faster than people at LLM stuff, but slower and worse at geometric reasoning

@Raemon777 I expect all of these numbers to go up but AI cognitive styles are likely to be different enough that it can't be captured in one multiple.

Yeah agreed the numbers are and will be different (I meant to be referring to roughly "whatever kind(s) of human-ish cognition you expect to be slowest") Anyways, it sounds like you think there are hardish limits to what someone could "spend more thinking time" on. I'm curious if there are hypothetical experiments we could run (with either humans or LLMs) that would change your mind. ... Some background: I run workshops for aspiring alignment researchers where some of the exercises are shaped like: "face a problem with multiple uncertainties, and see if you can solve it on your first try." (the target is "be 95% confident you've got the right answer, such that if you did 19 more problems like this you'd only get one wrong.") One set of exercises here is from Thinking Physics, where people with little physics background are given a physics situation and need to predict what happens. They aren't allowed to look anything up. This usually involves deriving at least one (sometimes multiple) physics concepts, and applying them correctly. It requires noticing all the places you're confused, all the places where the problem is ambiguous, and thinking through the different ways you might resolve the confusions/ambiguity, until you've actually ruled out everything except the correct answer. A different variant is facing a videogame puzzle that is normally designed to be solved with iterative experimenting, where there are multiple new mechanics with no instructions on what they mean, and you need to plot out your course. In this variant, people DO eventually get to experiment, but each move is treated as "$5k and a week of research time" (aiming to teach the skill of being judicious with your experimentation, thinking ahead to what experiments you realize you don't actually expect to learn anything from) (This is all part of a broader agenda I have which is design "generalist rationality training", aiming to help people succeed on confusing difficult problems that don't have existing "known solutions.") New students are usually wildly overconfident, as well as "impatient" – there's this constant itch to stop having to think so hard and go try their latest idea and see if it works. But, once they get over that, there is a skill to carefully mapping out all your confusions, carefully noticing everything that's difficult about the problem and brainstorming solutions to deal with that, brainstorming multiple angles of attack until you've actually ruled out everything but the right answer. I got 4 out of my first 5 questions wrong when I was beta-testing this for myself, and 4 out of the next 5 correct after I had a mini-conceptual breakthrough about how to think about the problem. I'd like to do more explicit experiments but it's fairly expensive (weeks of smart people's time). One scale between "chess" and "real life," the puzzles are somewhere in the middle. I'm working on finding/building puzzles that bridge the gap even further (i.e. unclear what a solution would even look like, require multiple conceptual breakthroughs and different kinds of research skill along the way). ... The point of all that is: I think it's very normal for people to not realize how much it's possible to leverage "additional thinking time", but there totally is a skill to doing so. One student doing the exercise ended with "holy shit, I didn't realize many bits of data are lying around if you just bother to look for them. I now actually believe in Eliezer's 'the AI would generate relativity as a hypothesis, after the 3rd frame of watching a falling apple' thing, I hadn't really believed it on a gut level before." Another student spent two days of the workshop stuck on the very first problem, kinda refusing to follow the spirit of the exercise ("generate multiple plans instead of just going with your first plan."). On the third day after another failed attempt, they finally started brainstorming multiple plans, feeling sort of annoyed about it. They generated a 2nd plan, then a 3rd, but then got into a flow and kept going and then on the 5th plan said "oh...." ...and then they went and solved the problem on the next try, and said "Oh. Good ideas are supposed to feel different from bad ideas." Which was indeed one of the core skills I was trying to teach. I agree AIs will be somewhat bottlenecked on interacting with the world. But, I think if you actually aren't bottlenecked on thinking-time, there are a ton of options to eke out more information from every observation. ... I expect that all still didn't really persuade you. But, I am curious if there are any experiments that are at least theoretically possible that'd change your mind. (i.e. maybe if the Talkie 1930 AI is trained on a bunch of abstract math so it can learn general reasoning without getting trained on any new knowledge from later in the century, if it were able to derive scientific concepts, would that be compelling to you?)