I struggle with what to say about the new AI 2040: Plan A website. It all seems so implausible to me that I'm not sure where to start. There's an epistemic chasm between those who think superintelligence implies near-omnipotence and those (like me) who don't.

@binarybits You should write your own concrete predictions (ideally a whole scenario, which involves a ton of predictions & would expose your thought process very clearly) so that people can get evidence about how well you understand the problem through how well you're able to predict it.

@gayestgaymer1 I don't know what's going to happen, so any specific concrete predictions I make more than ~2 years out are very likely to be either obvious (the sun will come up every morning) or wrong.

@binarybits I remember reading my first article written by you when I was in school & I've been consistently reading your writing in the ~19 years since then, so hopefully it's clear that I respect what you have to say even if I struggle to empathise with your position on AI.

@binarybits We need to be able to predict what will happen with AI, before we're irreversibly committed to the outcome of any given course of action. "We can't predict it" isn't acceptable, because one potential outcome is that we all die, something we can't gamble with.

@binarybits You don't think everyone dying is a possibility, okay, but that is a very confident prediction about the world 2+ years in the future & it seems to be the only prediction you're willing to make. That prediction underpins your entire argument & other smart people disagree.

@gayestgaymer1 I guess I don't see this as a binary question of "go forward" or "don't go forward." I think it would be very difficult to completely stop AI development even if the world's governments all wanted to.

@gayestgaymer1 And it's not obvious that slowing the pace of AI development, or driving development underground, would reduce the medium-term risks. Plausibly it would shift control to entities that are less likely to be careful.