I struggle with what to say about the new AI 2040: Plan A website. It all seems so implausible to me that I'm not sure where to start. There's an epistemic chasm between those who think superintelligence implies near-omnipotence and those (like me) who don't.

@binarybits @sebkrier Thank you for stating clearly your main crux. I'm one of the people who does believe superintelligence implies near-omnipotence.

@binarybits @sebkrier What do you think? Do you think we're going to build superintelligence, but stay in control of it indefinitely? What do you think it was about humanity that led us to become the dominant species on earth?

@JaEsf @sebkrier I think we achieved our dominance over nature by building an advanced material civilization over centuries via division of labor. I think it's a mistake to view AI as a separate species rather than part of our civilziation.

@binarybits @sebkrier OK, but why did we manage to build that and not one of the millions of other species? I don't think definitions of species are worth getting into. An individual AI instance can do amazing things, and will soon be able to do much more amazing things.

@JaEsf @binarybits @sebkrier While our intelligence was part of what enabled us to build our dominant civilization it now exists as a collective that transcends individual minds and capabilities. AIs don’t come into a virgin world they are born integrated into an existing superstructure.

@JaEsf @binarybits @sebkrier Importantly these civilizational structures have had a long history integrating and managing novel and powerful entities. Think states and corporations and the like. Modern liberal institutions are an extraordinary technology for maintaining pluralistic equilibria

@LeoBreston @binarybits @sebkrier This is all pretty abstract. Corporations aren't like a human being except much much smarter faster etc. They're just bureaucracies of humans. The best corporation in the world couldn't beat Magnus Carlsen once in 100 games (without a computer).

@LeoBreston @binarybits @sebkrier If something comes along that can beat Magnus 100/100 games, not just at chess but at essentially every other mental benchmark, that will be a massive challenge for the rest of us. And that's the road we're on here.

@JaEsf @LeoBreston @sebkrier Yes, a key question here is whether power-seeking activities (like running big companies and winning wars) are similar to chess. I say they are not in some important ways that make the DeepBlue/AlphaZero analogy inapplicable.

@binarybits @JaEsf @LeoBreston I always liked this paper, wish Danzig released a 2.0 update :) cset.georgetown.edu/publication/ma…

@binarybits @LeoBreston @sebkrier We can separate three components of power-seeking: 1. Having power as an objective. 2. Ability to correctly assess what steps are needed to gain power. 3. Ability to competently act on those steps.

@binarybits @JaEsf @sebkrier A key thing here is that power is not arbitrary or unconstrained; it’s mediated by our systems and institutions which are more robust than individual humans. Even if they are super human at those i don’t see why it necessarily implies they will suddenly breakout of the system.