I struggle with what to say about the new AI 2040: Plan A website. It all seems so implausible to me that I'm not sure where to start. There's an epistemic chasm between those who think superintelligence implies near-omnipotence and those (like me) who don't.

@binarybits Relative to cats, we are super powerful. It seems plausible that there is some similar OOM of power between us and ASI.

@MichaelTontchev If you sent a human 10 million years into the past, he'd be far smarter than any animals around him but he'd struggle to survive, to say nothing of achieving power over the other animals.

@binarybits @MichaelTontchev This is true, but what if you sent Kenya approx 2000 yrs into the past? A relatively average military and economy would collapse Carthage, burn the Roman Republic in the space of a few weeks. Wrt superintelligence(s), one of us is making a category error

@1carringtone @MichaelTontchev Right, that's my point — the power of modern nation-states comes from their technology, infrastructure, institutions, manpower, etc. more than raw intelligence.

@binarybits @MichaelTontchev Raw intelligence contributes to the ability to build all of these. It's likely the central feature

@binarybits @1carringtone @MichaelTontchev So where do you think all those facets you mentioned - technology, infrastructure, institutions and “manpower” come from? These are byproducts of raw intelligence. If they weren’t, there would be more than one species on Earth capable of these feats.

@binarybits @1carringtone @MichaelTontchev Well, but in the plan A scenario AI has all of these, it's not just a brain floating in a vacuum.

@binarybits @1carringtone @MichaelTontchev But isn't all that (except manpower) a direct consequence of the higher intelligence threshold compared to cats or gorillas? Wouldn't, in theory, a swarm of ASIs be capable of dramatically more ambitious feats?

@binarybits @1carringtone @MichaelTontchev today AI is raw intelligence, soon it's heavily integrated into our tech, infra, institutions, manpower, etc.

@binarybits @1carringtone @MichaelTontchev the question then is can you from inside a datacenter create technology, institutions, acquire 'manpower' (robots?), build institutions, etc. IMO this is the intuition Amodei wants to invoke with "country of geniuses in a datacenter" -- if you can, ASI is go