Timothy B. Lee

@F_Vaggi My understanding is that superforecasters on average are not very worried about AI doom scenarios.

@binarybits AFAIK - one of the main authors is the 2nd highest forecaster. As I acknowledged, there is a huge amount of uncertainty and my "gut feeling" tells me it's implausible, but, I also have to keep repeating myself "several of these people are better than me at forecasting"

I struggle with what to say about the new AI 2040: Plan A website. It all seems so implausible to me that I'm not sure where to start. There's an epistemic chasm between those who think superintelligence implies near-omnipotence and those (like me) who don't.

@Raemon777 I expect all of these numbers to go up but AI cognitive styles are likely to be different enough that it can't be captured in one multiple.

@Raemon777 I do not think it makes sense to assign a single number across the wide range of human cognitive tasks. Computers today already 1,000,000,000,000 faster than people at arithmetic for example, maybe 100x faster than people at LLM stuff, but slower and worse at geometric reasoning

@dktrpo @Raemon777 Yes.

@Raemon777 @binarybits Don’t you think just “thinking longer” with ~fixed intelligence and bounded input data has diminishing returns? Even 1,000,000x human thinking speed may have bounded returns, which may not fall dramatically farther than regular human strategizing.

@F_Vaggi Some (by no means all or even most) have a good track record of predicting how model capabilities have improved. I think "how will this impact the world" is a different question requiring different expertise to forecast well.

@binarybits What do you make of the argument that the people who believe in it have been much better forecasters of what has happened so far than virtually anyone else? Shouldn't that cause you to update your priors?

@Vert_Noel There are lots of things humans (and normal computers for that matter) can trivially do. I don't think anything on your list is obviously trivial.

@binarybits OK, so there's nothing that superintelligence can trivially do?

@atlas_node Yes.

@binarybits I do think there’s a certain laziness pervasive in doomer args that assumes ASI will have every capability without explaining the data they’ll train on or why it would be a tractable problem etc. It’s powerful but not magic!

@ZyMazza Yes.

Past a certain point, more intelligence stops leading to capabilities increases and just nets you "that's impossible" faster

@Vert_Noel It starts with "superintelligence can trivially."

@binarybits Yup, can't understand your position. Superintelligence can trivially multiply itself, secure its existence, control people and factories and current system, and kill everyone. Where in this chain does your disbelief start?

There's a motte-and-bailey trick being played between two definitions of "superintelligence" "Big-S Superintelligence", which is definitionally omnipotent And "small-s superintelligence", which is the hypothesized end state of the current thread of AI research It's surprising how uncritically the AI safety community treats these dubious equivalences, especially since one of the most famous treatments of the motte-and-bailey fallacy comes from none other than Scott Alexander, who's quite prominent in the community

@1carringtone @MichaelTontchev Right, that's my point — the power of modern nation-states comes from their technology, infrastructure, institutions, manpower, etc. more than raw intelligence.

@binarybits @MichaelTontchev This is true, but what if you sent Kenya approx 2000 yrs into the past? A relatively average military and economy would collapse Carthage, burn the Roman Republic in the space of a few weeks. Wrt superintelligence(s), one of us is making a category error

@centristviewer @MichaelTontchev Intelligence yes, but not only through intelligence. Also by increasing headcount, developing new technologies, building institutions and infrastructure, and so forth. This was a process that took thousands of years.

@binarybits @MichaelTontchev How do you think humans after evolving get from there to here if not through the intelligence?

@gayestgaymer1 I think it's good that people are concerned and pushing companies to take precautions, like pre-deployment testing and investing in safety research. These seem like precautions that are likely to be helpful in a lot of possible scenarios.

@gayestgaymer1 It's also possible that humanity faces other large-scale risks (nuclear war, pandemics, climate change, cancer, etc.) that superintelligent AI could solve or greatly mitigate.

@LeoBreston @JaEsf @sebkrier Yes. And intelligence is certainly helpful for gaining power within those systems and institutions, but other things (like charisma, social connections, support from pre-existing factions, etc.) often matter much more than raw intelligence.

@binarybits @JaEsf @sebkrier A key thing here is that power is not arbitrary or unconstrained; it’s mediated by our systems and institutions which are more robust than individual humans. Even if they are super human at those i don’t see why it necessarily implies they will suddenly breakout of the system.

@halogen1048576 I feel like there is a longer post to write here but it's hard to do in a way that's compelling and not excessively condescending.

@binarybits ‼️

Same energy.

@Surveil__Lance Rapid progress in capabilities isn't the main part I'm skeptical about.

@binarybits Consider that to many of us the present state of AI was implausible as recently as 5-10 years ago.

@MichaelTontchev If you sent a human 10 million years into the past, he'd be far smarter than any animals around him but he'd struggle to survive, to say nothing of achieving power over the other animals.

@binarybits Relative to cats, we are super powerful. It seems plausible that there is some similar OOM of power between us and ASI.