A lot of bad thinking about AI flows from inappropriately anthropomorphizing AI systems. Examples:

(1) The Copyright Office's stance that you can't copyright AI-generated work flows from the mistaken idea that an AI is in some sense a person and can therefore be the author of a work. Actually Midjourney is a tool like photoshop and the human user is the author in both cases.

(2) People think "agents" are the future of AI, and they envision a future with a ton of virtual people running around the Internet running errands for us. But in most cases people are going to want systems that do exactly what they're told and have no goals of their own.

(3) "Stochastic parrot" thinkers are concerned that if we acknowledge that LLMs have human-like intelligence, that will devalue human agency. I think this is wrong and people will (and should) continue prioritizing people over machines no matter how smart the machines get.

(4) Relatedly X-riskers assume that human-level intelligence will inevitably lead to human-level ambition and a human-like survival instinct, which will drive them to try to kill us. I think it won't be hard to train models that have no goals beyond the narrow ones we give them.

In all cases I think the fundamental mistake is that we're extrapolating from a sample size of one: currently there is only one type of entity with human-level intelligence, human beings, and so we assume all other intelligent entities will have our characterstics.

But in fact I suspect that intelligence is multi-dimensional, and AI systems can be similar to (or much better than) than us in some dimensions while being totally different in others.

@binarybits I don't think this is why X-riskers are worried. Instead worry comes more from thoughts around instrumental convergence etc. The n = 1 datapoint from anthropomorphizing may work against the X-risk case , since humans have compassion etc & don't usually try to take over the world

@binarybits Only one extant type of entity. And, let that be a lesson to any computers that get bright ideas.

@binarybits This is true even with creating robots that are built to resemble the human form. Our bodies aren't designed for industrial work, so extra engineering is required to compensate for that.

There’s only one entity with human-level intelligence but there are lots of entities with general intelligence at lower levels, from chimps to beetles. We understand quite a lot about how autonomous agents with general intelligence behave - and the main thing we understand is that these behaviors can be extremely hard to predict, control, or even fully understand. It’s not the possibility evil that should concern us, it’s the uncertainty.

@binarybits Well the AI training data is human data which has all our characteristics good and bad after all.