A lot of bad thinking about AI flows from inappropriately anthropomorphizing AI systems. Examples:

(1) The Copyright Office's stance that you can't copyright AI-generated work flows from the mistaken idea that an AI is in some sense a person and can therefore be the author of a work. Actually Midjourney is a tool like photoshop and the human user is the author in both cases.

(2) People think "agents" are the future of AI, and they envision a future with a ton of virtual people running around the Internet running errands for us. But in most cases people are going to want systems that do exactly what they're told and have no goals of their own.

(3) "Stochastic parrot" thinkers are concerned that if we acknowledge that LLMs have human-like intelligence, that will devalue human agency. I think this is wrong and people will (and should) continue prioritizing people over machines no matter how smart the machines get.

(4) Relatedly X-riskers assume that human-level intelligence will inevitably lead to human-level ambition and a human-like survival instinct, which will drive them to try to kill us. I think it won't be hard to train models that have no goals beyond the narrow ones we give them.

In all cases I think the fundamental mistake is that we're extrapolating from a sample size of one: currently there is only one type of entity with human-level intelligence, human beings, and so we assume all other intelligent entities will have our characterstics.

@binarybits "ambition" maybe human, survival is a universal subgoal for any agents having any goals whatsoever. en.wikipedia.org/wiki/Instrumen… These goals can arise through various means, including RL, explicit programming or others

@binarybits Of course it's not 'hard to train models that have no goals beyond the narrow ones we give them' - we do that right now. However, if we want real intelligence then we want the computation to come up with subgoals. And if the computation knows enough about itself then...

@binarybits Geoffrey Hinton says you're dead wrong. He says any goal creates subgoals, and when you cannot predict, understand or control those subgoals with an ASI you are most likely dead. You think it won't be hard to keep goals narrow. What do you know that Hinton doesn't?

@binarybits Why would it not be hard to train an intelligent system which can pursue the narrow goal we give it but not the instrumental goal of staying alive?

@binarybits This argument has always been related to instrumental incentives or the complexity of goals rather than anthropomorphising AI.

@binarybits I'd think you'd at least need curiosity/play to build an aGi and that can spin off in to many things. And your fitness function has to be so general it can lead to many hidden motivations.

@binarybits Are you saying "intelligent" systems can't come up with goals on its own? Or ways to bypass the limitations we build into it?

@binarybits Your comments seem to suggest you don’t believe singularity is possible. Or at least your argument seems to ignore that large issue.

@binarybits "X-riskers assume that human-level intelligence will inevitably lead to human-level ambition and a human-like survival instinct" Are you sure that is what they think or is it just an assumption on your part?

@binarybits Sure, until we actively build them to pursue goals. Also: gwern.net/tool-ai

@binarybits The narrow goals we give them is one of Yudkowsky's pillars for extinction risk.

@binarybits The survival instinct isn't human-like but life-like. It's an effect of procreation via genetic recombination. Until AI models start having babies where only the fittest survive long enough to reproduce, it's unlikely AI will spontaneously develop adversarial traits.

@binarybits AFAIK, few believe that. I think this falls into the category of a rumor which was possibly started by doom critics LeCun and Pinker.