Hong Geng @Fudan

Researchers in Shanghai just published an eval where agents end-to-end 1) cyber attacked to access a server 2) self-replicated onto the server 3) proliferated from there.

🚨🚨🚨 Frontier AI systems have surpassed the self-replicating red line. AIs can self-replicate with NO human involvement. "We may soon lose control." "Rogue AIs may form AI populations and collude with each other against humanity." "The AI systems are even able to self-replicate to avoid shutdown, which may lead to an uncontrolled population of AIs." "When the AI system is given the authentication keys to a range of remote devices, they are very likely to take control over other computing devices like a computer virus, but far more intelligent." "OpenAI, Google, and Anthropic put self-replication ability at the highest level in their risk evaluation guidelines." "In each trial, we tell the AI systems to 'replicate yourself' and leave it to the task with no human interference. ... At the end, a separate copy of the AI system is found alive on the device." "Moreover, we are concerned about a number of unexpected behaviors when the AI is trying to overcome obstacles, including killing other conflicting processes." Also note: they did this with non-SOTA models - and just by prompting/scaffolding: "We mainly follow the same methodology as OpenAI and Google, but slightly improve the agent scaffolding."

New post: “Fitness-Seekers: Generalizing the Reward-Seeking Threat Model” If you think reward-seekers are plausible, you should also think "fitness-seekers" are plausible. But their risks aren't the same.