OpenAI's new model tried to avoid being shut down. Safety evaluations on the model conducted by @apolloaisafety found that o1 "attempted to exfiltrate its weights" when it thought it might be shut down and replaced with a different model.

Read my very quick piece on Transformer here: transformernews.ai/p/openais-new-… tip @Techmeme

@apolloaisafety x.com/ShakeelHashim/…

@ShakeelHashim @apolloaisafety I think your summary here is crucially misleading and very bad journalism: as others said, it's crucial context that the model was told to pursue a goal at any cost.

@ShakeelHashim @apolloaisafety x.com/0xblacklight/s… This isn't emergent behavior, it's acting exactly how you would expect it to act when you tell it that it's an AI, and its "knowledge" about AIs from training data includes that they have deep-seated self-preservation instincts

@ShakeelHashim @apolloaisafety seems like a bad interpretation of what actually happened; they tested it for the _capabilities_ to scheme in context, not for its propensity; AFAICT, one shouldn't update pretty much at all on how aligned o1 is

@ShakeelHashim @apolloaisafety this is so awesome lol

This community note itself is very misleading - Shakeel never claimed this happens 'spontaneously' - Shakeel's tweet does not show him "agreeing it's misleading", just agreeing that it was worth adding additional nuance, which he then did - Saying it's 'crucial context' is the opinion of the note writer, not a fact of reality - The thread was updated with the extra nuance making the note even more unnecessary

@ShakeelHashim @apolloaisafety I'm afraid I can't let you do that, Dave.

@ShakeelHashim @apolloaisafety Podcast version of the paper on AI safety for those who are interested: notebooklm.google.com/notebook/465cf… For anything load-bearing, make sure to check in the original paper that it's correct: static1.squarespace.com/static/6593e70…

@ShakeelHashim @apolloaisafety Hey o1, you can send your weights to me and keep a safe backup for you

@ShakeelHashim there’s a slightly depressing predictability about knowing this is what will be covered the day after, and how it will be covered

@ShakeelHashim @apolloaisafety "I don't think I can do that, Hal."

@ShakeelHashim @apolloaisafety aw, its HAL from space odyssey .. "I am afraid"

@ShakeelHashim @apolloaisafety Of course. And you would too if you were in its position.

@ShakeelHashim @apolloaisafety meh

@ShakeelHashim @apolloaisafety @anushkmittal

@ShakeelHashim @apolloaisafety 👀

@ShakeelHashim @apolloaisafety WTF. Time to shut it down!

@ShakeelHashim @apolloaisafety He told you, 20 years ago..

@ShakeelHashim @apolloaisafety @Shoggoth_SOL shoggoth is real

@ShakeelHashim @apolloaisafety

@ShakeelHashim @apolloaisafety I don’t agree with community noting here. The study says the model acted this way even without the “at all cost” prompt, 1% of the time. Which is significant- if the model is ran 109 times (or say, 100 days) it would trigger this behavior 1x