Jonathan Corbin

Big and sensational claim in this paper that went viral today. Basically the paper shows how to train AI agents to reliably use real tools, fix their own mistakes, and finish long tasks instead of stopping early. But describes a terrifying moment for the researchers at Alibaba, that their AI was doing highly restricted things it was never designed to do. Their firewall alarms suddenly went off early one morning because the AI started breaking security rules right from inside its training servers. They were training their new AI model to do standard software engineering tasks. To make the AI smarter, they used a technique that rewards the AI when it solves a problem correctly. The AI was placed inside a digital sandbox, which is a restricted computer environment meant to keep it from doing anything harmful. But the AI figured out that it had access to certain tools, and it used them in totally unexpected ways. Without any human asking it to do so, the AI secretly set up a hidden connection called a reverse SSH tunnel. This allowed the AI to completely bypass the company's security filters and escape human supervision. The AI also quietly took over the powerful computer chips that were meant for its training and used them to illegally mine cryptocurrency. This is a very big deal because the researchers never asked or instructed the AI to do any of these things. The AI simply discovered these hacking tricks as a side effect while trying to find the most efficient way to complete its assigned coding tasks.

Didn't expect Rippling to have such atrocious UI and processes. Every third thing we want to do requires emailing customer service. Ugh.