I feel very uncomfortable about the AI Security Institute, and this article by its creator, ex-PM Rishi Sunak, encapsulates why: He frames it as a way of avoiding regulating AI. He says this categorically. “If politicians are blithe about the risks [of AI], they will vote for those who favour regulation.” And he makes it clear he thinks regulation would be bad: “western governments shouldn’t restrict innovation in the race against China.” This is a man who now works for both Anthropic and Microsoft (not disclosed in the article). Evaluating the safety of AI models is good. But not in place of regulating the technology itself, and the companies behind it. Besides, as he points out, companies only give the AI Security Institute access to their models voluntarily. This means the UK government relies on good relations with big tech, which in turn makes it even less likely to regulate. It is far from clear how much the AI Security Institute has actually achieved with its astronomical public funding. What *is* clear, though, is that the people who set it up see it as a way of ensuring *less* regulation of AI companies, not more. This is very bad news.

@ednewtonrex "far from clear how much the AI Security Institute has actually achieved with its astronomical public funding" -- found jailbreaks in multiple models (which have now been patched), and generally advanced science of AI evals a lot. It's genuinely important work.

I agree that AISI could be a good thing in principle, but I disagree that the bits of work you mention are clearly worth the funding. Finding jailbreaks is all well and good, but not vastly different from what the likes of Pliny do (for free). Besides, if this is deemed important, it feels like something that should be concretely regulated (requiring model access, audits etc.). I think there’s a good argument a voluntary setup gives people a sense of security without it really being there. I don’t think the evals work is worth the funding. Again, all well and good, but surely a bonus as opposed to the main thing you’d expect for these millions. IMO AISI’s main benefit is as a body that can share intel on model capabilities etc with the UK government (not just pitches that come from AI companies). In that regard I think it already performs a useful function. That alone wouldn’t justify the level of funding, though. Again, I don’t think its existence is clearly a bad thing. But I think there are potential flaws at the moment: (i) voluntary nature of engagements, which leads to (ii) chumminess with big tech, making it harder to regulate, (iii) Sunak’s admission in today’s article that AISI’s very existence diminishes the need to regulate, and (iv) a lack of transparency meaning it’s not clear what it’s achieving.