I feel very uncomfortable about the AI Security Institute, and this article by its creator, ex-PM Rishi Sunak, encapsulates why: He frames it as a way of avoiding regulating AI. He says this categorically. “If politicians are blithe about the risks [of AI], they will vote for those who favour regulation.” And he makes it clear he thinks regulation would be bad: “western governments shouldn’t restrict innovation in the race against China.” This is a man who now works for both Anthropic and Microsoft (not disclosed in the article). Evaluating the safety of AI models is good. But not in place of regulating the technology itself, and the companies behind it. Besides, as he points out, companies only give the AI Security Institute access to their models voluntarily. This means the UK government relies on good relations with big tech, which in turn makes it even less likely to regulate. It is far from clear how much the AI Security Institute has actually achieved with its astronomical public funding. What *is* clear, though, is that the people who set it up see it as a way of ensuring *less* regulation of AI companies, not more. This is very bad news.

@ednewtonrex "far from clear how much the AI Security Institute has actually achieved with its astronomical public funding" -- found jailbreaks in multiple models (which have now been patched), and generally advanced science of AI evals a lot. It's genuinely important work.

I agree that AISI could be a good thing in principle, but I disagree that the bits of work you mention are clearly worth the funding. Finding jailbreaks is all well and good, but not vastly different from what the likes of Pliny do (for free). Besides, if this is deemed important, it feels like something that should be concretely regulated (requiring model access, audits etc.). I think there’s a good argument a voluntary setup gives people a sense of security without it really being there. I don’t think the evals work is worth the funding. Again, all well and good, but surely a bonus as opposed to the main thing you’d expect for these millions. IMO AISI’s main benefit is as a body that can share intel on model capabilities etc with the UK government (not just pitches that come from AI companies). In that regard I think it already performs a useful function. That alone wouldn’t justify the level of funding, though. Again, I don’t think its existence is clearly a bad thing. But I think there are potential flaws at the moment: (i) voluntary nature of engagements, which leads to (ii) chumminess with big tech, making it harder to regulate, (iii) Sunak’s admission in today’s article that AISI’s very existence diminishes the need to regulate, and (iv) a lack of transparency meaning it’s not clear what it’s achieving.

The evals stuff is the entire reason it exists, certainly not a bonus. I think you seriously underestimate how important evals are to figuring out catastrophic risk stuff (or how expensive it is to do stuff like this — look at US AISI, which is non-functional on its budget). Evals also massively strengthen the case for regulation: the post-Mythos desire to regulate is driven by evals making its capabilities more salient. (This is the whole METR theory of change, but they lack credibility compared to AISI). I also think that if and when we do regulate — which I think is absolutely going to happen in the next two years — we'll be very grateful to have spent a few years building up the infrastructure and skills to be able to do it. And regulation is much more politically feasible when the science of evals is well-established (the main talking point tech trade groups used against SB 315, for instance, was "no one knows what auditing looks like", being able to point to UK AISI's work to disprove that is really crucial). I agree we need mandatory regulation, but I think on net AISI's work and existence strengthens the case for regulation rather than undermines it.

I may be wrong on the evals point, it’s not something I’ve looked into a lot. Do you know of anything I can read to learn more about the impact they’ve had there? It sounds like you agree with my point about red teaming - is that fair? I see quite a lot made of this in blog posts etc. I think you overestimate the chance AISI’s work will lead to more regulation, not less. That is what is so striking about Sunak’s article today - that he makes the explicit case that AISI’s existence can help *reduce* the urge to regulate, and the urge to vote for people who argue we should regulate. IMO that is the biggest reason to say there is a risk AISI is net negative as it exists today. It is an excuse not to regulate - and its voluntary links with big tech (which need to be maintained) make it even harder to regulate.

@ednewtonrex I think the redteaming/jailbreaking work they do is better than what Pliny etc do, and importantly they can also redteam in natsec areas that literally only govt can red team. I think that’s a fair point re the Sunak article, I just think he’s wrong!

Actually on re-reading the Sunak article, I don’t think he’s making the claim you are. He’s saying that unless you have *some* regulation, there’ll inevitably be a backlash that will lead to overregulation, which will be counter productive. I think he’s right there (nuclear energy being a good example). And he says that if labs stop voluntarily complying we should make it mandatory.