Ben Brooks

Nearly every AI firm is invoking frontier risk to justify a persistent gap between open and closed models. But this application of the precautionary principle deserves more scrutiny than it gets. Restricting access to useful technology—models that will, in their developers' own words, transform the economy—shouldn't be our primary response to uncertain risks. Check it out at @aif_media! There are already a bunch of reasons firms might not release their best models openly: cost recovery, competitive pressure, anxious investors. But we should be skeptical of efforts to freeze open-source behind the frontier under the guise of risk.

@egastfriend @DavidSacks Agree that a voluntary program is likely, and desirable. But as CAISI (and UKAISI) have shown, you can set up a voluntary arrangement with minimal fuss. Borrowing hard regulatory language to set up a soft collaborative framework via EO is certainly a signal, but not a good one.

You don’t *need* an EO to set up this kind of program, but it’s a very helpful way to clarify the administration’s position, coordinate the interagency process, and signal seriousness to industry to foster cooperation. A missed opportunity, but I expect something similar will eventually be set up regardless.

Here's the thing: you don't need an EO to set up a voluntary beta release, nor do you need to define a "covered frontier model" for a voluntary beta. As @DavidSacks & POTUS correctly surmised, this EO was just one adjective and one sentence away from endorsing "FDA for weights".

@SpaceX Not to mention the Pez satellite dispenser

The AV team at @SpaceX deserves a raise. Phenomenal views from Starship 12. Not long ago, it would've been unthinkable to livestream the engine bay, re-entry, and distant splashdown from outside the spacecraft

I gave a talk at the Global Free Speech Summit with @SpeechFuture about why AI is a fundamentally expressive technology, and how open weights can promote transparency in the systems that determine how we access, produce, and interpret information

The Biden folks didn't want to repeat the Bernstein culture wars, and specifically excluded open models in their export rule. It would be... a remarkable turn of events for this Administration to try. Thankfully, it sounds like a majority in this WH understand the significant downsides of trying to "FDA-ify" open weights.

An interesting line in Politico’s coverage of the proposed AI executive order, which, at 16 pages, is also much longer than expected. This is still under discussion and not yet finalized, and everything I'm about to write is conjecture, but it appears the administration intends to regulate US open-weight models. Here are the reasons why this will almost certainly happen in some form. Open-weight models are currently about nine months behind the frontier. Once the big labs are subjected to pre-release screening, development itself will not slow down, but the release cadence will. At that point, open-weight development will quickly close the gap - much faster than nine months. When those models surpass the big labs, everyone will switch to using open-weight alternatives. From the administration’s perspective, allowing this option defeats the entire purpose of regulation. If the government is restricting and vetting models beyond a certain capability level, and people can simply switch to open-weight models that are just as capable - and eventually even more capable as the big labs slow down their release schedules under the new rules - then the situation becomes even worse from the government's perspective. They will not allow this to happen. Second, the big labs themselves have almost certainly been covertly lobbying for open-weight models to be included in any new regulations. Allowing the public to switch to a superior, free alternative would completely destroy their business models, potentially bankrupting them all. Given the enormous scale of current investment in these companies and in AI infrastructure, the broader economy would also suffer "significant disruption". That leaves China. If the two dynamics above play out, the same pattern repeats: everyone switches to Chinese open-weight models, which now quickly surpass both US closed and open releases. This produces the same consequences for the big labs, and causes the same issues with regulation. The government therefore has only two realistic options: ban Chinese models from use in the West, or negotiate a deal with Xi Jinping to impose identical regulation and pre-release vetting on open-weight models in China. The first option would mean China pulls ahead and wins the AI race. So the administration will almost certainly pursue the second. Negotiations are likely already underway, because the ideal outcome for the admin would be to announce that China has agreed to similar restrictions to what they are announcing, thereby blunting domestic backlash. China will know it has the US over a barrel and will insist on compromises. Compromises such as lifting all export controls on NVIDIA GPUs.

@ashleevance Latent diffusion guy @robrombach

Who is the most prominent AI chief we have not heard from in a long, long time?

Update on the AI executive order: discussions are still ongoing, but a senior White House official who was granted anonymity told Politico that 'There's one or two people who are very intent on government regulations, but they're sort of the minority of the bunch.' and added that the White House is looking for 'partnership' with companies rather than pursuing 'government regulation.'

@ohlennart It could, but that would require capable open alternatives. If US / EU labs (other than e.g. Reflection, AI2, Mistral) no longer release strong open models, the options are increasingly limited.

@opensauceAI But why couldn't the next search engine, social platform, or everything app just swap the model under the hood at any point?

Some thoughts on the open weights debate. Most concerns I hear are based on misconceptions, and the actually worrying things barely come up. Genuinely keen to hear pushback. Not debating the ecosystem and economic benefits of open weights here, same as for open-source software (related but different debate). Mostly on the US-China open weights angle. A few things I think are misunderstood: 1. A common view is that China gets access to data or makes money. Roughly backwards for open models. The whole point is you download and run it on your own hardware. No revenue flows back, no data flows back. If anything, open weights reduce your exposure to the releaser, not increase it. Closed hosted APIs (like using DeepSeek on DeepSeek's site) are where data actually moves. 2. Are models actually that sticky? People swap them weekly, often with a couple lines of code. The brand and ecosystem case for the releaser is thinner than claimed. Risks I do see, but that get less airtime: 1. Sleeper-agent-style behaviour baked into weights you can't audit. A model could look fine in normal use, then trigger on a specific input and write insecure code or exfiltrate data. Gets worse with more autonomy. 2. Same with embedded values. Imagine education systems globally running on weights with someone else's framing of history. 3. License creep. Free initially, fees later once you cross some (user) threshold (Llama-style). 4. Stack lock-in (but same goes with closed ones). Selling you the model, the hardware, and the phone preloaded with it (then the data might flow back to a specific API in a specific country). But notably, I think most of these risks are low right now, or don't exist yet (licenses, stack lock-in). And, as said, IMO models aren't sticky. I think open releases are just a strong counter-strategy. If you can't capture API revenue at the closed frontier, open releases let you leverage US compute (to the degree it gets deployed there) and eat up the discourse. Personally, the sleeper-agent worry might be enough to tip the balance for me, which is why I'd want strong open models that are truth-seeking and ideologically neutral. (And that's why I'd take US models over the alternatives.) But that's not what I'm hearing in the debate, and instead lots of confusion around open source vs open weights.

Deja vu.

Looks like we'll need to resolve the "what" and "who" for pre-release risk acceptance sooner rather than later...

Mandatory audits feel a bit like Prop 65—this model is known to the state of California to cause birth defects and anxiety in Hayes Valley. They punt the real trillion dollar question: what happens after the audit? Consider three scenarios: 1. The results confirm the lab's claims about residual risk and meet minimum public expectations --> there's no problem. 2. The results debunk the lab's claims but still meet minimum public expectations --> the lab is in trouble with the FTC and SEC, and op-ed columnists have a field day, but industry red lines are fundamentally unchanged. 3. The results confirm the lab's claims but fall short of public expectations --> pressure grows to "do something" (the Mythos scenario), but we return to the unanswered question: what is the threshold for acceptable catastrophic risk, and who decides? The idea that a federal commission, state agency, or some kind of PwC-for-frontier-AI should make that determination is as troubling today as it was in 2023 (when none other than OpenAI was pushing for developer licensing). The idea that a court should make that determination in advance of actual harm is as troubling today as it was back when we were all debating proto-SB1047 and proto-RAISE. Mandatory audits may be a reasonable early warning system, and tighten up a few gaps in tort and criminal liability. But the real question—the one rightly holding back exotic reform—is: what happens after the audit, and what is the opportunity cost of getting it wrong?

In a narrow sense, this kind of research is just a new lens on system health. ChatGPT is more useful when you're polite and legal, not pranking it like @huskirl. But "wellbeing" is regularly invoked as evidence that the system has interests that are independent of human users, deployers, and bystanders. That is a *choice* not a *finding*. A Roomba can distinguish between when it's free to roam, and when it's stuck on a ledge. That it optimizes for the former over the latter, or that we can measure both its preference and its internal state, has zero implications for how we should treat Roombas. You help the Roomba because the floor needs cleaning and it's a trip hazard. Not because it deserves happiness. The paper itself ~mostly avoids conflating measurement with moral patienthood. But the comms around this announcement are hugely troubling, especially if they're a preview of advocacy to come.

Folks are reading too much into Mythos as a template for all frontier model releases. Withholding Mythos made sense on perfectly normal cybersecurity grounds. If you find a vulnerability, you give the vendor (or 50+) a heads up before public disclosure. Concrete vulnerabilities with specific actors and practical fixes. Not generalized anxiety about diffuse or speculative risk—which is how pre-release restrictions are usually justified. Put another way, nothing about the phased release of Mythos should change how, say, Llama was released. It might justify e.g. pre-market evaluation (so that developers can't turn a blind eye), but it doesn't justify pre-market clearance in the sense @TheEconomist describes here.

@Thomas_Woodside @Scott_Wiener Agree, verifiably silly. Love it or hate it, Scott Wiener delivered on more AI regulation in Sacramento than any single member of the House has delivered or could ever deliver in DC.

I can say with authority that this is complete nonsense. I've been advocating for AI regulation in California for two years. In 2024, I worked with @Scott_Wiener on SB 1047, an AI regulation bill that was vetoed. And I spent my 2025 labor day weekend at a Marriott in Sacramento so that I could advise him during the SB 53 negotiations referred to inaccurately in this video. Scott Wiener has fought incredibly hard on this issue. He burned huge amounts of political capital to ensure the strongest possible version of SB 53 was signed into law. It was, and we now have a good law on the books, which paved the way for others to follow. Previously, there were no laws in California regulating AI companies' management of catastrophic risk.

More from our conversation with @bfl_ml @opensauceAI on why restricting access to general-purpose technology comes at the expense of open research and innovation. Ben lays out a more targeted path to identify where existing law falls short, then introduce reforms where they are actually needed.

Restricting access to versatile technology should be a measure of last resort, not a measure of first resort. Great to chat with @MattPerault & @a16z about the importance of open weights, how we think about risk @bfl_ml, and how small labs can advocate for big communities.

@StephenKentX @DarioAmodei I frequently disagree with Anthropic, but this is the most sophomoric kind of intellectual snobbery. Literally the scene from Good Will Hunting, but it turns out Dario is Matt Damon and the "gotcha" people are the pony tail sweater guy who doesn't get Minnie Driver's number

Anthropic's @DarioAmodei did a roundtable with two dozen conservative intellectuals in Washington. DC was SURPRISED by questions on the Aristotelian / Platoist / Nietzschian / Chestertonian inclinations of his AI?? WHO IS ADVISING HIM?

@MattPerault @BlackForestLabs The spicy conversations are the most important (and most interesting). Thanks for bringing us together @MattPerault!

The team at @BlackForestLabs has distinguished themselves not only through their technical leadership in visual intelligence, but through their commitment to building openly. Ben Brooks @opensauceAI, BFL’s head of policy, explains how they are working to prove that developing cutting-edge frontier models can be both open and responsible. This and more in our latest a16z AI Policy Brief conversation, below.