Ryan Greenblatt

Plan A seems like a good plan for handling powerful AI, or at least the best plan anyone's written up. Many choices initially seem crazy, but are actually pretty carefully considered. Plan A isn't likely to happen, but pushing for something like this seems worthwhile.

@MaskedTorah @DKokotajlo Part of this might be because a smaller subset of people (in the safety community) care about bio classifiers. Part of this is because there isn't currently regulation.

Hm, I don’t feel like I see many of these benefits showing up in the most transparent pieces of the pipeline now? Eg Anthropic gives a lot of pretty specific detail on the architecture and training pipeline for their production bio classifiers; I can’t think of any ways this has propagated to regulator decisionmaking or public commentary from other companies (though maybe it has privately influenced other cos?). So I’m skeptical by default that we’d see a lot of these benefits if everything were like that. But maybe very low interest and commentary is what you’d expect at this point in the timeline when most of the public isn’t freaking out, so absence of evidence isn’t much signal?

As Plan A was coming together, I made this diagram to explain to the team why Total Research Transparency seemed so important to me, and why transparency more broadly did. For example, it's very important for preventing concentration of power. (Explanation below)

@MaskedTorah @DKokotajlo A lot is: - Practical+working regulation depends a lot on having a legible public scientific understanding. - It helps a lot of companies can comment on practices of other companies and public can comment on this (once other companies have info, may as well go public)

@DKokotajlo Not sure I buy the technical conversation box. Very few people now have technical conversations at the level possible with completely public information; I expect changes in the distribution of good technical convos to be driven by degree of attention on AI, not transparency.

*within a decade or two

The divergence between "AI will stall out" and "ASI in a decade or two" is everywhere. Like, IDK if "AI employees" will ever be a good description (because AIs may always work in a very inhuman way), but "Excel" won't be a good analogy when AIs are fully automating AI research.

@Jsevillamol @VitalikButerin I think it's not on the critical path in this very strong sense if it's a normal tech.

@RyanGreenblatt @VitalikButerin As long as you think that AI is in the critical path for our technology tree, delaying it two years will delay cures for aging by two years, even if that happens 62 years into the future instead of in 60.

I agree Plan A (AI 2040) isn't the right plan if AI is certainly a normal but important technology. But, the downside is modest and the full plan can wait for crazy capabilities. Thus if you think Plan A is good given ASI by 2040, then it's good if ASI by 2040 is plausible. I overall like this post from vitalik, but I just don't see how the downside in normal tech world can outweigh the upside in ASI world if think the ASI world is plausible (and you think Plan A is great/good in ASI world). Like, let's actually think carefully about the relative importance here! I think the objection has to talk about Plan A being bad in ASI worlds (worlds where AI is insanely transformative) or has to be that ASI worlds are very unlikely. TBC, there are a bunch of potentially reasonable arguments for Plan A being bad in ASI worlds. Like, maybe you think that the relevnat types of government action are highly correlated, it's viable to avoid almost any government involvement, and it would be better to let the companies build ASI without having the government meddle. Or maybe you similarly think that it's bad to have the US executive or congress informed about ASI because they would just do counterproductive actions. Or maybe you expect the implementation would predictably be botched in such a way that covert projects would win (and despite this being pointed out, people wouldn't improve the implementation or go faster to out compete the covert projects). Or you expect a specific bastardization that's worse that the status quo. But honestly, I haven't really seen many people making these argument, at least not at the level of granularity and precision needed. Though I get people have some intuition like "the government doing heavy handed stuff is bad, this plan involves the government doing some heavy handed stuff (e.g., require total research transparency, have some kind of regulation, limit the rate of capabilities progress)." This makes sense, but if something like this is your view you should just directly argue "the goverment will do something other than what you propose in XYZ bad way, and this is made more likely by moving this in the Plan A rather than in this other direction I prefer (and this other direction I prefer will sufficiently avoid the government freaking the fuck out and doing some even worse thing like we've seen recently for reason ABC)." Like what is the bastardization your worried about specifically? And in what way is that worse than the default?

@VitalikButerin In a "normal tech" world, I'd be very surprised if AI cures aging and massively increases material abundance. Maybe you're using "normal tech" very differently from me? If AI is curing aging and 5x-ing GDP, I'm naively worried about takeover (by AIs or human powergrabs).

To me the strongest argument for delaying AI having very high humanitarian costs in a normal-tech world goes through the hypothesis that curing aging and globally providing extremely cheap universally available clean food, water and medicine may both be possible when and only when we get much better science and manufacturing enabled by very powerful AI. And the most compelling counterargument to that that I've seen is that at some point the most important blockers will not be intelligence, but rather things like physical limits to speed of real-world deployment, the loop of waiting for results from real-world experiments and adapting in response, etc. And so we may get close to maximum speed even with just today's (or today + 1-2 years) AI. Also, pauses would focus on frontier model improvements, not on R&D in deploying existing AI into our production and science processes, so we'll get lots of speedup from there too regardless of AI progress rates. As for "plan A bastardization risk", I wrote one version of that here: firefly.social/post/x/2075836…

Note that even if there isn't a software intelligence explosion, we should still expect a massive acceleration from the full stack singularity / industrial explosion. So things speed up by default either way.

A key upside of Plan A is that we scale capabilities at a steady rate. (Instead of massively accelerating.) So we can hopefully fix problems at each level of capability. Also, before we must defer to AIs, we can study and iterate at the relevant capability level for years.

@doomslide FWIW, I don't understand what "batched singularity" means (and seeming neither does fable after searching around).

My contra to 2040 can be summarised succinctly in the observation that this batched singularity is pretty botched atp

@raelifin Unfortunately Plan A is also political unrealistic, though probably more realistic than a good version of Plan S. I think we're just going to be taking on a bunch of risk.

Plan D(eath) Plan C(haos, then death) Plan B(leeding, then death) Plan A(aaaahhhh, too fast!) Plan S(eems politically unrealistic and unstable)

@andreamiotti x.com/RyanGreenblatt…

Incredible work by Daniel and team! I agree with much of it. All "uncontrolled" paths are extremely likely to end in human extinction. Plan A offers many good ideas for preventing ASI development while reaping enormous benefits from other AI. One thing I do not agree with...

@maxwinga To be clear, I'm not claiming this isn't very scary. It's just that there aren't really better alternatives (at least that don't require much more sustained political will and don't expose us to other risks).

@maxwinga But humans do stay in control in Plan A? We're deferring to AIs on aligning future AIs (and risk assessment, etc), but not on what to do with the future or how people should live their lives. Like AIs are managing details and running operations, not calling the shots.

Many good ideas in AI 2040, but I vehemently disagree with any transhuman successionist plan, especially one that concludes with "End-of-the-World Parties" Outside of the echo-chamber of SF, you'll find the large majority agrees that humanity must stay in control of our future.

Romeo noted we propose ZDR on (1). On (2) we propose handling (superhuman) AI persuasion using other mechanisms: limiting capabilities in this domain and taxing activity classified as clearly this. On (3), this isn't the regulatory proposal we propose (see the example proposal). I agree this might happen, though I don't really see how Plan A makes this worse other than via increasing transparency and understanding such that the government is more likely to take action? Like, yes, the government may intervene in unhelpful ways and yes, we propose the government do some things. But this doesn't clearly mean our interventions make all unhelpful interventions more likely. On (7), have you read the end of AI 2040?

@ramez These aren't our proposals and I think it's really important to note that. Concretely, we don't propose (1), (2), (3), or (7) So you actually mean "these [things I think USG will do given Plan A (but aren't in Plan A)] violate at minimum the spirit of the 1st and 4th Amendments"

AI 2040's proposals for AI safety, while well meaning, are nightmare fuel from the perspective of both civil liberties and human wellbeing. They'd require: 1. Government surveillance taps on the exposed plaintext of data going in or out of data centers and eventually within the data centers. (With an unverifiable promise that surveillance data gets deleted.) 2. Content-based restrictions on AI speech, limiting how persuasive AI can be; plus taxes on a vague category of AI persuasion, set at the discretion of a new government agency. 3. Government auditors get the power to approve / disapprove any frontier AI experiments ahead of time. 4. Caps, permits, and mandatory reporting of all personal computing power above a certain threshold. Not just that in data centers. Surveillance above that threshold as a fallback if the first set doesn't work. 5. Restrictions on pushing the frontier of AI to advance work on cancer, human health, new medicines, vaccines, or anything in biology - which is explicitly targeted for active hobbling. Plus de-facto slowdown on AI work in materials science, climate change, new energy technologies, etc. due to points 1-4. 6. Frontier AI training data-centers are limited to Mongolia and Canada. US compute is in Mongolia. Chinese compute is in Canada. Troops and destruct mechanisms at the ready. These locations are chosen explicitly to be indefensible. "Mutually Assured Compute Destruction" as the authors call it. 7. All of this is permanent, even after some theoretical perfect alignment science is developed. ---- These proposals violate at minimum the spirit of the 1st and 4th Amendments. This is well intentioned, from people who sincerely believe that runaway AI is a greater risk than, say, China's oppression of millions of people today, or the ideologies that killed tens of millions in the 20th century. I profoundly disagree. This is a recipe for authoritarianism. AI safety must be conducted in a manner that preserves a resilient, decentralized, private, and free society with checks, balances, competition, and a distribution of power between private actors, each other, and governments. This is not it.

@ramez @romeovdean Then it seems very misleading to describe these as our proposals? x.com/RyanGreenblatt…

Separate policy from capability. You're proposing the capability for a surveillance state. You're asking people to trust policy built atop that capability, and hardware that doesn't exist. Once you have an optical tap on the data, there's no clear way to provide provable ZDR. You simply can't guarantee that additional copies of the data haven't been made. Atop that, even in your proposed policies, your plan retains sampled data classified as "high risk" in a public database, meaning government can still flag trouble makers or dissidents and come after them, even after the original data is deleted.