Raymond Arnold

@LeahLibresco Gotcha. If there's any more flavor to them (or demographic reference class) you feel like you can share I'm curious but no worries if not.

@Raemon777 They’re in private signal convos

I think one of the big shifts I’ve seen AI 2040 provoke is people realizing that the people they thought of as doomers anticipate being ruled by AI (positive) sooner than the people they thought of as e/acc

I was just surprised you-in-particular disagreed enough to make the comment you did. I think misaligned takeover on the timescale of months is still too fast for "iteratively fixing issues" to be the sort of response that makes sense. If it's on the timescale of years... I dunno I still think Plan A style interventions make sense in that world but feels more reasonable to disagree about. My actual position is "'Takeoff takes years' most likely means 'there is more calendar time before a sudden fast takeoff of at most months'." I agree my last paragraph was more handwavy.

@Raemon777 @ramez @rmushkatblat @thlarsen Also, I don't think you meant it this way, because I think we do largely agree, but "we have the same list of plausible-to-discuss scenarios but assign different probabilities to them" is basically how reasonable people are supposed to disagree about the future...

This critique seriously misrepresents Plan A. Some of the top things it gets wrong: - Plan A is iterative - Plan A doesn't involve nationalization - Plan A has a complex interaction of pretty novel proposals, very unlike previous "AI safety milieu since 2018", including total research transparency, cap and trade for Industrial explosion, and massive compute buildouts. (Read our detailed post on LW, link in thread)

I'm not sure what you think we disagree on? Or, I think if we both laid out the various scenarios we think are likely we'd be like "okay yeah those are pretty similar", with somewhat different likelihoods for scenarios. The two main ways I'm imagining fast takeoff being plausible are: – LLMs helping to find the next paradigm – Training LLMs on sufficiently good RLVR with sufficient scale eventually stumbles into more general/fast algorithms Are you like "nah" to both of those? If we get fast takeoff, I think it happens during training. Labs can barely respond if they've done a good job with control. If we get slowish takeoff, and it looks like the government (or labs) might actually take major action to change the status quo, AIs probably are fucking with us somehow-or-other in a way makes it more difficult to respond.

@Raemon777 @ramez @rmushkatblat @thlarsen I disagree that there won't be time to iterate - both because traditional versions of fast takeoff via LLMs isn't feasible, and because as things accelerate, public pressure to decelerate is a strong countervailing force. But plan A gives deceleration something useful to do.

fwiw I don't actually think Plan A as-written is particularly blatant about it's iterativeness. But, answering this question: The core assumption is that there won't be time to iteratively respond to superintelligence, it's just too much too fast. (I realize you probably disagree) But, given that assumption, it's kinda the point of Plan A is to slow things down enough that we _can_ be iterative. It just requires upfront preparation to have a shot of working.

Yes. It did not answer my question. Proponents of the plan describe it as iterative. To me that would mean that policies and AI restrictions would be put in place in response to real world learning and perhaps experimentation. That is not what the plan calls for so far as I can tell. It appears to call for a set of fairly broad restrictive policies to be put in place without any new evidence or any interation on those policies. Am I mistaken in my reading of it?

@rayefull Eliezer Yudkowsky, canonically

Are there any canonical examples of people going from deeply techno-optimist (like technology can solve all things let it rip) to classically ai safety pilled? I was telling someone of my path recently and they were like, uh

@DKokotajlo @1a3orn I think re: "decision procedure" he means "how does the Consortium actually make decisions, in practice?"

thanks! We didn't actually propose an international authority because we thought people would hate that ("concentrates power!") among other reasons. I don't think we've failed to offer the chief thing that international agreements are supposed to supply. The agreement on total research transparency improves the incentives and creates the foundation on which further agreements can happen. Re: the surveillance concern: If you agree that AI inference is going to be such a big deal (e.g. because the whole economy is going to be automated) then how would you propose it be organized? By default, one to three tech companies will see everything & then one government (the USG) will have the authority to demand to see everything too. They can make ZDR promises but we have to hope that they aren't backdooring it etc. Seems like Plan A is at least an improvement over that default, no? More generally I'm interested in what your recommendation would be. What parts of the plan would you keep and what parts would you change and how? I see you like the transparency and the incremental wishlist? Anything else?

Overall, "AI 2040" has some parts that I like. But it proposes an authority ("Consortium") that makes the right in-universe decisions -- with no decision procedure but "nations bargain, weighed by clout." This is a hope rather than an expectation of good decisions. (link blw)

Of the three arguments I list, I think the strongest is #1 – you probably don't actually get Plan A. I'm less confident about "Plan S will have more support among voters" (or rather, that this will matter). I'm basing it on a political model where it's pretty easy for special-interests (in this case the AI orgs) to lobby for things that voters don't care about, but actually pretty hard for them to get politicians to do massively unpopular things. I'm not sure if this is true, it's just a model I heard that sounded plausible to me.

I think Plan S might be more politically tractable than Plan A. (I distrust myself here because I separately think Plan S is much more likely to solve the underlying problem, and I risk halo-affecting myself). But, my reasons: 1. In most worlds where you try to implement Plan A, I think it gets massively distorted and ends up being more like Plan C. The president and Xi Jinping need to actually deeply "get it" in a way that seems unlikely to me, for the nuances are to survive. 2. Plan S is pretty simple and easy to explain: "don't train more advanced AIs until there is strong consensus that it's a good idea." 3. I put decent odds on "Plan S has more support among voters." Plan A requires continuously making nuanced judgment calls about what counts as safe, while generally maintaining momentum on training more powerful AIs (and leaving lots of dry tinder around). It's essentially an unsolved problem to make regulations careful enough to distinguish good vs bad safety cases, and I don't think the Plan A documents had particularly good ideas for how to do so. Meanwhile, I expect tons of voters and politicians to understand "this AI stuff is scary, it's going to get more freaky fast, it's going to turn the world upside down, take your job, maybe kill everyone." The thing they'll want is for the craziness to stop. Meanwhile, Plan A is actually kinda freaky sounding. Centuries of progress in years? I lose my job but it's okay somehow? We hand off control to godlike AIs at the end? Of the people who will actually understand it, I think the people who'd support it are niche futurists, niche libertarians (who maybe have good principles, but, sorry guys, don't actually have much political clout), and niche AI Safety-ists with particular combinations of beliefs about which parts of the problem are hard. (also sorry guys). The larger political group that might support A over S are people broadly wanting to Make The Economy Go Faster who don't really understand or believe in the stakes. Whether than counts as "supporting Plan A" is kinda ambiguous.

Is there any software that does this thing? (I'm indeed not that familiar with this domain, does Google Docs succeed here according to you?) From the OP it sounds like "no", but then I'm kind of unsure what your point is. (i.e. if this is a conceptually unsolved problem, seems like a weirdly high bar for "AI is useful to senior programmers.")

@Raemon777 CRDTs rarely resolve conflicts in a way that any user actually intended, which is the lesson you'd learn if you built something real and threw real users at it.

I'll believe it when i see a working high quality multiplayer-capable web app, with local undo/redo and offline support, coded by AI. Spoiler: there isn't anything for it to have trained on, and the difficult part actually requires you to understand domain-driven data modeling at a level that will make you weep.

I think this isn't really right – it's fairly different to get everyone agree to a temporary pause than an indefinite one. (I think it's reasonable to argue that "real" versions of Plan A actually have to involve getting agreement on indefinite pause, at least in principle, because otherwise they immediately devolve into Plan C. But, it's still a pretty different vibe on what people's mainline expectations of the pause are)

@Raemon777 Plan S MUST be more tractable than A for same reason "Carol is a violinist" must be more likely than "Carol is a violinist who loves music": the former includes the latter. The first step of Plan A is a global pause (Plan S)! ANY subsequent steps make it less tractable!

@unconed Any constraints on what sort of things the multiplayer webapp does? Any constraints on what sort of prompts would feel like cheating?

@freed_dfilan Not particularly more centralized than most regulations? Centralization doesn't seem to be the axis it's varying on.

@Raemon777 Putting aside "Orwellian", do you not think that a country having a national rule about how many computers people are allowed to make counts as centralized decision-making?

I feel very frustrated by people who are considering "any attempt to regulate AI at all" as Orwellian centralized decisionmaking. (Also, Plan A specifically avoids having a centralized agency - each individual country makes their own regulations, while negotiating/arguing about what regulations are a good idea. This is... just so reasonable?)

My objection was not to "it's strong regulation" (it is and should be), it's "Orwellian centralized decisionmaking." The piece at least attempts to deliberately not be those things. One can make arguments that in practice this will turn out to be Orwellian centralized decisionmaking anyway, or that all regulation this strong is Orwellian (if not centralized). But these are generally asserted, not argued for.

@Raemon777 Anyway if you think Plan A is really strong regulation and people complain that it's really strong regulation, it feels weird to respond "you would say that about anything" if they're in fact right

@1a3orn I think the basic answer to this bit is "negotation underpinned by realpolitik military power", which I realize is unsatisfying, but, feels more realistic to me than whatever treaty-words they could have written down?

@Raemon777 Yeah I realize it's not canon, it's just that presumably he's a > 95th percentile careful reader. I find confusing / underdefined what conflict resolution mechanism is being proposed for Plan A as a treaty, see what I wrote here in section 1. 1a3orn.com/sub/2026-some-…