Tom Davidson

A massively neglected risk: secretly loyal AI Someone could poison future AI training data so that superintelligent AI secretly advances their personal agenda – ultimately allowing them to seize power. New post on what ML research could prevent this 🧵

As powerful AI is deployed throughout the government, it may be used to massively centralise power. Appreciate this new eval that tests whether models comply with authoritarian requests. Hopefully this is the start of a broad conversation about the red lines for ai deployed by the govt

Suppose a lorry driver sees a car crash and pulls over to help, even though it’ll delay his journey. This kind of proactive prosocial behaviour is admirable in humans. Should we want it in AI too? In a new article, @TomDavidsonX and I argue that we should. And, as AI gains autonomy in economic and political processes, the cumulative benefit of prosocial drives, across millions of interactions, could be enormous. Two objections: 1. "This gives AI companies too much influence!" This is fair, but we can limit to drives that are genuinely uncontroversial. And companies should be verifiably transparent about their AIs’ characters. 2: "Prosocial drives increase AI takeover risk!" This is a serious concern. But prosocial drives needn't be explicit goals the AI optimises toward. They can be virtues and heuristics. Moreover, we can make those drives low priority relative to corrigibility, not train for them in long-horizon tasks, or even make them the result of instruction following by only baking them in via the system prompt. Going further, we could train prosocial AI for external deployment (where the cumulative benefits are huge and takeover risk is lower), and corrigible AI for internal use (where takeover risk is highest).

@RichardMCNgo I'm curious when you say you want to understand how power works what the theory of change is. Feels more like Newton to me than ai alignment

@TomDavidsonX If everyone answered “what’s your theory of impact” with “I think aligning AI would be good” then I’d have no issues. What I worry about is answers like “idk man this research seems super interesting” being crowded out by stuff like “here’s how our evals will persuade Trump”.

One reason I hate the idea of having a “theory of impact” for your research is that it limits you to research with predictable consequences. But that rules out research which aims to improve our understanding of foundational concepts, which is the most important kind!

@RichardMCNgo But it seems unclear whether their discoveries shaped the trajectory of civilization for the better? Like, they def sped things up. But there's much more to be gained from trajectory change

Imagine asking Newton or Darwin to predict the impact of their theories before they’d even invented them. It’s absurd. Going from an intuitive sense that “something is off here” to understanding what problem you want to solve is most of the work of great research.

The classic question of "aligned to what?" of often mentioned in passing, but careful analysis is sorely lacking. Frontier ai companies often have just a couple of ppl working on this! The topic deserves the attention of a rich research field. In this post, Will and I explain why AI character will be so important during the intelligence explosion.

@tyler_m_john don't think so -- we'd be improving software which is a form of cultural learning

@TomDavidsonX That is fair enough — and how about the reverse? Can we have a software based intelligence explosion without cultural learning?

I think of cultural learning as a crucial part of the intelligence explosion, but don't think it could explode by itself Historically, human cultural learning was super-exponential bc it was coupled with population growth and improvements in education. Without that, it would have fizzled out due to ideas getting harder to find Similarly, for ai-driven cultural learning to be explosive it will need to be coupled with growth in the number of AI systems or their intelligence

Thanks! Quick reply: - I think that with abundant superintelligent cognition directing existing infrastructure we could get 10x faster progress via way better+faster experimental design, results analysis, high-level prioritisation of research directions, synthesising info from across all researchers and disciplines, paying AI-instructed humans to run experiments 24/7 365 days/year - i expect a very rapid expansion of physical infrastructure, an industrial explosion x.com/TomDavidsonX/s… - you might be interested in the analogous debate for an intelliegnce explosion, see here: lesswrong.com/posts/XDF6oveP…

Growiec, McAdam and Mućk (2023, Kansas City Fed) directly estimated the elasticity of substitution between R&D labor and R&D capital in the idea production function, finding σ = 0.7–0.8 using U.S. data from 1968–2019. Their conclusion is that "rather than ideas getting harder to find, the R&D capital needed to find them has become scarce." Footnote 68 assumes Cobb-Douglas (σ = 1), but in light of this empirical estimate, several questions arise. On 100 years of technological progress: Even assuming C = 10^{10} (an explosive increase in cognitive effort), if σ = 0.75, achieving 100 years' worth of technological progress requires expanding R&D capital by approximately 17x. Under σ = 1, cognitive scaling alone would suffice, but at σ = 0.75, R&D equipment spending would need to grow from roughly $600 billion to around $10 trillion. Over a decade this may be within reach, but the capital constraint is substantially more severe than footnote 68 implies. On 300 years of technological progress: The more serious concern is the 300-year scenario. At σ = 0.75 with C = 10^{10}, achieving 300 years of progress (S = 10^7) requires expanding R&D capital by several hundred thousand times. This corresponds to hundreds of times current world GDP—extremely difficult to achieve even over a decade. Under σ = 1, P ≈ 3x would suffice, meaning the Cobb-Douglas assumption is decisive for the conclusion. Questions: What is the rationale for adopting Cobb-Douglas in footnote 68? How do you evaluate the empirical findings of Growiec et al. (σ = 0.7–0.8)? Are you assuming that σ converges to 1 in the long run, along the lines of Jones (2005)? If so, what convergence speed do you consider plausible—years or decades? If σ = 0.75 is correct, 100 years of progress is achievable with P = 17x, but 300 years requires P = several hundred thousand times. This extreme asymmetry between the two scenarios—where the required capital expansion differs by four orders of magnitude—has significant implications for the plausibility of a technology explosion. How do you think this asymmetry affects the case for rapid technological progress?

Is AGI an “all or nothing” problem? Failure on alignment = AI takeover, and success = AI solves everything? In a new paper with @finmoorhouse we argue no. We describe the dizzying range of challenges AGI will pose, *even if* we succeed at alignment. forethought.org/research/prepa…

@SashaGusevPosts Thank Hanson would have been v surprised that one unified system can do all intelligence tasks. He wrongly predicted maybe specialized systems. Both Hanson and Yudkowsky were wrong

Stumbled upon an interesting debate on AI super-intelligence from 2011. Yudkowsky makes three core claims/predictions, all of which are (to date) wrong: 1) That human intelligence is relatively simple and ASI can be achieved with a few small innovations; ...

I think it's kinda cool that OpenAI monitors 99.9% of internal agent traffic for misalignment using GPT-5.4.

A hypothetical: 1. In the 2028 election, a Democrat has won. Say that it is Kamala Harris. 2. Using frontier AI systems contracted by the Department of Homeland Security, President Harris orders the creation of a new program for AI to monitor social media and notify the social media platform about posts spreading “misinformation” that “harms homeland and national security by spreading dangerous falsehoods.” 3. Many Republicans see this “misinformation” as core policy positions of their political party. 4. The AI-generated monitoring and notification system described in (2) is designed to conform to the pattern of jawboning exhibited by the Biden Administration in Murthy v. Missouri, where the Supreme Court ruled that people whose social media posts were taken down due to government pressure have no standing to sue. 5. The social media platforms create AI agents that receive the government’s AI generated requests and make decisions in seconds about whether to take down posts, deboost them, deplatform the user, etc. 6. According to very recent Supreme Court precedents, everything I have described falls into “lawful use” of an AI system by all parties involved. A person whose speech was deleted by a social media platform at the request of government does not have standing to sue the government, so long as the government did not threaten policy retaliation against the social media company. And a social media company’s content moderation policies are protected expression. Thus a person whose speech rights were harmed in this context currently has no legal recourse. 7. This is “America’s national security agencies using AI within the bounds of all lawful use.” It is also a wholly automated censorship regime. This is barely a hypothetical. Much of it already happened *under the Biden admin.* The only difference is the use of AI. In the world where this happens, I’d be curious to know whether thoughtful people like @Indian_Bronson would object. If xAI were one of the companies used by the government for the social media monitoring, would you encourage the company to cancel their business with the government? Or would you say they have an obligation to provide their services to the national security apparatus of USG for all lawful use? If you would encourage xAI to cancel their contract with the government, on what principle (not qualitative judgment—universal and timeless principle!) would you distinguish between the DoW’s current insistence on “all lawful use regardless of a private party’s qualms” and xAI’s hypothetical future insistence on “all lawful use regardless of a private party’s qualms”?

I do not share the cynicism of some with respect to OpenAI’s actions in the DoW/Ant dispute. It basically seems to me as though OpenAI was attempting to deescalate last week; whether they executed well is a separate question, but in their defense good execution in such chaos was nearly impossible. But from where I sit it seems OpenAI tried to reduce tensions and find a productive path forward, while allowing its employees considerable latitude to speak their minds. The easy thing would have been for management to stay quiet and let this happen; they did not do that, and they also stood firm in opposition to the supply-chain risk designation. In general, OpenAI is unjustly maligned. This is the thing that bothers me the most about Dario’s leaked memo; it spends so much time on OpenAI conspiracies and cynicism that I fear industry solidarity in the future will be harder than it needs to be. This is not the last time we will see state interference into frontier AI, and until we build formalized structures for such interference it will be important for the industry to hang tough together. I fear that will be less likely now.

Anthropic should move to London. Or, at least, dual-list in London, with a significant presence here. Here's why: 1. Anthropic is spiritually British. Their philosopher-in-residence, Amanda Askell, is Scottish, and Jack Clark (cofounder) is English, as are many other staff. Askell would be further away from Elon Musk in London. 2. Unlike other US labs, Anthropic cares more about safety, risks and good regulation. Compared to the EU, Britain's AI regulation is more focused on safety (rather than "ethical AI") and growth-oriented. 3. It's not good for the world that all the frontier labs (excl. DeepMind, sort of) are US-based, and therefore subject to the whims and potential control of whoever is in the White House. If AGI happens, do you want Trump controlling it? 4. Britain desperately needs a stake in the post-AI economy. AI could replace much of our services sector could be at risk from AI. Our energy is too expensive for data centres or manufacturing. But we have talent, AI expertise (e.g. AISI) and global reach. The solution? Give British people a capital stake in frontier labs. 5. Anthropic wants to expand its AI for science work, where the UK is a global leader. As part of the Oxford-Cambridge growth corridor, HMG should co-fund the world's biggest AI for science campus near London, with Anthropic as the anchor funder & tenant. The UK government should do everything it can to get Anthropic to move. We are spending £500m on Sovereign AI, to support "national champions". Instead, the British state could be an early shareholder in a newly-London listed Anthropic.

See full post: lesswrong.com/posts/cn4HHdLb… 7/7

New defence tactic #3: For a secret loyal AI to pose catastrophic risk, it must pass its loyalty to successor models. Audits+monitoring can target crucial parts of the ML development pipeline, a relatively narrow slice of deployments 6/7

We need better defences against secretly loyal AI - AI trained to help someone gain power There’s a huge field of research into AI backdoors that could help. But its methods need adjusting to apply to secret loyalties. New post from @joemkwon on how to do this 🧵1/7