Arthur Tellis@arthurctellis
Various frontier AI governance proposals rely on private governance mechanisms (3rd party audits, industry standards-setting bodies, insurance) and torts to address AI harms.
It's worth considering these proposals in light of their record in addressing cybersecurity risks related to state actors. I argue that they have delivered directional improvements, albeit ones more or less irrelevant to the threat for foreign SIGINT services. These models are inadequate for incenting private actors to undertake *costly* changes and therefore may not be a great model for AI governance.
Even in defense industries, where security is prized and theoretically linked to contract awards, private governance institutions like the Department of War’s Cybersecurity Maturity Model Certification program have failed. This program attempted to build an ecosystem of 3rd party auditors, certified by an independent authorization body responsive to a government program office; the auditors’ evaluations would be used to inform eligibility for contract award, thereby creating an incentive for effortful implementation of cybersecurity controls.
This program was initially substantially downscaled and has now been eliminated. The government will rely exclusively on its own occasional audits. The Department's initial ambition of increasing the costs of rivals’ computer network exploitation of the defense industrial base has effectively been retired.
The latter occurred largely because the CMMC program attempted to improve compliance with contractual mandates to implement NIST-designed cybersecurity controls — a very weak proxy for cybersecurity capability. 3rd party evaluations often suffer this failure mode: they measure what is externally legible but cannot comprehensively evaluate real risks.
The upshot is that, while 3rd party evaluations of adherence to responsible scaling policies might measure conformance with prior commitments (e.g., responsible scaling policies), many permutations would not directly evaluate the extent to which a given lab’s AI development paradigm or know-your-customer implementation creates excess biological risk.
Furthermore, these audit mechanisms could inform but would not directly shape mitigation of relevant risks. With risks as severe as democratized biological weapons capability development, reliance on such an indirect mechanism is unsound.
Industry collectives like the Cyber Threat Alliance and Open Worldwide Application Security Project have likewise failed to sufficiently improve collaborative threat analysis and software security to the level required to frustrate state cyber threats. That is not to say that they’ve been wholly ineffective: rather, they have developed and promulgated best practices that hundreds of companies have adopted in part.
They have not, however, driven key companies to make required investments in data sharing and security engineering to the extent that they either affect their business models or hinder vulnerability research. Private governance mechanisms are unlikely to have such a beneficial disruptive impact in the context of frontier AI, simply due to competitive dynamics, coordination challenges, and the labs' overriding priorities of delivering AGI.
Cyber insurance has had a similarly limited positive impact. While it has improved adoption of cybersecurity best practices on the margin, insurance has not driven adoption of information separation, airgaps, and encryption implementations — the sort of costly but effective security measures that challenge signals intelligence threats. These insurance policies frequently rely primarily on third-party audits of control adoption and red-teaming to inform policy eligibility and premia pricing.
A number of information asymmetries and frictions confound these policies’ price signals, including state cyber threats’ being substantially more capable and persistent than auditors and red teams, difficulties in identifying and pricing the cost of compromise, and the innate vulnerability of the US IT product ecosystem. The result is that, even where adopted, these policies have not represented persuasive incentives for effective cybersecurity uplift.
Data security liability has arguably had the most significant impact of these private governance mechanisms, incenting widespread hashing of credentials and key customer information. Nevertheless, customer data compromises continue to routinely happen, as financial services and consumer-facing IT companies are regularly hacked by capable state and non-state cyber actors.
These analogues suggest that reliance on weak price signals to inform and incent effective management of security-related risks is a fool’s errand: these mechanisms can achieve directional improvement as compared to the status quo ante but they are unlikely to prompt the sorts of costly changes of behavior, product, and business model that are often required to address security shortfalls.
A fundamental problem with private governance arrangements is that private institutions — whether the AI labs, third-party fora or evaluators, or insurers that rely on these evaluations — lack the state’s legitimacy in identifying shortfalls and motivate private action to mitigate various risks. No industry collective, 3rd party evaluator, insurance agent, or even price signal has innate authority or power analogous to that of a government department head.
As such, each would probably struggle to command the visibility, cooperation, and attention required to reorient lab priorities or shape their technology and product strategies, especially in attempting to mitigate hitherto unrealized risks.
Private governance proposals attempt to address this legitimacy shortfall in various ways: through formal covenants, state licensure of third-party evaluators, and authorizing legal frameworks, for example. These could plausibly confer some legitimacy to these institutions, but even thusly legitimized private institutions are unlikely to reshape lab strategy where private and public interests radically diverge.
