Blaine Dillingham

Responses inline “For the voluntary insurance, that’s why I kept to normal risks, rather than tail risks.” >> Companies already account for these “normal” risks (assuming you mean below judgment-proof amount). If anything, purchasing insurance for these risks worsens moral hazard. “For scaling up, it almost certainly won’t work, the big cat bond players are an entirely separate set of firms with different structures, skills, and incentives than the small scale insurance (mostly driven by the risks being not just quantitatively, but qualitatively different).” >> Not sure I understand the argument here, just that AI insurance will be qualitatively different from small scale insurance like car insurance? “As for coinsurance, I agree it’s better than flat, but for it to work you have to preemptively pick the risk profile you are most worried about (100B/1T/more; 1/5/50%?)” >> I agree we’ll need to make choices “and, to get the insurances companies to agree to carry, explicitly define the risks.” >> They would cover some amount of tort liability beyond the judgment proof amount. For an example, policy could say “frontier labs must hold an insurance policy that will begin paying out after the frontier lab is liquidated, covering 10% of the first $100 billion in damage past the judgment-proof amount. “Otherwise they are very unlikely to be willing to carry on their books. There’s also a ton of other thorny questions about timelines,” >> what does the timelines piece refer to? “potential for an insurer to monopolize the market,” >> yeah seems bad. How is this different from other industries? “how an insurer would develop expertise to price the risks,” >> hire third party auditors. Seems like a great thing to have “distorting effects on global capital pools, etc.” >> say more? “Maybe I’m being pollyannaish, but I just don’t think providing further incentive for the big labs to avoid catastrophic risk is a priority. They have plenty!” >> Not a believer in the judgment-proof problem? “If your concern is actually race dynamics, regulation/verification and monitoring/mandatory staged deployment are better fits for the risk profile under discussion. I think all have potential massive downsides that are rarely grappled with, but they are at least better at solving the race issue with the insurance market as it exists in the world today.” >> This is a different and additional concern. The point of insurance above the judgment-proof amount is to prevent companies from externalizing massive costs onto other people

Have heard the idea of insurance mandates for AI labs thrown around a lot, and, candidly, I just think it’s a straightforwardly bad idea. If we are talking about quotidian risks, tort law (you can discuss between negligence and strict) is a better tool. It lets the companies determine whether they are better placed to bear the risk themselves, or, if they’d rather smooth out the lumps, they can voluntarily insure themselves (a field which is rapidly growing). But the types of risks that are routinely mentioned as justifying mandatory insurance (cybersecurity attacks, large scale biological attacks, etc.) would fall into the catastrophe risk category. Taking the latter, as it’s better understood, you’re probably talking about hundreds of billions to trillions of notional exposure to the company in question from a single event. The GLOBAL catastrophic risk insurance market (~600B in capital if you really stretch definition) could not take a blow like that, which means you would have to have government guarantees, which gets back to this just being a political/governance issue anyway. Insurance is one of the great feats of our civilization and a hugely underrated institution, but it is not magic, and it is not very fast moving. Frankly, if one of the labs happens to act so negligently that something terrible happens, we, as a society, will find a way to sent the leadership to jail and throw away the key (see: Enron), which is already a substantially superior incentive aligning mechanism that does not distort what is one of the worlds most important existing institutions.

Well, I know we’ve discussed this and disagreed, so will just go one by one. For the voluntary insurance, that’s why I kept to normal risks, rather than tail risks. For scaling up, it almost certainly won’t work, the big cat bond players are an entirely separate set of firms with different structures, skills, and incentives than the small scale insurance (mostly driven by the risks being not just quantitatively, but qualitatively different). As for coinsurance, I agree it’s better than flat, but for it to work you have to preemptively pick the risk profile you are most worried about (100B/1T/more; 1/5/50%?) and, to get the insurances companies to agree to carry, explicitly define the risks. Otherwise they are very unlikely to be willing to carry on their books. There’s also a ton of other thorny questions about timelines, potential for an insurer to monopolize the market, how an insurer would develop expertise to price the risks, distorting effects on global capital pools, etc. Maybe I’m being pollyannaish, but I just don’t think providing further incentive for the big labs to avoid catastrophic risk is a priority. They have plenty! If your concern is actually race dynamics, regulation/verification and monitoring/mandatory staged deployment are better fits for the risk profile under discussion. I think all have potential massive downsides that are rarely grappled with, but they are at least better at solving the race issue with the insurance market as it exists in the world today.

Everyone worries about AI agent enabled cyberattacks on grids, banks, and pipelines. But America's sewers are often forgotten even though they may be near perfect targets for agentic attacks because of their fragmentation and technical redundancy. Read my latest on why we need def/acc for sewage:

Agreed tort law is a good tool for quotidian risks. But re "they can voluntarily insure themselves", they would never do this for harms beyond the judgment proof amount. That's the purpose of a mandate. Re "you’re probably talking about hundreds of billions to trillions of notional exposure to the company in question from a single event" -- nope, that's why you start the mandate small, and why you can do percent coinsurance (see my tweet)

Not sure I see how any of this cuts against an insurance mandate for AI labs. Some very good points here about why cyber insurance is difficult. But re AI: "the basic problem with liability insurance in the AI context is captured well in the diversity of opinions about failure modes and p(doom)" This doesn't seem like a huge problem for insurance. 'Things could go wrong in multiple ways' is something insurance handles fine all the time. A stronger point for you is that "the risk models are pretty explicit" and "mitigating interventions are pretty obvious" in normal insurance and may not be for AI, but this is exactly the reason that insurance companies will have to hire good third party auditors. In cases with actuarial history like car accidents or health insurance, it's often cheaper for insurance companies to just do statistics rather than invest up front in expensive auditing of every policyholder's risky behaviors and invest in research into safety mitigations. But precisely because we don't have an actuarial history of catastrophic harms from AI, insurance companies can't just use quick heuristics to set premiums. They'll need to audit labs' technical internals and encourage mitigations. I also just want to clarify the specific kind of insurance I think could be good: a mandate for frontier labs to have liability insurance that kicks in after lab bankruptcy. I don't necessarily want labs to buy insurance for harms that they could pay off without going bankrupt -- this could create moral hazard. I just want to make them internalize costs above the judgment-proof amount. Currently, they treat the same all potential liabilities that would go beyond bankrupting them, whether it's them causing a $100b cyberattack or a $1 trillion one. We likely can't start off with a requirement to carry insurance for anywhere near that amount, so it should start small and scale up over time. We can also capture the large tail risks with coinsurance: rather that insurance covering, for example, the first $50 billion in damages after the frontier lab is liquidated, instead you could say that insurance has to cover 5% of the first $1 trillion above liquidation

In cybersecurity, 97% of compromises aren't detected, a persistent actor will almost always get in to most commercial networks, and premia would have to be super costly to incent fundamental IT ops changes -- there are also just problems that are beyond the capacity of insurance-motivated targets to address (e.g., edge device vulnerabilities, APTs' easy access to domestic VPS, network monitoring capabilities that aren't that performant, extreme competition for cybersecurity talent)? So it's really hard for companies to receive a real price signal that motivates productive changes or derive benefits from risk pooling. I think the basic problem with liability insurance in the AI context is captured well in the diversity of opinions about failure modes and p(doom). For car insurance, home fire insurance, theft recovery insurance, and most forms of financial asset insurance, the risk models are pretty explicit, the signals are pretty clear, and the mitigating interventions are pretty obvious. I am not convinced that this is the case in IT or AI risk.

Various frontier AI governance proposals rely on private governance mechanisms (3rd party audits, industry standards-setting bodies, insurance) and torts to address AI harms. It's worth considering these proposals in light of their record in addressing cybersecurity risks related to state actors. I argue that they have delivered directional improvements, albeit ones more or less irrelevant to the threat for foreign SIGINT services. These models are inadequate for incenting private actors to undertake *costly* changes and therefore may not be a great model for AI governance. Even in defense industries, where security is prized and theoretically linked to contract awards, private governance institutions like the Department of War’s Cybersecurity Maturity Model Certification program have failed. This program attempted to build an ecosystem of 3rd party auditors, certified by an independent authorization body responsive to a government program office; the auditors’ evaluations would be used to inform eligibility for contract award, thereby creating an incentive for effortful implementation of cybersecurity controls. This program was initially substantially downscaled and has now been eliminated. The government will rely exclusively on its own occasional audits. The Department's initial ambition of increasing the costs of rivals’ computer network exploitation of the defense industrial base has effectively been retired. The latter occurred largely because the CMMC program attempted to improve compliance with contractual mandates to implement NIST-designed cybersecurity controls — a very weak proxy for cybersecurity capability. 3rd party evaluations often suffer this failure mode: they measure what is externally legible but cannot comprehensively evaluate real risks. The upshot is that, while 3rd party evaluations of adherence to responsible scaling policies might measure conformance with prior commitments (e.g., responsible scaling policies), many permutations would not directly evaluate the extent to which a given lab’s AI development paradigm or know-your-customer implementation creates excess biological risk. Furthermore, these audit mechanisms could inform but would not directly shape mitigation of relevant risks. With risks as severe as democratized biological weapons capability development, reliance on such an indirect mechanism is unsound. Industry collectives like the Cyber Threat Alliance and Open Worldwide Application Security Project have likewise failed to sufficiently improve collaborative threat analysis and software security to the level required to frustrate state cyber threats. That is not to say that they’ve been wholly ineffective: rather, they have developed and promulgated best practices that hundreds of companies have adopted in part. They have not, however, driven key companies to make required investments in data sharing and security engineering to the extent that they either affect their business models or hinder vulnerability research. Private governance mechanisms are unlikely to have such a beneficial disruptive impact in the context of frontier AI, simply due to competitive dynamics, coordination challenges, and the labs' overriding priorities of delivering AGI. Cyber insurance has had a similarly limited positive impact. While it has improved adoption of cybersecurity best practices on the margin, insurance has not driven adoption of information separation, airgaps, and encryption implementations — the sort of costly but effective security measures that challenge signals intelligence threats. These insurance policies frequently rely primarily on third-party audits of control adoption and red-teaming to inform policy eligibility and premia pricing. A number of information asymmetries and frictions confound these policies’ price signals, including state cyber threats’ being substantially more capable and persistent than auditors and red teams, difficulties in identifying and pricing the cost of compromise, and the innate vulnerability of the US IT product ecosystem. The result is that, even where adopted, these policies have not represented persuasive incentives for effective cybersecurity uplift. Data security liability has arguably had the most significant impact of these private governance mechanisms, incenting widespread hashing of credentials and key customer information. Nevertheless, customer data compromises continue to routinely happen, as financial services and consumer-facing IT companies are regularly hacked by capable state and non-state cyber actors. These analogues suggest that reliance on weak price signals to inform and incent effective management of security-related risks is a fool’s errand: these mechanisms can achieve directional improvement as compared to the status quo ante but they are unlikely to prompt the sorts of costly changes of behavior, product, and business model that are often required to address security shortfalls. A fundamental problem with private governance arrangements is that private institutions — whether the AI labs, third-party fora or evaluators, or insurers that rely on these evaluations — lack the state’s legitimacy in identifying shortfalls and motivate private action to mitigate various risks. No industry collective, 3rd party evaluator, insurance agent, or even price signal has innate authority or power analogous to that of a government department head. As such, each would probably struggle to command the visibility, cooperation, and attention required to reorient lab priorities or shape their technology and product strategies, especially in attempting to mitigate hitherto unrealized risks. Private governance proposals attempt to address this legitimacy shortfall in various ways: through formal covenants, state licensure of third-party evaluators, and authorizing legal frameworks, for example. These could plausibly confer some legitimacy to these institutions, but even thusly legitimized private institutions are unlikely to reshape lab strategy where private and public interests radically diverge.